+ <listitem>
+ <para>If the MGS and MDS share a partition, start the MGS only:</para>
+ <screen>mount -t lustre <replaceable>MDT partition</replaceable> -o nosvc <replaceable>mount_point</replaceable></screen>
+ </listitem>
+ <listitem>
+ <para>Run the <literal>replace_nids</literal> command on the MGS:</para>
+ <screen>lctl replace_nids <replaceable>devicename</replaceable> <replaceable>nid1</replaceable>[,nid2,nid3 ...]</screen>
+ <para>where <replaceable>devicename</replaceable> is the Lustre target name, e.g.
+ <literal>testfs-OST0013</literal></para>
+ </listitem>
+ <listitem>
+ <para>If the MGS and MDS share a partition, stop the MGS:</para>
+ <screen>umount <replaceable>mount_point</replaceable></screen>
+ </listitem>
+ </orderedlist>
+ <note><para>The <literal>replace_nids</literal> command also cleans
+ all old, invalidated records out of the configuration log, while
+ preserving all other current settings.</para></note>
+ <note><para>The previous configuration log is backed up on the MGS
+ disk with the suffix <literal>'.bak'</literal>.</para></note>
+ </section>
+ <section xml:id="lustremaint.clear_conf" condition="l2B">
+ <title><indexterm>
+ <primary>maintenance</primary>
+ <secondary>Clearing a config</secondary>
+ </indexterm> Clearing configuration</title>
+ <para>
+ This command runs on MGS node having the MGS device mounted with
+ <literal>-o nosvc.</literal> It cleans up configuration files
+ stored in the CONFIGS/ directory of any records marked SKIP.
+ If the device name is given, then the specific logs for that
+ filesystem (e.g. testfs-MDT0000) are processed. Otherwise, if a
+ filesystem name is given then all configuration files are cleared.
+ The previous configuration log is backed up on the MGS disk with
+ the suffix 'config.timestamp.bak'. Eg: Lustre-MDT0000-1476454535.bak.
+ </para>
+ <para> To clear a configuration:</para>
+ <orderedlist>
+ <listitem>
+ <para>Shut down the file system in this order:</para>
+ <orderedlist>
+ <listitem>
+ <para>Unmount the clients.</para>
+ </listitem>
+ <listitem>
+ <para>Unmount the MDT.</para>
+ </listitem>
+ <listitem>
+ <para>Unmount all OSTs.</para>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para>
+ If the MGS and MDS share a partition, start the MGS only
+ using "nosvc" option.
+ </para>
+ <screen>mount -t lustre <replaceable>MDT partition</replaceable> -o nosvc <replaceable>mount_point</replaceable></screen>
+ </listitem>
+ <listitem>
+ <para>Run the <literal>clear_conf</literal> command on the MGS:
+ </para>
+ <screen>lctl clear_conf <replaceable>config</replaceable></screen>
+ <para>
+ Example: To clear the configuration for
+ <literal>MDT0000</literal> on a filesystem named
+ <literal>testfs</literal>
+ </para>
+ <screen>mgs# lctl clear_conf testfs-MDT0000</screen>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section xml:id="lustremaint.adding_new_mdt" condition='l24'>
+ <title><indexterm>
+ <primary>maintenance</primary>
+ <secondary>adding an MDT</secondary>
+ </indexterm>Adding a New MDT to a Lustre File System</title>
+ <para>Additional MDTs can be added using the DNE feature to serve one
+ or more remote sub-directories within a filesystem, in order to
+ increase the total number of files that can be created in the
+ filesystem, to increase aggregate metadata performance, or to isolate
+ user or application workloads from other users of the filesystem. It
+ is possible to have multiple remote sub-directories reference the
+ same MDT. However, the root directory will always be located on
+ MDT0. To add a new MDT into the file system:</para>
+ <orderedlist>
+ <listitem>
+ <para>Discover the maximum MDT index. Each MDT must have unique index.</para>
+<screen>
+client$ lctl dl | grep mdc
+36 UP mdc testfs-MDT0000-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
+37 UP mdc testfs-MDT0001-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
+38 UP mdc testfs-MDT0002-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
+39 UP mdc testfs-MDT0003-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
+</screen>
+ </listitem>
+ <listitem>
+ <para>Add the new block device as a new MDT at the next available
+ index. In this example, the next available index is 4.</para>
+<screen>
+mds# mkfs.lustre --reformat --fsname=<replaceable>testfs</replaceable> --mdt --mgsnode=<replaceable>mgsnode</replaceable> --index 4 <replaceable>/dev/mdt4_device</replaceable>
+</screen>
+ </listitem>
+ <listitem>
+ <para>Mount the MDTs.</para>
+<screen>
+mds# mount –t lustre <replaceable>/dev/mdt4_blockdevice</replaceable> /mnt/mdt4
+</screen>
+ </listitem>
+ <listitem>
+ <para>In order to start creating new files and directories on the
+ new MDT(s) they need to be attached into the namespace at one or
+ more subdirectories using the <literal>lfs mkdir</literal> command.
+ All files and directories below those created with
+ <literal>lfs mkdir</literal> will also be created on the same MDT
+ unless otherwise specified.
+ </para>
+<screen>
+client# lfs mkdir -i 3 /mnt/testfs/new_dir_on_mdt3
+client# lfs mkdir -i 4 /mnt/testfs/new_dir_on_mdt4
+client# lfs mkdir -c 4 /mnt/testfs/new_directory_striped_across_4_mdts
+</screen>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section xml:id="lustremaint.adding_new_ost">
+ <title><indexterm><primary>maintenance</primary><secondary>adding a OST</secondary></indexterm>
+Adding a New OST to a Lustre File System</title>
+ <para>A new OST can be added to existing Lustre file system on either
+ an existing OSS node or on a new OSS node. In order to keep client IO
+ load balanced across OSS nodes for maximum aggregate performance, it is
+ not recommended to configure different numbers of OSTs to each OSS node.
+ </para>
+ <orderedlist>
+ <listitem>
+ <para> Add a new OST by using <literal>mkfs.lustre</literal> as when
+ the filesystem was first formatted, see
+ <xref linkend="dbdoclet.format_ost" /> for details. Each new OST
+ must have a unique index number, use <literal>lctl dl</literal> to
+ see a list of all OSTs. For example, to add a new OST at index 12
+ to the <literal>testfs</literal> filesystem run following commands
+ should be run on the OSS:</para>
+ <screen>oss# mkfs.lustre --fsname=testfs --mgsnode=mds16@tcp0 --ost --index=12 /dev/sda
+oss# mkdir -p /mnt/testfs/ost12
+oss# mount -t lustre /dev/sda /mnt/testfs/ost12</screen>
+ </listitem>
+ <listitem>
+ <para>Balance OST space usage (possibly).</para>
+ <para>The file system can be quite unbalanced when new empty OSTs
+ are added to a relatively full filesystem. New file creations are
+ automatically balanced to favour the new OSTs. If this is a scratch
+ file system or files are pruned at regular intervals, then no further
+ work may be needed to balance the OST space usage as new files being
+ created will preferentially be placed on the less full OST(s). As old
+ files are deleted, they will release space on the old OST(s).</para>
+ <para>Files existing prior to the expansion can optionally be
+ rebalanced using the <literal>lfs_migrate</literal> utility.
+ This redistributes file data over the entire set of OSTs.</para>
+ <para>For example, to rebalance all files within the directory
+ <literal>/mnt/lustre/dir</literal>, enter:</para>
+ <screen>client# lfs_migrate /mnt/lustre/dir</screen>
+ <para>To migrate files within the <literal>/test</literal> file
+ system on <literal>OST0004</literal> that are larger than 4GB in
+ size to other OSTs, enter:</para>
+ <screen>client# lfs find /test --ost test-OST0004 -size +4G | lfs_migrate -y</screen>
+ <para>See <xref linkend="dbdoclet.lfs_migrate"/> for details.</para>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section xml:id="lustremaint.deactivating_mdt_ost">
+ <title><indexterm><primary>maintenance</primary><secondary>restoring an OST</secondary></indexterm>
+ <indexterm><primary>maintenance</primary><secondary>removing an OST</secondary></indexterm>
+Removing and Restoring MDTs and OSTs</title>
+ <para>OSTs and DNE MDTs can be removed from and restored to a Lustre
+ filesystem. Deactivating an OST means that it is temporarily or
+ permanently marked unavailable. Deactivating an OST on the MDS means
+ it will not try to allocate new objects there or perform OST recovery,
+ while deactivating an OST the client means it will not wait for OST
+ recovery if it cannot contact the OST and will instead return an IO
+ error to the application immediately if files on the OST are accessed.
+ An OST may be permanently deactivated from the file system,
+ depending on the situation and commands used.</para>
+ <note><para>A permanently deactivated MDT or OST still appears in the
+ filesystem configuration until the configuration is regenerated with
+ <literal>writeconf</literal> or it is replaced with a new MDT or OST
+ at the same index and permanently reactivated. A deactivated OST
+ will not be listed by <literal>lfs df</literal>.
+ </para></note>
+ <para>You may want to temporarily deactivate an OST on the MDS to
+ prevent new files from being written to it in several situations:</para>
+ <itemizedlist>
+ <listitem>
+ <para>A hard drive has failed and a RAID resync/rebuild is underway,
+ though the OST can also be marked <emphasis>degraded</emphasis> by
+ the RAID system to avoid allocating new files on the slow OST which
+ can reduce performance, see <xref linkend='dbdoclet.degraded_ost' />
+ for more details.
+ </para>
+ </listitem>
+ <listitem>
+ <para>OST is nearing its space capacity, though the MDS will already
+ try to avoid allocating new files on overly-full OSTs if possible,
+ see <xref linkend='dbdoclet.balancing_free_space' /> for details.
+ </para>
+ </listitem>
+ <listitem>
+ <para>MDT/OST storage or MDS/OSS node has failed, and will not
+ be available for some time (or forever), but there is still a
+ desire to continue using the filesystem before it is repaired.</para>
+ </listitem>
+ </itemizedlist>
+ <section condition="l24" xml:id="lustremaint.rmremotedir">
+ <title><indexterm><primary>maintenance</primary><secondary>removing an MDT</secondary></indexterm>Removing an MDT from the File System</title>
+ <para>If the MDT is permanently inaccessible,
+ <literal>lfs rm_entry {directory}</literal> can be used to delete the
+ directory entry for the unavailable MDT. Using <literal>rmdir</literal>
+ would otherwise report an IO error due to the remote MDT being inactive.
+ Please note that if the MDT <emphasis>is</emphasis> available, standard
+ <literal>rm -r</literal> should be used to delete the remote directory.
+ After the remote directory has been removed, the administrator should
+ mark the MDT as permanently inactive with:</para>
+ <screen>lctl conf_param {MDT name}.mdc.active=0</screen>
+ <para>A user can identify which MDT holds a remote sub-directory using
+ the <literal>lfs</literal> utility. For example:</para>
+<screen>client$ lfs getstripe --mdt-index /mnt/lustre/remote_dir1
+1
+client$ mkdir /mnt/lustre/local_dir0
+client$ lfs getstripe --mdt-index /mnt/lustre/local_dir0
+0
+</screen>
+ <para>The <literal>lfs getstripe --mdt-index</literal> command
+ returns the index of the MDT that is serving the given directory.</para>
+ </section>
+ <section xml:id="lustremaint.inactivemdt" condition='l24'>
+ <title>
+ <indexterm><primary>maintenance</primary></indexterm>
+ <indexterm><primary>maintenance</primary><secondary>inactive MDTs</secondary></indexterm>Working with Inactive MDTs</title>
+ <para>Files located on or below an inactive MDT are inaccessible until
+ the MDT is activated again. Clients accessing an inactive MDT will receive
+ an EIO error.</para>
+ </section>
+ <section remap="h3" xml:id="lustremaint.remove_ost">
+ <title><indexterm>
+ <primary>maintenance</primary>
+ <secondary>removing an OST</secondary>
+ </indexterm>Removing an OST from the File System</title>
+ <para>When deactivating an OST, note that the client and MDS each have
+ an OSC device that handles communication with the corresponding OST.
+ To remove an OST from the file system:</para>
+ <orderedlist>