-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremaintenance">
+<?xml version='1.0' encoding='UTF-8'?>
+<chapter xmlns="http://docbook.org/ns/docbook"
+ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
+ xml:id="lustremaintenance">
<title xml:id="lustremaintenance.title">Lustre Maintenance</title>
<para>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre maintenance tasks:</para>
<itemizedlist>
<listitem>
- <para><xref linkend="dbdoclet.50438199_42877"/></para>
+ <para><xref linkend="lustremaint.inactiveOST"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_15240"/></para>
+ <para><xref linkend="lustremaint.findingNodes"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_26070"/></para>
+ <para><xref linkend="lustremaint.mountingServerWithoutLustre"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_54623"/></para>
+ <para><xref linkend="lustremaint.regenerateConfigLogs"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_31353"/></para>
+ <para><xref linkend="lustremaint.changingservernid"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.addingamdt"/></para>
+ <para><xref linkend="lustremaint.clear_conf"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_22527"/></para>
+ <para><xref linkend="lustremaint.adding_new_mdt"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_14978"/></para>
+ <para><xref linkend="lustremaint.adding_new_ost"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.rmremotedir"/></para>
+ <para><xref linkend="lustremaint.deactivating_mdt_ost"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.inactivemdt"/></para>
+ <para><xref linkend="lustremaint.rmremotedir"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_77819"/></para>
+ <para><xref linkend="lustremaint.inactivemdt"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_12607"/></para>
+ <para><xref linkend="lustremaint.remove_ost"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_62333"/></para>
+ <para><xref linkend="lustremaint.ydg_pgt_tl"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_62545"/></para>
+ <para><xref linkend="lustremaint.restore_ost"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="lustremaint.ucf_qgt_tl"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="lustremaint.abortRecovery"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="lustremaint.determineOST"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="lustremaint.ChangeAddrFailoverNode"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="lustremaint.seperateCombinedMGSMDT"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="lustremaint.setMDTReadonly"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="lustremaint.tunefallocate"/></para>
</listitem>
</itemizedlist>
- <section xml:id="dbdoclet.50438199_42877">
+ <section xml:id="lustremaint.inactiveOST">
<title>
<indexterm><primary>maintenance</primary></indexterm>
<indexterm><primary>maintenance</primary><secondary>inactive OSTs</secondary></indexterm>
<para>To mount a client or an MDT with one or more inactive OSTs, run commands similar to this:</para>
<screen>client# mount -o exclude=testfs-OST0000 -t lustre \
uml1:/testfs /mnt/testfs
- client# cat /proc/fs/lustre/lov/testfs-clilov-*/target_obd</screen>
- <para>To activate an inactive OST on a live client or MDT, use the <literal>lctl activate</literal> command on the OSC device. For example:</para>
+ client# lctl get_param lov.testfs-clilov-*.target_obd</screen>
+ <para>To activate an inactive OST on a live client or MDT, use the
+ <literal>lctl activate</literal> command on the OSC device. For example:</para>
<screen>lctl --device 7 activate</screen>
<note>
- <para>A colon-separated list can also be specified. For example, <literal>exclude=testfs-OST0000:testfs-OST0001</literal>.</para>
+ <para>A colon-separated list can also be specified. For example,
+ <literal>exclude=testfs-OST0000:testfs-OST0001</literal>.</para>
</note>
</section>
- <section xml:id="dbdoclet.50438199_15240">
+ <section xml:id="lustremaint.findingNodes">
<title><indexterm><primary>maintenance</primary><secondary>finding nodes</secondary></indexterm>
Finding Nodes in the Lustre File System</title>
- <para>There may be situations in which you need to find all nodes in your Lustre file system or get the names of all OSTs.</para>
+ <para>There may be situations in which you need to find all nodes in
+ your Lustre file system or get the names of all OSTs.</para>
<para>To get a list of all Lustre nodes, run this command on the MGS:</para>
- <screen># cat /proc/fs/lustre/mgs/MGS/live/*</screen>
+ <screen># lctl get_param mgs.MGS.live.*</screen>
<note>
- <para>This command must be run on the MGS.
- </para>
+ <para>This command must be run on the MGS.</para>
</note>
- <para>In this example, file system lustre has three nodes, <literal>lustre-MDT0000</literal>, <literal>lustre-OST0000</literal>, and <literal>lustre-OST0001</literal>.</para>
- <screen>cfs21:/tmp# cat /proc/fs/lustre/mgs/MGS/live/*
- fsname: lustre
+ <para>In this example, file system <literal>testfs</literal> has three
+ nodes, <literal>testfs-MDT0000</literal>,
+ <literal>testfs-OST0000</literal>, and
+ <literal>testfs-OST0001</literal>.</para>
+ <screen>mgs:/root# lctl get_param mgs.MGS.live.*
+ fsname: testfs
flags: 0x0 gen: 26
- lustre-MDT0000
- lustre-OST0000
- lustre-OST0001 </screen>
+ testfs-MDT0000
+ testfs-OST0000
+ testfs-OST0001 </screen>
<para>To get the names of all OSTs, run this command on the MDS:</para>
- <screen># cat /proc/fs/lustre/lov/<replaceable>fsname</replaceable>-mdtlov/target_obd </screen>
+ <screen>mds:/root# lctl get_param lov.*-mdtlov.target_obd </screen>
<note>
- <para>This command must be run on the MDS.
- </para>
+ <para>This command must be run on the MDS.</para>
</note>
- <para>In this example, there are two OSTs, lustre-OST0000 and lustre-OST0001, which are both active.</para>
- <screen>cfs21:/tmp# cat /proc/fs/lustre/lov/lustre-mdtlov/target_obd
-0: lustre-OST0000_UUID ACTIVE
-1: lustre-OST0001_UUID ACTIVE </screen>
+ <para>In this example, there are two OSTs, testfs-OST0000 and
+ testfs-OST0001, which are both active.</para>
+ <screen>mgs:/root# lctl get_param lov.testfs-mdtlov.target_obd
+0: testfs-OST0000_UUID ACTIVE
+1: testfs-OST0001_UUID ACTIVE </screen>
</section>
- <section xml:id="dbdoclet.50438199_26070">
+ <section xml:id="lustremaint.mountingServerWithoutLustre">
<title><indexterm><primary>maintenance</primary><secondary>mounting a server</secondary></indexterm>
Mounting a Server Without Lustre Service</title>
<para>If you are using a combined MGS/MDT, but you only want to start the MGS and not the MDT, run this command:</para>
<para>In this example, the combined MGS/MDT is <literal>testfs-MDT0000</literal> and the mount point is <literal>/mnt/test/mdt</literal>.</para>
<screen>$ mount -t lustre -L testfs-MDT0000 -o nosvc /mnt/test/mdt</screen>
</section>
- <section xml:id="dbdoclet.50438199_54623">
+ <section xml:id="lustremaint.regenerateConfigLogs">
<title><indexterm><primary>maintenance</primary><secondary>regenerating config logs</secondary></indexterm>
Regenerating Lustre Configuration Logs</title>
- <para>If the Lustre system's configuration logs are in a state where the file system cannot be started, use the <literal>writeconf</literal> command to erase them. After the <literal>writeconf</literal> command is run and the servers restart, the configuration logs are re-generated and stored on the MGS (as in a new file system).</para>
+ <para>If the Lustre file system configuration logs are in a state where
+ the file system cannot be started, use the
+ <literal>tunefs.lustre --writeconf</literal> command to regenerate them.
+ After the <literal>writeconf</literal> command is run and the servers
+ restart, the configuration logs are re-generated and stored on the MGS
+ (as with a new file system).</para>
<para>You should only use the <literal>writeconf</literal> command if:</para>
<itemizedlist>
<listitem>
<para>A server NID is being changed</para>
</listitem>
</itemizedlist>
- <para>The <literal>writeconf</literal> command is destructive to some configuration items (i.e., OST pools information and items set via <literal>conf_param</literal>), and should be used with caution. To avoid problems:</para>
- <itemizedlist>
- <listitem>
- <para>Shut down the file system before running the <literal>writeconf</literal> command</para>
- </listitem>
- <listitem>
- <para>Run the <literal>writeconf</literal> command on all servers (MDT first, then OSTs)</para>
- </listitem>
- <listitem>
- <para>Start the file system in this order:</para>
- <itemizedlist>
- <listitem>
- <para>MGS (or the combined MGS/MDT)</para>
- </listitem>
- <listitem>
- <para>MDT</para>
- </listitem>
- <listitem>
- <para>OSTs</para>
- </listitem>
- <listitem>
- <para>Lustre clients</para>
- </listitem>
- </itemizedlist>
- </listitem>
- </itemizedlist>
+ <para>The <literal>writeconf</literal> command is destructive to some
+ configuration items (e.g. OST pools information and tunables set via
+ <literal>conf_param</literal>), and should be used with caution.</para>
<caution>
- <para>The OST pools feature enables a group of OSTs to be named for file striping purposes. If you use OST pools, be aware that running the <literal>writeconf</literal> command erases <emphasis role="bold">all</emphasis> pools information (as well as any other parameters set via <literal>lctl conf_param</literal>). We recommend that the pools definitions (and <literal>conf_param</literal> settings) be executed via a script, so they can be reproduced easily after a <literal>writeconf</literal> is performed.</para>
+ <para>The OST pools feature enables a group of OSTs to be named for
+ file striping purposes. If you use OST pools, be aware that running
+ the <literal>writeconf</literal> command erases
+ <emphasis role="bold">all</emphasis> pools information (as well as
+ any other parameters set via <literal>lctl conf_param</literal>).
+ We recommend that the pools definitions (and
+ <literal>conf_param</literal> settings) be executed via a script,
+ so they can be regenerated easily after <literal>writeconf</literal>
+ is performed. However, tunables saved with <literal>lctl set_param
+ -P</literal> are <emphasis>not</emphasis> erased in this case.</para>
</caution>
- <para>To regenerate Lustre's system configuration logs:</para>
+ <note>
+ <para>If the MGS still holds any configuration logs, it may be
+ possible to dump these logs to save any parameters stored with
+ <literal>lctl conf_param</literal> by dumping the config logs on
+ the MGS and saving the output (once for each MDT and OST device):
+ </para>
+<screen>
+mgs# lctl --device MGS llog_print <replaceable>fsname</replaceable>-client
+mgs# lctl --device MGS llog_print <replaceable>fsname</replaceable>-MDT0000
+mgs# lctl --device MGS llog_print <replaceable>fsname</replaceable>-OST0000
+</screen>
+ </note>
+ <para>To regenerate Lustre file system configuration logs:</para>
<orderedlist>
<listitem>
- <para>Shut down the file system in this order.</para>
+ <para>Stop the file system services in the following order before
+ running the <literal>tunefs.lustre --writeconf</literal> command:
+ </para>
<orderedlist>
<listitem>
<para>Unmount the clients.</para>
</listitem>
<listitem>
- <para>Unmount the MDT.</para>
+ <para>Unmount the MDT(s).</para>
</listitem>
<listitem>
- <para>Unmount all OSTs.</para>
+ <para>Unmount the OST(s).</para>
+ </listitem>
+ <listitem>
+ <para>If the MGS is separate from the MDT it can remain mounted
+ during this process.</para>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para>Make sure the the MDT and OST devices are available.</para>
+ <para>Make sure the MDT and OST devices are available.</para>
</listitem>
<listitem>
- <para>Run the <literal>writeconf</literal> command on all servers.</para>
- <para>Run writeconf on the MDT first, and then the OSTs.</para>
+ <para>Run the <literal>tunefs.lustre --writeconf</literal> command
+ on all target devices.</para>
+ <para>Run writeconf on the MDT(s) first, and then the OST(s).</para>
<orderedlist>
<listitem>
- <para>On the MDT, run:</para>
- <screen>mdt# tunefs.lustre --writeconf <replaceable>/dev/mdt_device</replaceable></screen>
+ <para>On each MDS, for each MDT run:</para>
+ <screen>mds# tunefs.lustre --writeconf <replaceable>/dev/mdt_device</replaceable></screen>
</listitem>
<listitem>
- <para>
- On each OST, run:
-
- <screen>ost# tunefs.lustre --writeconf <replaceable>/dev/ost_device</replaceable></screen>
+ <para> On each OSS, for each OST run:
+ <screen>oss# tunefs.lustre --writeconf <replaceable>/dev/ost_device</replaceable></screen>
</para>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para>Restart the file system in this order.</para>
+ <para>Restart the file system in the following order:</para>
<orderedlist>
<listitem>
- <para>Mount the MGS (or the combined MGS/MDT).</para>
+ <para>Mount the separate MGT, if it is not already mounted.</para>
</listitem>
<listitem>
- <para>Mount the MDT.</para>
+ <para>Mount the MDT(s) in order, starting with MDT0000.</para>
</listitem>
<listitem>
- <para>Mount the OSTs.</para>
+ <para>Mount the OSTs in order, starting with OST0000.</para>
</listitem>
<listitem>
<para>Mount the clients.</para>
</orderedlist>
</listitem>
</orderedlist>
- <para>After the <literal>writeconf</literal> command is run, the configuration logs are re-generated as servers restart.</para>
+ <para>After the <literal>tunefs.lustre --writeconf</literal> command is
+ run, the configuration logs are re-generated as servers connect to the
+ MGS.</para>
</section>
- <section xml:id="dbdoclet.50438199_31353">
+ <section xml:id="lustremaint.changingservernid">
<title><indexterm><primary>maintenance</primary><secondary>changing a NID</secondary></indexterm>
Changing a Server NID</title>
- <para>In Lustre 2.3 or earlier, the <literal>tunefs.lustre --writeconf</literal> command is used to rewrite all of the configuration files.</para>
- <para condition="l24">If you need to change the NID on the MDT or OST, a new <literal>replace_nids</literal> command was added in Lustre 2.4 to simplify this process.
- The <literal>replace_nids</literal> command differs from <literal>tunefs.lustre --writeconf</literal> in that it does not erase the entire configuration log, precluding the need the need to execute the <literal>writeconf</literal> command on all servers and re-specify all permanent parameter settings. However, the <literal>writeconf</literal> command can still be used if desired.</para>
+ <para>In order to totally rewrite the Lustre configuration, the
+ <literal>tunefs.lustre --writeconf</literal> command is used to
+ rewrite all of the configuration files.</para>
+ <para>If you need to change only the NID of the MDT or OST, the
+ <literal>replace_nids</literal> command can simplify this process.
+ The <literal>replace_nids</literal> command differs from
+ <literal>tunefs.lustre --writeconf</literal> in that it does not
+ erase the entire configuration log, precluding the need the need to
+ execute the <literal>writeconf</literal> command on all servers and
+ re-specify all permanent parameter settings. However, the
+ <literal>writeconf</literal> command can still be used if desired.
+ </para>
<para>Change a server NID in these situations:</para>
<itemizedlist>
<listitem>
<para>To change a server NID:</para>
<orderedlist>
<listitem>
- <para>Update the LNET configuration in the <literal>/etc/modprobe.conf</literal> file so the list of server NIDs is correct. Use <literal>lctl list_nids</literal> to view the list of server NIDS.</para>
- <para>The <literal>lctl list_nids</literal> command indicates which network(s) are configured to work with Lustre.</para>
+ <para>Update the LNet configuration in the <literal>/etc/modprobe.conf</literal> file so the list of server NIDs is correct. Use <literal>lctl list_nids</literal> to view the list of server NIDS.</para>
+ <para>The <literal>lctl list_nids</literal> command indicates which network(s) are
+ configured to work with the Lustre file system.</para>
</listitem>
<listitem>
<para>Shut down the file system in this order:</para>
<listitem>
<para>Run the <literal>replace_nids</literal> command on the MGS:</para>
<screen>lctl replace_nids <replaceable>devicename</replaceable> <replaceable>nid1</replaceable>[,nid2,nid3 ...]</screen>
- <para>where <replaceable>devicename</replaceable> is the Lustre target name, e.g. <literal>myfs-OST0013</literal></para>
+ <para>where <replaceable>devicename</replaceable> is the Lustre target name, e.g.
+ <literal>testfs-OST0013</literal></para>
+ </listitem>
+ <listitem>
+ <para>If the MGS and MDS share a partition, stop the MGS:</para>
+ <screen>umount <replaceable>mount_point</replaceable></screen>
</listitem>
- <listitem>
- <para>If the MGS and MDS share a partition, stop the MGS:</para>
- <screen>umount <replaceable>mount_point</replaceable></screen>
- </listitem>
</orderedlist>
- <note><para>The <literal>replace_nids</literal> command also cleans all old, invalidated records out of the configuration log, while preserving all other current settings.</para></note>
- <note><para>The previous configuration log is backed up on the MGS disk with the suffix <literal>'.bak'</literal>.</para></note>
+ <note><para>The <literal>replace_nids</literal> command also cleans
+ all old, invalidated records out of the configuration log, while
+ preserving all other current settings.</para></note>
+ <note><para>The previous configuration log is backed up on the MGS
+ disk with the suffix <literal>'.bak'</literal>.</para></note>
</section>
- <section xml:id="dbdoclet.addingamdt" condition='l24'>
- <title><indexterm><primary>maintenance</primary><secondary>adding an MDT</secondary></indexterm>Adding a new MDT to a Lustre file system</title>
- <para>Additional MDTs can be added to serve one or more remote sub-directories within the filesystem. It is possible to have multiple remote sub-directories reference the same MDT. However, the root directory will always be located on MDT0. To add a new MDT into the file system:</para>
+ <section xml:id="lustremaint.clear_conf" condition="l2B">
+ <title><indexterm>
+ <primary>maintenance</primary>
+ <secondary>Clearing a config</secondary>
+ </indexterm> Clearing configuration</title>
+ <para>
+ This command runs on MGS node having the MGS device mounted with
+ <literal>-o nosvc.</literal> It cleans up configuration files
+ stored in the CONFIGS/ directory of any records marked SKIP.
+ If the device name is given, then the specific logs for that
+ filesystem (e.g. testfs-MDT0000) are processed. Otherwise, if a
+ filesystem name is given then all configuration files are cleared.
+ The previous configuration log is backed up on the MGS disk with
+ the suffix 'config.timestamp.bak'. Eg: Lustre-MDT0000-1476454535.bak.
+ </para>
+ <para> To clear a configuration:</para>
+ <orderedlist>
+ <listitem>
+ <para>Shut down the file system in this order:</para>
+ <orderedlist>
+ <listitem>
+ <para>Unmount the clients.</para>
+ </listitem>
+ <listitem>
+ <para>Unmount the MDT.</para>
+ </listitem>
+ <listitem>
+ <para>Unmount all OSTs.</para>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para>
+ If the MGS and MDS share a partition, start the MGS only
+ using "nosvc" option.
+ </para>
+ <screen>mount -t lustre <replaceable>MDT partition</replaceable> -o nosvc <replaceable>mount_point</replaceable></screen>
+ </listitem>
+ <listitem>
+ <para>Run the <literal>clear_conf</literal> command on the MGS:
+ </para>
+ <screen>lctl clear_conf <replaceable>config</replaceable></screen>
+ <para>
+ Example: To clear the configuration for
+ <literal>MDT0000</literal> on a filesystem named
+ <literal>testfs</literal>
+ </para>
+ <screen>mgs# lctl clear_conf testfs-MDT0000</screen>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section xml:id="lustremaint.adding_new_mdt">
+ <title><indexterm>
+ <primary>maintenance</primary>
+ <secondary>adding an MDT</secondary>
+ </indexterm>Adding a New MDT to a Lustre File System</title>
+ <para>Additional MDTs can be added using the DNE feature to serve one
+ or more remote sub-directories within a filesystem, in order to
+ increase the total number of files that can be created in the
+ filesystem, to increase aggregate metadata performance, or to isolate
+ user or application workloads from other users of the filesystem. It
+ is possible to have multiple remote sub-directories reference the
+ same MDT. However, the root directory will always be located on
+ MDT0000. To add a new MDT into the file system:</para>
<orderedlist>
<listitem>
- <para>Discover the maximum MDT index. Each MDTs must have unique index.</para>
- <screen>
+ <para>Discover the maximum MDT index. Each MDT must have unique index.</para>
+<screen>
client$ lctl dl | grep mdc
-36 UP mdc lustre-MDT0000-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
-37 UP mdc lustre-MDT0001-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
-38 UP mdc lustre-MDT0002-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
-39 UP mdc lustre-MDT0003-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
- </screen>
+36 UP mdc testfs-MDT0000-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
+37 UP mdc testfs-MDT0001-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
+38 UP mdc testfs-MDT0002-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
+39 UP mdc testfs-MDT0003-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
+</screen>
</listitem>
<listitem>
- <para>Add the new block device as a new MDT at the next available index. In this example, the next available index is 4.</para>
- <screen>
-mds# mkfs.lustre --reformat --fsname=<replaceable>filesystem_name</replaceable> --mdt --mgsnode=<replaceable>mgsnode</replaceable> --index 4 <replaceable>/dev/mdt4_device</replaceable>
- </screen>
+ <para>Add the new block device as a new MDT at the next available
+ index. In this example, the next available index is 4.</para>
+<screen>
+mds# mkfs.lustre --reformat --fsname=<replaceable>testfs</replaceable> --mdt --mgsnode=<replaceable>mgsnode</replaceable> --index 4 <replaceable>/dev/mdt4_device</replaceable>
+</screen>
</listitem>
<listitem>
-
<para>Mount the MDTs.</para>
- <screen>
+ <para>Mount the MDTs.</para>
+<screen>
mds# mount –t lustre <replaceable>/dev/mdt4_blockdevice</replaceable> /mnt/mdt4
- </screen>
+</screen>
+ </listitem>
+ <listitem>
+ <para>In order to start creating new files and directories on the
+ new MDT(s) they need to be attached into the namespace at one or
+ more subdirectories using the <literal>lfs mkdir</literal> command.
+ All files and directories below those created with
+ <literal>lfs mkdir</literal> will also be created on the same MDT
+ unless otherwise specified.
+ </para>
+<screen>
+client# lfs mkdir -i 3 /mnt/testfs/new_dir_on_mdt3
+client# lfs mkdir -i 4 /mnt/testfs/new_dir_on_mdt4
+client# lfs mkdir -c 4 /mnt/testfs/project/new_large_dir_striped_over_4_mdts
+</screen>
</listitem>
</orderedlist>
- </section>
- <section xml:id="dbdoclet.50438199_22527">
+ </section>
+ <section xml:id="lustremaint.adding_new_ost">
<title><indexterm><primary>maintenance</primary><secondary>adding a OST</secondary></indexterm>
Adding a New OST to a Lustre File System</title>
- <para>To add an OST to existing Lustre file system:</para>
+ <para>A new OST can be added to existing Lustre file system on either
+ an existing OSS node or on a new OSS node. In order to keep client IO
+ load balanced across OSS nodes for maximum aggregate performance, it is
+ not recommended to configure different numbers of OSTs to each OSS node.
+ </para>
<orderedlist>
<listitem>
- <para> Add a new OST by passing on the following commands, run:</para>
- <screen>oss# mkfs.lustre --fsname=spfs --mgsnode=mds16@tcp0 --ost --index=12 /dev/sda
-oss# mkdir -p /mnt/test/ost12
-oss# mount -t lustre /dev/sda /mnt/test/ost12</screen>
+ <para> Add a new OST by using <literal>mkfs.lustre</literal> as when
+ the filesystem was first formatted, see
+ <xref linkend="format_ost" /> for details. Each new OST
+ must have a unique index number, use <literal>lctl dl</literal> to
+ see a list of all OSTs. For example, to add a new OST at index 12
+ to the <literal>testfs</literal> filesystem run following commands
+ should be run on the OSS:</para>
+ <screen>oss# mkfs.lustre --fsname=testfs --mgsnode=mds16@tcp0 --ost --index=12 /dev/sda
+oss# mkdir -p /mnt/testfs/ost12
+oss# mount -t lustre /dev/sda /mnt/testfs/ost12</screen>
</listitem>
<listitem>
- <para> Migrate the data (possibly).</para>
- <para>The file system is quite unbalanced when new empty OSTs are added. New file creations are automatically balanced. If this is a scratch file system or files are pruned at a regular interval, then no further work may be needed.</para>
- <para>New files being created will preferentially be placed on the empty OST. As old files are deleted, they will release space on the old OST.</para>
- <para>Files existing prior to the expansion can optionally be rebalanced with an in-place copy, which can be done with a simple script. The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
- <para>For example, to rebalance all files within <literal>/mnt/lustre/dir</literal>, enter:</para>
- <screen>client# lfs_migrate /mnt/lustre/file</screen>
- <para>To migrate files within the <literal>/test</literal> filesystem on <literal>OST0004</literal> that are larger than 4GB in size, enter:</para>
- <screen>client# lfs find /test -obd test-OST0004 -size +4G | lfs_migrate -y</screen>
- <para>See <xref linkend="dbdoclet.50438206_42260"/> for more details.</para>
+ <para>Balance OST space usage (possibly).</para>
+ <para>The file system can be quite unbalanced when new empty OSTs
+ are added to a relatively full filesystem. New file creations are
+ automatically balanced to favour the new OSTs. If this is a scratch
+ file system or files are pruned at regular intervals, then no further
+ work may be needed to balance the OST space usage as new files being
+ created will preferentially be placed on the less full OST(s). As old
+ files are deleted, they will release space on the old OST(s).</para>
+ <para>Files existing prior to the expansion can optionally be
+ rebalanced using the <literal>lfs_migrate</literal> utility.
+ This redistributes file data over the entire set of OSTs.</para>
+ <para>For example, to rebalance all files within the directory
+ <literal>/mnt/lustre/dir</literal>, enter:</para>
+ <screen>client# lfs_migrate /mnt/lustre/dir</screen>
+ <para>To migrate files within the <literal>/test</literal> file
+ system on <literal>OST0004</literal> that are larger than 4GB in
+ size to other OSTs, enter:</para>
+ <screen>client# lfs find /test --ost test-OST0004 -size +4G | lfs_migrate -y</screen>
+ <para>See <xref linkend="lfs_migrate"/> for details.</para>
</listitem>
</orderedlist>
</section>
- <section xml:id="dbdoclet.50438199_14978">
- <title><indexterm><primary>maintenance</primary><secondary>restoring a OST</secondary></indexterm>
- <indexterm><primary>maintenance</primary><secondary>removing a OST</secondary></indexterm>
-Removing and Restoring OSTs</title>
- <para>OSTs can be removed from and restored to a Lustre file system. Currently in Lustre, removing a OST means the OST is 'deactivated' in the file system, not permanently removed.</para>
- <note><para>A removed OST still appears in the file system; do not create a new OST with the same name.</para></note>
- <para>You may want to remove (deactivate) an OST and prevent new files from being written to it in several situations:</para>
+ <section xml:id="lustremaint.deactivating_mdt_ost">
+ <title><indexterm><primary>maintenance</primary><secondary>restoring an OST</secondary></indexterm>
+ <indexterm><primary>maintenance</primary><secondary>removing an OST</secondary></indexterm>
+Removing and Restoring MDTs and OSTs</title>
+ <para>OSTs and DNE MDTs can be removed from and restored to a Lustre
+ filesystem. Deactivating an OST means that it is temporarily or
+ permanently marked unavailable. Deactivating an OST on the MDS means
+ it will not try to allocate new objects there or perform OST recovery,
+ while deactivating an OST the client means it will not wait for OST
+ recovery if it cannot contact the OST and will instead return an IO
+ error to the application immediately if files on the OST are accessed.
+ An OST may be permanently deactivated from the file system,
+ depending on the situation and commands used.</para>
+ <note><para>A permanently deactivated MDT or OST still appears in the
+ filesystem configuration until the configuration is regenerated with
+ <literal>writeconf</literal> or it is replaced with a new MDT or OST
+ at the same index and permanently reactivated. A deactivated OST
+ will not be listed by <literal>lfs df</literal>.
+ </para></note>
+ <para>You may want to temporarily deactivate an OST on the MDS to
+ prevent new files from being written to it in several situations:</para>
<itemizedlist>
<listitem>
- <para>Hard drive has failed and a RAID resync/rebuild is underway</para>
+ <para>A hard drive has failed and a RAID resync/rebuild is underway,
+ though the OST can also be marked <emphasis>degraded</emphasis> by
+ the RAID system to avoid allocating new files on the slow OST which
+ can reduce performance, see <xref linkend='degraded_ost' />
+ for more details.
+ </para>
</listitem>
<listitem>
- <para>OST is nearing its space capacity</para>
+ <para>OST is nearing its space capacity, though the MDS will already
+ try to avoid allocating new files on overly-full OSTs if possible,
+ see <xref linkend='balancing_free_space' /> for details.
+ </para>
</listitem>
<listitem>
- <para>OST storage has failed permanently</para>
- </listitem>
+ <para>MDT/OST storage or MDS/OSS node has failed, and will not
+ be available for some time (or forever), but there is still a
+ desire to continue using the filesystem before it is repaired.</para>
+ </listitem>
</itemizedlist>
- <section condition="l24" xml:id="dbdoclet.rmremotedir">
- <title><indexterm><primary>maintenance</primary><secondary>removing a MDT</secondary></indexterm>Removing a MDT from the File System</title>
- <para>If the MDT is permanently inaccessible, <literal>lfs rmdir {directory}</literal> can be used to delete the directory entry. A normal <literal>rmdir</literal> will report an IO error due to the remote MDT being inactive. After the remote directory has been removed, the administrator should mark the MDT as permanently inactive with:</para>
-<screen>lctl conf_param {MDT name}.mdc.active=0
-</screen>
-<para>A user can identify the location of a remote sub-directory using the <literal>lfs</literal> utility. For example:</para>
-<screen>client$ lfs getstripe -M /mnt/lustre/remote_dir1
+ <section xml:id="lustremaint.rmremotedir">
+ <title><indexterm><primary>maintenance</primary><secondary>removing an MDT</secondary></indexterm>Removing an MDT from the File System</title>
+ <para>If the MDT is permanently inaccessible,
+ <literal>lfs rm_entry {directory}</literal> can be used to delete the
+ directory entry for the unavailable MDT. Using <literal>rmdir</literal>
+ would otherwise report an IO error due to the remote MDT being inactive.
+ Please note that if the MDT <emphasis>is</emphasis> available, standard
+ <literal>rm -r</literal> should be used to delete the remote directory.
+ After the remote directory has been removed, the administrator should
+ mark the MDT as permanently inactive with:</para>
+ <screen>lctl conf_param {MDT name}.mdc.active=0</screen>
+ <para>A user can identify which MDT holds a remote sub-directory using
+ the <literal>lfs</literal> utility. For example:</para>
+<screen>client$ lfs getstripe --mdt-index /mnt/lustre/remote_dir1
1
client$ mkdir /mnt/lustre/local_dir0
-client$ lfs getstripe -M /mnt/lustre/local_dir0
+client$ lfs getstripe --mdt-index /mnt/lustre/local_dir0
0
</screen>
- <para>The <literal>getstripe -M</literal> parameters return the index of the MDT that is serving the given directory.</para>
- </section>
- <section xml:id="dbdoclet.inactivemdt" condition='l24'>
+ <para>The <literal>lfs getstripe --mdt-index</literal> command
+ returns the index of the MDT that is serving the given directory.</para>
+ </section>
+ <section xml:id="lustremaint.inactivemdt">
<title>
<indexterm><primary>maintenance</primary></indexterm>
<indexterm><primary>maintenance</primary><secondary>inactive MDTs</secondary></indexterm>Working with Inactive MDTs</title>
- <para>Files located on or below an inactive MDT are inaccessible until the MDT is activated again. Clients accessing an inactive MDT will receive an EIO error.</para></section>
- <section remap="h3">
- <title><indexterm><primary>maintenance</primary><secondary>removing a OST</secondary></indexterm>
- Removing an OST from the File System</title>
- <para>When removing an OST, remember that the MDT does not communicate directly with OSTs. Rather, each OST has a corresponding OSC which communicates with the MDT. It is necessary to determine the device number of the OSC that corresponds to the OST. Then, you use this device number to deactivate the OSC on the MDT.</para>
- <para>To remove an OST from the file system:</para>
- <orderedlist>
- <listitem>
- <para>For the OST to be removed, determine the device number of the corresponding OSC on the MDT.</para>
- <orderedlist>
- <listitem>
- <para>List all OSCs on the node, along with their device numbers. Run:</para>
- <screen>lctl dl | grep osc</screen>
- <para>For example: <literal>lctl dl | grep</literal></para>
- <screen>11 UP osc lustre-OST-0000-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
-12 UP osc lustre-OST-0001-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
-13 IN osc lustre-OST-0000-osc lustre-MDT0000-mdtlov_UUID 5
-14 UP osc lustre-OST-0001-osc lustre-MDT0000-mdtlov_UUID 5</screen>
- </listitem>
- <listitem>
- <para>Determine the device number of the OSC that corresponds to the OST to be removed.</para>
- </listitem>
- </orderedlist>
- </listitem>
- <listitem>
- <para>Temporarily deactivate the OSC on the MDT. On the MDT, run: </para>
- <screen>mds# lctl --device <replaceable>lustre_devno</replaceable> deactivate</screen>
- <para>For example, based on the command output in Step 1, to deactivate device 13 (the MDT’s OSC for <literal>OST-0000</literal>), the command would be:
-</para>
- <screen>mds# lctl --device 13 deactivate</screen>
- <para>This marks the OST as inactive on the MDS, so no new objects are assigned to the OST. This does not prevent use of existing objects for reads or writes.
-</para>
- <note>
- <para>Do not deactivate the OST on the clients. Do so causes errors (EIOs), and the copy out to fail. </para>
- </note>
- <caution>
- <para>Do not use <literal>lctl conf_param</literal> to deactivate the OST. It permanently sets a parameter in the file system configuration.</para>
- </caution>
- </listitem>
- <listitem>
- <para>Discover all files that have objects residing on the deactivated OST. </para>
- <para>Depending on whether the deactivated OST is available or not, the data from that OST may be migrated to other OSTs, or may need to be restored from backup.
-</para>
- <orderedlist>
- <listitem>
- <para>If the OST is still online and available, find all files with objects on the deactivated OST, and copy them to other OSTs in the file system to:
-</para>
- <screen>client# lfs find --obd <replaceable>ost_name</replaceable> <replaceable>/mount/point</replaceable> | lfs_migrate -y</screen>
- </listitem>
- <listitem>
- <para>If the OST is no longer available, delete the files on that OST and restore them from backup:
- <screen>client# lfs find --obd <replaceable>ost_uuid</replaceable> -print0 <replaceable>/mount/point</replaceable> | \
- tee /tmp/files_to_restore | xargs -0 -n 1 unlink</screen>
- The list of files that need to be restored from backup is stored in <literal>/tmp/files_to_restore</literal>. Restoring these files is beyond the scope of this document.</para>
- </listitem>
- </orderedlist>
- </listitem>
- <listitem>
- <para>Deactivate the OST.</para>
- <orderedlist>
- <listitem>
- <para>To temporarily disable the deactivated OST, enter: <screen>[client]# lctl set_param osc.<replaceable>fsname</replaceable>-<replaceable>OSTnumber</replaceable>-*.active=0</screen>If there is expected to be a replacement OST in some short time (a few days), the OST can temporarily be deactivated on the clients: <note>
- <para>This setting is only temporary and will be reset if the clients or MDS are rebooted. It needs to be run on all clients.</para>
- </note></para>
- </listitem>
- </orderedlist>
- <para>If there is not expected to be a replacement for this OST in the near future, permanently deactivate the OST on all clients and the MDS: <screen>[mgs]# lctl conf_param <replaceable>ost_name</replaceable>.osc.active=0</screen></para>
- <note>
- <para>A removed OST still appears in the file system; do not create a new OST with the same name.</para>
- </note>
- </listitem>
- </orderedlist>
+ <para>Files located on or below an inactive MDT are inaccessible until
+ the MDT is activated again. Clients accessing an inactive MDT will receive
+ an EIO error.</para>
</section>
- <section remap="h3">
- <title><indexterm><primary>maintenance</primary><secondary>backing up OST config</secondary></indexterm>
-<indexterm><primary>backup</primary><secondary>OST config</secondary></indexterm>
-Backing Up OST Configuration Files</title>
- <para>If the OST device is still accessible, then the Lustre configuration files on the OST should be backed up and saved for future use in order to avoid difficulties when a replacement OST is returned to service. These files rarely change, so they can and should be backed up while the OST is functional and accessible. If the deactivated OST is still available to mount (i.e. has not permanently failed or is unmountable due to severe corruption), an effort should be made to preserve these files. </para>
- <orderedlist>
- <listitem>
- <para>Mount the OST filesystem.
- <screen>oss# mkdir -p /mnt/ost
-[oss]# mount -t ldiskfs <replaceable>/dev/ost_device</replaceable> /mnt/ost</screen>
+ <section remap="h3" xml:id="lustremaint.remove_ost">
+ <title><indexterm>
+ <primary>maintenance</primary>
+ <secondary>removing an OST</secondary>
+ </indexterm>Removing an OST from the File System</title>
+ <para>When deactivating an OST, note that the client and MDS each have
+ an OSC device that handles communication with the corresponding OST.
+ To remove an OST from the file system:</para>
+ <orderedlist>
+ <listitem>
+ <para>If the OST is functional, and there are files located on
+ the OST that need to be migrated off of the OST, the file creation
+ for that OST should be temporarily deactivated on the MDS (each MDS
+ if running with multiple MDS nodes in DNE mode).
</para>
- </listitem>
- <listitem>
- <para>Back up the OST configuration files.
- <screen>oss# tar cvf <replaceable>ost_name</replaceable>.tar -C /mnt/ost last_rcvd \
- CONFIGS/ O/0/LAST_ID</screen>
+ <orderedlist>
+ <listitem>
+ <para condition="l29">With Lustre 2.9 and later, the MDS should be
+ set to only disable file creation on that OST by setting
+ <literal>max_create_count</literal> to zero:
+ <screen>mds# lctl set_param osp.<replaceable>osc_name</replaceable>.max_create_count=0</screen>
+ This ensures that files deleted or migrated off of the OST
+ will have their corresponding OST objects destroyed, and the space
+ will be freed. For example, to disable <literal>OST0000</literal>
+ in the filesystem <literal>testfs</literal>, run:
+ <screen>mds# lctl set_param osp.testfs-OST0000-osc-MDT*.max_create_count=0</screen>
+ on each MDS in the <literal>testfs</literal> filesystem.</para>
+ </listitem>
+ <listitem>
+ <para>With older versions of Lustre, to deactivate the OSC on the
+ MDS node(s) use:
+ <screen>mds# lctl set_param osp.<replaceable>osc_name</replaceable>.active=0</screen>
+ This will prevent the MDS from attempting any communication with
+ that OST, including destroying objects located thereon. This is
+ fine if the OST will be removed permanently, if the OST is not
+ stable in operation, or if it is in a read-only state. Otherwise,
+ the free space and objects on the OST will not decrease when
+ files are deleted, and object destruction will be deferred until
+ the MDS reconnects to the OST.</para>
+ <para>For example, to deactivate <literal>OST0000</literal> in
+ the filesystem <literal>testfs</literal>, run:
+ <screen>mds# lctl set_param osp.testfs-OST0000-osc-MDT*.active=0</screen>
+ Deactivating the OST on the <emphasis>MDS</emphasis> does not
+ prevent use of existing objects for read/write by a client.</para>
+ <note>
+ <para>If migrating files from a working OST, do not deactivate
+ the OST on clients. This causes IO errors when accessing files
+ located there, and migrating files on the OST would fail.</para>
+ </note>
+ <caution>
+ <para>Do not use <literal>lctl set_param -P</literal> or
+ <literal>lctl conf_param</literal> to
+ deactivate the OST if it is still working, as this immediately
+ and permanently deactivates it in the file system configuration
+ on both the MDS and all clients.</para>
+ </caution>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para>Discover all files that have objects residing on the
+ deactivated OST. Depending on whether the deactivated OST is
+ available or not, the data from that OST may be migrated to
+ other OSTs, or may need to be restored from backup.</para>
+ <orderedlist>
+ <listitem>
+ <para>If the OST is still online and available, find all
+ files with objects on the deactivated OST, and copy them
+ to other OSTs in the file system to: </para>
+ <screen>client# lfs find --ost <replaceable>ost_name</replaceable> <replaceable>/mount/point</replaceable> | lfs_migrate -y</screen>
+ <para>Note that if multiple OSTs are being deactivated at one
+ time, the <literal>lfs find</literal> command can take multiple
+ <literal>--ost</literal> arguments, and will return files that
+ are located on <emphasis>any</emphasis> of the specified OSTs.
+ </para>
+ </listitem>
+ <listitem>
+ <para>If the OST is no longer available, delete the files
+ on that OST and restore them from backup:
+ <screen>client# lfs find --ost <replaceable>ost_uuid</replaceable> -print0 <replaceable>/mount/point</replaceable> |
+ tee /tmp/files_to_restore | xargs -0 -n 1 unlink</screen>
+ The list of files that need to be restored from backup is
+ stored in <literal>/tmp/files_to_restore</literal>. Restoring
+ these files is beyond the scope of this document.</para>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para>Deactivate the OST.</para>
+ <orderedlist>
+ <listitem>
+ <para>
+ If there is expected to be a replacement OST in some short
+ time (a few days), the OST can temporarily be deactivated on
+ the clients using:
+ <screen>client# lctl set_param osc.<replaceable>fsname</replaceable>-OST<replaceable>number</replaceable>-*.active=0</screen>
+ <note><para>This setting is only temporary and will be reset
+ if the clients are remounted or rebooted. It needs to be run
+ on all clients.</para>
+ </note>
+ </para>
+ </listitem>
+ <listitem>
+ <para>If there is not expected to be a replacement for this OST in
+ the near future, permanently deactivate it on all clients and
+ the MDS by running the following command on the MGS:
+ <screen>mgs# lctl conf_param <replaceable>ost_name</replaceable>.osc.active=0</screen></para>
+ <note><para>A deactivated OST still appears in the file system
+ configuration, though a replacement OST can be created that
+ re-uses the same OST index with the
+ <literal>mkfs.lustre --replace</literal> option, see
+ <xref linkend="lustremaint.restore_ost"/>.
+ </para></note>
+ <para condition="l2G">
+ In Lustre 2.16 and later, it is possible to run the command
+ "<literal>lctl del_ost --target <replaceable>fsname</replaceable>-OST<replaceable>xxxx</replaceable></literal>"
+ on the MGS to totally remove an OST from the MGS configuration
+ logs. This will cancel the configuration logs for that OST in
+ the client and MDT configuration logs for the named filesystem.
+ This permanently removes the configuration records for that OST
+ from the filesystem, so that it will not be visible on later
+ client and MDT mounts, and should only be run after earlier
+ steps to migrate files off the OST.
</para>
- </listitem>
- <listitem>
- <para>
- Unmount the OST filesystem.
- <screen>oss# umount /mnt/ost</screen>
+ <para>If the <literal>del_ost</literal> command is not available,
+ the OST configuration records should be found in the startup
+ logs by running the command
+ "<literal>lctl --device MGS llog_print <replaceable>fsname</replaceable>-client</literal>"
+ on the MGS (and also
+ "<literal>... <replaceable>$fsname</replaceable>-MDT<replaceable>xxxx</replaceable></literal>"
+ for all the MDTs) to list all <literal>attach</literal>,
+ <literal>setup</literal>, <literal>add_osc</literal>,
+ <literal>add_pool</literal>, and other records related to the
+ removed OST(s). Once the <literal>index</literal> value is
+ known for each configuration record, the command
+ "<literal>lctl --device MGS llog_cancel <replaceable>llog_name</replaceable> -i <replaceable>index</replaceable> </literal>"
+ will drop that record from the configuration log
+ <replaceable>llog_name</replaceable>. This is needed for each
+ of <literal><replaceable>fsname</replaceable>-client</literal>
+ and <literal><replaceable>fsname</replaceable>-MDTxxxx</literal>
+ configuration logs so that new mounts will no longer process it.
+ If a whole OSS is being removed, the<literal>add_uuid</literal>
+ records for the OSS should similarly be canceled.
+ <screen>
+mgs# lctl --device MGS llog_print testfs-client | egrep "192.168.10.99@tcp|OST0003"
+- { index: 135, event: add_uuid, nid: 192.168.10.99@tcp(0x20000c0a80a63), node: 192.168.10.99@tcp }
+- { index: 136, event: attach, device: testfs-OST0003-osc, type: osc, UUID: testfs-clilov_UUID }
+- { index: 137, event: setup, device: testfs-OST0003-osc, UUID: testfs-OST0003_UUID, node: 192.168.10.99@tcp }
+- { index: 138, event: add_osc, device: testfs-clilov, ost: testfs-OST0003_UUID, index: 3, gen: 1 }
+mgs# lctl --device MGS llog_cancel testfs-client -i 138
+mgs# lctl --device MGS llog_cancel testfs-client -i 137
+mgs# lctl --device MGS llog_cancel testfs-client -i 136
+ </screen>
</para>
- </listitem>
- </orderedlist>
- </section>
- <section>
- <title><indexterm><primary>maintenance</primary><secondary>restoring OST config</secondary></indexterm>
-<indexterm><primary>backup</primary><secondary>restoring OST config</secondary></indexterm>
-Restoring OST Configuration Files</title>
- <para>If the original OST is still available, it is best to follow the OST backup and restore procedure given in either <xref linkend="dbdoclet.50438207_71633"/>, or <xref linkend="dbdoclet.50438207_21638"/> and <xref linkend="dbdoclet.50438207_22325"/>.
- </para>
- <para>To replace an OST that was removed from service due to corruption or hardware failure, the file system needs to be formatted for Lustre, and the Lustre configuration should be restored, if available.
-
- </para>
- <para>If the OST configuration files were not backed up, due to the OST file system being completely inaccessible, it is still possible to replace the failed OST with a new one at the same OST index. </para>
- <orderedlist>
- <listitem>
- <para>
- Format the OST file system.
- <screen>oss# mkfs.lustre --ost --index=<replaceable>old_ost_index</replaceable> <replaceable>other_options</replaceable> \
- <replaceable>/dev/new_ost_dev</replaceable></screen>
- </para>
- </listitem>
- <listitem>
- <para>
- Mount the OST filesystem.
+ </listitem>
+ </orderedlist>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section remap="h3" xml:id="lustremaint.ydg_pgt_tl">
+ <title><indexterm>
+ <primary>maintenance</primary>
+ <secondary>backing up OST config</secondary>
+ </indexterm>
+ <indexterm>
+ <primary>backup</primary>
+ <secondary>OST config</secondary>
+ </indexterm> Backing Up OST Configuration Files</title>
+ <para>If the OST device is still accessible, then the Lustre
+ configuration files on the OST should be backed up and saved for
+ future use in order to avoid difficulties when a replacement OST is
+ returned to service. These files rarely change, so they can and
+ should be backed up while the OST is functional and accessible. If
+ the deactivated OST is still available to mount (i.e. has not
+ permanently failed or is unmountable due to severe corruption), an
+ effort should be made to preserve these files. </para>
+ <orderedlist>
+ <listitem>
+ <para>Mount the OST file system.
+ <screen>oss# mkdir -p /mnt/ost
+oss# mount -t ldiskfs <replaceable>/dev/ost_device</replaceable> /mnt/ost</screen>
+ </para>
+ </listitem>
+ <listitem>
+ <para>Back up the OST configuration files.
+ <screen>oss# tar cvf <replaceable>ost_name</replaceable>.tar -C /mnt/ost last_rcvd \
+ CONFIGS/ O/0/LAST_ID</screen>
+ </para>
+ </listitem>
+ <listitem>
+ <para> Unmount the OST file system. <screen>oss# umount /mnt/ost</screen>
+ </para>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section xml:id="lustremaint.restore_ost">
+ <title><indexterm>
+ <primary>maintenance</primary>
+ <secondary>restoring OST config</secondary>
+ </indexterm>
+ <indexterm>
+ <primary>backup</primary>
+ <secondary>restoring OST config</secondary>
+ </indexterm> Restoring OST Configuration Files</title>
+ <para>If the original OST is still available, it is best to follow the
+ OST backup and restore procedure given in either
+ <xref linkend="backup_device"/>, or
+ <xref linkend="backup_fs_level"/> and
+ <xref linkend="backup_fs_level.restore"/>.</para>
+ <para>To replace an OST that was removed from service due to corruption
+ or hardware failure, the replacement OST needs to be formatted using
+ <literal>mkfs.lustre</literal>, and the Lustre file system configuration
+ should be restored, if available. Any objects stored on the OST will
+ be permanently lost, and files using the OST should be deleted and/or
+ restored from backup.</para>
+ <para condition="l25">With Lustre 2.5 and later, it is possible to
+ replace an OST to the same index without restoring the configuration
+ files, using the <literal>--replace</literal> option at format time.
+ <screen>oss# mkfs.lustre --ost --reformat --replace --index=<replaceable>old_ost_index</replaceable> \
+ <replaceable>other_options</replaceable> <replaceable>/dev/new_ost_dev</replaceable></screen>
+ The MDS and OSS will negotiate the <literal>LAST_ID</literal> value
+ for the replacement OST.
+ </para>
+ <para>If the OST configuration files were not backed up, due to the
+ OST file system being completely inaccessible, it is still possible to
+ replace the failed OST with a new one at the same OST index. </para>
+ <orderedlist>
+ <listitem>
+ <para>For older versions, format the OST file system without the
+ <literal>--replace</literal> option and restore the saved
+ configuration:
+ <screen>oss# mkfs.lustre --ost --reformat --index=<replaceable>old_ost_index</replaceable> \
+ <replaceable>other_options</replaceable> <replaceable>/dev/new_ost_dev</replaceable></screen>
+ </para>
+ </listitem>
+ <listitem>
+ <para> Mount the OST file system.
<screen>oss# mkdir /mnt/ost
oss# mount -t ldiskfs <replaceable>/dev/new_ost_dev</replaceable> <replaceable>/mnt/ost</replaceable></screen>
- </para>
- </listitem>
- <listitem>
- <para>Restore the OST configuration files, if available.
+ </para>
+ </listitem>
+ <listitem>
+ <para>Restore the OST configuration files, if available.
<screen>oss# tar xvf <replaceable>ost_name</replaceable>.tar -C /mnt/ost</screen></para>
- </listitem>
- <listitem>
- <para>Recreate the OST configuration files, if unavailable.
- </para>
- <para>Follow the procedure in <xref linkend="dbdoclet.50438198_69657"/> to recreate the
- LAST_ID file for this OST index. The <literal>last_rcvd</literal> file will be recreated
- when the OST is first mounted using the default parameters, which are normally correct
- for all file systems. The <literal>CONFIGS/mountdata</literal> file is created by
- <literal>mkfs.lustre</literal> at format time, but has flags set that request it to
- register itself with the MGS. It is possible to copy these flags from another working
- OST (which should be the same):
- <screen>oss1# debugfs -c -R "dump CONFIGS/mountdata /tmp/ldd" <replaceable>/dev/other_osdev</replaceable>
-oss1# scp /tmp/ldd oss0:/tmp/ldd
-oss0# dd if=/tmp/ldd of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 skip=5 conv=notrunc</screen></para>
- </listitem>
- <listitem>
- <para>
- Unmount the OST filesystem.
- <screen>oss# umount /mnt/ost</screen>
- </para>
- </listitem>
- </orderedlist>
- </section>
- <section>
- <title><indexterm><primary>maintenance</primary><secondary>reintroducing an OSTs</secondary></indexterm>
-Returning a Deactivated OST to Service</title>
- <para> If the OST was permanently deactivated, it needs to be reactivated in the MGS configuration. <screen>mgs# lctl conf_param <replaceable>ost_name</replaceable>.osc.active=1</screen> If the OST was temporarily deactivated, it needs to be reactivated on the MDS and clients. <screen>mds# lctl --device <replaceable>lustre_devno</replaceable> activate
- client# lctl set_param osc.<replaceable>fsname</replaceable>-<replaceable>OSTnumber</replaceable>-*.active=0</screen></para>
- </section>
+ </listitem>
+ <listitem>
+ <para>Recreate the OST configuration files, if unavailable. </para>
+ <para>Follow the procedure in
+ <xref linkend="repair_ost_lastid"/> to recreate the LAST_ID
+ file for this OST index. The <literal>last_rcvd</literal> file
+ will be recreated when the OST is first mounted using the default
+ parameters, which are normally correct for all file systems. The
+ <literal>CONFIGS/mountdata</literal> file is created by
+ <literal>mkfs.lustre</literal> at format time, but has flags set
+ that request it to register itself with the MGS. It is possible to
+ copy the flags from another working OST (which should be the same):
+ <screen>oss1# debugfs -c -R "dump CONFIGS/mountdata /tmp" <replaceable>/dev/other_osdev</replaceable>
+oss1# scp /tmp/mountdata oss0:/tmp/mountdata
+oss0# dd if=/tmp/mountdata of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 skip=5 conv=notrunc</screen></para>
+ </listitem>
+ <listitem>
+ <para> Unmount the OST file system.
+ <screen>oss# umount /mnt/ost</screen>
+ </para>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section xml:id="lustremaint.ucf_qgt_tl">
+ <title><indexterm>
+ <primary>maintenance</primary>
+ <secondary>reintroducing an OSTs</secondary>
+ </indexterm>Returning a Deactivated OST to Service</title>
+ <para>If the OST was permanently deactivated, it needs to be
+ reactivated in the MGS configuration.
+ <screen>mgs# lctl conf_param <replaceable>ost_name</replaceable>.osc.active=1</screen>
+ If the OST was temporarily deactivated, it needs to be reactivated on
+ the MDS and clients.
+ <screen>mds# lctl set_param osp.<replaceable>fsname</replaceable>-OST<replaceable>number</replaceable>-*.active=1
+client# lctl set_param osc.<replaceable>fsname</replaceable>-OST<replaceable>number</replaceable>-*.active=1</screen></para>
</section>
- <section xml:id="dbdoclet.50438199_77819">
+ </section>
+ <section xml:id="lustremaint.abortRecovery">
<title><indexterm><primary>maintenance</primary><secondary>aborting recovery</secondary></indexterm>
<indexterm><primary>backup</primary><secondary>aborting recovery</secondary></indexterm>
Aborting Recovery</title>
<para>The recovery process is blocked until all OSTs are available. </para>
</note>
</section>
- <section xml:id="dbdoclet.50438199_12607">
+ <section xml:id="lustremaint.determineOST">
<title><indexterm><primary>maintenance</primary><secondary>identifying OST host</secondary></indexterm>
Determining Which Machine is Serving an OST </title>
- <para>In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the NID that is serving a specific OST, run one of the following commands on a client (you do not need to be a root user): <screen>client$ lctl get_param osc.<replaceable>fsname</replaceable>-<replaceable>OSTnumber</replaceable>*.ost_conn_uuid</screen>For example: <screen>client$ lctl get_param osc.*-OST0000*.ost_conn_uuid
-osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen>- OR - <screen>client$ lctl get_param osc.*.ost_conn_uuid
-osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-osc.lustre-OST0001-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-osc.lustre-OST0002-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-osc.lustre-OST0003-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-osc.lustre-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen></para>
+ <para>In the course of administering a Lustre file system, you may need to determine which
+ machine is serving a specific OST. It is not as simple as identifying the machine’s IP
+ address, as IP is only one of several networking protocols that the Lustre software uses and,
+ as such, LNet does not use IP addresses as node identifiers, but NIDs instead. To identify the
+ NID that is serving a specific OST, run one of the following commands on a client (you do not
+ need to be a root user):
+ <screen>client$ lctl get_param osc.<replaceable>fsname</replaceable>-OST<replaceable>number</replaceable>*.ost_conn_uuid</screen>
+ For example:
+ <screen>client$ lctl get_param osc.*-OST0000*.ost_conn_uuid
+osc.testfs-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen>-
+ OR -
+ <screen>client$ lctl get_param osc.*.ost_conn_uuid
+osc.testfs-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
+osc.testfs-OST0001-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
+osc.testfs-OST0002-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
+osc.testfs-OST0003-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
+osc.testfs-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen></para>
</section>
- <section xml:id="dbdoclet.50438199_62333">
+ <section xml:id="lustremaint.ChangeAddrFailoverNode">
<title><indexterm><primary>maintenance</primary><secondary>changing failover node address</secondary></indexterm>
Changing the Address of a Failover Node</title>
- <para>To change the address of a failover node (e.g, to use node X instead of node Y), run this command on the OSS/OST partition:
- <screen>oss# tunefs.lustre --erase-params --failnode=<replaceable>NID</replaceable> <replaceable>/dev/ost_device</replaceable></screen>
- or
- <screen>oss# tunefs.lustre --erase-params --servicenode=<replaceable>NID</replaceable> <replaceable>/dev/ost_device</replaceable></screen>
- </para>
+ <para>To change the address of a failover node (e.g, to use node X instead of node Y), run
+ this command on the OSS/OST partition (depending on which option was used to originally
+ identify the NID):
+ <screen>oss# tunefs.lustre --erase-params --servicenode=<replaceable>NID</replaceable> <replaceable>/dev/ost_device</replaceable></screen>
+ or
+ <screen>oss# tunefs.lustre --erase-params --failnode=<replaceable>NID</replaceable> <replaceable>/dev/ost_device</replaceable></screen>
+ For more information about the <literal>--servicenode</literal> and
+ <literal>--failnode</literal> options, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="configuringfailover"/>.</para>
</section>
- <section xml:id="dbdoclet.50438199_62545">
- <title><indexterm><primary>maintenance</primary><secondary>separate a combined MGS/MDT</secondary></indexterm>
-Separate a combined MGS/MDT</title>
- <para>These instructions assume the MGS node will be the same as the MDS node. For instructions on how to move MGS to a different node, see <xref linkend="dbdoclet.50438199_31353"/>.</para>
- <para>These instructions are for doing the split without shutting down other servers and clients.</para>
+ <section xml:id="lustremaint.seperateCombinedMGSMDT">
+ <title><indexterm><primary>maintenance</primary><secondary>separate a
+ combined MGS/MDT</secondary></indexterm>
+ Separate a combined MGS/MDT</title>
+ <para>These instructions assume the MGS node will be the same as the MDS
+ node. For instructions on how to move MGS to a different node, see
+ <xref linkend="lustremaint.changingservernid"/>.</para>
+ <para>These instructions are for doing the split without shutting down
+ other servers and clients.</para>
<orderedlist>
<listitem>
<para>Stop the MDS.</para>
<screen>mds# cp -r <replaceable>/mdt_mount_point</replaceable>/CONFIGS/<replaceable>filesystem_name</replaceable>-* <replaceable>/mgs_mount_point</replaceable>/CONFIGS/. </screen>
<screen>mds# umount <replaceable>/mgs_mount_point</replaceable></screen>
<screen>mds# umount <replaceable>/mdt_mount_point</replaceable></screen>
- <para>See <xref linkend="dbdoclet.50438199_54623"/> for alternative method.</para>
+ <para>See <xref linkend="lustremaint.regenerateConfigLogs"/> for alternative method.</para>
</listitem>
<listitem>
<para>Start the MGS.</para>
<screen>mgs# mount -t lustre <replaceable>/dev/mgs_device</replaceable> <replaceable>/mgs_mount_point</replaceable></screen>
- <para>Check to make sure it knows about all your filesystem</para>
- <screen>cat /proc/fs/lustre/mgs/MGS/filesystems</screen>
+ <para>Check to make sure it knows about all your file system</para>
+ <screen>mgs:/root# lctl get_param mgs.MGS.filesystems</screen>
</listitem>
<listitem>
<para>Remove the MGS option from the MDT, and set the new MGS nid.</para>
<listitem>
<para>Start the MDT.</para>
<screen>mds# mount -t lustre <replaceable>/dev/mdt_device /mdt_mount_point</replaceable></screen>
- <para>Check to make sure the MGS configuration look right</para>
- <screen>mds# cat /proc/fs/lustre/mgs/MGS/live/<replaceable>filesystem_name</replaceable></screen>
+ <para>Check to make sure the MGS configuration looks right:</para>
+ <screen>mgs# lctl get_param mgs.MGS.live.<replaceable>filesystem_name</replaceable></screen>
</listitem>
</orderedlist>
</section>
+ <section xml:id="lustremaint.setMDTReadonly" condition="l2D">
+ <title><indexterm><primary>maintenance</primary>
+ <secondary>set an MDT to readonly</secondary></indexterm>
+ Set an MDT to read-only</title>
+ <para>It is sometimes desirable to be able to mark the filesystem
+ read-only directly on the server, rather than remounting the clients and
+ setting the option there. This can be useful if there is a rogue client
+ that is deleting files, or when decommissioning a system to prevent
+ already-mounted clients from modifying it anymore.</para>
+ <para>Set the <literal>mdt.*.readonly</literal> parameter to
+ <literal>1</literal> to immediately set the MDT to read-only. All future
+ MDT access will immediately return a "Read-only file system" error
+ (<literal>EROFS</literal>) until the parameter is set to
+ <literal>0</literal> again.</para>
+ <para>Example of setting the <literal>readonly</literal> parameter to
+ <literal>1</literal>, verifying the current setting, accessing from a
+ client, and setting the parameter back to <literal>0</literal>:</para>
+ <screen>mds# lctl set_param mdt.fs-MDT0000.readonly=1
+mdt.fs-MDT0000.readonly=1
+
+mds# lctl get_param mdt.fs-MDT0000.readonly
+mdt.fs-MDT0000.readonly=1
+
+client$ touch test_file
+touch: cannot touch ‘test_file’: Read-only file system
+
+mds# lctl set_param mdt.fs-MDT0000.readonly=0
+mdt.fs-MDT0000.readonly=0</screen>
+ </section>
+ <section xml:id="lustremaint.tunefallocate" condition="l2E">
+ <title><indexterm><primary>maintenance</primary>
+ <secondary>Tune fallocate</secondary></indexterm>
+ Tune Fallocate for ldiskfs</title>
+ <para>This section shows how to tune/enable/disable fallocate for
+ ldiskfs OSTs.</para>
+ <para>The default <literal>mode=0</literal> is the standard
+ "allocate unwritten extents" behavior used by ext4. This is by far the
+ fastest for space allocation, but requires the unwritten extents to be
+ split and/or zeroed when they are overwritten.</para>
+ <para> The OST fallocate <literal>mode=1</literal> can also be set to use
+ "zeroed extents", which may be handled by "WRITE SAME", "TRIM zeroes data",
+ or other low-level functionality in the underlying block device.</para>
+ <para><literal>mode=-1</literal> completely disables fallocate.</para>
+ <para>Example: To completely disable fallocate</para>
+ <screen>lctl set_param osd-ldiskfs.*.fallocate_zero_blocks=-1</screen>
+ <para>Example: To enable fallocate to use 'zeroed extents'</para>
+ <screen>lctl set_param osd-ldiskfs.*.fallocate_zero_blocks=1</screen>
+ </section>
</chapter>
+<!--
+ vim:expandtab:shiftwidth=2:tabstop=8:
+ -->
git://git.whamcloud.com / doc / manual.git / blobdiff