<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='configurationfilesmodulesparamters'>
+<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='configurationfilesmoduleparameters'>
<info>
- <title xml:id='configurationfilesmodulesparamters.title'>Configuration Files and Module Parameters</title>
+ <title xml:id='configurationfilesmoduleparameters.title'>Configuration Files and Module Parameters</title>
</info>
<para><anchor xml:id="dbdoclet.50438293_pgfId-1293284" xreflabel=""/>This section describes configuration files and module parameters and includes the following sections:</para>
<itemizedlist><listitem>
<?xml version="1.0" encoding="UTF-8"?>
<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='configuringfailover'>
<info>
- <title>Configuring Lustre Failover</title>
+ <title xml:id='configuringfailover.title'>Configuring Lustre Failover</title>
</info>
<para><anchor xml:id="dbdoclet.50438188_pgfId-1292189" xreflabel=""/>This chapter describes how to configure Lustre failover using the Heartbeat cluster infrastructure daemon. It includes:</para>
<para>By default, Lustre ignores the loopback (lo0) interface. Lustre does not ignore IP addresses aliased to the loopback. If you alias IP addresses to the loopback interface, you must specify all Lustre networks using the LNET networks parameter.</para></note>
<note>
- <para>If the server has multiple interfaces on the same subnet, the Linux kernel will send all traffic using the first configured interface. This is a limitation of Linux, not Lustre. In this case, network interface bonding should be used. For more information about network interface bonding, see <xref linkend='settinglupbonding'/>.</para>
+ <para>If the server has multiple interfaces on the same subnet, the Linux kernel will send all traffic using the first configured interface. This is a limitation of Linux, not Lustre. In this case, network interface bonding should be used. For more information about network interface bonding, see <xref linkend='settingupbonding'/>.</para>
</note>
</section>
<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='confirugrelustre'>
+<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='configuringlustre'>
<info>
- <title>Configuring Lustre</title>
+ <title xml:id='configuringlustre.title'>Configuring Lustre</title>
</info>
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438267_pgfId-1294840" xreflabel=""/>For information about configuring LNET, see <xref linkend='configuringlnet'/>. For information about testing LNET, see <xref linkend='lnetselftest'/>.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1294602" xreflabel=""/><emphasis>Run the benchmark script sgpdd_survey to determine baseline performance of your hardware.</emphasis> Benchmarking your hardware will simplify debugging performance issues that are unrelated to Lustre and ensure you are getting the best possible performance with your installation. For information about running sgpdd_survey, see <xref linkend='benchmarkingtets'/>.</para>
+ <para><anchor xml:id="dbdoclet.50438267_pgfId-1294602" xreflabel=""/><emphasis>Run the benchmark script sgpdd_survey to determine baseline performance of your hardware.</emphasis> Benchmarking your hardware will simplify debugging performance issues that are unrelated to Lustre and ensure you are getting the best possible performance with your installation. For information about running sgpdd_survey, see <xref linkend='benchmarkingtests'/>.</para>
</listitem>
</itemizedlist>
</orderedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293656" xreflabel=""/> 4. Create and mount ost2.</para>
+ <para><anchor xml:id="dbdoclet.50438267_pgfId-1293656" xreflabel=""/>Create and mount ost2.</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293645" xreflabel=""/>a. Create ost2. On oss2 node, run:</para>
+ <para><anchor xml:id="dbdoclet.50438267_pgfId-1293645" xreflabel=""/>Create ost2. On oss2 node, run:</para>
<screen><anchor xml:id="dbdoclet.50438267_pgfId-1291196" xreflabel=""/>[root@oss2 /]# mkfs.lustre --ost --fsname=temp --mgsnode=10.2.0.1@tcp0 /dev\
/sdd
</screen>
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291233" xreflabel=""/>b. Mount ost2 on the OSS on which it was created. On oss2 node, run:</para>
+ <para><anchor xml:id="dbdoclet.50438267_pgfId-1291233" xreflabel=""/>Mount ost2 on the OSS on which it was created. On oss2 node, run:</para>
<screen><anchor xml:id="dbdoclet.50438267_pgfId-1291234" xreflabel=""/>root@oss2 /] mount -t lustre /dev/sdd /mnt/ost2
</screen>
<para><anchor xml:id="dbdoclet.50438267_pgfId-1291235" xreflabel=""/>The command generates this output:</para>
</orderedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291243" xreflabel=""/> 5. Mount the Lustre file system on the client. On the client node, run:</para>
+ <para><anchor xml:id="dbdoclet.50438267_pgfId-1291243" xreflabel=""/>Mount the Lustre file system on the client. On the client node, run:</para>
<screen><anchor xml:id="dbdoclet.50438267_pgfId-1291244" xreflabel=""/>root@client1 /] mount -t lustre 10.2.0.1@tcp0:/temp /lustre
</screen>
<para><anchor xml:id="dbdoclet.50438267_pgfId-1291245" xreflabel=""/>This command generates this output:</para>
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291247" xreflabel=""/> 6. Verify that the file system started and is working by running the df, dd and ls commands on the client node.</para>
+ <para><anchor xml:id="dbdoclet.50438267_pgfId-1291247" xreflabel=""/>Verify that the file system started and is working by running the df, dd and ls commands on the client node.</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291248" xreflabel=""/>a. Run the lfsdf -h command:</para>
+ <para><anchor xml:id="dbdoclet.50438267_pgfId-1291248" xreflabel=""/>Run the lfsdf -h command:</para>
<screen><anchor xml:id="dbdoclet.50438267_pgfId-1291249" xreflabel=""/>[root@client1 /] lfs df -h
</screen>
<para><anchor xml:id="dbdoclet.50438267_pgfId-1293698" xreflabel=""/>The lfsdf-h command lists space usage per OST and the MDT in human-readable format. This command generates output similar to this:</para>
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293715" xreflabel=""/>b. Run the lfsdf-ih command.</para>
+ <para><anchor xml:id="dbdoclet.50438267_pgfId-1293715" xreflabel=""/>Run the lfsdf-ih command.</para>
<screen><anchor xml:id="dbdoclet.50438267_pgfId-1293716" xreflabel=""/>[root@client1 /] lfs df -ih
</screen>
<para><anchor xml:id="dbdoclet.50438267_pgfId-1293717" xreflabel=""/>The lfsdf-ih command lists inode usage per OST and the MDT. This command generates output similar to this:</para>
<para><anchor xml:id="dbdoclet.50438267_pgfId-1292458" xreflabel=""/>This section describes how to scale the Lustre file system or make configuration changes using the Lustre configuration utilities.</para>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438267_pgfId-1292441" xreflabel=""/>10.2.1 Scaling the <anchor xml:id="dbdoclet.50438267_marker-1292440" xreflabel=""/>Lustre File System</title>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1292442" xreflabel=""/>A Lustre file system can be scaled by adding OSTs or clients. For instructions on creating additional OSTs repeat <xref linkend="dbdoclet.50438267_81555"/>Step 3 and <xref linkend="dbdoclet.50438267_24062"/>Step 4 above. For mounting additional clients, repeat <xref linkend="dbdoclet.50438267_43365"/>Step 5 for each client.</para>
+ <para><anchor xml:id="dbdoclet.50438267_pgfId-1292442" xreflabel=""/>A Lustre file system can be scaled by adding OSTs or clients. For instructions on creating additional OSTs repeat <xref linkend="dbdoclet.50438267_pgfId-1291170"/>Step 3 and <xref linkend="dbdoclet.50438267_pgfId-1293955"/>Step 4 above. For mounting additional clients, repeat <xref linkend="dbdoclet.50438267_pgfId-1290934"/>Step 5 for each client.</para>
</section>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438267_pgfId-1292798" xreflabel=""/>10.2.2 <anchor xml:id="dbdoclet.50438267_50212" xreflabel=""/>Changing Striping Defaults</title>
</tbody>
</tgroup>
</table>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1292907" xreflabel=""/>Use the lfs setstripe command described in <xref linkend='managiningstripingfreespace'/> to change the file layout configuration.</para>
+ <para><anchor xml:id="dbdoclet.50438267_pgfId-1292907" xreflabel=""/>Use the lfs setstripe command described in <xref linkend='managingstripingfreespace'/> to change the file layout configuration.</para>
</section>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438267_pgfId-1292908" xreflabel=""/>10.2.3 Using the Lustre Configuration Utilities</title>
</listitem>
<listitem>
<para>
- <link linkend='benchmarkingtests' endterm='benchmarking.title'/>
+ <link linkend='benchmarkingtests' endterm='benchmarkingtests.title'/>
</para>
</listitem>
<listitem>
<?xml version="1.0" encoding="UTF-8"?>
<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='installinglustre'>
<info>
- <title>Installing the Lustre Software</title>
+ <title xml:id='installinglustre.title'>Installing the Lustre Software</title>
</info>
<para><anchor xml:id="dbdoclet.50438261_pgfId-1292576" xreflabel=""/>This chapter describes how to install the Lustre software. It includes:</para>
</listitem></orderedlist>
</listitem><listitem>
-4. Deactivate the OST.
+Deactivate the OST.
<orderedlist><listitem>
-a. To temporarily disable the deactivated OST, enter:
+To temporarily disable the deactivated OST, enter:
<screen>
[client]# lctl set_param osc.<fsname>-<OST name>-*.active=0
</screen>
If there is expected to be a replacement OST in some short time (a few days), the OST can temporarily be deactivated on the clients:
-Note - This setting is only temporary and will be reset if the clients or MDS are rebooted. It needs to be run on all clients.
+<note><para> This setting is only temporary and will be reset if the clients or MDS are rebooted. It needs to be run on all clients.</para></note>
</listitem><listitem>
b. To permanently disable the deactivated OST, enter:
</listitem></orderedlist>
If there is not expected to be a replacement for this OST in the near future, permanently deactivate the OST on all clients and the MDS:
-Note - A removed OST still appears in the file system; do not create a new OST with the same name.
-14.7.2 Backing Up OST Configuration Files
+<note><para>A removed OST still appears in the file system; do not create a new OST with the same name.</para></note>
+</listitem></orderedlist>
+
+</section>
+<section remap="h3">
+ <title>14.7.2 Backing Up OST Configuration Files</title>
If the OST device is still accessible, then the Lustre configuration files on the OST should be backed up and saved for future use in order to avoid difficulties when a replacement OST is returned to service. These files rarely change, so they can and should be backed up while the OST is functional and accessible. If the deactivated OST is still available to mount (i.e. has not permanently failed or is unmountable due to severe corruption), an effort should be made to preserve these files.
</screen>
</listitem></orderedlist>
-14.7.3 Restoring OST Configuration Files
+</section>
+<section>
+ <title>14.7.3 Restoring OST Configuration Files</title>
If the original OST is still available, it is best to follow the OST backup and restore procedure given in either Backing Up and Restoring an MDS or OST (Device Level), or Making a File-Level Backup of an OST File System and Restoring a File-Level Backup.
</screen>
</listitem></orderedlist>
-14.7.4 Returning a Deactivated OST to Service
+</section>
+<section>
+ <title>14.7.4 Returning a Deactivated OST to Service</title>
If the OST was permanently deactivated, it needs to be reactivated in the MGS configuration.
[client]# lctl set_param osc.<fsname>-<OST name>-*.active=0
</screen>
-14.8 Aborting Recovery
+</section>
+</section>
+ <section xml:id="dbdoclet.50438199_77819">
+ <title>14.8 Aborting Recovery</title>
You can abort recovery with either the lctl utility or by mounting the target with the abort_recov option (mount -o abort_recov). When starting a target, run:
</screen>
Note - The recovery process is blocked until all OSTs are available.
-14.9 Determining Which Machine is Serving an OST
+
+</section>
+ <section xml:id="dbdoclet.50438199_12607">
+ <title>14.9 Determining Which Machine is Serving an OST </title>
In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead.
osc.lustre-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
</screen>
-14.10 Changing the Address of a Failover Node
+</section>
+ <section xml:id="dbdoclet.50438199_62333">
+ <title>14.10 Changing the Address of a Failover Node</title>
To change the address of a failover node (e.g, to use node X instead of node Y), run this command on the OSS/OST partition:
tunefs.lustre --erase-params --failnode=<NID> <device>
</screen>
-</listitem></orderedlist>
- </section>
- </section>
- </section>
- </chapter>
+ </section>
+ </section>
+</chapter>
</screen>
<section remap="h4">
<title><anchor xml:id="dbdoclet.50438271_pgfId-1290921" xreflabel=""/>31.3.1.1 Interpreting OST Statistics</title>
- <note><para>See also <xref linkend="dbdoclet.50438219_84890"/> (llobdstat) and <xref linend="dbdoclet.50438273_80593"/> (CollectL).</para></note>
+ <note><para>See also <xref linkend="dbdoclet.50438219_84890"/> (llobdstat) and <xref linkend="dbdoclet.50438273_80593"/> (CollectL).</para></note>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1301139" xreflabel=""/>The OST .../stats files can be used to track client statistics (client activity) for each OST. It is possible to get a periodic dump of values from these file (for example, every 10 seconds), that show the RPC rates (similar to iostat) by using the llstat.pl tool:</para>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1290922" xreflabel=""/># llstat /proc/fs/lustre/osc/lustre-OST0000-osc/stats
</itemizedlist>
-<note><para>Usually the Lustre recovery process is transparent. For information about troubleshooting recovery when something goes wrong, see <xref linkend="dbdoclet.50438268_93053"/>.</para></note>
-
<section xml:id="dbdoclet.50438268_58047">
<title>30.1 Recovery Overview</title>
<para><anchor xml:id="dbdoclet.50438268_pgfId-1291584" xreflabel=""/>Lustre's recovery feature is responsible for dealing with node or network failure and returning the cluster to a consistent, performant state. Because Lustre allows servers to perform asynchronous update operations to the on-disk file system (i.e., the server can reply without waiting for the update to synchronously commit to disk), the clients may have state in memory that is newer than what the server can recover from disk after a crash.</para>
<para><anchor xml:id="dbdoclet.50438203_pgfId-999824" xreflabel=""/>This chapter describes some tools for managing Lustre Networking (LNET) and includes the following sections:</para>
<itemizedlist><listitem>
- <para><xref linkend="50438203_51732"/>Updating the Health Status of a Peer or Router</para>
+ <para><xref linkend="dbdoclet.50438203_51732"/>Updating the Health Status of a Peer or Router</para>
</listitem>
<listitem>
- <para><xref linkend="50438203_48703"/>Starting and Stopping LNET</para>
+ <para><xref linkend="dbdoclet.50438203_48703"/>Starting and Stopping LNET</para>
</listitem>
<listitem>
- <para><xref linkend="50438203_82542"/>Multi-Rail Configurations with LNET</para>
+ <para><xref linkend="dbdoclet.50438203_82542"/>Multi-Rail Configurations with LNET</para>
</listitem>
<listitem>
- <para><xref linkend="50438203_78227"/>Load Balancing with InfiniBand</para>
+ <para><xref linkend="dbdoclet.50438203_78227"/>Load Balancing with InfiniBand</para>
</listitem>
</itemizedlist>
</section>
<section remap="h5">
<title><anchor xml:id="dbdoclet.50438219_pgfId-1318212" xreflabel=""/>See Also</title>
- <para><xref linkend="troubleshootinglustre"/></para>
+ <para><xref linkend="dbdoclet.50438219_39574"/></para>
</section>
</section>
<section xml:id="dbdoclet.50438219_23232">
<section remap="h5">
<title><anchor xml:id="dbdoclet.50438219_pgfId-1294873" xreflabel=""/>lr_reader</title>
<para><anchor xml:id="dbdoclet.50438219_pgfId-1294874" xreflabel=""/>The lr_reader utility translates a last received (last_rcvd) file into human-readable form.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318389" xreflabel=""/>The following utilites are part of the Lustre I/O kit. For more information, see <xref linkend='benchmarkinglustreperformance'/>.</para>
+ <para><anchor xml:id="dbdoclet.50438219_pgfId-1318389" xreflabel=""/>The following utilites are part of the Lustre I/O kit. For more information, see <xref linkend='benchmarkingtests'/>.</para>
</section>
<section remap="h5">
<title><anchor xml:id="dbdoclet.50438219_pgfId-1318396" xreflabel=""/>sgpdd_survey</title>
</itemizedlist>
-<note><para>For a description of how recovery is implemented in Lustre, see <xref linkend="dbdoclet.50438225_28147"/>.</para></note>
-
<section xml:id="dbdoclet.50438225_71141">
<title>27.1 Recovering from Errors or <anchor xml:id="dbdoclet.50438225_marker-1292184" xreflabel=""/>Corruption on a Backing File System</title>
<para><anchor xml:id="dbdoclet.50438225_pgfId-1291241" xreflabel=""/>When an OSS, MDS, or MGS server crash occurs, it is not necessary to run e2fsck on the file system. ldiskfs journaling ensures that the file system remains coherent. The backing file systems are never accessed directly from the client, so client crashes are not relevant.</para>
<para><anchor xml:id="dbdoclet.50540653_pgfId-1304341" xreflabel=""/><emphasis role="bold">Health monitoring</emphasis> - Verifies the availability of hardware and network resources and responds to health indications provided by Lustre.</para>
</listitem>
</itemizedlist>
-<para><anchor xml:id="dbdoclet.50540653_pgfId-1305091" xreflabel=""/>These capabilities can be provided by a variety of software and/or hardware solutions. For more information about using power management software or hardware and high availability (HA) software with Lustre, see <xref linkend="configuringlustrefailover.title"/>.</para>
+<para><anchor xml:id="dbdoclet.50540653_pgfId-1305091" xreflabel=""/>These capabilities can be provided by a variety of software and/or hardware solutions. For more information about using power management software or hardware and high availability (HA) software with Lustre, see <xref linkend="configuringfailover.title"/>.</para>
<para><anchor xml:id="dbdoclet.50540653_pgfId-1305272" xreflabel=""/>HA software is responsible for detecting failure of the primary Lustre server node and controlling the failover. Lustre works with any HA software that includes resource (I/O) fencing. For proper resource fencing, the HA software must be able to completely power off the failed server or disconnect it from the shared storage device. If two active nodes have access to the same storage device, data may be severely corrupted.</para>
</section>
<section remap="h3">
</tbody>
</tgroup>
</table>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293938" xreflabel=""/>For additional hardware requirements and considerations, see <xref linkend='settinguplustre.title'/> .</para>
+ <para><anchor xml:id="dbdoclet.50438250_pgfId-1293938" xreflabel=""/>For additional hardware requirements and considerations, see <xref linkend='settinguplustresystem.title'/> .</para>
</section>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438250_pgfId-1295546" xreflabel=""/>1.2.3 Lustre Networking (LNET)</title>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293939" xreflabel=""/>Lustre Networking (LNET) is a custom networking API that provides the communication infrastructure that handles metadata and file I/O data for the Lustre file system servers and clients. For more information about LNET, see <xref linked='understandinglustrenetworking.title'/>.</para>
+ <para><anchor xml:id="dbdoclet.50438250_pgfId-1293939" xreflabel=""/>Lustre Networking (LNET) is a custom networking API that provides the communication infrastructure that handles metadata and file I/O data for the Lustre file system servers and clients. For more information about LNET, see <xref linkend='understandinglustrenetworking.title'/>.</para>
</section>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438250_pgfId-1293940" xreflabel=""/>1.2.4 Lustre Cluster</title>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293941" xreflabel=""/>At scale, the Lustre cluster can include hundreds of OSSs and thousands of clients (see <xref linkend='understandinglustre.fig.lustrescale/'/>). More than one type of network can be used in a Lustre cluster. Shared storage between OSSs enables failover capability. For more details about OSS failover, see <xref linkend='understandingfailover.title'/>.</para>
+ <para><anchor xml:id="dbdoclet.50438250_pgfId-1293941" xreflabel=""/>At scale, the Lustre cluster can include hundreds of OSSs and thousands of clients (see <xref linkend='understandinglustre.fig.lustrescale'/>). More than one type of network can be used in a Lustre cluster. Shared storage between OSSs enables failover capability. For more details about OSS failover, see <xref linkend='understandingfailover.title'/>.</para>
<figure>
<title xml:id='understandinglustre.fig.lustrescale'>Lustre cluster at scale</title>
<mediaobject>
git://git.whamcloud.com / doc / manual.git / commitdiff