<?xml version='1.0' encoding='UTF-8'?>
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="backupandrestore">
<title xml:id="backupandrestore.title">Backing Up and Restoring a File System</title>
- <para>This chapter describes how to backup and restore on Lustre, at the file system-level,
- device-level and file-level. Each backup approach is described in the the following
+ <para>This chapter describes how to backup and restore at the file system-level, device-level and
+ file-level in a Lustre file system. Each backup approach is described in the the following
sections:</para>
<itemizedlist>
<listitem>
<para>File system backups are performed from a Lustre client (or many clients working parallel in different directories) rather than on individual server nodes; this is no different than backing up any other file system.</para>
<para>However, due to the large size of most Lustre file systems, it is not always possible to get a complete backup. We recommend that you back up subsets of a file system. This includes subdirectories of the entire file system, filesets for a single user, files incremented by date, and so on.</para>
<note>
- <para>In order to allow Lustre to scale the file system namespace for future applications,
- Lustre 2.x internally uses a 128-bit file identifier for all files. To interface with user
- applications, Lustre presents 64-bit inode numbers for the <literal>stat()</literal>,
- <literal>fstat()</literal>, and <literal>readdir()</literal> system calls on 64-bit
- applications, and 32-bit inode numbers to 32-bit applications.</para>
- <para>Some 32-bit applications accessing Lustre filesystems (on both 32-bit and 64-bit CPUs) may experience problems with the <literal>stat()</literal>, <literal>fstat()</literal> or<literal> readdir()</literal> system calls under certain circumstances, though the Lustre client should return 32-bit inode numbers to these applications.</para>
+ <para>In order to allow the file system namespace to scale for future applications, Lustre
+ software release 2.x internally uses a 128-bit file identifier for all files. To interface
+ with user applications, the Lustre software presents 64-bit inode numbers for the
+ <literal>stat()</literal>, <literal>fstat()</literal>, and <literal>readdir()</literal>
+ system calls on 64-bit applications, and 32-bit inode numbers to 32-bit applications.</para>
+ <para>Some 32-bit applications accessing Lustre file systems (on both 32-bit and 64-bit CPUs)
+ may experience problems with the <literal>stat()</literal>, <literal>fstat()</literal>
+ or<literal> readdir()</literal> system calls under certain circumstances, though the
+ Lustre client should return 32-bit inode numbers to these applications.</para>
<para>In particular, if the Lustre file system is exported from a 64-bit client via NFS to a
32-bit client, the Linux NFS server will export 64-bit inode numbers to applications running
on the NFS client. If the 32-bit applications are not compiled with Large File Support
<section remap="h4">
<title><indexterm><primary>backup</primary><secondary>rsync</secondary><tertiary>examples</tertiary></indexterm><literal>lustre_rsync</literal> Examples</title>
<para>Sample <literal>lustre_rsync</literal> commands are listed below.</para>
- <para>Register a changelog user for an MDT (e.g. <literal>lustre-MDT0000</literal>).</para>
- <screen># lctl --device lustre-MDT0000 changelog_register lustre-MDT0000
+ <para>Register a changelog user for an MDT (e.g. <literal>testfs-MDT0000</literal>).</para>
+ <screen># lctl --device testfs-MDT0000 changelog_register testfs-MDT0000
Registered changelog userid 'cl1'</screen>
<para>Synchronize a Lustre file system (<literal>/mnt/lustre</literal>) to a target file system (<literal>/mnt/target</literal>).</para>
<screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target \
- --mdt=lustre-MDT0000 --user=cl1 --statuslog sync.log --verbose
-Lustre filesystem: lustre
-MDT device: lustre-MDT0000
+ --mdt=testfs-MDT0000 --user=cl1 --statuslog sync.log --verbose
+Lustre filesystem: testfs
+MDT device: testfs-MDT0000
Source: /mnt/lustre
Target: /mnt/target
Statuslog: sync.log
Changelog records consumed: 22</screen>
<para>After the file system undergoes changes, synchronize the changes onto the target file system. Only the <literal>statuslog</literal> name needs to be specified, as it has all the parameters passed earlier.</para>
<screen>$ lustre_rsync --statuslog sync.log --verbose
-Replicating Lustre filesystem: lustre
-MDT device: lustre-MDT0000
+Replicating Lustre filesystem: testfs
+MDT device: testfs-MDT0000
Source: /mnt/lustre
Target: /mnt/target
Statuslog: sync.log
Changelog records consumed: 42</screen>
<para>To synchronize a Lustre file system (<literal>/mnt/lustre</literal>) to two target file systems (<literal>/mnt/target1</literal> and <literal>/mnt/target2</literal>).</para>
<screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target1 \
- --target=/mnt/target2 --mdt=lustre-MDT0000 --user=cl1 \
+ --target=/mnt/target2 --mdt=testfs-MDT0000 --user=cl1 \
--statuslog sync.log</screen>
</section>
</section>
<para>Keeping an updated full backup of the MDT is especially important because a permanent failure of the MDT file system renders the much larger amount of data in all the OSTs largely inaccessible and unusable.</para>
</note>
<warning condition='l23'>
- <para>In Lustre 2.0 through 2.2, the only successful way to backup and
- restore an MDT is to do a device-level backup as is described in this
- section. File-level restore of an MDT is not possible before Lustre 2.3,
- as the Object Index (OI) file cannot be rebuilt after restore without
- the OI Scrub functionality.
- <emphasis role='bold'>Since Lustre 2.3</emphasis>, Object Index files
- are automatically rebuilt at first mount after a restore is detected
- (see <link xl:href='http://jira.hpdd.intel.com/browse/LU-957'>LU-957</link>),
- and file-level backup is supported
- (see <xref linkend='dbdoclet.50438207_21638'/>).</para>
+ <para>In Lustre software release 2.0 through 2.2, the only successful way to backup and
+ restore an MDT is to do a device-level backup as is described in this section. File-level
+ restore of an MDT is not possible before Lustre software release 2.3, as the Object Index
+ (OI) file cannot be rebuilt after restore without the OI Scrub functionality. <emphasis
+ role="bold">Since Lustre software release 2.3</emphasis>, Object Index files are
+ automatically rebuilt at first mount after a restore is detected (see <link
+ xl:href="http://jira.hpdd.intel.com/browse/LU-957">LU-957</link>), and file-level backup
+ is supported (see <xref linkend="dbdoclet.50438207_21638"/>).</para>
</warning>
<para>If hardware replacement is the reason for the backup or if a spare storage device is available, it is possible to do a raw copy of the MDT or OST from one block device to the other, as long as the new device is at least as large as the original device. To do this, run:</para>
<screen>dd if=/dev/{original} of=/dev/{newdev} bs=1M</screen>
<title><indexterm><primary>backup</primary><secondary>OST file system</secondary></indexterm><indexterm><primary>backup</primary><secondary>MDT file system</secondary></indexterm>Making a File-Level Backup of an OST or MDT File System</title>
<para>This procedure provides an alternative to backup or migrate the data of an OST or MDT at the file level. At the file-level, unused space is omitted from the backed up and the process may be completed quicker with smaller total backup size. Backing up a single OST device is not necessarily the best way to perform backups of the Lustre file system, since the files stored in the backup are not usable without metadata stored on the MDT and additional file stripes that may be on other OSTs. However, it is the preferred method for migration of OST devices, especially when it is desirable to reformat the underlying file system with different configuration options or to reduce fragmentation.</para>
<note>
- <para>Prior to Lustre 2.3, the only successful way to perform an MDT
- backup and restore is to do a device-level backup as is described in
- <xref linkend='dbdoclet.50438207_71633'/>. The ability to do MDT
- file-level backups is not available for Lustre 2.0 thru 2.2, because
- restoration of the Object Index (OI) file does not return the MDT to
- a functioning state. <emphasis role='bold'>Since Lustre 2.3</emphasis>,
- Object Index files are automatically rebuilt at first mount after a
- restore is detected (see
- <link xl:href='http://jira.hpdd.intel.com/browse/LU-957'>LU-957</link>),
- so file-level MDT restore is supported.</para>
+ <para>Prior to Lustre software release 2.3, the only successful way to perform an MDT backup
+ and restore is to do a device-level backup as is described in <xref
+ linkend="dbdoclet.50438207_71633"/>. The ability to do MDT file-level backups is not
+ available for Lustre software release 2.0 through 2.2, because restoration of the Object
+ Index (OI) file does not return the MDT to a functioning state. <emphasis role="bold">Since
+ Lustre software release 2.3</emphasis>, Object Index files are automatically rebuilt at
+ first mount after a restore is detected (see <link
+ xl:href="http://jira.hpdd.intel.com/browse/LU-957">LU-957</link>), so file-level MDT
+ restore is supported.</para>
</note>
- <para>For Lustre 2.3 and newer with MDT file-level backup support, substitute <literal>mdt</literal> for <literal>ost</literal> in the instructions below.</para>
+ <para>For Lustre software release 2.3 and newer with MDT file-level backup support, substitute
+ <literal>mdt</literal> for <literal>ost</literal> in the instructions below.</para>
<orderedlist>
<listitem>
<para><emphasis role="bold">Make a mountpoint for the file system.</emphasis></para>
<para>If the file system was used between the time the backup was made and when it was restored, then the <literal>lfsck</literal> tool (part of Lustre <literal>e2fsprogs</literal>) can optionally be run to ensure the file system is coherent. If all of the device file systems were backed up at the same time after the entire Lustre file system was stopped, this is not necessary. In either case, the file system should be immediately usable even if <literal>lfsck</literal> is not run, though there may be I/O errors reading from files that are present on the MDT but not the OSTs, and files that were created after the MDT backup will not be accessible/visible.</para>
</section>
<section xml:id="dbdoclet.50438207_31553">
- <title><indexterm><primary>backup</primary><secondary>using LVM</secondary></indexterm>Using LVM Snapshots with Lustre</title>
+ <title><indexterm>
+ <primary>backup</primary>
+ <secondary>using LVM</secondary>
+ </indexterm>Using LVM Snapshots with the Lustre File System</title>
<para>If you want to perform disk-based backups (because, for example, access to the backup system needs to be as fast as to the primary Lustre file system), you can use the Linux LVM snapshot tool to maintain multiple, incremental file system backups.</para>
<para>Because LVM snapshots cost CPU cycles as new files are written, taking snapshots of the main Lustre file system will probably result in unacceptable performance losses. You should create a new, backup Lustre file system and periodically (e.g., nightly) back up new/changed files to it. Periodic snapshots can be taken of this backup file system to create a series of "full" backups.</para>
<note>
</listitem>
<listitem>
<para>Format the LVM volumes as Lustre targets.</para>
- <para>In this example, the backup file system is called 'main' and designates the current, most up-to-date backup.</para>
+ <para>In this example, the backup file system is called <literal>main</literal> and
+ designates the current, most up-to-date backup.</para>
<screen>cfs21:~# mkfs.lustre --fsname=main --mdt --index=0 /dev/vgmain/MDT0
No management node specified, adding MGS to this MDT.
Permanent disk data:
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: mgsnode=192.168.0.21@tcp
Writing CONFIGS/mountdata</screen>
- <para>When renaming an FS, we must also erase the last_rcvd file from the snapshots</para>
+ <para>When renaming a file system, we must also erase the last_rcvd file from the
+ snapshots</para>
<screen>cfs21:~# mount -t ldiskfs /dev/vgmain/MDT0.b1 /mnt/mdtback
cfs21:~# rm /mnt/mdtback/last_rcvd
cfs21:~# umount /mnt/mdtback
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="benchmarkingtests">
- <title xml:id="benchmarkingtests.title">Benchmarking Lustre Performance (Lustre I/O Kit)</title>
+ <title xml:id="benchmarkingtests.title">Benchmarking Lustre File System Performance (Lustre I/O
+ Kit)</title>
<para>This chapter describes the Lustre I/O kit, a collection of I/O benchmarking tools for a
Lustre cluster, and PIOS, a parallel I/O simulator for Linux and
Solaris<superscript>*</superscript> operating systems. It includes:</para>
<indexterm><primary>performance</primary><see>benchmarking</see></indexterm>
Using Lustre I/O Kit Tools</title>
- <para>The tools in the Lustre I/O Kit are used to benchmark Lustre hardware and validate that it is working as expected before you install the Lustre software. It can also be used to to validate the performance of the various hardware and software layers in the cluster and also to find and troubleshoot I/O issues.</para>
+ <para>The tools in the Lustre I/O Kit are used to benchmark Lustre file system hardware and
+ validate that it is working as expected before you install the Lustre software. It can also be
+ used to to validate the performance of the various hardware and software layers in the cluster
+ and also to find and troubleshoot I/O issues.</para>
<para>Typically, performance is measured starting with single raw devices and then proceeding to groups of devices. Once raw performance has been established, other software layers are then added incrementally and tested.</para>
<section remap="h3">
<title>Contents of the Lustre I/O Kit</title>
- <para>The I/O kit contains three tests, each of which tests a progressively higher layer in the Lustre stack:</para>
+ <para>The I/O kit contains three tests, each of which tests a progressively higher layer in
+ the Lustre software stack:</para>
<itemizedlist>
<listitem>
<para><literal>sgpdd-survey</literal> - Measure basic 'bare metal' performance
issues.</para>
</listitem>
</itemizedlist>
- <para>Typically with these tests, Lustre should deliver 85-90% of the raw device performance.</para>
+ <para>Typically with these tests, a Lustre file system should deliver 85-90% of the raw device
+ performance.</para>
<para>A utility <literal>stats-collect</literal> is also provided to collect application profiling information from Lustre clients and servers. See <xref linkend="dbdoclet.50438212_58201"/> for more information.</para>
</section>
<section remap="h3">
<para>Password-free remote access to nodes in the system (provided by <literal>ssh</literal> or <literal>rsh</literal>).</para>
</listitem>
<listitem>
- <para>LNET self-test completed to test that Lustre Networking has been properly installed and configured. See <xref linkend="lnetselftest"/>.</para>
+ <para>LNET self-test completed to test that Lustre networking has been properly installed
+ and configured. See <xref linkend="lnetselftest"/>.</para>
</listitem>
<listitem>
<para>Lustre file system software installed.</para>
<para>The <literal>plot-obdfilter</literal> script generates from the output of the
<literal>obdfilter-survey</literal> a CSV file and parameters for importing into a
spreadsheet or gnuplot to visualize the data.</para>
- <para>To run the <literal>obdfilter-survey</literal> script, create a standard Lustre
- configuration; no special setup is needed.</para>
+ <para>To run the <literal>obdfilter-survey</literal> script, create a standard Lustre file
+ system configuration; no special setup is needed.</para>
<para><emphasis role="bold">To perform an automatic run:</emphasis></para>
<orderedlist>
<listitem>
<title><indexterm><primary>benchmarking</primary><secondary>network</secondary></indexterm>Testing Network Performance</title>
<para>The <literal>obdfilter-survey</literal> script can only be run automatically against a
network; no manual test is provided.</para>
- <para>To run the network test, a specific Lustre setup is needed. Make sure that these configuration requirements have been met.</para>
+ <para>To run the network test, a specific Lustre file system setup is needed. Make sure that
+ these configuration requirements have been met.</para>
<para><emphasis role="bold">To perform an automatic run:</emphasis></para>
<orderedlist>
<listitem>
network hardware.</para>
</note>
<para>To run the <literal>ost-survey</literal> script, supply a file size (in KB) and the Lustre
- mount point. For example, run:</para>
+ file system mount point. For example, run:</para>
<screen>$ ./ost-survey.sh 10 /mnt/lustre
</screen>
<para>Typical output is:</para>
<section xml:id="mds_survey_ref">
<title><indexterm><primary>benchmarking</primary><secondary>MDS
performance</secondary></indexterm>Testing MDS Performance (<literal>mds-survey</literal>)</title>
- <para><literal>mds-survey</literal> is available in Lustre 2.2 and beyond. The <literal>mds-survey</literal> script tests the local metadata performance using the echo_client to drive different layers of the MDS stack: mdd, mdt, osd (current lustre version only supports mdd stack). It can be used with the following classes of operations:</para>
+ <para><literal>mds-survey</literal> is available in Lustre software release 2.2 and beyond. The
+ <literal>mds-survey</literal> script tests the local metadata performance using the
+ echo_client to drive different layers of the MDS stack: mdd, mdt, osd (the Lustre software
+ only supports mdd stack). It can be used with the following classes of operations:</para>
<itemizedlist>
<listitem>
<para>The <literal>stats-collect</literal> utility requires:</para>
<itemizedlist>
<listitem>
- <para>Lustre to be installed and set up on your cluster</para>
+ <para>Lustre software to be installed and set up on your cluster</para>
</listitem>
<listitem>
<para>SSH and SCP access to these nodes without requiring a password</para>
<para>A separate <literal>lustre.conf</literal> file makes distributing the configuration much easier.</para>
</listitem>
<listitem>
- <para>If you set <literal>config_on_load=1</literal>, LNET starts at <literal>modprobe</literal> time rather than waiting for Lustre to start. This ensures routers start working at module load time.</para>
+ <para>If you set <literal>config_on_load=1</literal>, LNET starts at
+ <literal>modprobe</literal> time rather than waiting for the Lustre file system to
+ start. This ensures routers start working at module load time.</para>
</listitem>
</itemizedlist>
<screen># lctl
</tgroup>
</informaltable>
<note>
- <para>Lustre ignores the loopback interface (<literal>lo0</literal>), but Lustre use any IP addresses aliased to the loopback (by default). When in doubt, explicitly specify networks.</para>
+ <para>The Lustre software ignores the loopback interface (<literal>lo0</literal>), but the
+ Lustre file system uses any IP addresses aliased to the loopback (by default). When in
+ doubt, explicitly specify networks.</para>
</note>
<para><literal>ip2nets</literal> ("") is a string that lists globally-available networks, each with a set of IP address ranges. LNET determines the locally-available networks from this list by matching the IP address ranges with the local IPs of a node. The purpose of this option is to be able to use the same <literal>modules.conf</literal> file across a variety of nodes on different networks. The string has the following syntax.</para>
<screen><ip2nets> :== <net-match> [ <comment> ] { <net-sep> <net-match> }
<para><literal>n_waitd(1)</literal> sets the number of threads that process completed MX requests (sends and receives).</para>
<para><literal>max_peers(1024)</literal> tells MXLND the upper limit of machines that it will need to communicate with. This affects how many receives it will pre-post and each receive will use one page of memory. Ideally, on clients, this value will be equal to the total number of Lustre servers (MDS and OSS). On servers, it needs to equal the total number of machines in the storage system. cksum (0) turns on small message checksums. It can be used to aid in troubleshooting. MX also provides an optional checksumming feature which can check all messages (large and small). For details, see the MX README.</para>
<para><literal>ntx(256)</literal> is the number of total sends in flight from this machine. In actuality, MXLND reserves half of them for connect messages so make this value twice as large as you want for the total number of sends in flight.</para>
- <para><literal>credits(8)</literal> is the number of in-flight messages for a specific peer. This is part of the flow-control system in Lustre. Increasing this value may improve performance but it requires more memory because each message requires at least one page.</para>
+ <para><literal>credits(8)</literal> is the number of in-flight messages for a specific peer.
+ This is part of the flow-control system in provided by the Lustre software. Increasing this
+ value may improve performance but it requires more memory because each message requires at
+ least one page.</para>
<para><literal>board(0)</literal> is the index of the Myricom NIC. Hosts can have multiple Myricom NICs and this identifies which one MXLND should use. This value must match the board value in your MXLND hosts file for this host.</para>
<para><literal>ep_id(3)</literal> is the MX endpoint ID. Each process that uses MX is required to have at least one MX endpoint to access the MX library and NIC. The ID is a simple index starting at zero (0). This value must match the endpoint ID value in your MXLND hosts file for this host.</para>
<para><literal>polling(0)</literal> determines whether this host will poll or block for MX request completions. A value of 0 blocks and any positive value will poll that many times before blocking. Since polling increases CPU usage, we suggest that you set this to zero (0) on the client and experiment with different values for servers.</para>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlustre">
- <title xml:id="configuringlustre.title">Configuring Lustre</title>
- <para>This chapter shows how to configure a simple Lustre system comprised of a combined MGS/MDT, an OST and a client. It includes:</para>
+ <title xml:id="configuringlustre.title">Configuring a Lustre File System</title>
+ <para>This chapter shows how to configure a simple Lustre file system comprised of a combined
+ MGS/MDT, an OST and a client. It includes:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438267_50692"/>
<title>
<indexterm><primary>Lustre</primary><secondary>configuring</secondary></indexterm>
Configuring a Simple Lustre File System</title>
- <para>A Lustre system can be set up in a variety of configurations by using the administrative utilities provided with Lustre. The procedure below shows how to configure a simple Lustre file system consisting of a combined MGS/MDS, one OSS with two OSTs, and a client. For an overview of the entire Lustre installation procedure, see <xref linkend="installoverview"/>.</para>
+ <para>A Lustre file system can be set up in a variety of configurations by using the
+ administrative utilities provided with the Lustre software. The procedure below shows how to
+ configure a simple Lustre file system consisting of a combined MGS/MDS, one OSS with two OSTs,
+ and a client. For an overview of the entire Lustre installation procedure, see <xref
+ linkend="installoverview"/>.</para>
<para>This configuration procedure assumes you have completed the following:</para>
<itemizedlist>
<listitem>
</listitem>
<listitem>
<para><emphasis>Set</emphasis> lnet <emphasis>module parameters to specify how Lustre
- Networking (LNET) is to be configured to work with Lustre and test the LNET
- configuration.</emphasis> LNET will, by default, use the first TCP/IP interface it
+ Networking (LNET) is to be configured to work with a Lustre file system and test the
+ LNET configuration.</emphasis> LNET will, by default, use the first TCP/IP interface it
discovers on a system. If this network configuration is sufficient, you do not need to
configure LNET. LNET configuration is required if you are using InfiniBand or multiple
Ethernet interfaces.</para>
<listitem>
<para><emphasis>Run the benchmark script <literal>sgpdd-survey</literal> to determine
baseline performance of your hardware.</emphasis> Benchmarking your hardware will
- simplify debugging performance issues that are unrelated to Lustre and ensure you are
- getting the best possible performance with your installation. For information about
- running <literal>sgpdd-survey</literal>, see <xref linkend="benchmarkingtests"/>.</para>
+ simplify debugging performance issues that are unrelated to the Lustre software and ensure
+ you are getting the best possible performance with your installation. For information
+ about running <literal>sgpdd-survey</literal>, see <xref linkend="benchmarkingtests"
+ />.</para>
</listitem>
</itemizedlist>
<note>
</note>
</listitem>
<listitem xml:id="dbdoclet.addmdtindex">
- <para>Optional for Lustre 2.4 and later. Add in additional MDTs.</para>
+ <para>Optional for Lustre software release 2.4 and later. Add in additional MDTs.</para>
<screen>mkfs.lustre --fsname=<replaceable>fsname</replaceable> --mgsnode=<replaceable>nid</replaceable> --mdt --index=1 <replaceable>/dev/block_device</replaceable></screen>
<note><para>Up to 4095 additional MDTs can be added.</para></note>
</listitem>
<para>You should specify the OST index number at format time in order to simplify translating the OST number in error messages or file striping to the OSS node and block device later on.</para>
<para>If you are using block devices that are accessible from multiple OSS nodes, ensure that you mount the OSTs from only one OSS node at at time. It is strongly recommended that multiple-mount protection be enabled for such devices to prevent serious data corruption. For more information about multiple-mount protection, see <xref linkend="managingfailover"/>.</para>
<note>
- <para>Lustre currently supports block devices up to 128 TB on Red Hat Enterprise Linux 5
- and 6 (up to 8 TB on other distributions). If the device size is only slightly larger
- that 16 TB, it is recommended that you limit the file system size to 16 TB at format
- time. We recommend that you not place DOS partitions on top of RAID 5/6 block devices
- due to negative impacts on performance, but instead format the whole disk for the file
- system.</para>
+ <para>The Lustre software currently supports block devices up to 128 TB on Red Hat
+ Enterprise Linux 5 and 6 (up to 8 TB on other distributions). If the device size is only
+ slightly larger that 16 TB, it is recommended that you limit the file system size to 16
+ TB at format time. We recommend that you not place DOS partitions on top of RAID 5/6
+ block devices due to negative impacts on performance, but instead format the whole disk
+ for the file system.</para>
</note>
</listitem>
<listitem xml:id="dbdoclet.50438267_pgfId-1293955">
<title>
<indexterm><primary>Lustre</primary><secondary>configuring</secondary><tertiary>simple example</tertiary></indexterm>
Simple Lustre Configuration Example</title>
- <para>To see the steps in a simple Lustre configuration, follow this example in which a combined MGS/MDT and two OSTs are created. Three block devices are used, one for the combined MGS/MDS node and one for each OSS node. Common parameters used in the example are listed below, along with individual node parameters.</para>
+ <para>To see the steps to complete for a simple Lustre file system configuration, follow this
+ example in which a combined MGS/MDT and two OSTs are created to form a file system called
+ <literal>temp</literal>. Three block devices are used, one for the combined MGS/MDS node
+ and one for each OSS node. Common parameters used in the example are listed below, along
+ with individual node parameters.</para>
<informaltable frame="all">
<tgroup cols="4">
<colspec colname="c1" colwidth="2*"/>
<para> <literal>TCP/IP</literal></para>
</entry>
<entry>
- <para>Network type used for Lustre file system temp</para>
+ <para>Network type used for Lustre file system <literal>temp</literal></para>
</entry>
</row>
</tbody>
<para> <literal>mdt0</literal></para>
</entry>
<entry>
- <para>MDS in Lustre file system temp</para>
+ <para>MDS in Lustre file system <literal>temp</literal></para>
</entry>
</row>
<row>
<para><literal> oss0</literal></para>
</entry>
<entry>
- <para>First OSS node in Lustre file system temp</para>
+ <para>First OSS node in Lustre file system <literal>temp</literal></para>
</entry>
</row>
<row>
<para><literal>ost0</literal></para>
</entry>
<entry>
- <para>First OST in Lustre file system temp</para>
+ <para>First OST in Lustre file system <literal>temp</literal></para>
</entry>
</row>
<row>
<para><literal>oss1</literal></para>
</entry>
<entry>
- <para>Second OSS node in Lustre file system temp</para>
+ <para>Second OSS node in Lustre file system <literal>temp</literal></para>
</entry>
</row>
<row>
<para> <literal>ost1</literal></para>
</entry>
<entry>
- <para>Second OST in Lustre file system temp</para>
+ <para>Second OST in Lustre file system <literal>temp</literal></para>
</entry>
</row>
<row>
<para> <literal>client1</literal></para>
</entry>
<entry>
- <para>Client in Lustre file system temp</para>
+ <para>Client in Lustre file system <literal>temp</literal></para>
</entry>
</row>
<row>
<para> <literal>/lustre</literal></para>
</entry>
<entry>
- <para>Mount point for Lustre file system temp on the <literal>client1</literal> node</para>
+ <para>Mount point for Lustre file system <literal>temp</literal> on the
+ <literal>client1</literal> node</para>
</entry>
</row>
</tbody>
<para><literal>tunefs.lustre</literal> - Use to modify configuration information on a Lustre target disk.</para>
</listitem>
<listitem>
- <para><literal>lctl</literal> - Use to directly control Lustre via an ioctl interface, allowing various configuration, maintenance and debugging features to be accessed.</para>
+ <para><literal>lctl</literal> - Use to directly control Lustre features via an
+ <literal>ioctl</literal> interface, allowing various configuration, maintenance and
+ debugging features to be accessed.</para>
</listitem>
<listitem>
<para><literal>mount.lustre</literal> - Use to start a Lustre client or target service.</para>
<para>For examples using these utilities, see the topic <xref linkend="systemconfigurationutilities"/></para>
<para>The <literal>lfs</literal> utility is useful for configuring and querying a variety of options related to files. For more information, see <xref linkend="userutilities"/>.</para>
<note>
- <para>Some sample scripts are included in the directory where Lustre is installed. If you have installed the Lustre source code, the scripts are located in the <literal>lustre/tests</literal> sub-directory. These scripts enable quick setup of some simple standard Lustre configurations.</para>
+ <para>Some sample scripts are included in the directory where the Lustre software is
+ installed. If you have installed the Lustre source code, the scripts are located in the
+ <literal>lustre/tests</literal> sub-directory. These scripts enable quick setup of some
+ simple standard Lustre configurations.</para>
</note>
</section>
</section>
<para>Quotas are administered via the <literal>lfs</literal> and <literal>lctl</literal> commands (post-mount).</para>
</listitem>
<listitem>
- <para>Quotas are distributed (as Lustre is a distributed file system), which has several ramifications.</para>
+ <para>Quotas are distributed (as the Lustre file system is a distributed file system), which
+ has several ramifications.</para>
</listitem>
<listitem>
<para>Quotas are allocated and consumed in a quantized fashion.</para>
</listitem>
<listitem>
<para>Client does not set the <literal>usrquota</literal> or <literal>grpquota</literal>
- options to mount. As of Lustre 2.4, space accounting is always enabled by default and
- quota enforcement can be enabled/disabled on a per-file system basis with <literal>lctl
- conf_param</literal>. It is worth noting that both <literal>lfs quotaon</literal> and
- <literal>quota_type</literal> are deprecated as of Lustre 2.4.0.</para>
+ options to mount. As of Lustre software release 2.4, space accounting is always enabled by
+ default and quota enforcement can be enabled/disabled on a per-file system basis with
+ <literal>lctl conf_param</literal>. It is worth noting that both <literal>lfs
+ quotaon</literal> and <literal>quota_type</literal> are deprecated as of Lustre software
+ release 2.4.0.</para>
</listitem>
</itemizedlist>
<caution>
- <para>Although quotas are available in Lustre, root quotas are NOT enforced.</para>
+ <para>Although a quota feature is available in the Lustre software, root quotas are NOT
+ enforced.</para>
<para><literal>lfs setquota -u root</literal> (limits are not enforced)</para>
<para><literal>lfs quota -u root</literal> (usage includes internal Lustre data that is dynamic in size and does not accurately reflect mount point visible block and inode usage).</para>
</caution>
</section>
<section xml:id="enabling_disk_quotas">
<title><indexterm><primary>Quotas</primary><secondary>enabling disk</secondary></indexterm>Enabling Disk Quotas</title>
- <para>Prior to Lustre 2.4.0, enabling quota involved a full file system scan via <literal>lfs
- quotacheck</literal>. All file systems formatted with Lustre 2.4.0 or newer no longer
- require quotacheck to be run since up-to-date accounting information are now always maintained
- by the OSD layer, regardless of the quota enforcement status. </para>
+ <para>Prior to Lustre software release 2.4.0, enabling quota involved a full file system scan
+ via <literal>lfs quotacheck</literal>. All file systems formatted with Lustre software release
+ 2.4.0 or newer no longer require quotacheck to be run since up-to-date accounting information
+ are now always maintained by the OSD layer, regardless of the quota enforcement status. </para>
<section remap="h3" condition="l24">
- <title>Enabling Disk Quotas (Lustre Release 2.4 and later)</title>
- <para>Although quota enforcement is managed by Lustre itself, each OSD implementation relies
- on the backend file system to maintain per-user/group block and inode usage:</para>
+ <title>Enabling Disk Quotas (Lustre Software Release 2.4 and later)</title>
+ <para>Although quota enforcement is managed by the Lustre software, each OSD implementation
+ relies on the backend file system to maintain per-user/group block and inode usage:</para>
<itemizedlist>
<listitem>
- <para>For ldiskfs backend, mkfs.lustre now creates empty quota files and enables the QUOTA
- feature flag in the superblock which turns quota accounting on at mount time
- automatically. e2fsck was also modified to fix the quota files when the QUOTA feature
- flag is present.</para>
+ <para>For ldiskfs backend, <literal>mkfs.lustre</literal> now creates empty quota files
+ and enables the QUOTA feature flag in the superblock which turns quota accounting on at
+ mount time automatically. e2fsck was also modified to fix the quota files when the QUOTA
+ feature flag is present.</para>
</listitem>
<listitem>
<para>For ZFS backend, accounting ZAPs are created and maintained by the ZFS file system
</listitem>
</itemizedlist>
<para>As a result, <literal>lfs quotacheck</literal> is now deprecated and not required any
- more when running Lustre 2.4 on the servers.</para>
- <para>Lustre filesystems formatted with a Lustre version prior to 2.4.0 can be still safely
- upgraded to 2.4.0, but won't have functional space usage report until <literal>tunefs.lustre
- --quota</literal> is run against all targets. This command sets the QUOTA feature flag in
- the superblock and runs e2fsck (as a result, the target must be offline) to build the
- per-UID/GID disk usage database. </para>
+ more when running Lustre software release 2.4 on the servers.</para>
+ <para>Lustre file systems formatted with a Lustre release prior to 2.4.0 can be still safely
+ upgraded to release 2.4.0, but won't have functional space usage report until
+ <literal>tunefs.lustre --quota</literal> is run against all targets. This command sets the
+ QUOTA feature flag in the superblock and runs e2fsck (as a result, the target must be
+ offline) to build the per-UID/GID disk usage database. </para>
<caution>
- <para>Lustre 2.4 and beyond requires a version of e2fsprogs that supports quota (i.e. newer
- or equal to 1.42.3.wc1) to be installed on the server nodes using ldiskfs backend
- (e2fsprogs isn't needed with ZFS backend). In general, we recommend to use the latest
- e2fsprogs version available on <link xl:href="http://downloads.hpdd.intel.com/e2fsprogs/">http://downloads.hpdd.intel.com/public/e2fsprogs/</link>.</para>
+ <para>Lustre software release 2.4 and beyond requires a version of e2fsprogs that supports
+ quota (i.e. newer or equal to 1.42.3.wc1) to be installed on the server nodes using
+ ldiskfs backend (e2fsprogs isn't needed with ZFS backend). In general, we recommend to use
+ the latest e2fsprogs version available on <link
+ xl:href="http://downloads.hpdd.intel.com/e2fsprogs/"
+ >http://downloads.hpdd.intel.com/public/e2fsprogs/</link>.</para>
<para>The ldiskfs OSD relies on the standard Linux quota to maintain accounting information
on disk. As a consequence, the Linux kernel running on the Lustre servers using ldiskfs
backend must have <literal>CONFIG_QUOTA</literal>, <literal>CONFIG_QUOTACTL</literal> and
<literal>CONFIG_QFMT_V2</literal> enabled.</para>
</caution>
- <para>As of Lustre 2.4.0, quota enforcement is thus turned on/off independently of space
- accounting which is always enabled. <literal>lfs
- quota<replaceable>on|off</replaceable></literal> as well as the per-target
+ <para>As of Lustre software release 2.4.0, quota enforcement is thus turned on/off
+ independently of space accounting which is always enabled. <literal>lfs
+ quota<replaceable>on|off</replaceable></literal> as well as the per-target
<literal>quota_type</literal> parameter are deprecated in favor of a single per-file
system quota parameter controlling inode/block quota enforcement. Like all permanent
parameters, this quota parameter can be set via <literal>lctl conf_param</literal> on the
user uptodate: glb[1],slv[1],reint[0]
group uptodate: glb[1],slv[1],reint[0]</screen>
<caution>
- <para>Lustre 2.4 comes with a new quota protocol and a new on-disk format, be sure to check
- the Interoperability section below (see <xref linkend="quota_interoperability"/>.) when
- migrating to 2.4</para>
+ <para>Lustre software release 2.4 comes with a new quota protocol and a new on-disk format,
+ be sure to check the Interoperability section below (see <xref
+ linkend="quota_interoperability"/>.) when migrating to release 2.4</para>
</caution>
</section>
<section remap="h3">
<title>Enabling Disk Quotas (Lustre Releases Previous to Release 2.4 )</title>
<para><note>
<?oxy_custom_start type="oxy_content_highlight" color="255,64,0"?>
- <para><?oxy_custom_end?>In Lustre software releases previous to Release 2.4, when new OSTs
+ <para><?oxy_custom_end?>In Lustre software releases previous to release 2.4, when new OSTs
are added to the file system, quotas are not automatically propagated to the new OSTs.
As a workaround, clear and then reset quotas for each user or group using the
<literal>lfs setquota</literal> command. In the example below, quotas are cleared and
reset for user <literal>bob</literal> on file system <literal>testfs</literal>:
<screen>$ lfs setquota -u bob -b 0 -B 0 -i 0 -I 0 /mnt/testfs
$ lfs setquota -u bob -b 307200 -B 309200 -i 10000 -I 11000 /mnt/testfs</screen></para>
- </note>For Lustre versions older than 2.4, <literal>lfs quotacheck</literal> must be first
- run from a client node to create quota files on the Lustre targets (i.e. the MDT and OSTs).
- <literal>lfs quotacheck</literal> requires the file system to be quiescent (i.e. no
- modifying operations like write, truncate, create or delete should run concurrently).
- Failure to follow this caution may result in inaccurate user/group disk usage. Operations
- that do not change Lustre files (such as read or mount) are okay to run. <literal>lfs
- quotacheck</literal> performs a scan on all the Lustre targets to calculates the
- block/inode usage for each user/group. If the Lustre file system has many files,
- <literal>quotacheck</literal> may take a long time to complete. Several options can be
- passed to <literal>lfs quotacheck</literal>:</para>
+ </note>For Lustre software releases older than release 2.4, <literal>lfs
+ quotacheck</literal> must be first run from a client node to create quota files on the
+ Lustre targets (i.e. the MDT and OSTs). <literal>lfs quotacheck</literal> requires the file
+ system to be quiescent (i.e. no modifying operations like write, truncate, create or delete
+ should run concurrently). Failure to follow this caution may result in inaccurate user/group
+ disk usage. Operations that do not change Lustre files (such as read or mount) are okay to
+ run. <literal>lfs quotacheck</literal> performs a scan on all the Lustre targets to
+ calculates the block/inode usage for each user/group. If the Lustre file system has many
+ files, <literal>quotacheck</literal> may take a long time to complete. Several options can
+ be passed to <literal>lfs quotacheck</literal>:</para>
<screen># lfs quotacheck -ug /mnt/testfs</screen>
<itemizedlist>
<listitem>
this setting isn't persistent and quota will have to be enabled again (via <literal>lfs
quotaon</literal>) if one of the Lustre targets is restarted. <literal>lfs
quotaoff</literal> is used to turn off quota.</para>
- <para>To enable quota permanently with a Lustre version older than 2.4, the
+ <para>To enable quota permanently with a Lustre software release older than release 2.4, the
<literal>quota_type</literal> parameter must be used. This requires setting
<literal>mdd.quota_type</literal> and <literal>ost.quota_type</literal>, respectively, on
the MDT and OSTs. <literal>quota_type</literal> can be set to the string
<para>The grace period applies to all users. The user block soft limit is for all users who are using a blocks quota.</para>
<para><emphasis role="bold">Soft limit</emphasis> -- The grace timer is started once the soft limit is exceeded. At this point, the user/group can still allocate block/inode. When the grace time expires and if the user is still above the soft limit, the soft limit becomes a hard limit and the user/group can't allocate any new block/inode any more. The user/group should then delete files to be under the soft limit. The soft limit MUST be smaller than the hard limit. If the soft limit is not needed, it should be set to zero (0).</para>
<para><emphasis role="bold">Hard limit</emphasis> -- Block or inode allocation will fail with <literal>EDQUOT</literal> (i.e. quota exceeded) when the hard limit is reached. The hard limit is the absolute limit. When a grace period is set, one can exceed the soft limit within the grace period if under the hard limit.</para>
- <para>Due to the distributed nature of Lustre and the need to mainain performance under load, those quota parameters may not be 100% accurate. The quota settings can be manipulated via the <literal>lfs</literal> command which includes several options to work with quotas:</para>
+ <para>Due to the distributed nature of a Lustre file system and the need to mainain performance
+ under load, those quota parameters may not be 100% accurate. The quota settings can be
+ manipulated via the <literal>lfs</literal> command which includes several options to work with
+ quotas:</para>
<itemizedlist>
<listitem>
<para><varname>quota</varname> -- displays general quota information (disk usage and limits)</para>
</section>
<section xml:id="quota_allocation">
<title><indexterm><primary>Quotas</primary><secondary>allocating</secondary></indexterm>Quota Allocation</title>
- <para>In Lustre, quota must be properly allocated or users may experience unnecessary failures. The file system block quota is divided up among the OSTs within the file system. Each OST requests an allocation which is increased up to the quota limit. The quota allocation is then <emphasis role="italic">quantized</emphasis> to reduce the number of quota-related request traffic.</para>
+ <para>In a Lustre file system, quota must be properly allocated or users may experience
+ unnecessary failures. The file system block quota is divided up among the OSTs within the file
+ system. Each OST requests an allocation which is increased up to the quota limit. The quota
+ allocation is then <emphasis role="italic">quantized</emphasis> to reduce the number of
+ quota-related request traffic.</para>
<para>The Lustre quota system distributes quotas from the Quota Master Target (aka QMT). Only one QMT instance is supported for now and only runs on the same node as MDT0000. All OSTs and MDTs set up a Quota Slave Device (aka QSD) which connects to the QMT to allocate/release quota space. The QSD is setup directly from the OSD layer.</para>
<para>To reduce quota requests, quota space is initially allocated to QSDs in very large chunks. How much unused quota space can be hold by a target is controlled by the qunit size. When quota space for a given ID is close to exhaustion on the QMT, the qunit size is reduced and QSDs are notified of the new qunit size value via a glimpse callback. Slaves are then responsible for releasing quota space above the new qunit value. The qunit size isn't shrunk indefinitely and there is a minimal value of 1MB for blocks and 1,024 for inodes. This means that the quota space rebalancing process will stop when this mininum value is reached. As a result, quota exceeded can be returned while many slaves still have 1MB or 1,024 inodes of spare quota space.</para>
<para>If we look at the <literal>setquota</literal> example again, running this <literal>lfs quota</literal> command:</para>
<para>Setting the quota limit below the minimal qunit size may prevent the user/group from all file creation. It is thus recommended to use soft/hard limits which are a multiple of the number of OSTs * the minimal qunit size.</para>
</note>
<para> To determine the total number of inodes, use <literal>lfs df -i</literal> (and also <literal>lctl get_param *.*.filestotal</literal>). For more information on using the <literal>lfs df -i</literal> command and the command output, see <xref linkend="dbdoclet.50438209_35838"/>.</para>
- <para>Unfortunately, the <literal>statfs</literal> interface does not report the free inode count directly, but instead reports the total inode and used inode counts. The free inode count is calculated for <literal>df</literal> from (total inodes - used inodes). It is not critical to know a file system's total inode count. Instead, you should know (accurately), the free inode count and the used inode count for a file system. Lustre manipulates the total inode count in order to accurately report the other two values.</para>
+ <para>Unfortunately, the <literal>statfs</literal> interface does not report the free inode
+ count directly, but instead reports the total inode and used inode counts. The free inode
+ count is calculated for <literal>df</literal> from (total inodes - used inodes). It is not
+ critical to know the total inode count for a file system. Instead, you should know
+ (accurately), the free inode count and the used inode count for a file system. The Lustre
+ software manipulates the total inode count in order to accurately report the other two
+ values.</para>
</section>
<section xml:id="quota_interoperability">
<title><indexterm><primary>Quotas</primary><secondary>Interoperability</secondary></indexterm>Interoperability</title>
- <para>The new quota protocol introduced in Lustre 2.4.0 <emphasis role="bold">isn't
- compatible</emphasis> with the old one. As a consequence, <emphasis role="bold">all Lustre
- servers must be upgraded to 2.4.0 for quota to be functional</emphasis>. Quota limits set on
- the Lustre file system prior to the upgrade will be automatically migrated to the new quota
- index format. As for accounting information with ldiskfs backend, they will be regenerated by
- running <literal>tunefs.lustre --quota</literal> against all targets. It is worth noting that
- running <literal>tunefs.lustre --quota</literal> is <emphasis role="bold">mandatory</emphasis>
- for all targets formatted with a Lustre version older than 2.4.0, otherwise quota enforcement
- as well as accounting won't be functional.</para>
- <para>Besides, the 2.4 quota protocol takes for granted that the Lustre client supports the <literal>OBD_CONNECT_EINPROGRESS</literal> connect flag. Clients supporting this flag will retry indefinitely when the server returns <literal>EINPROGRESS</literal> in a reply. Here is the list of Lustre client version which are compatible with 2.4:</para>
+ <para>The new quota protocol introduced in Lustre software release 2.4.0 <emphasis role="bold"
+ >isn't compatible</emphasis> with the old one. As a consequence, <emphasis role="bold">all
+ Lustre servers must be upgraded to release 2.4.0 for quota to be functional</emphasis>.
+ Quota limits set on the Lustre file system prior to the upgrade will be automatically migrated
+ to the new quota index format. As for accounting information with ldiskfs backend, they will
+ be regenerated by running <literal>tunefs.lustre --quota</literal> against all targets. It is
+ worth noting that running <literal>tunefs.lustre --quota</literal> is <emphasis role="bold"
+ >mandatory</emphasis> for all targets formatted with a Lustre software release older than
+ release 2.4.0, otherwise quota enforcement as well as accounting won't be functional.</para>
+ <para>Besides, the quota protocol in release 2.4 takes for granted that the Lustre client
+ supports the <literal>OBD_CONNECT_EINPROGRESS</literal> connect flag. Clients supporting this
+ flag will retry indefinitely when the server returns <literal>EINPROGRESS</literal> in a
+ reply. Here is the list of Lustre client version which are compatible with release 2.4:</para>
<itemizedlist>
<listitem>
- <para>2.3-based clients and beyond</para>
+ <para>Release 2.3-based clients and beyond</para>
</listitem>
<listitem>
- <para>1.8 clients newer or equal to 1.8.9-wc1</para>
+ <para>Release 1.8 clients newer or equal to release 1.8.9-wc1</para>
</listitem>
<listitem>
- <para>2.1 clients newer or equal to 2.1.4</para>
+ <para>Release 2.1 clients newer or equal to release 2.1.4</para>
</listitem>
</itemizedlist>
</section>
<section xml:id="granted_cache_and_quota_limits">
<title><indexterm><primary>Quotas</primary><secondary>known issues</secondary></indexterm>Granted Cache and Quota Limits</title>
- <para>In Lustre, granted cache does not respect quota limits. In this situation, OSTs grant cache to Lustre client to accelerate I/O. Granting cache causes writes to be successful in OSTs, even if they exceed the quota limits, and will overwrite them.</para>
+ <para>In a Lustre file system, granted cache does not respect quota limits. In this situation,
+ OSTs grant cache to a Lustre client to accelerate I/O. Granting cache causes writes to be
+ successful in OSTs, even if they exceed the quota limits, and will overwrite them.</para>
<para>The sequence is:</para>
<orderedlist>
<listitem>
- <para>A user writes files to Lustre.</para>
+ <para>A user writes files to the Lustre file system.</para>
</listitem>
<listitem>
<para>If the Lustre client has enough granted cache, then it returns 'success' to users and arranges the writes to the OSTs.</para>
</section>
<section xml:id="lustre_quota_statistics">
<title><indexterm><primary>Quotas</primary><secondary>statistics</secondary></indexterm>Lustre Quota Statistics</title>
- <para>Lustre includes statistics that monitor quota activity, such as the kinds of quota RPCs sent during a specific period, the average time to complete the RPCs, etc. These statistics are useful to measure performance of a Lustre file system.</para>
+ <para>The Lustre software includes statistics that monitor quota activity, such as the kinds of
+ quota RPCs sent during a specific period, the average time to complete the RPCs, etc. These
+ statistics are useful to measure performance of a Lustre file system.</para>
<para>Each quota statistic consists of a quota event and <literal>min_time</literal>, <literal>max_time</literal> and <literal>sum_time</literal> values for the event.</para>
<informaltable frame="all">
<tgroup cols="2">
</informaltable>
<section remap="h3">
<title>Interpreting Quota Statistics</title>
- <para>Quota statistics are an important measure of a Lustre file system's performance. Interpreting these statistics correctly can help you diagnose problems with quotas, and may indicate adjustments to improve system performance.</para>
+ <para>Quota statistics are an important measure of the performance of a Lustre file system.
+ Interpreting these statistics correctly can help you diagnose problems with quotas, and may
+ indicate adjustments to improve system performance.</para>
<para>For example, if you run this command on the OSTs:</para>
<screen>lctl get_param lquota.testfs-OST0000.stats</screen>
<para>You will get a result similar to this:</para>
</listitem>
</itemizedlist>
<note>
- <para><emphasis role="bold">It is strongly recommended that hardware RAID be used with Lustre.</emphasis> Lustre currently does not support any redundancy at the file system level and RAID is required to protect against disk failure.</para>
+ <para><emphasis role="bold">It is strongly recommended that storage used in a Lustre file system
+ be configured with hardware RAID.</emphasis> The Lustre software does not support redundancy
+ at the file system level and RAID is required to protect against disk failure.</para>
</note>
<section xml:id="dbdoclet.50438208_60972">
<title>
</section>
<section xml:id="dbdoclet.50438208_51921">
<title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>RAID options</tertiary></indexterm>Formatting Options for RAID Devices</title>
- <para>When formatting a file system on a RAID device, it is beneficial to ensure that I/O requests are aligned with the underlying RAID geometry. This ensures that the Lustre RPCs do not generate unnecessary disk operations which may reduce performance dramatically. Use the <literal>--mkfsoptions</literal> parameter to specify additional parameters when formatting the OST or MDT.</para>
+ <para>When formatting a file system on a RAID device, it is beneficial to ensure that I/O
+ requests are aligned with the underlying RAID geometry. This ensures that Lustre RPCs do not
+ generate unnecessary disk operations which may reduce performance dramatically. Use the
+ <literal>--mkfsoptions</literal> parameter to specify additional parameters when formatting
+ the OST or MDT.</para>
<para>For RAID 5, RAID 6, or RAID 1+0 storage, specifying the following option to the <literal>--mkfsoptions</literal> parameter option improves the layout of the file system metadata, ensuring that no single disk contains all of the allocation bitmaps:</para>
<screen>-E stride = <replaceable>chunk_blocks</replaceable> </screen>
<para>The <literal><replaceable>chunk_blocks</replaceable></literal> variable is in units of 4096-byte blocks and represents the amount of contiguous data written to a single disk before moving to the next disk. This is alternately referred to as the RAID stripe size. This is applicable to both MDT and OST file systems.</para>
<section remap="h3">
<title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>external journal</tertiary></indexterm>Choosing Parameters for an External Journal</title>
<para>If you have configured a RAID array and use it directly as an OST, it contains both data and metadata. For better performance, we recommend putting the OST journal on a separate device, by creating a small RAID 1 array and using it as an external journal for the OST.</para>
- <para>Lustre's default journal size is 400 MB. A journal size of up to 1 GB has shown increased performance but diminishing returns are seen for larger journals. Additionally, a copy of the journal is kept in RAM. Therefore, make sure you have enough memory available to hold copies of all the journals.</para>
+ <para>In a Lustre file system, the default journal size is 400 MB. A journal size of up to 1
+ GB has shown increased performance but diminishing returns are seen for larger journals.
+ Additionally, a copy of the journal is kept in RAM. Therefore, make sure you have enough
+ memory available to hold copies of all the journals.</para>
<para>The file system journal options are specified to <literal>mkfs.lustre</literal> using
the <literal>--mkfsoptions</literal> parameter. For example:</para>
<screen>--mkfsoptions "<replaceable>other_options</replaceable> -j -J device=/dev/mdJ" </screen>
<para>Depending on your cluster size and workload, you may want to connect a SAN to a Lustre file system. Before making this connection, consider the following:</para>
<itemizedlist>
<listitem>
- <para>In many SAN file systems without Lustre, clients allocate and lock blocks or inodes individually as they are updated. The Lustre design avoids the high contention that some of these blocks and inodes may have.</para>
+ <para>In many SAN file systems, clients allocate and lock blocks or inodes individually as
+ they are updated. The design of the Lustre file system avoids the high contention that
+ some of these blocks and inodes may have.</para>
</listitem>
<listitem>
- <para>Lustre is highly scalable and can have a very large number of clients. SAN switches do not scale to a large number of nodes, and the cost per port of a SAN is generally higher than other networking.</para>
+ <para>The Lustre file system is highly scalable and can have a very large number of clients.
+ SAN switches do not scale to a large number of nodes, and the cost per port of a SAN is
+ generally higher than other networking.</para>
</listitem>
<listitem>
<para>File systems that allow direct-to-SAN access from the clients have a security risk because clients can potentially read any data on the SAN disks, and misbehaving clients can corrupt the file system for many reasons like improper file system, network, or other kernel software, bad cabling, bad memory, and so on. The risk increases with increase in the number of clients directly accessing the storage.</para>
<glossterm>lfs
</glossterm>
<glossdef>
- <para>The Lustre File System command-line utility that allows end users to interact with
- Lustre-specific features, such as setting or checking file striping or per-target free
+ <para>The Lustre file system command-line utility that allows end users to interact with
+ Lustre software features, such as setting or checking file striping or per-target free
space. For more details, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
linkend="dbdoclet.50438206_94597"/>.</para>
</glossdef>
<glossterm>LNET
</glossterm>
<glossdef>
- <para>Lustre Networking. A message passing network protocol capable of running and routing through various physical layers. LNET forms the underpinning of LNETrpc.</para>
+ <para>Lustre networking. A message passing network protocol capable of running and routing
+ through various physical layers. LNET forms the underpinning of LNETrpc.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lockclient">
<glossterm>MDT0
</glossterm>
<glossdef>
- <para>The metadata target for the file system root. Since Lustre release 2.4, multiple
- metadata targets are possible in the same file system. MDT0 is the root of the file
- system, which must be available for the file system to be accessible.</para>
+ <para>The metadata target for the file system root. Since Lustre software release 2.4,
+ multiple metadata targets are possible in the same file system. MDT0 is the root of the
+ file system, which must be available for the file system to be accessible.</para>
</glossdef>
</glossentry>
<glossentry xml:id="mgs">
<glossterm>Obdfilter </glossterm>
<glossdef>
<para>An older name for the OBD API data object operation device driver that sits between
- the OST and the OSD. In Lustre 2.4 this device has been renamed OFD."</para>
+ the OST and the OSD. In Lustre software release 2.4 this device has been renamed
+ OFD."</para>
</glossdef>
</glossentry>
<glossentry xml:id="objectstorage">
<para>Refers to a storage-device API or protocol involving storage objects. The two most
well known instances of object storage are the T10 iSCSI storage object protocol and the
Lustre object storage protocol (a network implementation of the Lustre object API). The
- principal difference between the Lustre and T10 protocols is that the Lustre protocol
- includes locking and recovery control in the protocol and is not tied to a SCSI transport
- layer.</para>
+ principal difference between the Lustre protocol and T10 protocol is that the Lustre
+ protocol includes locking and recovery control in the protocol and is not tied to a SCSI
+ transport layer.</para>
</glossdef>
</glossentry>
<glossentry xml:id="opencache">
</itemizedlist>
<tip>
- <para>The starting point for administering Lustre is to monitor all logs and console logs for system health:</para>
+ <para>The starting point for administering a Lustre file system is to monitor all logs and
+ console logs for system health:</para>
<para> - Monitor logs on all servers and all clients.</para>
<para> - Invest in tools that allow you to condense logs from multiple systems.</para>
<para> - Use the logging resources provided in the Linux distribution.</para>
<?xml version="1.0" encoding="UTF-8"?>
<part version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink">
<info>
- <title>Tuning Lustre for Performance</title>
+ <title>Tuning a Lustre File System for Performance</title>
</info>
<partintro>
<para>Part IV describes tools and procedures used to tune a Lustre file system for optimum performance. You will find information in this section about:</para>
<title xml:id="installoverview.title">Installation Overview</title>
<para>This chapter provides on overview of the procedures required to set up, install and configure a Lustre file system.</para>
<note>
- <para>
- If you are new to Lustre, you may find it helpful to refer to <xref linkend="part.intro"/> for a description of the Lustre architecture, file system components and terminology before proceeding with the installation procedure.</para>
+ <para> If the Lustre file system is new to you, you may find it helpful to refer to <xref
+ linkend="part.intro"/> for a description of the Lustre architecture, file system components
+ and terminology before proceeding with the installation procedure.</para>
</note>
<section remap="h2">
<title>
- <indexterm><primary>installing</primary></indexterm>
- Steps to Installing Lustre</title>
+ <indexterm>
+ <primary>installing</primary>
+ </indexterm> Steps to Installing the Lustre Software</title>
<para>To set up Lustre file system hardware and install and configure the Lustre software, refer the the chapters below in the order listed:</para>
<orderedlist>
<listitem>
- <para> <emphasis>(Required)</emphasis> <emphasis role="bold">Set up your Lustre file System hardware.</emphasis></para>
+ <para>
+ <emphasis>(Required)</emphasis>
+ <emphasis role="bold">Set up your Lustre file system hardware.</emphasis></para>
<para>See <xref linkend="settinguplustresystem"/> - Provides guidelines for configuring hardware for a Lustre file system including storage, memory, and networking requirements.</para>
</listitem>
<listitem>
<para>See <xref linkend="installinglustre"/> - Describes preparation steps and a procedure for installing the Lustre software.</para>
</listitem>
<listitem>
- <para><emphasis>(Optional)</emphasis> <emphasis role="bold">Configure Lustre Networking (LNET).</emphasis></para>
+ <para><emphasis>(Optional)</emphasis>
+ <emphasis role="bold">Configure Lustre networking (LNET).</emphasis></para>
<para>See <xref linkend="configuringlnet"/> - Describes how to configure LNET if the default
configuration is not sufficient. By default, LNET will use the first TCP/IP interface it
discovers on a system. LNET configuration is required if you are using InfiniBand or
multiple Ethernet interfaces.</para>
</listitem>
<listitem>
- <para><emphasis>(Required)</emphasis> <emphasis role="bold">Configure Lustre.</emphasis></para>
+ <para><emphasis>(Required)</emphasis>
+ <emphasis role="bold">Configure the Lustre file system.</emphasis></para>
<para>See <xref linkend="configuringlustre"/> - Provides an example of a simple Lustre configuration procedure and points to tools for completing more complex configurations.</para>
</listitem>
<listitem>
- <para><emphasis>(Optional)</emphasis> <emphasis role="bold">Configure Lustre Failover.</emphasis></para>
- <para>See <xref linkend="configuringfailover"/> - Describes how to configure Lustre failover using the Heartbeat cluster infrastructure daemon.</para>
+ <para><emphasis>(Optional)</emphasis>
+ <emphasis role="bold">Configure Lustre failover.</emphasis></para>
+ <para>See <xref linkend="configuringfailover"/> - Describes how to configure Lustre
+ failover.</para>
</listitem>
</orderedlist>
</section>
<colspec colname="c3" colnum="3" colwidth="1.46*"/>
<thead>
<row>
- <entry>Lustre Version</entry>
+ <entry>Lustre Release</entry>
<entry>Servers Tested<superscript>1</superscript></entry>
<entry>Clients Tested</entry>
</row>
<tbody>
<row>
<entry><code>kernel-</code><replaceable>ver</replaceable><code>_lustre.</code><replaceable>arch</replaceable></entry>
- <entry>Linux kernel with Lustre patches (often referred to as "patched
- kernel")</entry>
+ <entry>Linux kernel with Lustre software patches (often referred to as
+ "patched kernel")</entry>
</row>
<row>
<entry><code>lustre-</code><replaceable>ver</replaceable><code>_lustre.</code><replaceable>arch</replaceable></entry>
- <entry>Lustre command line tools</entry>
+ <entry>Lustre software command line tools</entry>
</row>
<row>
<entry><code>lustre-modules-</code><replaceable>ver</replaceable><code>_lustre.</code><replaceable>arch</replaceable></entry>
<row>
<entry><code>lustre-tests-</code><replaceable>ver</replaceable><code>_lustre.</code><replaceable>arch</replaceable></entry>
<entry>Lustre I/O Kit benchmarking tools<emphasis role="italic"> (Included in
- Lustre software as of Version 2.2)</emphasis></entry>
+ Lustre software as of release 2.2)</emphasis></entry>
</row>
</tbody>
</tgroup>
Releases</link> repository.</para>
<para>
<table frame="all" xml:id="table_j3r_ym3_gk">
- <title>Packages Installed in Lustre Clients</title>
+ <title>Packages Installed on Lustre Clients</title>
<tgroup cols="2">
<colspec colname="c1" colnum="1" colwidth="1*"/>
<colspec colname="c2" colnum="2" colwidth="1.02*"/>
<row>
<entry><code>lustre-client-tests-</code><replaceable>ver</replaceable></entry>
<entry>Lustre I/O Kit <emphasis role="italic">(Included in Lustre software as
- of Version 2.2)</emphasis>
+ of release 2.2)</emphasis>
</entry>
</row>
</tbody>
used when compiling user-space, kernel-related code.</para>
</listitem>
<listitem>
- <para><code>lustre-source</code> - Source code for Lustre.</para>
+ <para><code>lustre-source</code> - Lustre software source code.</para>
</listitem>
<listitem>
<para><emphasis role="italic">(Recommended) </emphasis><code>perf</code>,
</section>
</section>
<section xml:id="dbdoclet.50438261_41900">
- <title>Lustre Installation Procedure</title>
+ <title>Lustre Software Installation Procedure</title>
<caution>
<para>Before installing the Lustre software, back up ALL data. The Lustre software contains
kernel modifications that interact with storage devices and may introduce security issues
<secondary>from source</secondary>
<tertiary>prerequisites</tertiary>
</indexterm>Overview and Prerequisites</title>
- <para>To install Lustre from source code, the following are required:</para>
+ <para>To install the Lustre software from source code, the following are required:</para>
<itemizedlist>
<listitem>
<para>A x86_64 machine with a fresh installation of a Linux operating system.</para>
<para>(Recommended) Access to a recent version of <link xmlns:xlink="http://www.w3.org/1999/xlink"
xlink:href="http://fedoraproject.org/wiki/EPEL">EPEL</link> containing the
<literal>quilt</literal> utility used for managing a series of patches. </para>
- <note><para>The use of <literal>quilt</literal> is optional for building Lustre, and is detected at configure time if it is installed. Otherwise the <literal>patch</literal> utility is used to apply <literal>ldiskfs</literal> patches to the kernel source. Quilt is useful if you intend to modify the Lustre kernel patches.</para></note>
+ <note><para>The use of <literal>quilt</literal> is optional for building the Lustre software, and is
+ detected at configure time if it is installed. Otherwise the <literal>patch</literal>
+ utility is used to apply <literal>ldiskfs</literal> patches to the kernel source. Quilt
+ is useful if you intend to modify the Lustre kernel patches.</para></note>
</listitem>
<listitem>
<para><emphasis role="italic">(Recommended)</emphasis> At least 1 GB memory on the machine
(Optional)</title>
<para>When using third-party network hardware, you must follow a specific process to install
and recompile the Lustre software. This section provides an installation example that shows
- how to build Lustre release 2.4.x when using the OpenFabrics Enterprise Distribution (OFED*)
- version 3.5-1 open-source software. The same process can be used for other third-party
- network stacks by replacing the OFED installation procedure and using the appropriate
- <literal>--with</literal> option when configuring the Lustre source code.<orderedlist>
+ how to build Lustre software release 2.4.x when using the OpenFabrics Enterprise
+ Distribution (OFED*) version 3.5-1 open-source software. The same process can be used for
+ other third-party network stacks by replacing the OFED installation procedure and using the
+ appropriate <literal>--with</literal> option when configuring the Lustre source code.<orderedlist>
<listitem>
<para>Install the Lustre kernel (see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
linkend="InstallingLustreKernel"/>).</para>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lnetselftest">
<title xml:id="lnetselftest.title">Testing Lustre Network Performance (LNET Self-Test)</title>
- <para><anchor xml:id="dbdoclet.50438223_36273" xreflabel=""/>This chapter describes the LNET self-test, which is used by site administrators to confirm that Lustre Networking (LNET) has been properly installed and configured, and that underlying network software and hardware are performing according to expectations. The chapter includes:</para>
+ <para><anchor xml:id="dbdoclet.50438223_36273" xreflabel=""/>This chapter describes the LNET
+ self-test, which is used by site administrators to confirm that Lustre networking (LNET) has
+ been properly installed and configured, and that underlying network software and hardware are
+ performing according to expectations. The chapter includes:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438223_91742"/></para>
</itemizedlist>
<para>After you have obtained performance results for your Lustre network, refer to <xref linkend="lustretuning"/> for information about parameters that can be used to tune LNET for optimum performance.</para>
<note>
- <para>Apart from the performance impact, LNET self-test is invisible to Lustre.</para>
+ <para>Apart from the performance impact, LNET self-test is invisible to the Lustre file
+ system.</para>
</note>
<para>An LNET self-test cluster includes two types of nodes:</para>
<itemizedlist>
</section>
<section xml:id="dbdoclet.50438223_42848">
<title>Defining and Running the Tests</title>
- <para>A <emphasis>test</emphasis> generates a network load between two groups of nodes, a source group identified using the <literal>--from</literal> parameter and a target group identified using the <literal>--to</literal> parameter. When a test is running, each node in the <literal>--from <replaceable>group</replaceable></literal> simulates a client by sending requests to nodes in the <literal>--to <replaceable>group</replaceable></literal>, which are simulating a set of servers, and then receives responses in return. This activity is designed to mimic Lustre RPC traffic.</para>
+ <para>A <emphasis>test</emphasis> generates a network load between two groups of nodes, a
+ source group identified using the <literal>--from</literal> parameter and a target group
+ identified using the <literal>--to</literal> parameter. When a test is running, each node in
+ the <literal>--from <replaceable>group</replaceable></literal> simulates a client by sending
+ requests to nodes in the <literal>--to <replaceable>group</replaceable></literal>, which are
+ simulating a set of servers, and then receives responses in return. This activity is
+ designed to mimic Lustre file system RPC traffic.</para>
<para>A <emphasis>batch</emphasis> is a collection of tests that are started and stopped together and run in parallel. A test must always be run as part of a batch, even if it is just a single test. Users can only run or stop a test batch, not individual tests.</para>
- <para>Tests in a batch are non-destructive to the file system, and can be run in a normal Lustre environment (provided the performance impact is acceptable).</para>
+ <para>Tests in a batch are non-destructive to the file system, and can be run in a normal
+ Lustre file system environment (provided the performance impact is acceptable).</para>
<para>A simple batch might contain a single test, for example, to determine whether the network bandwidth presents an I/O bottleneck. In this example, the <literal>--to <replaceable>group</replaceable></literal> could be comprised of Lustre OSSs and <literal>--from <replaceable>group</replaceable></literal> the compute nodes. A second test could be added to perform pings from a login node to the MDS to see how checkpointing affects the <literal>ls -l</literal> process.</para>
<para>Two types of tests are available:</para>
<itemizedlist>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustredebugging">
- <title xml:id="lustredebugging.title">Lustre Debugging</title>
- <para>This chapter describes tips and information to debug Lustre, and includes the following sections:</para>
+ <title xml:id="lustredebugging.title">Debugging a Lustre File System</title>
+ <para>This chapter describes tips and information to debug a Lustre file system, and includes the
+ following sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438274_15874"/></para>
linkend="dbdoclet.50438219_38274"/>.</para>
</listitem>
<listitem>
- <para>Lustre subsystem asserts - A panic-style assertion (LBUG) in the kernel causes
- Lustre to dump the debug log to the file
+ <para>Lustre subsystem asserts - A panic-style assertion (LBUG) in the kernel causes the
+ Lustre file system to dump the debug log to the file
<literal>/tmp/lustre-log.<replaceable>timestamp</replaceable></literal> where it can
be retrieved after a reboot. For more information, see <xref
linkend="dbdoclet.50438198_40669"/>.</para>
</emphasis> . <literal>syslogd</literal> prints fatal or serious messages at this log.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Crash dumps</emphasis> . On crash-dump enabled kernels, sysrq c produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64 KB of the log) to the console.</para>
+ <para><emphasis role="bold">Crash dumps</emphasis> . On crash-dump enabled kernels,
+ sysrq c produces a crash dump. The Lustre software enhances this crash dump with a log
+ dump (the last 64 KB of the log) to the console.</para>
</listitem>
<listitem>
<para><emphasis role="bold">
</section>
<section remap="h4">
<title><indexterm><primary>debugging</primary><secondary>developer tools</secondary></indexterm>Tools for Developers</title>
- <para>The tools described in this section may be useful for debugging Lustre in a development environment.</para>
+ <para>The tools described in this section may be useful for debugging a Lustre file system
+ in a development environment.</para>
<para>Of general interest is:</para>
<itemizedlist>
<listitem>
<para><literal>
<replaceable>leak_finder.pl</replaceable>
- </literal> . This program provided with Lustre is useful for finding memory leaks in the code.</para>
+ </literal> . This program provided with the Lustre software is useful for finding
+ memory leaks in the code.</para>
</listitem>
</itemizedlist>
<para>A virtual machine is often used to create an isolated development and test environment. Some commonly-used virtual machines are:</para>
<title><indexterm><primary>debugging</primary><secondary>message format</secondary></indexterm>Understanding the Lustre Debug Messaging Format</title>
<para>Lustre debug messages are categorized by originating subsystem, message type, and location in the source code. For a list of subsystems and message types, see <xref linkend="dbdoclet.50438274_57603"/>.</para>
<note>
- <para>For a current list of subsystems and debug message types, see <literal>libcfs/include/libcfs/libcfs_debug.h</literal> in the Lustre tree</para>
+ <para>For a current list of subsystems and debug message types, see
+ <literal>libcfs/include/libcfs/libcfs_debug.h</literal> in the Lustre software
+ tree</para>
</note>
<para>The elements of a Lustre debug message are described in <xref linkend="dbdoclet.50438274_57177"/> Format of Lustre Debug Messages.</para>
<section xml:id="dbdoclet.50438274_57603">
<title>Lustre Debug Messages</title>
- <para>Each Lustre debug message has the tag of the subsystem it originated in, the message type, and the location in the source code. The subsystems and debug types used in Lustre are as follows:</para>
+ <para>Each Lustre debug message has the tag of the subsystem it originated in, the message
+ type, and the location in the source code. The subsystems and debug types used are as
+ follows:</para>
<itemizedlist>
<listitem>
<para> Standard Subsystems:</para>
</section>
<section xml:id="dbdoclet.50438274_57177">
<title>Format of Lustre Debug Messages</title>
- <para>Lustre uses the <literal>CDEBUG()</literal> and <literal>CERROR()</literal> macros to print the debug or error messages. To print the message, the <literal>CDEBUG()</literal> macro uses the function <literal>libcfs_debug_msg()</literal> (<literal>libcfs/libcfs/tracefile.c</literal>). The message format is described below, along with an example.</para>
+ <para>The Lustre software uses the <literal>CDEBUG()</literal> and
+ <literal>CERROR()</literal> macros to print the debug or error messages. To print the
+ message, the <literal>CDEBUG()</literal> macro uses the function
+ <literal>libcfs_debug_msg()</literal> (<literal>libcfs/libcfs/tracefile.c</literal>).
+ The message format is described below, along with an example.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
</section>
<section remap="h3">
<title><indexterm><primary>debugging</primary><secondary>disk contents</secondary></indexterm>Looking at Disk Content</title>
- <para>In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The <literal>lfs</literal> tool can be used to obtain this information for a given file using the <literal>getstripe</literal> subcommand. Use a corresponding <literal>lfs setstripe</literal> command to specify striping attributes for a new file or directory.</para>
- <para>The <literal>lfs getstripe</literal> command takes a Lustre filename as input and lists all the objects that form a part of this file. To obtain this information for the file <literal>/mnt/lustre/frog</literal> in Lustre file system, run:</para>
+ <para>In a Lustre file system, the inodes on the metadata server contain extended attributes
+ (EAs) that store information about file striping. EAs contain a list of all object IDs and
+ their locations (that is, the OST that stores them). The <literal>lfs</literal> tool can be
+ used to obtain this information for a given file using the <literal>getstripe</literal>
+ subcommand. Use a corresponding <literal>lfs setstripe</literal> command to specify striping
+ attributes for a new file or directory.</para>
+ <para>The <literal>lfs getstripe</literal> command takes a Lustre filename as input and lists
+ all the objects that form a part of this file. To obtain this information for the file
+ <literal>/mnt/lustre/frog</literal> in a Lustre file system, run:</para>
<screen>$ lfs getstripe /mnt/lustre/frog
lmm_stripe_count: 2
lmm_stripe_size: 1048576
2 818855 0xc7ea7 0
0 873123 0xd52a3 0
</screen>
- <para>The <literal>debugfs</literal> tool is provided in the <literal>e2fsprogs</literal> package. It can be used for interactive debugging of an <literal>ldiskfs</literal> file system. The <literal>debugfs</literal> tool can either be used to check status or modify information in the file system. In Lustre, all objects that belong to a file are stored in an underlying <literal>ldiskfs</literal> file system on the OSTs. The file system uses the object IDs as the file names. Once the object IDs are known, use the <literal>debugfs</literal> tool to obtain the attributes of all objects from different OSTs.</para>
+ <para>The <literal>debugfs</literal> tool is provided in the <literal>e2fsprogs</literal>
+ package. It can be used for interactive debugging of an <literal>ldiskfs</literal> file
+ system. The <literal>debugfs</literal> tool can either be used to check status or modify
+ information in the file system. In a Lustre file system, all objects that belong to a file
+ are stored in an underlying <literal>ldiskfs</literal> file system on the OSTs. The file
+ system uses the object IDs as the file names. Once the object IDs are known, use the
+ <literal>debugfs</literal> tool to obtain the attributes of all objects from different
+ OSTs.</para>
<para>A sample run for the <literal>/mnt/lustre/frog</literal> file used in the above example is shown here:</para>
<screen>$ debugfs -c -R "stat O/0/d$((818855 % 32))/818855" /dev/vgmyth/lvmythost2
strings /tmp/last_rcvd | head -1
myth-OST0004_UUID
</screen>
- <para>It is also possible (and easier) to extract this from the filesystem label using the <literal>dumpe2fs</literal> command:</para>
+ <para>It is also possible (and easier) to extract this from the file system label using the
+ <literal>dumpe2fs</literal> command:</para>
<screen>dumpe2fs -h /dev/sdc | grep volume
dumpe2fs 1.41.90.wc3 (28-May-2011)
Filesystem volume name: myth-OST0004
</section>
<section remap="h3">
<title>Tracing Lock Traffic</title>
- <para>Lustre has a specific debug type category for tracing lock traffic. Use:</para>
+ <para>The Lustre software provides a specific debug type category for tracing lock traffic.
+ Use:</para>
<screen>lctl> filter all_types
lctl> show dlmtrace
lctl> debug_kernel [<replaceable>filename</replaceable>] </screen>
</section>
<section xml:id="dbdoclet.50438274_80443">
<title><indexterm><primary>debugging</primary><secondary>developers tools</secondary></indexterm>Lustre Debugging for Developers</title>
- <para>The procedures in this section may be useful to developers debugging Lustre code.</para>
+ <para>The procedures in this section may be useful to developers debugging Lustre source
+ code.</para>
<section remap="h3">
<title>Adding Debugging to the Lustre Source Code</title>
<para>The debugging infrastructure provides a number of macros that can be used in Lustre source code to aid in debugging or reporting serious errors.</para>
</emphasis></para>
</entry>
<entry>
- <para>A panic-style assertion in the kernel which causes Lustre to dump its circular log to the <literal>/tmp/lustre-log</literal> file. This file can be retrieved after a reboot. <literal>LBUG()</literal> freezes the thread to allow capture of the panic stack. A system reboot is needed to clear the thread.</para>
+ <para>A panic-style assertion in the kernel which causes the Lustre file system to
+ dump its circular log to the <literal>/tmp/lustre-log</literal> file. This file
+ can be retrieved after a reboot. <literal>LBUG()</literal> freezes the thread to
+ allow capture of the panic stack. A system reboot is needed to clear the
+ thread.</para>
</entry>
</row>
<row>
</emphasis></para>
</entry>
<entry>
- <para>Allows insertion of failure points into the Lustre code. This is useful to generate regression tests that can hit a very specific sequence of events. This works in conjunction with "<literal>lctl set_param fail_loc=<replaceable>fail_loc</replaceable></literal>" to set a specific failure point for which a given <literal>OBD_FAIL_CHECK()</literal> will test.</para>
+ <para>Allows insertion of failure points into the Lustre source code. This is useful
+ to generate regression tests that can hit a very specific sequence of events. This
+ works in conjunction with "<literal>lctl set_param
+ fail_loc=<replaceable>fail_loc</replaceable></literal>" to set a specific
+ failure point for which a given <literal>OBD_FAIL_CHECK()</literal> will
+ test.</para>
</entry>
</row>
<row>
<para>This command must be run on the MGS.
</para>
</note>
- <para>In this example, file system lustre has three nodes, <literal>lustre-MDT0000</literal>, <literal>lustre-OST0000</literal>, and <literal>lustre-OST0001</literal>.</para>
+ <para>In this example, file system <literal>testfs</literal> has three nodes,
+ <literal>testfs-MDT0000</literal>, <literal>testfs-OST0000</literal>, and
+ <literal>testfs-OST0001</literal>.</para>
<screen>cfs21:/tmp# cat /proc/fs/lustre/mgs/MGS/live/*
- fsname: lustre
+ fsname: testfs
flags: 0x0 gen: 26
- lustre-MDT0000
- lustre-OST0000
- lustre-OST0001 </screen>
+ testfs-MDT0000
+ testfs-OST0000
+ testfs-OST0001 </screen>
<para>To get the names of all OSTs, run this command on the MDS:</para>
<screen># cat /proc/fs/lustre/lov/<replaceable>fsname</replaceable>-mdtlov/target_obd </screen>
<note>
<para>This command must be run on the MDS.
</para>
</note>
- <para>In this example, there are two OSTs, lustre-OST0000 and lustre-OST0001, which are both active.</para>
- <screen>cfs21:/tmp# cat /proc/fs/lustre/lov/lustre-mdtlov/target_obd
-0: lustre-OST0000_UUID ACTIVE
-1: lustre-OST0001_UUID ACTIVE </screen>
+ <para>In this example, there are two OSTs, testfs-OST0000 and testfs-OST0001, which are both
+ active.</para>
+ <screen>cfs21:/tmp# cat /proc/fs/lustre/lov/testfs-mdtlov/target_obd
+0: testfs-OST0000_UUID ACTIVE
+1: testfs-OST0001_UUID ACTIVE </screen>
</section>
<section xml:id="dbdoclet.50438199_26070">
<title><indexterm><primary>maintenance</primary><secondary>mounting a server</secondary></indexterm>
<section xml:id="dbdoclet.50438199_54623">
<title><indexterm><primary>maintenance</primary><secondary>regenerating config logs</secondary></indexterm>
Regenerating Lustre Configuration Logs</title>
- <para>If the Lustre system's configuration logs are in a state where the file system cannot be started, use the <literal>writeconf</literal> command to erase them. After the <literal>writeconf</literal> command is run and the servers restart, the configuration logs are re-generated and stored on the MGS (as in a new file system).</para>
+ <para>If the Lustre file system configuration logs are in a state where the file system cannot
+ be started, use the <literal>writeconf</literal> command to erase them. After the
+ <literal>writeconf</literal> command is run and the servers restart, the configuration logs
+ are re-generated and stored on the MGS (as in a new file system).</para>
<para>You should only use the <literal>writeconf</literal> command if:</para>
<itemizedlist>
<listitem>
<caution>
<para>The OST pools feature enables a group of OSTs to be named for file striping purposes. If you use OST pools, be aware that running the <literal>writeconf</literal> command erases <emphasis role="bold">all</emphasis> pools information (as well as any other parameters set via <literal>lctl conf_param</literal>). We recommend that the pools definitions (and <literal>conf_param</literal> settings) be executed via a script, so they can be reproduced easily after a <literal>writeconf</literal> is performed.</para>
</caution>
- <para>To regenerate Lustre's system configuration logs:</para>
+ <para>To regenerate Lustre file system configuration logs:</para>
<orderedlist>
<listitem>
<para>Shut down the file system in this order.</para>
<section xml:id="dbdoclet.50438199_31353">
<title><indexterm><primary>maintenance</primary><secondary>changing a NID</secondary></indexterm>
Changing a Server NID</title>
- <para>In Lustre 2.3 or earlier, the <literal>tunefs.lustre --writeconf</literal> command is used to rewrite all of the configuration files.</para>
- <para condition="l24">If you need to change the NID on the MDT or OST, a new <literal>replace_nids</literal> command was added in Lustre 2.4 to simplify this process.
- The <literal>replace_nids</literal> command differs from <literal>tunefs.lustre --writeconf</literal> in that it does not erase the entire configuration log, precluding the need the need to execute the <literal>writeconf</literal> command on all servers and re-specify all permanent parameter settings. However, the <literal>writeconf</literal> command can still be used if desired.</para>
+ <para>In Lustre software release 2.3 or earlier, the <literal>tunefs.lustre
+ --writeconf</literal> command is used to rewrite all of the configuration files.</para>
+ <para condition="l24">If you need to change the NID on the MDT or OST, a new
+ <literal>replace_nids</literal> command was added in Lustre software release 2.4 to simplify
+ this process. The <literal>replace_nids</literal> command differs from <literal>tunefs.lustre
+ --writeconf</literal> in that it does not erase the entire configuration log, precluding the
+ need the need to execute the <literal>writeconf</literal> command on all servers and
+ re-specify all permanent parameter settings. However, the <literal>writeconf</literal> command
+ can still be used if desired.</para>
<para>Change a server NID in these situations:</para>
<itemizedlist>
<listitem>
<orderedlist>
<listitem>
<para>Update the LNET configuration in the <literal>/etc/modprobe.conf</literal> file so the list of server NIDs is correct. Use <literal>lctl list_nids</literal> to view the list of server NIDS.</para>
- <para>The <literal>lctl list_nids</literal> command indicates which network(s) are configured to work with Lustre.</para>
+ <para>The <literal>lctl list_nids</literal> command indicates which network(s) are
+ configured to work with the Lustre file system.</para>
</listitem>
<listitem>
<para>Shut down the file system in this order:</para>
<listitem>
<para>Run the <literal>replace_nids</literal> command on the MGS:</para>
<screen>lctl replace_nids <replaceable>devicename</replaceable> <replaceable>nid1</replaceable>[,nid2,nid3 ...]</screen>
- <para>where <replaceable>devicename</replaceable> is the Lustre target name, e.g. <literal>myfs-OST0013</literal></para>
+ <para>where <replaceable>devicename</replaceable> is the Lustre target name, e.g.
+ <literal>testfs-OST0013</literal></para>
</listitem>
<listitem>
<para>If the MGS and MDS share a partition, stop the MGS:</para>
<note><para>The previous configuration log is backed up on the MGS disk with the suffix <literal>'.bak'</literal>.</para></note>
</section>
<section xml:id="dbdoclet.addingamdt" condition='l24'>
- <title><indexterm><primary>maintenance</primary><secondary>adding an MDT</secondary></indexterm>Adding a new MDT to a Lustre file system</title>
+ <title><indexterm>
+ <primary>maintenance</primary>
+ <secondary>adding an MDT</secondary>
+ </indexterm>Adding a New MDT to a Lustre File System</title>
<para>Additional MDTs can be added to serve one or more remote sub-directories within the
file system. It is possible to have multiple remote sub-directories reference the same MDT.
However, the root directory will always be located on MDT0. To add a new MDT into the file
<para>Discover the maximum MDT index. Each MDTs must have unique index.</para>
<screen>
client$ lctl dl | grep mdc
-36 UP mdc lustre-MDT0000-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
-37 UP mdc lustre-MDT0001-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
-38 UP mdc lustre-MDT0002-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
-39 UP mdc lustre-MDT0003-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
+36 UP mdc testfs-MDT0000-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
+37 UP mdc testfs-MDT0001-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
+38 UP mdc testfs-MDT0002-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
+39 UP mdc testfs-MDT0003-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
</screen>
</listitem>
<listitem>
<title><indexterm><primary>maintenance</primary><secondary>restoring a OST</secondary></indexterm>
<indexterm><primary>maintenance</primary><secondary>removing a OST</secondary></indexterm>
Removing and Restoring OSTs</title>
- <para>OSTs can be removed from and restored to a Lustre file system. Currently in Lustre, removing a OST means the OST is 'deactivated' in the file system, not permanently removed.</para>
+ <para>OSTs can be removed from and restored to a Lustre file system. Removing a OST means the
+ OST is <emphasis role="italic">deactivated</emphasis> in the file system, not permanently
+ removed.</para>
<note><para>A removed OST still appears in the file system; do not create a new OST with the same name.</para></note>
<para>You may want to remove (deactivate) an OST and prevent new files from being written to it in several situations:</para>
<itemizedlist>
<para>List all OSCs on the node, along with their device numbers. Run:</para>
<screen>lctl dl | grep osc</screen>
<para>For example: <literal>lctl dl | grep</literal></para>
- <screen>11 UP osc lustre-OST-0000-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
-12 UP osc lustre-OST-0001-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
-13 IN osc lustre-OST-0000-osc lustre-MDT0000-mdtlov_UUID 5
-14 UP osc lustre-OST-0001-osc lustre-MDT0000-mdtlov_UUID 5</screen>
+ <screen>11 UP osc testfs-OST-0000-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
+12 UP osc testfs-OST-0001-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
+13 IN osc testfs-OST-0000-osc testfs-MDT0000-mdtlov_UUID 5
+14 UP osc testfs-OST-0001-osc testfs-MDT0000-mdtlov_UUID 5</screen>
</listitem>
<listitem>
<para>Determine the device number of the OSC that corresponds to the OST to be
procedure given in either <xref linkend="dbdoclet.50438207_71633"/>, or <xref
linkend="dbdoclet.50438207_21638"/> and <xref linkend="dbdoclet.50438207_22325"/>. </para>
<para>To replace an OST that was removed from service due to corruption or hardware failure,
- the file system needs to be formatted for Lustre, and the Lustre configuration should be
- restored, if available. </para>
+ the file system needs to be formatted using <literal>mkfs.lustre</literal>, and the Lustre
+ file system configuration should be restored, if available. </para>
<para>If the OST configuration files were not backed up, due to the OST file system being
completely inaccessible, it is still possible to replace the failed OST with a new one at
the same OST index. </para>
<section xml:id="dbdoclet.50438199_12607">
<title><indexterm><primary>maintenance</primary><secondary>identifying OST host</secondary></indexterm>
Determining Which Machine is Serving an OST </title>
- <para>In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the NID that is serving a specific OST, run one of the following commands on a client (you do not need to be a root user): <screen>client$ lctl get_param osc.<replaceable>fsname</replaceable>-<replaceable>OSTnumber</replaceable>*.ost_conn_uuid</screen>For example: <screen>client$ lctl get_param osc.*-OST0000*.ost_conn_uuid
-osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen>- OR - <screen>client$ lctl get_param osc.*.ost_conn_uuid
-osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-osc.lustre-OST0001-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-osc.lustre-OST0002-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-osc.lustre-OST0003-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-osc.lustre-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen></para>
+ <para>In the course of administering a Lustre file system, you may need to determine which
+ machine is serving a specific OST. It is not as simple as identifying the machine’s IP
+ address, as IP is only one of several networking protocols that the Lustre software uses and,
+ as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the
+ NID that is serving a specific OST, run one of the following commands on a client (you do not
+ need to be a root user):
+ <screen>client$ lctl get_param osc.<replaceable>fsname</replaceable>-<replaceable>OSTnumber</replaceable>*.ost_conn_uuid</screen>For
+ example:
+ <screen>client$ lctl get_param osc.*-OST0000*.ost_conn_uuid
+osc.testfs-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen>-
+ OR -
+ <screen>client$ lctl get_param osc.*.ost_conn_uuid
+osc.testfs-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
+osc.testfs-OST0001-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
+osc.testfs-OST0002-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
+osc.testfs-OST0003-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
+osc.testfs-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen></para>
</section>
<section xml:id="dbdoclet.50438199_62333">
<title><indexterm><primary>maintenance</primary><secondary>changing failover node address</secondary></indexterm>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremonitoring">
- <title xml:id="lustremonitoring.title">Lustre Monitoring</title>
- <para>This chapter provides information on monitoring Lustre and includes the following sections:</para>
+ <title xml:id="lustremonitoring.title">Monitoring a Lustre File System</title>
+ <para>This chapter provides information on monitoring a Lustre file system and includes the
+ following sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438273_18711"/>Lustre Changelogs</para>
<indexterm><primary>monitoring</primary><secondary>jobstats</secondary></indexterm>
Lustre Jobstats</title>
- <para>The Lustre Jobstats feature is available starting in Lustre version 2.3. It collects file
- system operation statistics for the jobs running on Lustre clients, and exposes them via
- procfs on the server. Job schedulers known to be able to work with jobstats include: SLURM,
- SGE, LSF, Loadleveler, PBS and Maui/MOAB.</para>
- <para>Since Jobstats is implemented in a scheduler-agnostic manner, it is likely that it will be able to work with other schedulers also.</para>
+ <para>The Lustre jobstats feature is available starting in Lustre software release 2.3. It
+ collects file system operation statistics for the jobs running on Lustre clients, and exposes
+ them via procfs on the server. Job schedulers known to be able to work with jobstats include:
+ SLURM, SGE, LSF, Loadleveler, PBS and Maui/MOAB.</para>
+ <para>Since jobstats is implemented in a scheduler-agnostic manner, it is likely that it will be
+ able to work with other schedulers also.</para>
<section remap="h3">
<title><indexterm><primary>monitoring</primary><secondary>jobstats</secondary></indexterm>
Enable/Disable Jobstats</title>
$ lctl get_param jobid_var
jobid_var=disable
</screen>
- <para>The Lustre Jobstats code extracts the job identifier from an environment variable set by
+ <para>The Lustre jobstats code extracts the job identifier from an environment variable set by
the scheduler when the job is started. To enable jobstats, specify the
<literal>jobid_var</literal> to name the environment variable set by the scheduler. For
example, SLURM sets the <literal>SLURM_JOB_ID</literal> environment variable with the unique
- job ID on each client. To permanently enable Jobstats on the <literal>testfs</literal> file
+ job ID on each client. To permanently enable jobstats on the <literal>testfs</literal> file
system:</para>
<screen>$ lctl conf_param testfs.sys.jobid_var=SLURM_JOB_ID</screen>
<para>The value of <literal>jobid_var</literal> can be:</para>
<section remap="h3">
<title><indexterm><primary>monitoring</primary><secondary>jobstats</secondary></indexterm>
Check Job Stats</title>
- <para>Metadata operation statistics are collected on MDTs. These statistics can be accessed for all filesystems and all jobs on the MDT via the <literal>lctl get_param mdt.*.job_stats</literal>. For example, clients running with <literal>jobid_var=procname_uid</literal>:</para>
+ <para>Metadata operation statistics are collected on MDTs. These statistics can be accessed for
+ all file systems and all jobs on the MDT via the <literal>lctl get_param
+ mdt.*.job_stats</literal>. For example, clients running with
+ <literal>jobid_var=procname_uid</literal>:</para>
<screen>
$ lctl get_param mdt.*.job_stats
job_stats:
<title>
<literal>CollectL</literal>
</title>
- <para><literal>CollectL</literal> is another tool that can be used to monitor Lustre. You can run <literal>CollectL</literal> on a Lustre system that has any combination of MDSs, OSTs and clients. The collected data can be written to a file for continuous logging and played back at a later time. It can also be converted to a format suitable for plotting.</para>
+ <para><literal>CollectL</literal> is another tool that can be used to monitor a Lustre file
+ system. You can run <literal>CollectL</literal> on a Lustre system that has any combination of
+ MDSs, OSTs and clients. The collected data can be written to a file for continuous logging and
+ played back at a later time. It can also be converted to a format suitable for
+ plotting.</para>
<para>For more information about <literal>CollectL</literal>, see:</para>
<para><link xl:href="http://collectl.sourceforge.net">http://collectl.sourceforge.net</link></para>
<para>Lustre-specific documentation is also available. See:</para>
</itemizedlist></para>
<para>Another option is to script a simple monitoring solution that looks at various reports
from <literal>ipconfig</literal>, as well as the <literal>procfs</literal> files generated by
- Lustre.</para>
+ the Lustre software.</para>
</section>
</chapter>
</screen></para>
</listitem>
<listitem>
- <para>For Lustre 2.x filesystems, the parent FID will be of the form [0x200000400:0x122:0x0]
- and can be resolved directly using the <literal>lfs fid2path [0x200000404:0x122:0x0]
- /mnt/lustre</literal> command on any Lustre client, and the process is complete.</para>
+ <para>For Lustre software release 2.x file systems, the parent FID will be of the form
+ [0x200000400:0x122:0x0] and can be resolved directly using the <literal>lfs fid2path
+ [0x200000404:0x122:0x0] /mnt/lustre</literal> command on any Lustre client, and the
+ process is complete.</para>
</listitem>
<listitem>
<para>In this example the parent inode FID is an upgraded 1.x inode
statahead reads metadata into memory. When readahead and statahead work well, a process that
accesses data finds that the information it needs is available immediately when requested in
memory without the delay of network I/O.</para>
- <para condition="l22">In Lustre release 2.2.0, the directory statahead feature was improved to
- enhance directory traversal performance. The improvements primarily addressed two
- issues:
- <orderedlist>
- <listitem>
- <para>A race condition existed between the statahead thread and other VFS operations while
- processing asynchronous <literal>getattr</literal> RPC replies, causing duplicate
- entries in dcache. This issue was resolved by using statahead local dcache. </para>
- </listitem>
- <listitem>
- <para>File size/block attributes pre-fetching was not supported, so the traversing thread
- had to send synchronous glimpse size RPCs to OST(s). This issue was resolved by using
- asynchronous glimpse lock (AGL) RPCs to pre-fetch file size/block attributes from
- OST(s).</para>
- </listitem>
- </orderedlist>
- </para>
+ <para condition="l22">In Lustre software release 2.2.0, the directory statahead feature was
+ improved to enhance directory traversal performance. The improvements primarily addressed
+ two issues: <orderedlist>
+ <listitem>
+ <para>A race condition existed between the statahead thread and other VFS operations
+ while processing asynchronous <literal>getattr</literal> RPC replies, causing
+ duplicate entries in dcache. This issue was resolved by using statahead local dcache.
+ </para>
+ </listitem>
+ <listitem>
+ <para>File size/block attributes pre-fetching was not supported, so the traversing
+ thread had to send synchronous glimpse size RPCs to OST(s). This issue was resolved by
+ using asynchronous glimpse lock (AGL) RPCs to pre-fetch file size/block attributes
+ from OST(s).</para>
+ </listitem>
+ </orderedlist>
+ </para>
<section remap="h4">
<title>Tuning File Readahead</title>
<para>File readahead is triggered when two or more sequential reads by an application fail
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustreprogramminginterfaces">
- <title xml:id="lustreprogramminginterfaces.title">Lustre Programming Interfaces</title>
+ <title xml:id="lustreprogramminginterfaces.title">Programming Interfaces</title>
<para>This chapter describes public programming interfaces to that can be used to control various
aspects of a Lustre file system from userspace. This chapter includes the following
sections:</para>
</listitem>
</itemizedlist>
<itemizedlist>
- <para>A Lustre administrator can specify permissions for a specific UID by configuring
- <literal>/etc/lustre/perm.conf</literal> on the MDS. The
+ <para>A Lustre file system administrator can specify permissions for a specific UID by
+ configuring <literal>/etc/lustre/perm.conf</literal> on the MDS. The
<literal>/usr/sbin/l_getidentity</literal> utility parses
<literal>/etc/lustre/perm.conf</literal> to obtain the permission mask for a specified
UID.</para>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustrerecovery">
- <title xml:id="lustrerecovery.title">Lustre Recovery</title>
- <para>This chapter describes how recovery is implemented in Lustre and includes the following sections:</para>
+ <title xml:id="lustrerecovery.title">Lustre File System Recovery</title>
+ <para>This chapter describes how recovery is implemented in a Lustre file system and includes the
+ following sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="recoveryoverview"/></para>
<indexterm><primary>recovery</primary><secondary>commit on share</secondary><see>commit on share</see></indexterm>
<indexterm><primary>lustre</primary><secondary>recovery</secondary><see>recovery</see></indexterm>
Recovery Overview</title>
- <para>Lustre's recovery feature is responsible for dealing with node or network failure and returning the cluster to a consistent, performant state. Because Lustre allows servers to perform asynchronous update operations to the on-disk file system (i.e., the server can reply without waiting for the update to synchronously commit to disk), the clients may have state in memory that is newer than what the server can recover from disk after a crash.</para>
+ <para>The recovery feature provided in the Lustre software is responsible for dealing with node
+ or network failure and returning the cluster to a consistent, performant state. Because the
+ Lustre software allows servers to perform asynchronous update operations to the on-disk file
+ system (i.e., the server can reply without waiting for the update to synchronously commit to
+ disk), the clients may have state in memory that is newer than what the server can recover
+ from disk after a crash.</para>
<para>A handful of different types of failures can cause recovery to occur:</para>
<itemizedlist>
<listitem>
<para> Transient network partition</para>
</listitem>
</itemizedlist>
- <para>For Lustre 2.1.x and all earlier releases, all Lustre failure and recovery operations are based on the concept of connection failure; all imports or exports associated with a given connection are considered to fail if any of them fail. Lustre 2.2.x adds the <xref linkend="imperativerecovery"/> feature which enables the MGS to actively inform clients when a target restarts after a failure, failover or other interruption.</para>
- <para>For information on Lustre recovery, see <xref linkend="metadatereplay"/>. For information on recovering from a corrupt file system, see <xref linkend="commitonshare"/>. For information on resolving orphaned objects, a common issue after recovery, see <xref linkend="dbdoclet.50438225_13916"/>. For information on imperative recovery see <xref linkend="imperativerecovery"/> </para>
+ <para>For Lustre software release 2.1.x and all earlier releases, all Lustre file system failure
+ and recovery operations are based on the concept of connection failure; all imports or exports
+ associated with a given connection are considered to fail if any of them fail. Lustre software
+ release 2.2.x adds the <xref linkend="imperativerecovery"/> feature which enables the MGS to
+ actively inform clients when a target restarts after a failure, failover or other
+ interruption.</para>
+ <para>For information on Lustre file system recovery, see <xref linkend="metadatereplay"/>. For
+ information on recovering from a corrupt file system, see <xref linkend="commitonshare"/>. For
+ information on resolving orphaned objects, a common issue after recovery, see <xref
+ linkend="dbdoclet.50438225_13916"/>. For information on imperative recovery see <xref
+ linkend="imperativerecovery"/>
+ </para>
<section remap="h3">
<title><indexterm><primary>recovery</primary><secondary>client failure</secondary></indexterm>Client Failure</title>
- <para>Recovery from client failure in Lustre is based on lock revocation and other resources, so surviving clients can continue their work uninterrupted. If a client fails to timely respond to a blocking lock callback from the Distributed Lock Manager (DLM) or fails to communicate with the server in a long period of time (i.e., no pings), the client is forcibly removed from the cluster (evicted). This enables other clients to acquire locks blocked by the dead client's locks, and also frees resources (file handles, export data) associated with that client. Note that this scenario can be caused by a network partition, as well as an actual client node system failure. <xref linkend="networkpartition"/> describes this case in more detail.</para>
+ <para>Recovery from client failure in a Lustre file system is based on lock revocation and
+ other resources, so surviving clients can continue their work uninterrupted. If a client
+ fails to timely respond to a blocking lock callback from the Distributed Lock Manager (DLM)
+ or fails to communicate with the server in a long period of time (i.e., no pings), the
+ client is forcibly removed from the cluster (evicted). This enables other clients to acquire
+ locks blocked by the dead client's locks, and also frees resources (file handles,
+ export data) associated with that client. Note that this scenario can be caused by a network
+ partition, as well as an actual client node system failure. <xref linkend="networkpartition"
+ /> describes this case in more detail.</para>
</section>
<section xml:id="clientevictions">
<title><indexterm><primary>recovery</primary><secondary>client eviction</secondary></indexterm>Client Eviction</title>
</section>
<section remap="h3">
<title><indexterm><primary>recovery</primary><secondary>MDS failure</secondary></indexterm>MDS Failure (Failover)</title>
- <para>Highly-available (HA) Lustre operation requires that the metadata server have a peer configured for failover, including the use of a shared storage device for the MDT backing file system. The actual mechanism for detecting peer failure, power off (STONITH) of the failed peer (to prevent it from continuing to modify the shared disk), and takeover of the Lustre MDS service on the backup node depends on external HA software such as Heartbeat. It is also possible to have MDS recovery with a single MDS node. In this case, recovery will take as long as is needed for the single MDS to be restarted.</para>
+ <para>Highly-available (HA) Lustre file system operation requires that the metadata server
+ have a peer configured for failover, including the use of a shared storage device for the
+ MDT backing file system. The actual mechanism for detecting peer failure, power off
+ (STONITH) of the failed peer (to prevent it from continuing to modify the shared disk), and
+ takeover of the Lustre MDS service on the backup node depends on external HA software such
+ as Heartbeat. It is also possible to have MDS recovery with a single MDS node. In this case,
+ recovery will take as long as is needed for the single MDS to be restarted.</para>
<para>When <xref linkend="imperativerecovery"/> is enabled, clients are notified of an MDS restart (either the backup or a restored primary). Clients always may detect an MDS failure either by timeouts of in-flight requests or idle-time ping messages. In either case the clients then connect to the new backup MDS and use the Metadata Replay protocol. Metadata Replay is responsible for ensuring that the backup MDS re-acquires state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk.</para>
<para>The reconnection to a new (or restarted) MDS is managed by the file system configuration loaded by the client when the file system is first mounted. If a failover MDS has been configured (using the <literal>--failnode=</literal> option to <literal>mkfs.lustre</literal> or <literal>tunefs.lustre</literal>), the client tries to reconnect to both the primary and backup MDS until one of them responds that the failed MDT is again available. At that point, the client begins recovery. For more information, see <xref linkend="metadatereplay"/>.</para>
- <para>Transaction numbers are used to ensure that operations are replayed in the order they were originally performed, so that they are guaranteed to succeed and present the same filesystem state as before the failure. In addition, clients inform the new server of their existing lock state (including locks that have not yet been granted). All metadata and lock replay must complete before new, non-recovery operations are permitted. In addition, only clients that were connected at the time of MDS failure are permitted to reconnect during the recovery window, to avoid the introduction of state changes that might conflict with what is being replayed by previously-connected clients.</para>
- <para condition='l24'>Lustre 2.4 introduces multiple metadata targets. If multiple metadata targets are in use, active-active failover is possible. See <xref linkend='dbdoclet.mdtactiveactive'/> for more information.</para>
+ <para>Transaction numbers are used to ensure that operations are replayed in the order they
+ were originally performed, so that they are guaranteed to succeed and present the same file
+ system state as before the failure. In addition, clients inform the new server of their
+ existing lock state (including locks that have not yet been granted). All metadata and lock
+ replay must complete before new, non-recovery operations are permitted. In addition, only
+ clients that were connected at the time of MDS failure are permitted to reconnect during the
+ recovery window, to avoid the introduction of state changes that might conflict with what is
+ being replayed by previously-connected clients.</para>
+ <para condition="l24">Lustre software release 2.4 introduces multiple metadata targets. If
+ multiple metadata targets are in use, active-active failover is possible. See <xref
+ linkend="dbdoclet.mdtactiveactive"/> for more information.</para>
</section>
<section remap="h3">
<title><indexterm><primary>recovery</primary><secondary>OST failure</secondary></indexterm>OST Failure (Failover)</title>
<para>When an OST fails or has communication problems with the client, the default action is that the corresponding OSC enters recovery, and I/O requests going to that OST are blocked waiting for OST recovery or failover. It is possible to administratively mark the OSC as <emphasis>inactive</emphasis> on the client, in which case file operations that involve the failed OST will return an IO error (<literal>-EIO</literal>). Otherwise, the application waits until the OST has recovered or the client process is interrupted (e.g. ,with <emphasis>CTRL-C</emphasis>).</para>
<para>The MDS (via the LOV) detects that an OST is unavailable and skips it when assigning objects to new files. When the OST is restarted or re-establishes communication with the MDS, the MDS and OST automatically perform orphan recovery to destroy any objects that belong to files that were deleted while the OST was unavailable. For more information, see <xref linkend="troubleshootingrecovery"/> (Working with Orphaned Objects).</para>
- <para>While the OSC to OST operation recovery protocol is the same as that between the MDC and MDT using the Metadata Replay protocol, typically the OST commits bulk write operations to disk synchronously and each reply indicates that the request is already committed and the data does not need to be saved for recovery. In some cases, the OST replies to the client before the operation is committed to disk (e.g. truncate, destroy, setattr, and I/O operations in very new versions of Lustre), and normal replay and resend handling is done, including resending of the bulk writes. In this case, the client keeps a copy of the data available in memory until the server indicates that the write has committed to disk.</para>
+ <para>While the OSC to OST operation recovery protocol is the same as that between the MDC and
+ MDT using the Metadata Replay protocol, typically the OST commits bulk write operations to
+ disk synchronously and each reply indicates that the request is already committed and the
+ data does not need to be saved for recovery. In some cases, the OST replies to the client
+ before the operation is committed to disk (e.g. truncate, destroy, setattr, and I/O
+ operations in newer releases of the Lustre software), and normal replay and resend handling
+ is done, including resending of the bulk writes. In this case, the client keeps a copy of
+ the data available in memory until the server indicates that the write has committed to
+ disk.</para>
<para>To force an OST recovery, unmount the OST and then mount it again. If the OST was connected to clients before it failed, then a recovery process starts after the remount, enabling clients to reconnect to the OST and replay transactions in their queue. When the OST is in recovery mode, all new client connections are refused until the recovery finishes. The recovery is complete when either all previously-connected clients reconnect and their transactions are replayed or a client connection attempt times out. If a connection attempt times out, then all clients waiting to reconnect (and their transactions) are lost.</para>
<note>
<para>If you know an OST will not recover a previously-connected client (if, for example, the client has crashed), you can manually abort the recovery using this command:</para>
</section>
<section xml:id="metadatereplay">
<title><indexterm><primary>recovery</primary><secondary>metadata replay</secondary></indexterm>Metadata Replay</title>
- <para>Highly available Lustre operation requires that the MDS have a peer configured for failover, including the use of a shared storage device for the MDS backing file system. When a client detects an MDS failure, it connects to the new MDS and uses the metadata replay protocol to replay its requests.</para>
+ <para>Highly available Lustre file system operation requires that the MDS have a peer configured
+ for failover, including the use of a shared storage device for the MDS backing file system.
+ When a client detects an MDS failure, it connects to the new MDS and uses the metadata replay
+ protocol to replay its requests.</para>
<para>Metadata replay ensures that the failover MDS re-accumulates state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk.</para>
<section remap="h3">
<title>XID Numbers</title>
<section remap="h3">
<title>Transaction Numbers</title>
<para>Each client request processed by the server that involves any state change (metadata update, file open, write, etc., depending on server type) is assigned a transaction number by the server that is a target-unique, monotonically increasing, server-wide 64-bit integer. The transaction number for each file system-modifying request is sent back to the client along with the reply to that client request. The transaction numbers allow the client and server to unambiguously order every modification to the file system in case recovery is needed.</para>
- <para>Each reply sent to a client (regardless of request type) also contains the last committed transaction number that indicates the highest transaction number committed to the file system. The <literal>ldiskfs</literal> backing file system that Lustre uses enforces the requirement that any earlier disk operation will always be committed to disk before a later disk operation, so the last committed transaction number also reports that any requests with a lower transaction number have been committed to disk.</para>
+ <para>Each reply sent to a client (regardless of request type) also contains the last
+ committed transaction number that indicates the highest transaction number committed to the
+ file system. The <literal>ldiskfs</literal> backing file system that the Lustre software
+ uses enforces the requirement that any earlier disk operation will always be committed to
+ disk before a later disk operation, so the last committed transaction number also reports
+ that any requests with a lower transaction number have been committed to disk.</para>
</section>
<section remap="h3">
<title>Replay and Resend</title>
- <para>Lustre recovery can be separated into two distinct types of operations: <emphasis>replay</emphasis> and <emphasis>resend</emphasis>.</para>
+ <para>Lustre file system recovery can be separated into two distinct types of operations:
+ <emphasis>replay</emphasis> and <emphasis>resend</emphasis>.</para>
<para><emphasis>Replay</emphasis> operations are those for which the client received a reply from the server that the operation had been successfully completed. These operations need to be redone in exactly the same manner after a server restart as had been reported before the server failed. Replay can only happen if the server failed; otherwise it will not have lost any state in memory.</para>
<para><emphasis>Resend</emphasis> operations are those for which the client never received a reply, so their final state is unknown to the client. The client sends unanswered requests to the server again in XID order, and again awaits a reply for each one. In some cases, resent requests have been handled and committed to disk by the server (possibly also having dependent operations committed), in which case, the server performs reply reconstruction for the lost reply. In other cases, the server did not receive the lost request at all and processing proceeds as with any normal request. These are what happen in the case of a network interruption. It is also possible that the server received the request, but was unable to reply or commit it to disk before failure.</para>
</section>
</section>
<section remap="h3">
<title><indexterm><primary>recovery</primary><secondary>locks</secondary></indexterm>Lock Recovery</title>
- <para>If all requests were replayed successfully and all clients reconnected, clients then do lock replay locks -- that is, every client sends information about every lock it holds from this server and its state (whenever it was granted or not, what mode, what properties and so on), and then recovery completes successfully. Currently, Lustre does not do lock verification and just trusts clients to present an accurate lock state. This does not impart any security concerns since Lustre 1.x clients are trusted for other information (e.g. user ID) during normal operation also.</para>
+ <para>If all requests were replayed successfully and all clients reconnected, clients then do
+ lock replay locks -- that is, every client sends information about every lock it holds from
+ this server and its state (whenever it was granted or not, what mode, what properties and so
+ on), and then recovery completes successfully. Currently, the Lustre software does not do
+ lock verification and just trusts clients to present an accurate lock state. This does not
+ impart any security concerns since Lustre software release 1.x clients are trusted for other
+ information (e.g. user ID) during normal operation also.</para>
<para>After all of the saved requests and locks have been replayed, the client sends an <literal>MDS_GETSTATUS</literal> request with last-replay flag set. The reply to that request is held back until all clients have completed replay (sent the same flagged getstatus request), so that clients don't send non-recovery requests before recovery is complete.</para>
</section>
<section remap="h3">
</section>
<section xml:id="versionbasedrecovery">
<title><indexterm><primary>Version-based recovery (VBR)</primary></indexterm>Version-based Recovery</title>
- <para>The Version-based Recovery (VBR) feature improves Lustre reliability in cases where client requests (RPCs) fail to replay during recovery
- <footnote>
- <para>There are two scenarios under which client RPCs are not replayed: (1) Non-functioning or isolated clients do not reconnect, and they cannot replay their RPCs, causing a gap in the replay sequence. These clients get errors and are evicted. (2) Functioning clients connect, but they cannot replay some or all of their RPCs that occurred after the gap caused by the non-functioning/isolated clients. These clients get errors (caused by the failed clients). With VBR, these requests have a better chance to replay because the "gaps" are only related to specific files that the missing client(s) changed.</para>
+ <para>The Version-based Recovery (VBR) feature improves Lustre file system reliability in cases
+ where client requests (RPCs) fail to replay during recovery <footnote>
+ <para>There are two scenarios under which client RPCs are not replayed: (1) Non-functioning
+ or isolated clients do not reconnect, and they cannot replay their RPCs, causing a gap in
+ the replay sequence. These clients get errors and are evicted. (2) Functioning clients
+ connect, but they cannot replay some or all of their RPCs that occurred after the gap
+ caused by the non-functioning/isolated clients. These clients get errors (caused by the
+ failed clients). With VBR, these requests have a better chance to replay because the
+ "gaps" are only related to specific files that the missing client(s)
+ changed.</para>
</footnote>.</para>
- <para>In pre-VBR versions of Lustre, if the MGS or an OST went down and then recovered, a recovery process was triggered in which clients attempted to replay their requests. Clients were only allowed to replay RPCs in serial order. If a particular client could not replay its requests, then those requests were lost as well as the requests of clients later in the sequence. The ''downstream'' clients never got to replay their requests because of the wait on the earlier client's RPCs. Eventually, the recovery period would time out (so the component could accept new requests), leaving some number of clients evicted and their requests and data lost.</para>
+ <para>In pre-VBR releases of the Lustre software, if the MGS or an OST went down and then
+ recovered, a recovery process was triggered in which clients attempted to replay their
+ requests. Clients were only allowed to replay RPCs in serial order. If a particular client
+ could not replay its requests, then those requests were lost as well as the requests of
+ clients later in the sequence. The ''downstream'' clients never got to
+ replay their requests because of the wait on the earlier client's RPCs. Eventually, the
+ recovery period would time out (so the component could accept new requests), leaving some
+ number of clients evicted and their requests and data lost.</para>
<para>With VBR, the recovery mechanism does not result in the loss of clients or their data, because changes in inode versions are tracked, and more clients are able to reintegrate into the cluster. With VBR, inode tracking looks like this:</para>
<itemizedlist>
<listitem>
<para>VBR recovery is fully transparent to users. It may lead to slightly longer recovery times if the cluster loses several clients during server recovery.</para>
<section remap="h3">
<title><indexterm><primary>Version-based recovery (VBR)</primary><secondary>messages</secondary></indexterm>VBR Messages</title>
- <para>The VBR feature is built into the Lustre recovery functionality. It cannot be disabled. These are some VBR messages that may be displayed:</para>
+ <para>The VBR feature is built into the Lustre file system recovery functionality. It cannot
+ be disabled. These are some VBR messages that may be displayed:</para>
<screen>DEBUG_REQ(D_WARNING, req, "Version mismatch during replay\n");</screen>
<para>This message indicates why the client was evicted. No action is needed.</para>
<screen>CWARN("%s: version recovery fails, reconnecting\n");</screen>
</section>
<section xml:id="commitonshare">
<title><indexterm><primary>commit on share</primary></indexterm>Commit on Share</title>
- <para>The commit-on-share (COS) feature makes Lustre recovery more reliable by preventing missing clients from causing cascading evictions of other clients. With COS enabled, if some Lustre clients miss the recovery window after a reboot or a server failure, the remaining clients are not evicted.</para>
+ <para>The commit-on-share (COS) feature makes Lustre file system recovery more reliable by
+ preventing missing clients from causing cascading evictions of other clients. With COS
+ enabled, if some Lustre clients miss the recovery window after a reboot or a server failure,
+ the remaining clients are not evicted.</para>
<note>
<para>The commit-on-share feature is enabled, by default.</para>
</note>
</section>
<section xml:id="imperativerecovery">
<title><indexterm><primary>imperative recovery</primary></indexterm>Imperative Recovery</title>
- <para>Imperative Recovery (IR) was first introduced in Lustre 2.2.0</para>
- <para>Large-scale lustre implementations have historically experienced problems recovering in a timely manner after a server failure. This is due to the way that clients detect the server failure and how the servers perform their recovery. Many of the processes are driven by the RPC timeout, which must be scaled with system size to prevent false diagnosis of server death. The purpose of imperative recovery is to reduce the recovery window by actively informing clients of server failure. The resulting reduction in the recovery window will minimize target downtime and therefore increase overall system availability. Imperative Recovery does not remove previous recovery mechanisms, and client timeout-based recovery actions can occur in a cluster when IR is enabled as each client can still independently disconnect and reconnect from a target. In case of a mix of IR and non-IR clients connecting to an OST or MDT, the server cannot reduce its recovery timeout window, because it cannot be sure that all clients have been notified of the server restart in a timely manner. Even in such mixed environments the time to complete recovery may be reduced, since IR-enabled clients will still be notified to reconnect to the server promptly and allow recovery to complete as soon as the last non-IR client detects the server failure.</para>
+ <para>Imperative Recovery (IR) was first introduced in Lustre software release 2.2.0.</para>
+ <para>Large-scale Lustre file system implementations have historically experienced problems
+ recovering in a timely manner after a server failure. This is due to the way that clients
+ detect the server failure and how the servers perform their recovery. Many of the processes
+ are driven by the RPC timeout, which must be scaled with system size to prevent false
+ diagnosis of server death. The purpose of imperative recovery is to reduce the recovery window
+ by actively informing clients of server failure. The resulting reduction in the recovery
+ window will minimize target downtime and therefore increase overall system availability.
+ Imperative Recovery does not remove previous recovery mechanisms, and client timeout-based
+ recovery actions can occur in a cluster when IR is enabled as each client can still
+ independently disconnect and reconnect from a target. In case of a mix of IR and non-IR
+ clients connecting to an OST or MDT, the server cannot reduce its recovery timeout window,
+ because it cannot be sure that all clients have been notified of the server restart in a
+ timely manner. Even in such mixed environments the time to complete recovery may be reduced,
+ since IR-enabled clients will still be notified to reconnect to the server promptly and allow
+ recovery to complete as soon as the last non-IR client detects the server failure.</para>
<section remap="h3">
<title><indexterm><primary>imperative recovery</primary><secondary>MGS role</secondary></indexterm>MGS role</title>
- <para>The MGS now holds additional information about Lustre targets, in the form of a Target Status Table. Whenever a target registers with the MGS, there is a corresponding entry in this table identifying the target. This entry includes NID information, and state/version information for the target. When a client mounts the filesystem, it caches a locked copy of this table, in the form of a Lustre configuration log. When a target restart occurs, the MGS revokes the client lock, forcing all clients to reload the table. Any new targets will have an updated version number, the client detects this and reconnects to the restarted target. Since successful IR notification of server restart depends on all clients being registered with the MGS, and there is no other node to notify clients in case of MGS restart, the MGS will disable IR for a period when it first starts. This interval is configurable, as shown in <xref linkend="imperativerecoveryparameters"/></para>
+ <para>The MGS now holds additional information about Lustre targets, in the form of a Target Status
+ Table. Whenever a target registers with the MGS, there is a corresponding entry in this
+ table identifying the target. This entry includes NID information, and state/version
+ information for the target. When a client mounts the file system, it caches a locked copy of
+ this table, in the form of a Lustre configuration log. When a target restart occurs, the MGS
+ revokes the client lock, forcing all clients to reload the table. Any new targets will have
+ an updated version number, the client detects this and reconnects to the restarted target.
+ Since successful IR notification of server restart depends on all clients being registered
+ with the MGS, and there is no other node to notify clients in case of MGS restart, the MGS
+ will disable IR for a period when it first starts. This interval is configurable, as shown
+ in <xref linkend="imperativerecoveryparameters"/></para>
<para>Because of the increased importance of the MGS in recovery, it is strongly recommended that the MGS node be separate from the MDS. If the MGS is co-located on the MDS node, then in case of MDS/MGS failure there will be no IR notification for the MDS restart, and clients will always use timeout-based recovery for the MDS. IR notification would still be used in the case of OSS failure and recovery.</para>
<para>Unfortunately, it’s impossible for the MGS to know how many clients have been successfully notified or whether a specific client has received the restarting target information. The only thing the MGS can do is tell the target that, for example, all clients are imperative recovery-capable, so it is not necessary to wait as long for all clients to reconnect. For this reason, we still require a timeout policy on the target side, but this timeout value can be much shorter than normal recovery. </para>
</section>
</section>
<section remap="h5">
<title>Checking Imperative Recovery State - client</title>
- <para>A `client’ in IR means a lustre client or a MDT. You can get the IR state on any node which running client or MDT, those nodes will always have an MGC running. An example from a client:</para>
+ <para>A `client’ in IR means a Lustre client or a MDT. You can get the IR state on any node which
+ running client or MDT, those nodes will always have an MGC running. An example from a
+ client:</para>
<screen>
[client]$ lctl get_param mgc.*.ir_state
mgc.MGC192.168.127.6@tcp.ir_state=
</section>
<section remap="h3" xml:id="imperativerecoveryrecomendations">
<title><indexterm><primary>imperative recovery</primary><secondary>Configuration Suggestions</secondary></indexterm>Configuration Suggestions for Imperative Recovery</title>
-<para>We used to build the MGS and MDT0 on the same target to save a server node. However, to make IR work efficiently, we strongly recommend running the MGS node on a separate node for any significant Lustre installation. There are three main advantages of doing this: </para>
+<para>We used to build the MGS and MDT0 on the same target to save a server node. However, to make
+ IR work efficiently, we strongly recommend running the MGS node on a separate node for any
+ significant Lustre file system installation. There are three main advantages of doing this: </para>
<orderedlist>
<listitem><para>Be able to notify clients if the MDT0 is dead</para></listitem>
<listitem><para>Load balance. The load on the MDS may be very high which may make the MGS unable to notify the clients in time</para></listitem>
<section xml:id="suppressingpings">
<title><indexterm><primary>suppressing pings</primary></indexterm>Suppressing Pings</title>
- <para>On clusters with large numbers of clients and OSTs, OBD_PING messages may impose significant performance overheads. As an intermediate solution before a more self-contained one is built, Lustre 2.4 introduces an option to suppress pings, allowing ping overheads to be considerably reduced. Before turning on this option, administrators should consider the following requirements and understand the trade-offs involved:</para>
+ <para>On clusters with large numbers of clients and OSTs, OBD_PING messages may impose
+ significant performance overheads. As an intermediate solution before a more self-contained
+ one is built, Lustre software release 2.4 introduces an option to suppress pings, allowing
+ ping overheads to be considerably reduced. Before turning on this option, administrators
+ should consider the following requirements and understand the trade-offs involved:</para>
<itemizedlist>
<listitem>
- <para>When suppressing pings, a target can not detect client deaths, since clients do not send pings that are only to keep their connections alive. Therefore, a mechanism external to Lustre shall be set up to notify Lustre targets of client deaths in a timely manner, so that stale connections do not exist for too long and lock callbacks to dead clients do not always have to wait for timeouts.</para>
+ <para>When suppressing pings, a target can not detect client deaths, since clients do not
+ send pings that are only to keep their connections alive. Therefore, a mechanism external
+ to the Lustre file system shall be set up to notify Lustre targets of client deaths in a
+ timely manner, so that stale connections do not exist for too long and lock callbacks to
+ dead clients do not always have to wait for timeouts.</para>
</listitem>
<listitem>
<para>Without pings, a client has to rely on Imperative Recovery to notify it of target failures, in order to join recoveries in time. This dictates that the client shall eargerly keep its MGS connection alive. Thus, a highly available standalone MGS is recommended and, on the other hand, MGS pings are always sent regardless of how the option is set.</para>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretroubleshooting">
- <title xml:id="lustretroubleshooting.title">Lustre Troubleshooting</title>
- <para>This chapter provides information to troubleshoot Lustre, submit a Lustre bug, and Lustre performance tips. It includes the following sections:</para>
+ <title xml:id="lustretroubleshooting.title">Lustre File System Troubleshooting</title>
+ <para>This chapter provides information about troubleshooting a Lustre file system, submitting a
+ bug to the Jira bug tracking system, and Lustre file system performance tips. It includes the
+ following sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438198_11171"/></para>
<indexterm><primary>lustre</primary><secondary>errors</secondary><see>troubleshooting</see></indexterm>
<indexterm><primary>errors</primary><see>troubleshooting</see></indexterm>
Lustre Error Messages</title>
- <para>Several resources are available to help troubleshoot Lustre. This section describes error numbers, error messages and logs.</para>
+ <para>Several resources are available to help troubleshoot an issue in a Lustre file system.
+ This section describes error numbers, error messages and logs.</para>
<section remap="h3">
<title><indexterm><primary>troubleshooting</primary><secondary>error numbers</secondary></indexterm>Error Numbers</title>
<para>Error numbers are generated by the Linux operating system and are located in
<literal>/usr/include/asm-generic/errno.h</literal>. The Lustre software does not use all
of the available Linux error numbers. The exact meaning of an error number depends on where
- it is used. Here is a summary of the basic errors that Lustre users may encounter.</para>
+ it is used. Here is a summary of the basic errors that Lustre file system users may
+ encounter.</para>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
</section>
<section xml:id="dbdoclet.50438198_40669">
<title><indexterm><primary>troubleshooting</primary><secondary>error messages</secondary></indexterm>Viewing Error Messages</title>
- <para>As Lustre code runs on the kernel, single-digit error codes display to the application; these error codes are an indication of the problem. Refer to the kernel console log (dmesg) for all recent kernel messages from that node. On the node, <literal>/var/log/messages</literal> holds a log of all messages for at least the past day.</para>
+ <para>As Lustre software code runs on the kernel, single-digit error codes display to the
+ application; these error codes are an indication of the problem. Refer to the kernel console
+ log (dmesg) for all recent kernel messages from that node. On the node,
+ <literal>/var/log/messages</literal> holds a log of all messages for at least the past
+ day.</para>
<para>The error message initiates with "LustreError" in the console log and provides a short description of:</para>
<itemizedlist>
<listitem>
</itemizedlist>
<para>Lustre logs are dumped to <literal>/proc/sys/lnet/debug_path</literal>.</para>
<para>Collect the first group of messages related to a problem, and any messages that precede "LBUG" or "assertion failure" errors. Messages that mention server nodes (OST or MDS) are specific to that server; you must collect similar messages from the relevant server console logs.</para>
- <para>Another Lustre debug log holds information for Lustre action for a short period of time which, in turn, depends on the processes on the node to use Lustre. Use the following command to extract debug logs on each of the nodes, run</para>
+ <para>Another Lustre debug log holds information for a short period of time for action by the
+ Lustre software, which, in turn, depends on the processes on the Lustre node. Use the
+ following command to extract debug logs on each of the nodes, run</para>
<screen>$ lctl dk <replaceable>filename</replaceable></screen>
<note>
<para>LBUG freezes the thread to allow capture of the panic stack. A system reboot is needed to clear the thread.</para>
</section>
</section>
<section xml:id="dbdoclet.50438198_30989">
- <title><indexterm><primary>troubleshooting</primary><secondary>reporting bugs</secondary></indexterm><indexterm><primary>reporting bugs</primary><see>troubleshooting</see></indexterm>
- Reporting a Lustre Bug</title>
+ <title><indexterm>
+ <primary>troubleshooting</primary>
+ <secondary>reporting bugs</secondary>
+ </indexterm><indexterm>
+ <primary>reporting bugs</primary>
+ <see>troubleshooting</see>
+ </indexterm> Reporting a Lustre File System Bug</title>
<para>If you cannot resolve a problem by troubleshooting your Lustre file system, other options are:<itemizedlist>
<listitem>
<para>Post a question to the <link xmlns:xlink="http://www.w3.org/1999/xlink"
<para>Submit a ticket to the <link xmlns:xlink="http://www.w3.org/1999/xlink"
xlink:href="https://jira.hpdd.intel.com/secure/Dashboard.jspa"
>Jira</link><abbrev><superscript>*</superscript></abbrev> bug tracking and project
- management tool used for the Lustre project. If you are a first-time user, you'll need
- to open an account by clicking on <emphasis role="bold">Sign up</emphasis> on the
- Welcome page.</para>
+ management tool used for the Lustre software project. If you are a first-time user,
+ you'll need to open an account by clicking on <emphasis role="bold">Sign up</emphasis>
+ on the Welcome page.</para>
</listitem>
</itemizedlist> To submit a Jira ticket, follow these steps:<orderedlist>
<listitem>
</listitem>
<listitem>
<para><emphasis role="italic">Affects version(s)</emphasis> - Select your Lustre
- version.</para>
+ release.</para>
</listitem>
<listitem>
<para><emphasis role="italic">Environment</emphasis> - Enter your kernel with
</section>
</section>
<section xml:id="dbdoclet.50438198_93109">
- <title><indexterm><primary>troubleshooting</primary><secondary>common problems</secondary></indexterm>Common Lustre Problems</title>
- <para>This section describes how to address common issues encountered with Lustre.</para>
+ <title><indexterm>
+ <primary>troubleshooting</primary>
+ <secondary>common problems</secondary>
+ </indexterm>Common Lustre File System Problems</title>
+ <para>This section describes how to address common issues encountered with a Lustre file
+ system.</para>
<section remap="h3">
<title>OST Object is Missing or Damaged</title>
<para>If the OSS fails to find an object or finds a damaged object, this message appears:</para>
<para>If the reported error is -2 (<literal>-ENOENT</literal>, or "No such file or directory"), then the object is missing. This can occur either because the MDS and OST are out of sync, or because an OST object was corrupted and deleted.</para>
<para>If you have recovered the file system from a disk failure by using e2fsck, then unrecoverable objects may have been deleted or moved to /lost+found on the raw OST partition. Because files on the MDS still reference these objects, attempts to access them produce this error.</para>
<para>If you have recovered a backup of the raw MDS or OST partition, then the restored partition is very likely to be out of sync with the rest of your cluster. No matter which server partition you restored from backup, files on the MDS may reference objects which no longer exist (or did not exist when the backup was taken); accessing those files produces this error.</para>
- <para>If neither of those descriptions is applicable to your situation, then it is possible that you have discovered a programming error that allowed the servers to get out of sync. Please report this condition to the Lustre group, and we will investigate.</para>
+ <para>If neither of those descriptions is applicable to your situation, then it is possible
+ that you have discovered a programming error that allowed the servers to get out of sync.
+ Please submit a Jira ticket (see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="dbdoclet.50438198_30989"/>).</para>
<para>If the reported error is anything else (such as -5, "<literal>I/O error</literal>"), it likely indicates a storage failure. The low-level file system returns this error if it is unable to read from the storage device.</para>
<para><emphasis role="bold">Suggested Action</emphasis></para>
<para>If the reported error is -2, you can consider checking in <literal>/lost+found</literal> on your raw OST device, to see if the missing object is there. However, it is likely that this object is lost forever, and that the file that references the object is now partially or completely lost. Restore this file from backup, or salvage what you can and delete it.</para>
</section>
<section remap="h3">
<title>OSTs Become Read-Only</title>
- <para>If the SCSI devices are inaccessible to Lustre at the block device level, then <literal>ldiskfs</literal> remounts the device read-only to prevent file system corruption. This is a normal behavior. The status in <literal>/proc/fs/lustre/health_check</literal> also shows "not healthy" on the affected nodes.</para>
+ <para>If the SCSI devices are inaccessible to the Lustre file system at the block device
+ level, then <literal>ldiskfs</literal> remounts the device read-only to prevent file system
+ corruption. This is a normal behavior. The status in
+ <literal>/proc/fs/lustre/health_check</literal> also shows "not healthy" on the
+ affected nodes.</para>
<para>To determine what caused the "not healthy" condition:</para>
<itemizedlist>
<listitem>
<para>However, in the case where lov_objid < LAST_ID, bad things can happen as the MDS is not aware of objects that have already been allocated on the OST, and it reallocates them to new files, overwriting their existing contents.</para>
<para>Here is the rule to avoid this scenario:</para>
<para>LAST_ID >= lov_objid and LAST_ID == last_physical_object and lov_objid >= last_used_object</para>
- <para>Although the lov_objid value should be equal to the last_used_object value, the above rule suffices to keep Lustre happy at the expense of a few leaked objects.</para>
+ <para>Although the lov_objid value should be equal to the last_used_object value, the above
+ rule suffices to keep the Lustre file system happy at the expense of a few leaked
+ objects.</para>
<para>In situations where there is on-disk corruption of the OST, for example caused by running with write cache enabled on the disks, the LAST_ID value may become inconsistent and result in a message similar to:</para>
<screen>"filter_precreate()) HOME-OST0003: Serious error:
objid 3478673 already exists; is this filesystem corrupt?"</screen>
</section>
<section remap="h3">
<title><indexterm><primary>troubleshooting</primary><secondary>'Address already in use'</secondary></indexterm>Handling/Debugging "<literal>Bind: Address already in use</literal>" Error</title>
- <para>During startup, Lustre may report a <literal>bind: Address already in use</literal> error and reject to start the operation. This is caused by a portmap service (often NFS locking) which starts before Lustre and binds to the default port 988. You must have port 988 open from firewall or IP tables for incoming connections on the client, OSS, and MDS nodes. LNET will create three outgoing connections on available, reserved ports to each client-server pair, starting with 1023, 1022 and 1021.</para>
+ <para>During startup, the Lustre software may report a <literal>bind: Address already in
+ use</literal> error and reject to start the operation. This is caused by a portmap service
+ (often NFS locking) that starts before the Lustre file system and binds to the default port
+ 988. You must have port 988 open from firewall or IP tables for incoming connections on the
+ client, OSS, and MDS nodes. LNET will create three outgoing connections on available,
+ reserved ports to each client-server pair, starting with 1023, 1022 and 1021.</para>
<para>Unfortunately, you cannot set sunprc to avoid port 988. If you receive this error, do the following:</para>
<itemizedlist>
<listitem>
- <para>Start Lustre before starting any service that uses sunrpc.</para>
+ <para>Start the Lustre file system before starting any service that uses sunrpc.</para>
</listitem>
<listitem>
- <para>Use a port other than 988 for Lustre. This is configured in <literal>/etc/modprobe.d/lustre.conf</literal> as an option to the LNET module. For example:</para>
+ <para>Use a port other than 988 for the Lustre file system. This is configured in
+ <literal>/etc/modprobe.d/lustre.conf</literal> as an option to the LNET module. For
+ example:</para>
<screen>options lnet accept_port=988</screen>
</listitem>
</itemizedlist>
<itemizedlist>
<listitem>
- <para>Add modprobe ptlrpc to your system startup scripts before the service that uses sunrpc. This causes Lustre to bind to port 988 and sunrpc to select a different port.</para>
+ <para>Add modprobe ptlrpc to your system startup scripts before the service that uses
+ sunrpc. This causes the Lustre file system to bind to port 988 and sunrpc to select a
+ different port.</para>
</listitem>
</itemizedlist>
<note>
</itemizedlist>
<para>A Linux error -28 (<literal>ENOSPC</literal>) that occurs when a new file is being created may indicate that the MDS has run out of inodes and needs to be made larger. Newly created files do not written to full OSTs, while existing files continue to reside on the OST where they were initially created. To view inode information on the MDS, enter:</para>
<screen>lfs df -i</screen>
- <para>Typically, Lustre reports this error to your application. If the application is checking the return code from its function calls, then it decodes it into a textual error message such as <literal>No space left on device</literal>. Both versions of the error message also appear in the system log.</para>
+ <para>Typically, the Lustre software reports this error to your application. If the
+ application is checking the return code from its function calls, then it decodes it into a
+ textual error message such as <literal>No space left on device</literal>. Both versions of
+ the error message also appear in the system log.</para>
<para>For more information about the <literal>lfs df</literal> command, see <xref linkend="dbdoclet.50438209_35838"/>.</para>
<para>Although it is less efficient, you can also use the grep command to determine which OST or MDS is running out of space. To check the free space and inodes on a client, enter:</para>
<screen>grep '[0-9]' /proc/fs/lustre/osc/*/kbytes{free,avail,total}
</screen>
</section>
<section remap="h3">
- <title><indexterm><primary>troubleshooting</primary><secondary>timeouts on setup</secondary></indexterm>Handling Timeouts on Initial Lustre Setup</title>
- <para>If you come across timeouts or hangs on the initial setup of your Lustre system, verify that name resolution for servers and clients is working correctly. Some distributions configure <literal>/etc/hosts</literal> so the name of the local machine (as reported by the 'hostname' command) is mapped to local host (127.0.0.1) instead of a proper IP address.</para>
+ <title><indexterm>
+ <primary>troubleshooting</primary>
+ <secondary>timeouts on setup</secondary>
+ </indexterm>Handling Timeouts on Initial Lustre File System Setup</title>
+ <para>If you come across timeouts or hangs on the initial setup of your Lustre file system,
+ verify that name resolution for servers and clients is working correctly. Some distributions
+ configure <literal>/etc/hosts</literal> so the name of the local machine (as reported by the
+ 'hostname' command) is mapped to local host (127.0.0.1) instead of a proper IP
+ address.</para>
<para>This might produce this error:</para>
<screen>LustreError:(ldlm_handle_cancel()) received cancel for unknown lock cookie
0xe74021a4b41b954e from nid 0x7f000001 (0:127.0.0.1)
</section>
<section remap="h3">
<title>Handling/Debugging "LustreError: xxx went back in time"</title>
- <para>Each time Lustre changes the state of the disk file system, it records a unique transaction number. Occasionally, when committing these transactions to the disk, the last committed transaction number displays to other nodes in the cluster to assist the recovery. Therefore, the promised transactions remain absolutely safe on the disappeared disk.</para>
+ <para>Each time the Lustre software changes the state of the disk file system, it records a
+ unique transaction number. Occasionally, when committing these transactions to the disk, the
+ last committed transaction number displays to other nodes in the cluster to assist the
+ recovery. Therefore, the promised transactions remain absolutely safe on the disappeared
+ disk.</para>
<para>This situation arises when:</para>
<itemizedlist>
<listitem>
- <para>You are using a disk device that claims to have data written to disk before it actually does, as in case of a device with a large cache. If that disk device crashes or loses power in a way that causes the loss of the cache, there can be a loss of transactions that you believe are committed. This is a very serious event, and you should run e2fsck against that storage before restarting Lustre.</para>
+ <para>You are using a disk device that claims to have data written to disk before it
+ actually does, as in case of a device with a large cache. If that disk device crashes or
+ loses power in a way that causes the loss of the cache, there can be a loss of
+ transactions that you believe are committed. This is a very serious event, and you
+ should run e2fsck against that storage before restarting the Lustre file system.</para>
</listitem>
<listitem>
- <para>As per the Lustre requirement, the shared storage used for failover is completely cache-coherent. This ensures that if one server takes over for another, it sees the most up-to-date and accurate copy of the data. In case of the failover of the server, if the shared storage does not provide cache coherency between all of its ports, then Lustre can produce an error.</para>
+ <para>As required by the Lustre software, the shared storage used for failover is
+ completely cache-coherent. This ensures that if one server takes over for another, it
+ sees the most up-to-date and accurate copy of the data. In case of the failover of the
+ server, if the shared storage does not provide cache coherency between all of its ports,
+ then the Lustre software can produce an error.</para>
</listitem>
</itemizedlist>
<para>If you know the exact reason for the error, then it is safe to proceed with no further action. If you do not know the reason, then this is a serious issue and you should explore it with your disk vendor.</para>
</itemizedlist>
</section>
<section remap="h3">
- <title><indexterm><primary>troubleshooting</primary><secondary>slowdown during startup</secondary></indexterm>Slowdown Occurs During Lustre Startup</title>
- <para>When Lustre starts, the Lustre file system needs to read in data from the disk. For the very first mdsrate run after the reboot, the MDS needs to wait on all the OSTs for object pre-creation. This causes a slowdown to occur when Lustre starts up.</para>
+ <title><indexterm>
+ <primary>troubleshooting</primary>
+ <secondary>slowdown during startup</secondary>
+ </indexterm>Slowdown Occurs During Lustre File System Startup</title>
+ <para>When a Lustre file system starts, it needs to read in data from the disk. For the very
+ first mdsrate run after the reboot, the MDS needs to wait on all the OSTs for object
+ pre-creation. This causes a slowdown to occur when the file system starts up.</para>
<para>After the file system has been running for some time, it contains more data in cache and hence, the variability caused by reading critical metadata from disk is mostly eliminated. The file system now reads data from the cache.</para>
</section>
<section remap="h3">
<title><indexterm><primary>troubleshooting</primary><secondary>OST out of memory</secondary></indexterm>Log Message <literal>'Out of Memory</literal>' on OST</title>
- <para>When planning the hardware for an OSS node, consider the memory usage of several components in the Lustre system. If insufficient memory is available, an 'out of memory' message can be logged.</para>
+ <para>When planning the hardware for an OSS node, consider the memory usage of several
+ components in the Lustre file system. If insufficient memory is available, an 'out of
+ memory' message can be logged.</para>
<para>During normal operation, several conditions indicate insufficient RAM on a server node:</para>
<itemizedlist>
<listitem>
</section>
<section remap="h3">
<title>Setting SCSI I/O Sizes</title>
- <para>Some SCSI drivers default to a maximum I/O size that is too small for good Lustre performance. we have fixed quite a few drivers, but you may still find that some drivers give unsatisfactory performance with Lustre. As the default value is hard-coded, you need to recompile the drivers to change their default. On the other hand, some drivers may have a wrong default set.</para>
- <para>If you suspect bad I/O performance and an analysis of Lustre statistics indicates that I/O is not 1 MB, check <literal>/sys/block/<replaceable>device</replaceable>/queue/max_sectors_kb</literal>. If the <literal>max_sectors_kb</literal> value is less than 1024, set it to at least 1024 to improve performance. If changing <literal>max_sectors_kb</literal> does not change the I/O size as reported by Lustre, you may want to examine the SCSI driver code.</para>
+ <para>Some SCSI drivers default to a maximum I/O size that is too small for good Lustre file
+ system performance. we have fixed quite a few drivers, but you may still find that some
+ drivers give unsatisfactory performance with the Lustre file system. As the default value is
+ hard-coded, you need to recompile the drivers to change their default. On the other hand,
+ some drivers may have a wrong default set.</para>
+ <para>If you suspect bad I/O performance and an analysis of Lustre file system statistics
+ indicates that I/O is not 1 MB, check
+ <literal>/sys/block/<replaceable>device</replaceable>/queue/max_sectors_kb</literal>. If
+ the <literal>max_sectors_kb</literal> value is less than 1024, set it to at least 1024 to
+ improve performance. If changing <literal>max_sectors_kb</literal> does not change the I/O
+ size as reported by the Lustre software, you may want to examine the SCSI driver
+ code.</para>
</section>
</section>
</chapter>
</listitem>
</itemizedlist>
<para>If the leading <literal><replaceable>fsname</replaceable></literal> and/or ending <literal>_UUID</literal> are missing, they are automatically added.</para>
- <para> For example, to add even-numbered OSTs to pool1 on file system <literal>lustre</literal>, run a single command (add) to add many OSTs to the pool at one time:</para>
+ <para> For example, to add even-numbered OSTs to <literal>pool1</literal> on file system
+ <literal>lustre</literal>, run a single command (<literal>pool_add</literal>) to add many
+ OSTs to the pool at one time:</para>
<para><screen>lctl pool_add lustre.pool1 OST[0-10/2]</screen></para>
<note>
<para>Each time an OST is added to a pool, a new <literal>llog</literal> configuration record is created. For convenience, you can run a single command.</para>
</itemizedlist>
</listitem>
</orderedlist>
- <para>If a Lustre administrator wants to explore this approach further, per-OST disk-usage statistics can be found under <literal>/proc/fs/lustre/osc/*/rpc_stats</literal></para>
+ <para>If a Lustre file system administrator wants to explore this approach further, per-OST
+ disk-usage statistics can be found under
+ <literal>/proc/fs/lustre/osc/*/rpc_stats</literal></para>
</section>
<section xml:id="dbdoclet.50438211_80295">
<title><indexterm><primary>I/O</primary><secondary>direct</secondary></indexterm>Performing Direct I/O</title>
- <para>Lustre supports the <literal>O_DIRECT</literal> flag to open.</para>
+ <para>The Lustre software supports the <literal>O_DIRECT</literal> flag to open.</para>
<para>Applications using the <literal>read()</literal> and <literal>write()</literal> calls must supply buffers aligned on a page boundary (usually 4 K). If the alignment is not correct, the call returns <literal>-EINVAL</literal>. Direct I/O may help performance in cases where the client is doing a large amount of I/O and is CPU-bound (CPU utilization 100%).</para>
<section remap="h3">
<title>Making File System Objects Immutable</title>
<screen>lctl get_param osc.*.checksums</screen>
<section remap="h4">
<title>Changing Checksum Algorithms</title>
- <para>By default, Lustre uses the adler32 checksum algorithm, because it is robust and has a lower impact on performance than crc32. The Lustre administrator can change the checksum algorithm via <literal>lctl get_param</literal>, depending on what is supported in the kernel.</para>
- <para>To check which checksum algorithm is being used by Lustre, run:</para>
+ <para>By default, the Lustre software uses the adler32 checksum algorithm, because it is
+ robust and has a lower impact on performance than crc32. The Lustre file system
+ administrator can change the checksum algorithm via <literal>lctl get_param</literal>,
+ depending on what is supported in the kernel.</para>
+ <para>To check which checksum algorithm is being used by the Lustre software, run:</para>
<screen>$ lctl get_param osc.*.checksum_type</screen>
- <para>To change the wire checksum algorithm used by Lustre, run:</para>
+ <para>To change the wire checksum algorithm, run:</para>
<screen>$ lctl set_param osc.*.checksum_type=<replaceable>algorithm</replaceable></screen>
<note>
<para>The in-memory checksum always uses the adler32 algorithm, if available, and only falls back to crc32 if adler32 cannot be used.</para>
</note>
- <para>In the following example, the <literal>lctl get_param</literal> command is used to determine that Lustre is using the adler32 checksum algorithm. Then the <literal>lctl set_param</literal> command is used to change the checksum algorithm to crc32. A second <literal>lctl get_param</literal> command confirms that the crc32 checksum algorithm is now in use.</para>
+ <para>In the following example, the <literal>lctl get_param</literal> command is used to
+ determine that the Lustre software is using the adler32 checksum algorithm. Then the
+ <literal>lctl set_param</literal> command is used to change the checksum algorithm to
+ crc32. A second <literal>lctl get_param</literal> command confirms that the crc32 checksum
+ algorithm is now in use.</para>
<screen>$ lctl get_param osc.*.checksum_type
osc.lustre-OST0000-osc-ffff81012b2c48e0.checksum_type=crc32 [adler]
$ lctl set_param osc.*.checksum_type=crc32
</section>
<section remap="h3">
<title>Ptlrpc Thread Pool </title>
-<para>Releases prior to Lustre 2.2 used two portal RPC daemons for each client/server pair. One daemon handled all synchronous IO requests, and the second daemon handled all asynchronous (non-IO) RPCs. The increasing use of large SMP nodes for Lustre servers exposed some scaling issues. The lack of threads for large SMP nodes resulted in cases where a single CPU would be 100% utilized and other CPUs would be relativity idle. This is especially noticeable when a single client traverses a large directory. </para>
-<para>Lustre 2.2.x implements a ptlrpc thread pool, so that multiple threads can be created to serve asynchronous RPC requests. The number of threads spawned is controlled at module load time using module options. By default one thread is spawned per CPU, with a minimum of 2 threads spawned irrespective of module options. </para>
+<para>Releases prior to Lustre software release 2.2 used two portal RPC daemons for each
+ client/server pair. One daemon handled all synchronous IO requests, and the second daemon
+ handled all asynchronous (non-IO) RPCs. The increasing use of large SMP nodes for Lustre
+ servers exposed some scaling issues. The lack of threads for large SMP nodes resulted in
+ cases where a single CPU would be 100% utilized and other CPUs would be relativity idle.
+ This is especially noticeable when a single client traverses a large directory. </para>
+<para>Lustre software release 2.2.x implements a ptlrpc thread pool, so that multiple threads can be
+ created to serve asynchronous RPC requests. The number of threads spawned is controlled at
+ module load time using module options. By default one thread is spawned per CPU, with a
+ minimum of 2 threads spawned irrespective of module options. </para>
<para>One of the issues with thread operations is the cost of moving a thread context from one CPU to another with the resulting loss of CPU cache warmth. To reduce this cost, ptlrpc threads can be bound to a CPU. However, if the CPUs are busy, a bound thread may not be able to respond quickly, as the bound CPU may be busy with other tasks and the thread must wait to schedule. </para>
<para>Because of these considerations, the pool of ptlrpc threads can be a mixture of bound and unbound threads. The system operator can balance the thread mixture based on system size and workload. </para>
<section>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managinglnet">
<title xml:id="managinglnet.title">Managing Lustre Networking (LNET)</title>
- <para>This chapter describes some tools for managing Lustre Networking (LNET) and includes the following sections:</para>
+ <para>This chapter describes some tools for managing Lustre networking (LNET) and includes the
+ following sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438203_51732"/></para>
</section>
<section xml:id="dbdoclet.50438203_48703">
<title><indexterm><primary>LNET</primary><secondary>starting/stopping</secondary></indexterm>Starting and Stopping LNET</title>
- <para>Lustre automatically starts and stops LNET, but it can also be manually started in a standalone manner. This is particularly useful to verify that your networking setup is working correctly before you attempt to start Lustre.</para>
+ <para>The Lustre software automatically starts and stops LNET, but it can also be manually
+ started in a standalone manner. This is particularly useful to verify that your networking
+ setup is working correctly before you attempt to start the Lustre file system.</para>
<section remap="h3">
<title>Starting LNET</title>
<para>To start LNET, run:</para>
$ lctl network up</screen>
<para>To see the list of local NIDs, run:</para>
<screen>$ lctl list_nids</screen>
- <para>This command tells you the network(s) configured to work with Lustre</para>
+ <para>This command tells you the network(s) configured to work with the Lustre file
+ system.</para>
<para>If the networks are not correctly setup, see the <literal>modules.conf</literal> "<literal>networks=</literal>" line and make sure the network layer modules are correctly installed and configured.</para>
<para>To get the best remote NID, run:</para>
<screen>$ lctl which_nid <replaceable>NIDs</replaceable></screen>
</section>
<section remap="h3">
<title>Stopping LNET</title>
- <para>Before the LNET modules can be removed, LNET references must be removed. In general, these references are removed automatically when Lustre is shut down, but for standalone routers, an explicit step is needed to stop LNET. Run:</para>
+ <para>Before the LNET modules can be removed, LNET references must be removed. In general,
+ these references are removed automatically when the Lustre file system is shut down, but for
+ standalone routers, an explicit step is needed to stop LNET. Run:</para>
<screen>lctl network unconfigure</screen>
<note>
- <para>Attempting to remove Lustre modules prior to stopping the network may result in a crash or an LNET hang. if this occurs, the node must be rebooted (in most cases). Make sure that the Lustre network and Lustre are stopped prior to unloading the modules. Be extremely careful using rmmod -f.</para>
+ <para>Attempting to remove Lustre modules prior to stopping the network may result in a
+ crash or an LNET hang. If this occurs, the node must be rebooted (in most cases). Make
+ sure that the Lustre network and Lustre file system are stopped prior to unloading the
+ modules. Be extremely careful using <literal>rmmod -f</literal>.</para>
</note>
<para>To unconfigure the LNET network, run:</para>
<screen>modprobe -r <replaceable>lnd_and_lnet_modules</replaceable></screen>
<para>LNET can work with multiple rails, however, it does not load balance across them. The actual rail used for any communication is determined by the peer NID.</para>
</listitem>
<listitem>
- <para>Multi-rail LNET configurations do not provide an additional level of network fault tolerance. The configurations described below are for bandwidth aggregation only. Network interface failover is planned as an upcoming Lustre feature.</para>
+ <para>Multi-rail LNET configurations do not provide an additional level of network fault
+ tolerance. The configurations described below are for bandwidth aggregation only. </para>
</listitem>
<listitem>
<para>A Lustre node always uses the same local NID to communicate with a given peer NID. The criteria used to determine the local NID are:</para>
#even clients;\
o2ib1(ib0),o2ib2(ib1) 192.168.[2-253].[1-253/2) \
#odd clients"</screen>
- <para>This configuration includes two additional proxy o2ib networks to work around Lustre's simplistic NID selection algorithm. It connects "even" clients to "even" servers with <literal>o2ib0</literal> on <literal>rail0</literal>, and "odd" servers with <literal>o2ib3</literal> on <literal>rail1</literal>. Similarly, it connects "odd" clients to "odd" servers with <literal>o2ib1</literal> on <literal>rail0</literal>, and "even" servers with <literal>o2ib2</literal> on <literal>rail1</literal>.</para>
+ <para>This configuration includes two additional proxy o2ib networks to work around the
+ simplistic NID selection algorithm in the Lustre software. It connects "even"
+ clients to "even" servers with <literal>o2ib0</literal> on
+ <literal>rail0</literal>, and "odd" servers with <literal>o2ib3</literal> on
+ <literal>rail1</literal>. Similarly, it connects "odd" clients to
+ "odd" servers with <literal>o2ib1</literal> on <literal>rail0</literal>, and
+ "even" servers with <literal>o2ib2</literal> on <literal>rail1</literal>.</para>
</section>
</section>
<section xml:id="managinglnet.configuringroutes" condition='l24'>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingsecurity">
- <title xml:id="managingsecurity.title">Managing Lustre Security</title>
- <para>This chapter describes Lustre security and includes the following sections:</para>
+ <title xml:id="managingsecurity.title">Managing Security in a Lustre File System</title>
+ <para>This chapter describes security features of the Lustre file system and includes the
+ following sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438221_16221"/></para>
schemes. For a detailed explanation of ACLs on a Linux operating system, refer to the SUSE
Labs article, <emphasis>Posix Access Control Lists on Linux</emphasis>:</para>
<para><link xl:href="http://www.suse.de/~agruen/acl/linux-acls/online/">http://www.suse.de/~agruen/acl/linux-acls/online/</link></para>
- <para>We have implemented ACLs according to this model. Lustre works with the standard Linux ACL tools, setfacl, getfacl, and the historical chacl, normally installed with the ACL package.</para>
+ <para>We have implemented ACLs according to this model. The Lustre software works with the
+ standard Linux ACL tools, setfacl, getfacl, and the historical chacl, normally installed
+ with the ACL package.</para>
<note>
<para>ACL support is a system-range feature, meaning that all clients have ACL enabled or not. You cannot specify which clients should enable ACL.</para>
</note>
</section>
<section remap="h3">
- <title><indexterm><primary>Access Control List (ACL)</primary><secondary>using</secondary></indexterm>Using ACLs with Lustre</title>
- <para>POSIX Access Control Lists (ACLs) can be used with Lustre. An ACL consists of file entries representing permissions based on standard POSIX file system object permissions that define three classes of user (owner, group and other). Each class is associated with a set of permissions [read (r), write (w) and execute (x)].</para>
+ <title><indexterm>
+ <primary>Access Control List (ACL)</primary>
+ <secondary>using</secondary>
+ </indexterm>Using ACLs with the Lustre Software</title>
+ <para>POSIX Access Control Lists (ACLs) can be used with the Lustre software. An ACL consists
+ of file entries representing permissions based on standard POSIX file system object
+ permissions that define three classes of user (owner, group and other). Each class is
+ associated with a set of permissions [read (r), write (w) and execute (x)].</para>
<itemizedlist>
<listitem>
<para>Owner class permissions define access privileges of the file owner.</para>
<screen>$ lctl get_param -n mdc.home-MDT0000-mdc-*.connect_flags | grep acl acl</screen>
<para>To mount the client with no ACLs:</para>
<screen>$ mount -t lustre -o noacl ibmds2@o2ib:/home /home</screen>
- <para>ACLs are enabled in Lustre on a system-wide basis; either all clients enable ACLs or none do. Activating ACLs is controlled by MDS mount options <literal>acl</literal> / <literal>noacl</literal> (enable/disable ACLs). Client-side mount options acl/noacl are ignored. You do not need to change the client configuration, and the 'acl' string will not appear in the client /etc/mtab. The client acl mount option is no longer needed. If a client is mounted with that option, then this message appears in the MDS syslog:</para>
+ <para>ACLs are enabled in a Lustre file system on a system-wide basis; either all clients
+ enable ACLs or none do. Activating ACLs is controlled by MDS mount options
+ <literal>acl</literal> / <literal>noacl</literal> (enable/disable ACLs). Client-side mount
+ options acl/noacl are ignored. You do not need to change the client configuration, and the
+ 'acl' string will not appear in the client /etc/mtab. The client acl mount option
+ is no longer needed. If a client is mounted with that option, then this message appears in
+ the MDS syslog:</para>
<screen>...MDS requires ACL support but client does not</screen>
<para>The message is harmless but indicates a configuration issue, which should be corrected.</para>
<para>If ACLs are not enabled on the MDS, then any attempts to reference an ACL on a client return an Operation not supported error.</para>
</section>
<section xml:id="dbdoclet.50438221_64726">
<title><indexterm><primary>root squash</primary></indexterm>Using Root Squash</title>
- <para>Root squash is a security feature which restricts super-user access rights to a Lustre file system. Without the root squash feature enabled, Lustre users on untrusted clients could access or modify files owned by root on the filesystem, including deleting them. Using the root squash feature restricts file access/modifications as the root user to only the specified clients. Note, however, that this does <emphasis>not</emphasis> prevent users on insecure clients from accessing files owned by <emphasis>other</emphasis> users.</para>
- <para>The root squash feature works by re-mapping the user ID (UID) and group ID (GID) of the root user to a UID and GID specified by the system administrator, via the Lustre configuration management server (MGS). The root squash feature also enables the Lustre administrator to specify a set of client for which UID/GID re-mapping does not apply.</para>
+ <para>Root squash is a security feature which restricts super-user access rights to a Lustre
+ file system. Without the root squash feature enabled, Lustre file system users on untrusted
+ clients could access or modify files owned by root on the file system, including deleting
+ them. Using the root squash feature restricts file access/modifications as the root user to
+ only the specified clients. Note, however, that this does <emphasis>not</emphasis> prevent
+ users on insecure clients from accessing files owned by <emphasis>other</emphasis>
+ users.</para>
+ <para>The root squash feature works by re-mapping the user ID (UID) and group ID (GID) of the
+ root user to a UID and GID specified by the system administrator, via the Lustre configuration
+ management server (MGS). The root squash feature also enables the Lustre fle system
+ administrator to specify a set of client for which UID/GID re-mapping does not apply.</para>
<section remap="h3">
<title><indexterm><primary>root squash</primary><secondary>configuring</secondary></indexterm>Configuring Root Squash</title>
<para>Root squash functionality is managed by two configuration parameters, <literal>root_squash</literal> and <literal>nosquash_nids</literal>.</para>
objects on OSTs with more free space. (This can reduce I/O performance until space usage is
rebalanced again.) For a more detailed description of how striping is allocated, see <xref
linkend="dbdoclet.50438209_10424"/>.</para>
- <para condition="l22">Files can only be striped over a finite number of OSTs. Prior to the
- Lustre 2.2 release, the maximum number of OSTs that a file could be striped across was limited
- to 160. As of the Lustre 2.2 release, the maximum number of OSTs is 2000. For more
- information, see <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_syy_gcl_qk"
- />.</para>
+ <para condition="l22">Files can only be striped over a finite number of OSTs. Prior to Lustre
+ software release 2.2, the maximum number of OSTs that a file could be striped across was
+ limited to 160. As of Lustre software release 2.2, the maximum number of OSTs is 2000. For
+ more information, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="section_syy_gcl_qk"/>.</para>
</section>
<section xml:id="dbdoclet.50438209_48033">
<title><indexterm>
<primary>striping</primary>
<secondary>remote directories</secondary>
</indexterm>Locating the MDT for a remote directory</title>
- <para condition="l24">Lustre release 2.4 can be configured with multiple MDTs in the same
- file system. Each sub-directory can have a different MDT. To identify on which MDT a given
- subdirectory is located, pass the <literal>getstripe -M</literal> parameters to
+ <para condition="l24">Lustre software release 2.4 can be configured with multiple MDTs in the
+ same file system. Each sub-directory can have a different MDT. To identify on which MDT a
+ given subdirectory is located, pass the <literal>getstripe -M</literal> parameters to
<literal>lfs</literal>. An example of this command is provided in the section <xref
linkend="dbdoclet.rmremotedir"/>.</para>
</section>
</indexterm><indexterm>
<primary>wide striping</primary>
</indexterm>Lustre Striping Internals</title>
- <para>For Lustre releases prior to Lustre release 2.2, files can be striped across a maximum of
- 160 OSTs. Lustre inodes use an extended attribute to record the location of each object (the
- object ID and the number of the OST on which it is stored). The size of the extended attribute
- limits the maximum stripe count to 160 objects.</para>
- <para condition="l22">In Lustre release 2.2 and subsequent releases, the maximum number of OSTs
- over which files can be striped has been raised to 2000 by allocating a new block on which to
- store the extended attribute that holds the object information. This feature, known as "wide
- striping," only allocates the additional extended attribute data block if the file is striped
- with a stripe count greater than 160. The file layout (object ID, OST number) is stored on the
- new data block with a pointer to this block stored in the original Lustre inode for the file.
- For files smaller than 160 objects, the Lustre inode is used to store the file layout.</para>
+ <para>For Lustre releases prior to Lustre software release 2.2, files can be striped across a
+ maximum of 160 OSTs. Lustre inodes use an extended attribute to record the location of each
+ object (the object ID and the number of the OST on which it is stored). The size of the
+ extended attribute limits the maximum stripe count to 160 objects.</para>
+ <para condition="l22">In Lustre software release 2.2 and subsequent releases, the maximum number
+ of OSTs over which files can be striped has been raised to 2000 by allocating a new block on
+ which to store the extended attribute that holds the object information. This feature, known
+ as "wide striping," only allocates the additional extended attribute data block if the file is
+ striped with a stripe count greater than 160. The file layout (object ID, OST number) is
+ stored on the new data block with a pointer to this block stored in the original Lustre inode
+ for the file. For files smaller than 160 objects, the Lustre inode is used to store the file
+ layout.</para>
</section>
</chapter>
<?xml version='1.0' encoding='UTF-8'?>
<preface xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US">
<title>Preface</title>
- <para>The <emphasis role="italic">Lustre<superscript>*</superscript> Operations Manual</emphasis>
- provides detailed information and procedures to install, configure and tune a Lustre file
- system. The manual covers topics such as failover, quotas, striping, and bonding. This manual
- also contains troubleshooting information and tips to improve the operation and performance of a
- Lustre file system.</para>
+ <para>The <emphasis role="italic">Lustre<superscript>*</superscript>Software Release 2.x
+ Operations Manual</emphasis> provides detailed information and procedures to install,
+ configure and tune a Lustre file system. The manual covers topics such as failover, quotas,
+ striping, and bonding. This manual also contains troubleshooting information and tips to improve
+ the operation and performance of a Lustre file system.</para>
<section remap="h2">
<title>About this Document</title>
<para>This document is maintained by Intel in Docbook format. The
</entry>
<entry>
<para>
- <emphasis>Lustre 2.0 Release Notes</emphasis></para>
+ <emphasis>Lustre Software Release 2.x Change Logs</emphasis></para>
</entry>
<entry>
- <para> PDF</para>
+ <para> Wiki page</para>
</entry>
<entry>
<para> Online at <link
</entry>
<entry>
<para>
- <emphasis>Lustre 2.0 Operations Manual</emphasis></para>
+ <emphasis>Lustre Software Release 2.x Operations Manual</emphasis></para>
</entry>
<entry>
<para> PDF</para>
<?xml version="1.0" encoding="UTF-8"?>
<section xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US">
<title>Revisions</title>
- <para>Note that the <emphasis role="italic">Lustre Operations Manual 2.x</emphasis> is intended
- to be relevant for all 2.x releases of the Lustre software. Most of the manual content is
- relevant to all of the releases. However, any features and content that are added for new
- versions of the Lustre software are clearly marked with the version in which this
- functionality is added. Similarly, features that are no longer available are marked with the
- version in which they were deprecated or removed. <revhistory>
+ <para>The <emphasis role="italic">Lustre Software Release 2.x Operations Manual</emphasis>
+ contains content relevant for all 2.x releases of the Lustre software. <note>
+ <para>Updates to the manual are made as the changes become available. When features are
+ added with new releases of the Lustre software, they are clearly marked with the
+ version of the release in which the functionality was added. Features that are
+ deprecated by a release or removed from the software are also marked with the
+ version of the release in which they became deprecated or were removed.</para>
+ </note></para>
+ <para>
+ <revhistory>
+ <revision>
+ <revnumber>2.x</revnumber>
+ <date>Ongoing (see note above)</date>
+ <author>
+ <orgname>Intel Corporation</orgname>
+ </author>
+ <revremark>Release of manual for Lustre software release 2.x</revremark>
+ </revision>
<revision>
<revnumber>2.3</revnumber>
<date>November 2012</date>
<author>
<orgname>Intel Corporation</orgname>
</author>
- <revremark>Release of Lustre 2.3 manual</revremark>
+ <revremark>Release of manual for Lustre software release 2.3</revremark>
</revision>
<revision>
<revnumber>2.1</revnumber>
<author>
<orgname>Whamcloud, Inc</orgname>
</author>
- <revremark>First release of Lustre 2.1 manual</revremark>
+ <revremark>First release of manual with Lustre software release 2.1</revremark>
</revision>
</revhistory>
</para>
</section>
<section remap="h5">
<title>Description</title>
- <para>The <literal>llapi_file_create()</literal> function sets a file descriptor's Lustre striping information. The file descriptor is then accessed with <literal>open()</literal>.</para>
+ <para>The <literal>llapi_file_create()</literal> function sets a file descriptor's Lustre
+ file system striping information. The file descriptor is then accessed with
+ <literal>open()</literal>.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<para> <literal>lmm_pattern</literal></para>
</entry>
<entry>
- <para>Holds the striping pattern. Only <literal>LOV_PATTERN_RAID0</literal> is possible in this Lustre version.</para>
+ <para>Holds the striping pattern. Only <literal>LOV_PATTERN_RAID0</literal> is
+ possible in this Lustre software release.</para>
</entry>
</row>
<row>
<title>
<literal>llapi_file_open</literal>
</title>
- <para>The <literal>llapi_file_open</literal> command opens (or creates) a file or device on a Lustre filesystem.</para>
+ <para>The <literal>llapi_file_open</literal> command opens (or creates) a file or device on a
+ Lustre file system.</para>
<section remap="h5">
<title>Synopsis</title>
<screen>#include <lustre/lustreapi.h>
<section remap="h5">
<title>Description</title>
<para>The <literal>llapi_file_create()</literal> call is equivalent to the <literal>llapi_file_open</literal> call with <emphasis>flags</emphasis> equal to <literal>O_CREAT|O_WRONLY</literal> and <emphasis>mode</emphasis> equal to <literal>0644</literal>, followed by file close.</para>
- <para><literal>llapi_file_open()</literal> opens a file with a given name on a Lustre filesystem.</para>
+ <para><literal>llapi_file_open()</literal> opens a file with a given name on a Lustre file
+ system.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<para> <literal>stripe_pattern</literal></para>
</entry>
<entry>
- <para>Specifies the striping pattern. In this version of Lustre, only <literal>LOV_PATTERN_RAID0</literal> is available. The default value is 0.</para>
+ <para>Specifies the striping pattern. In this release of the Lustre software, only
+ <literal>LOV_PATTERN_RAID0</literal> is available. The default value is
+ 0.</para>
</entry>
</row>
</tbody>
<para> <literal>ENOTTY</literal></para>
</entry>
<entry>
- <para> <literal>name</literal> may not point to a Lustre filesystem.</para>
+ <para>
+ <literal>name</literal> may not point to a Lustre file system.</para>
</entry>
</row>
</tbody>
<para> <literal>LUSTRE_Q_QUOTAON</literal></para>
</entry>
<entry>
- <para>Turns on quotas for a Lustre file system. Deprecated as of 2.4.0. <emphasis>qc_type</emphasis> is <literal>USRQUOTA</literal>, <literal>GRPQUOTA</literal> or <literal>UGQUOTA</literal> (both user and group quota). The quota files must exist. They are normally created with the <literal>llapi_quotacheck</literal> call. This call is restricted to the super user privilege. As of 2.4.0, quota is now enabled on a per-filesystem basis via <literal>lctl conf_param</literal> (see <xref linkend="enabling_disk_quotas"/>) on the MGS node and quotacheck isn't needed any more.</para>
+ <para>Turns on quotas for a Lustre file system. Deprecated as of 2.4.0.
+ <emphasis>qc_type</emphasis> is <literal>USRQUOTA</literal>,
+ <literal>GRPQUOTA</literal> or <literal>UGQUOTA</literal> (both user and group
+ quota). The quota files must exist. They are normally created with the
+ <literal>llapi_quotacheck</literal> call. This call is restricted to the super
+ user privilege. As of 2.4.0, quota is now enabled on a per file system basis via
+ <literal>lctl conf_param</literal> (see <xref linkend="enabling_disk_quotas"/>)
+ on the MGS node and quotacheck isn't needed any more.</para>
</entry>
</row>
<row>
<para> <literal>LUSTRE_Q_GETINFO</literal></para>
</entry>
<entry>
- <para>Gets information about quotas. <emphasis>qc_type</emphasis> is either <literal>USRQUOTA</literal> or <literal>GRPQUOTA</literal>. On return, <emphasis>dqi_igrace</emphasis> is inode grace time (in seconds), <emphasis>dqi_bgrace</emphasis> is block grace time (in seconds), <emphasis>dqi_flags</emphasis> is not used by the current Lustre version.</para>
+ <para>Gets information about quotas. <emphasis>qc_type</emphasis> is either
+ <literal>USRQUOTA</literal> or <literal>GRPQUOTA</literal>. On return,
+ <emphasis>dqi_igrace</emphasis> is inode grace time (in seconds),
+ <emphasis>dqi_bgrace</emphasis> is block grace time (in seconds),
+ <emphasis>dqi_flags</emphasis> is not used by the current release of the Lustre
+ software.</para>
</entry>
</row>
<row>
<para> <literal>LUSTRE_Q_SETINFO</literal></para>
</entry>
<entry>
- <para>Sets quota information (like grace times). <emphasis>qc_type</emphasis> is either <literal>USRQUOTA</literal> or <literal>GRPQUOTA</literal>. <emphasis>dqi_igrace</emphasis> is inode grace time (in seconds), <emphasis>dqi_bgrace</emphasis> is block grace time (in seconds), <emphasis>dqi_flags</emphasis> is not used by the current Lustre version and must be zeroed.</para>
+ <para>Sets quota information (like grace times). <emphasis>qc_type</emphasis> is
+ either <literal>USRQUOTA</literal> or <literal>GRPQUOTA</literal>.
+ <emphasis>dqi_igrace</emphasis> is inode grace time (in seconds),
+ <emphasis>dqi_bgrace</emphasis> is block grace time (in seconds),
+ <emphasis>dqi_flags</emphasis> is not used by the current release of the Lustre
+ software and must be zeroed.</para>
</entry>
</row>
</tbody>
</section>
<section xml:id="dbdoclet.50438215_marker-1297700">
<title>Example Using the <literal>llapi</literal> Library</title>
- <para>Use <literal>llapi_file_create</literal> to set Lustre properties for a new file. For a synopsis and description of llapi_file_create and examples of how to use it, see <xref linkend="configurationfilesmoduleparameters"/>.</para>
+ <para>Use <literal>llapi_file_create</literal> to set Lustre software properties for a new file.
+ For a synopsis and description of <literal>llapi_file_create</literal> and examples of how to
+ use it, see <xref linkend="configurationfilesmoduleparameters"/>.</para>
<para>You can set striping from inside programs like <literal>ioctl</literal>. To compile the sample program, you need to install the Lustre client source RPM.</para>
<para><emphasis role="bold">A simple C program to demonstrate striping API - libtest.c</emphasis></para>
<programlisting>
<section xml:id="dbdoclet.50438258_25353">
<title>Bonding Module Parameters</title>
<para>Bonding module parameters control various aspects of bonding.</para>
- <para>Outgoing traffic is mapped across the slave interfaces according to the transmit hash policy. For Lustre, we recommend that you set the <literal>xmit_hash_policy</literal> option to the layer3+4 option for bonding. This policy uses upper layer protocol information if available to generate the hash. This allows traffic to a particular network peer to span multiple slaves, although a single connection does not span multiple slaves.</para>
+ <para>Outgoing traffic is mapped across the slave interfaces according to the transmit hash
+ policy. We recommend that you set the <literal>xmit_hash_policy</literal> option to the
+ layer3+4 option for bonding. This policy uses upper layer protocol information if available to
+ generate the hash. This allows traffic to a particular network peer to span multiple slaves,
+ although a single connection does not span multiple slaves.</para>
<screen>$ xmit_hash_policy=layer3+4</screen>
- <para>The miimon option enables users to monitor the link status. (The parameter is a time interval in milliseconds.) It makes an interface failure transparent to avoid serious network degradation during link failures. A reasonable default setting is 100 milliseconds; run:</para>
+ <para>The <literal>miimon</literal> option enables users to monitor the link status. (The
+ parameter is a time interval in milliseconds.) It makes an interface failure transparent to
+ avoid serious network degradation during link failures. A reasonable default setting is 100
+ milliseconds; run:</para>
<screen>$ miimon=100</screen>
<para>For a busy network, increase the timeout.</para>
</section>
</section>
</section>
<section xml:id="dbdoclet.50438258_54769">
- <title>Configuring Lustre with Bonding</title>
- <para>Lustre uses the IP address of the bonded interfaces and requires no special configuration. It treats the bonded interface as a regular TCP/IP interface. If needed, specify bond0 using the Lustre networks parameter in /etc/modprobe.</para>
+ <title>Configuring a Lustre File System with Bonding</title>
+ <para>The Lustre software uses the IP address of the bonded interfaces and requires no special
+ configuration. The bonded interface is treated as a regular TCP/IP interface. If needed,
+ specify <literal>bond0</literal> using the Lustre <literal>networks</literal> parameter in
+ <literal>/etc/modprobe</literal>.</para>
<screen>options lnet networks=tcp(bond0)</screen>
</section>
<section xml:id="dbdoclet.50438258_92244">
<para>We recommend the following bonding references:</para>
<itemizedlist>
<listitem>
- <para>In the Linux kernel source tree, see documentation/networking/bonding.txt</para>
+ <para>In the Linux kernel source tree, see
+ <literal>documentation/networking/bonding.txt</literal></para>
</listitem>
<listitem>
- <para><link xl:href="http://linux-ip.net/html/ether-bonding.html">http://linux-ip.net/html/ether-bonding.html</link></para>
+ <para><link xl:href="http://linux-ip.net/html/ether-bonding.html"
+ >http://linux-ip.net/html/ether-bonding.html</link>.</para>
</listitem>
<listitem>
- <para><link xl:href="http://www.sourceforge.net/projects/bonding">http://www.sourceforge.net/projects/bonding</link></para>
+ <para><link xl:href="http://www.sourceforge.net/projects/bonding"
+ >http://www.sourceforge.net/projects/bonding</link>.</para>
</listitem>
<listitem>
- <para>Linux Foundation bonding website: <link xl:href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding">http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding</link></para>
+ <para>Linux Foundation bonding website: <link
+ xl:href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding"
+ >http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding</link>. This
+ is the most extensive reference and we highly recommend it. This website includes
+ explanations of more complicated setups, including the use of DHCP with bonding.</para>
</listitem>
</itemizedlist>
- <para>This is the most extensive reference and we highly recommend it. This website includes explanations of more complicated setups, including the use of DHCP with bonding.</para>
</section>
</chapter>
<para>Only servers running on 64-bit CPUs are tested and supported. 64-bit CPU clients are
typically used for testing to match expected customer usage and avoid limitations due to the 4
GB limit for RAM size, 1 GB low-memory limitation, and 16 TB file size limit of 32-bit CPUs.
- Also, due to kernel API limitations, performing backups of Lustre 2.x. file systems on 32-bit
- clients may cause backup tools to confuse files that have the same 32-bit inode number.</para>
+ Also, due to kernel API limitations, performing backups of Lustre software release 2.x. file
+ systems on 32-bit clients may cause backup tools to confuse files that have the same 32-bit
+ inode number.</para>
<para>The storage attached to the servers typically uses RAID to provide fault tolerance and can
optionally be organized with logical volume management (LVM), which is then formatted as a
Lustre file system. Lustre OSS and MDS servers read, write and modify data in the format
Subsequent directories may also be configured to have their own MDT. If an MDT serving a
subdirectory becomes unavailable this subdirectory and all directories beneath it will
also become unavailable. Configuring multiple levels of MDTs is an experimental feature
- for the Lustre 2.4 release.</para></note>
+ for the Lustre software release 2.4.</para></note>
</section>
<section remap="h3">
<title><indexterm><primary>setup</primary><secondary>OST</secondary></indexterm>OST Storage Hardware Considerations</title>
<secondary>determining MGT requirements</secondary>
</indexterm> Determining MGT Space Requirements</title>
<para>Less than 100 MB of space is required for the MGT. The size is determined by the number
- of servers in the Lustre cluster(s) that are managed by the MGS.</para>
+ of servers in the Lustre file system cluster(s) that are managed by the MGS.</para>
</section>
<section xml:id="dbdoclet.50438256_87676">
<title><indexterm>
<primary>space</primary>
<secondary>determining MDT requirements</secondary>
</indexterm> Determining MDT Space Requirements</title>
- <para>When calculating the MDT size, the important factor to consider is the number of files to be stored in the file system. This determines the number of inodes needed, which drives the MDT sizing. To be on the safe side, plan for 2 KB per inode on the MDT, which is the default value. Attached storage required for Lustre metadata is typically 1-2 percent of the file system capacity depending upon file size.</para>
+ <para>When calculating the MDT size, the important factor to consider is the number of files
+ to be stored in the file system. This determines the number of inodes needed, which drives
+ the MDT sizing. To be on the safe side, plan for 2 KB per inode on the MDT, which is the
+ default value. Attached storage required for Lustre file system metadata is typically 1-2
+ percent of the file system capacity depending upon file size.</para>
<para>For example, if the average file size is 5 MB and you have 100 TB of usable OST space, then you can calculate the minimum number of inodes as follows:</para>
<informalexample>
<para>(100 TB * 1024 GB/TB * 1024 MB/GB) / 5 MB/inode = 20 million inodes</para>
<para condition='l24'>4096</para>
</entry>
<entry>
- <para>The Lustre 2.3 release and earlier allows a maximum of 1 MDT per file system,
- but a single MDS can host multiple MDTs, each one for a separate file
+ <para>The Lustre software release 2.3 and earlier allows a maximum of 1 MDT per file
+ system, but a single MDS can host multiple MDTs, each one for a separate file
system.</para>
- <para condition="l24">The Lustre 2.4 release and later requires one MDT for the
- root. Upto 4095 additional MDTs can be added to the file system and attached into
- the namespace with remote directories.</para>
+ <para condition="l24">The Lustre software release 2.4 and later requires one MDT for
+ the root. Upto 4095 additional MDTs can be added to the file system and attached
+ into the namespace with remote directories.</para>
</entry>
</row>
<row>
</table>
<para> </para>
<note>
- <para condition="l22">In Lustre releases prior to 2.2, the maximum stripe count for a single
- file was limited to 160 OSTs. In Lustre release 2.2, the large <literal>xattr</literal>
- feature ("wide striping") was added to support up to 2000 OSTs. This feature is disabled
- by default at <literal>mkfs.lustre</literal> time. In order to enable this feature, set
- the "<literal>-O large_xattr</literal>" option on the MDT either by using
- <literal>--mkfsoptions</literal> at format time or by using
- <literal>tune2fs</literal>.</para>
+ <para condition="l22">In Lustre software releases prior to release 2.2, the maximum stripe
+ count for a single file was limited to 160 OSTs. In Lustre software release 2.2, the large
+ <literal>xattr</literal> feature ("wide striping") was added to support up to 2000 OSTs.
+ This feature is disabled by default at <literal>mkfs.lustre</literal> time. In order to
+ enable this feature, set the "<literal>-O large_xattr</literal>" option on the MDT either
+ by using <literal>--mkfsoptions</literal> at format time or by using
+ <literal>tune2fs</literal>.</para>
</note>
</section>
</section>
</section>
<section remap="h5">
<title>Description</title>
- <para>The ll_decode_filter_fid utility decodes and prints the Lustre OST object ID, MDT FID, stripe index for the specified OST object(s), which is stored in the "trusted.fid" attribute on each OST object. This is accessible to ll_decode_filter_fid when the OST filesystem is mounted locally as type ldiskfs for maintenance.</para>
+ <para>The ll_decode_filter_fid utility decodes and prints the Lustre OST object ID, MDT FID,
+ stripe index for the specified OST object(s), which is stored in the "trusted.fid"
+ attribute on each OST object. This is accessible to <literal>ll_decode_filter_fid</literal>
+ when the OST file system is mounted locally as type ldiskfs for maintenance.</para>
<para>The "trusted.fid" extended attribute is stored on each OST object when it is first modified (data written or attributes set), and is not accessed or modified by Lustre after that time.</para>
<para>The OST object ID (objid) is useful in case of OST directory corruption, though normally the ll_recover_lost_found_objs(8) utility is able to reconstruct the entire OST object directory hierarchy. The MDS FID can be useful to determine which MDS inode an OST object is (or was) used by. The stripe index can be used in conjunction with other OST objects to reconstruct the layout of a file even if the MDT inode was lost.</para>
</section>
<para> <literal>-t <replaceable>timestamp</replaceable></literal></para>
</entry>
<entry>
- <para> Sets the test start time as printed at the start of a previously-interrupted test to ensure that validation data is the same across the entire filesystem (default value is the current time()).</para>
+ <para> Sets the test start time as printed at the start of a previously-interrupted
+ test to ensure that validation data is the same across the entire file system
+ (default value is the current time()).</para>
</entry>
</row>
<row>
</itemizedlist>
<section xml:id="dbdoclet.50438225_71141">
<title><indexterm><primary>recovery</primary><secondary>corruption of backing file system</secondary></indexterm>Recovering from Errors or Corruption on a Backing File System</title>
- <para>When an OSS, MDS, or MGS server crash occurs, it is not necessary to run e2fsck on the file system. <literal>ldiskfs</literal> journaling ensures that the file system remains consistent over a system crash. The backing file systems are never accessed directly from the client, so client crashes are not relevant for server filesystem consistency.</para>
+ <para>When an OSS, MDS, or MGS server crash occurs, it is not necessary to run e2fsck on the
+ file system. <literal>ldiskfs</literal> journaling ensures that the file system remains
+ consistent over a system crash. The backing file systems are never accessed directly
+ from the client, so client crashes are not relevant for server file system
+ consistency.</para>
<para>The only time it is REQUIRED that <literal>e2fsck</literal> be run on a device is when an event causes problems that ldiskfs journaling is unable to handle, such as a hardware device failure or I/O error. If the ldiskfs kernel code detects corruption on the disk, it mounts the file system as read-only to prevent further corruption, but still allows read access to the device. This appears as error "-30" (<literal>EROFS</literal>) in the syslogs on the server, e.g.:</para>
<screen>Dec 29 14:11:32 mookie kernel: LDISKFS-fs error (device sdz):
- ldiskfs_lookup: unlinked inode 5384166 in dir #145170469</screen>
- <para>Dec 29 14:11:32 mookie kernel: Remounting filesystem read-only</para>
+ ldiskfs_lookup: unlinked inode 5384166 in dir #145170469
+Dec 29 14:11:32 mookie kernel: Remounting filesystem read-only</screen>
<para>In such a situation, it is normally required that e2fsck only be run on the bad device before placing the device back into service.</para>
- <para>In the vast majority of cases, Lustre can cope with any inconsistencies it finds on the disk and between other devices in the file system.</para>
+ <para>In the vast majority of cases, the Lustre software can cope with any inconsistencies
+ found on the disk and between other devices in the file system.</para>
<note>
- <para>The offline LFSCK tool included with e2fsprogs is rarely required for Lustre operation.</para>
+ <para>The offline LFSCK tool included with e2fsprogs is rarely required for Lustre file
+ system operation.</para>
</note>
<para>For problem analysis, it is strongly recommended that <literal>e2fsck</literal> be run under a logger, like script, to record all of the output and changes that are made to the file system in case this information is needed later.</para>
<para>If time permits, it is also a good idea to first run <literal>e2fsck</literal> in non-fixing mode (-n option) to assess the type and extent of damage to the file system. The drawback is that in this mode, <literal>e2fsck</literal> does not recover the file system journal, so there may appear to be file system corruption when none really exists.</para>
- <para>To address concern about whether corruption is real or only due to the journal not being replayed, you can briefly mount and unmount the <literal>ldiskfs</literal> filesystem directly on the node with Lustre stopped (NOT via Lustre), using a command similar to:</para>
+ <para>To address concern about whether corruption is real or only due to the journal not
+ being replayed, you can briefly mount and unmount the <literal>ldiskfs</literal> file
+ system directly on the node with the Lustre file system stopped, using a command similar
+ to:</para>
<screen>mount -t ldiskfs /dev/{ostdev} /mnt/ost; umount /mnt/ost</screen>
<para>This causes the journal to be recovered.</para>
- <para>The <literal>e2fsck</literal> utility works well when fixing file system corruption (better than similar file system recovery tools and a primary reason why <literal>ldiskfs</literal> was chosen over other file systems for Lustre). However, it is often useful to identify the type of damage that has occurred so an <literal>ldiskfs</literal> expert can make intelligent decisions about what needs fixing, in place of <literal>e2fsck</literal>.</para>
+ <para>The <literal>e2fsck</literal> utility works well when fixing file system corruption
+ (better than similar file system recovery tools and a primary reason why
+ <literal>ldiskfs</literal> was chosen over other file systems). However, it is often
+ useful to identify the type of damage that has occurred so an <literal>ldiskfs</literal>
+ expert can make intelligent decisions about what needs fixing, in place of
+ <literal>e2fsck</literal>.</para>
<screen>root# {stop lustre services for this device, if running}
root# script /tmp/e2fsck.sda
Script started, file is /tmp/e2fsck.sda
:
root# e2fsck -fp /dev/sda # fix errors with prudent answers (usually <literal>yes</literal>)
</screen>
- <para>In addition, the <literal>e2fsprogs</literal> package contains the LFSCK tool, which does distributed coherency checking for the Lustre file system after <literal>e2fsck</literal> has been run. Running LFSCK is NOT required in a large majority of cases, at a small risk of having some leaked space in the file system. To avoid a lengthy downtime, it can be run (with care) after Lustre is started.</para>
+ <para>In addition, the <literal>e2fsprogs</literal> package contains the LFSCK tool, which
+ does distributed coherency checking for the Lustre file system after
+ <literal>e2fsck</literal> has been run. Running LFSCK is NOT required in a large
+ majority of cases, at a small risk of having some leaked space in the file system. To
+ avoid a lengthy downtime, it can be run (with care) after the Lustre file system is
+ started.</para>
</section>
<section xml:id="dbdoclet.50438225_37365">
<title><indexterm><primary>recovery</primary><secondary>corruption of Lustre file system</secondary></indexterm>Recovering from Corruption in the Lustre File System</title>
</section>
<section xml:id="dbdoclet.50438225_12316">
<title><indexterm><primary>recovery</primary><secondary>unavailable OST</secondary></indexterm>Recovering from an Unavailable OST</title>
- <para>One of the most common problems encountered in a Lustre environment is when an OST becomes unavailable, because of a network partition, OSS node crash, etc. When this happens, the OST's clients pause and wait for the OST to become available again, either on the primary OSS or a failover OSS. When the OST comes back online, Lustre starts a recovery process to enable clients to reconnect to the OST. Lustre servers put a limit on the time they will wait in recovery for clients to reconnect. The timeout length is determined by the <literal>obd_timeout</literal> parameter.</para>
+ <para>One of the most common problems encountered in a Lustre file system environment is
+ when an OST becomes unavailable due to a network partition, OSS node crash, etc. When
+ this happens, the OST's clients pause and wait for the OST to become available
+ again, either on the primary OSS or a failover OSS. When the OST comes back online, the
+ Lustre file system starts a recovery process to enable clients to reconnect to the OST.
+ Lustre servers put a limit on the time they will wait in recovery for clients to
+ reconnect. The timeout length is determined by the <literal>obd_timeout</literal>
+ parameter.</para>
<para>During recovery, clients reconnect and replay their requests serially, in the same order they were done originally. Until a client receives a confirmation that a given transaction has been written to stable storage, the client holds on to the transaction, in case it needs to be replayed. Periodically, a progress message prints to the log, stating how_many/expected clients have reconnected. If the recovery is aborted, this log shows how many clients managed to reconnect. When all clients have completed recovery, or if the recovery timeout is reached, the recovery period ends and the OST resumes normal request processing.</para>
<para>If some clients fail to replay their requests during the recovery period, this will not stop the recovery from completing. You may have a situation where the OST recovers, but some clients are not able to participate in recovery (e.g. network problems or client failure), so they are evicted and their requests are not replayed. This would result in any operations on the evicted clients failing, including in-progress writes, which would cause cached writes to be lost. This is a normal outcome; the recovery cannot wait indefinitely, or the file system would be hung any time a client failed. The lost transactions are an unfortunate result of the recovery process.</para>
<note>
</section>
<section xml:id="dbdoclet.lfsckadmin" condition='l23'>
<title><indexterm><primary>recovery</primary><secondary>oiscrub</secondary></indexterm><indexterm><primary>recovery</primary><secondary>lfsck</secondary></indexterm>Checking the file system with LFSCK</title>
- <para>LFSCK is an administrative tool introduced in Lustre 2.3 for checking and repair of the Lustre-specific attributes of a mounted Lustre filesystem. It is similar in concept to the offline LFSCK Lustre repair tool that is included with the Lustre <literal>e2fsprogs</literal> package (see <xref linkend='dbdoclet.50438225_37365'/>), but LFSCK is implemented to run as part of Lustre while the filesystem is mounted and in use. This allows Lustre consistency checking and repair without unnecessary downtime, and can be run on the largest Lustre filesystems.</para>
- <para>In Lustre 2.3, LFSCK can verify and repair the Object Index (OI) table that is used internally to map Lustre File Identifiers (FIDs) to MDT internal inode numbers, through a process called OI Scrub. An OI Scrub is required after restoring from a file-level MDT backup (<xref linkend='dbdoclet.50438207_71633'/>), or in case the OI table is otherwise corrupted. Later phases of LFSCK will add further checks to the Lustre distributed filesystem state.</para>
- <para>Control and monitoring of LFSCK is through LFSCK and the <literal>/proc filesystem</literal> interfaces. LFSCK supports three types of interface: switch interfaces, status interfaces and adjustments interfaces. These interfaces are detailed below.</para>
+ <para>LFSCK is an administrative tool introduced in Lustre software release 2.3 for checking
+ and repair of the attributes specific to a mounted Lustre file system. It is similar in
+ concept to the offline LFSCK Lustre repair tool that is included with the Lustre
+ <literal>e2fsprogs</literal> package (see <xref linkend="dbdoclet.50438225_37365"
+ />), but LFSCK is implemented to run as part of the Lustre file system while the file
+ system is mounted and in use. This allows consistency of checking and repair by the
+ Lustre software without unnecessary downtime, and can be run on the largest Lustre file
+ systems.</para>
+ <para>In Lustre software release 2.3, LFSCK can verify and repair the Object Index (OI)
+ table that is used internally to map Lustre File Identifiers (FIDs) to MDT internal
+ inode numbers, through a process called OI Scrub. An OI Scrub is required after
+ restoring from a file-level MDT backup (<xref linkend="dbdoclet.50438207_71633"/>), or
+ in case the OI table is otherwise corrupted. Later phases of LFSCK will add further
+ checks to the Lustre distributed file system state.</para>
+ <para>Control and monitoring of LFSCK is through LFSCK and the <literal>/proc</literal> file system
+ interfaces. LFSCK supports three types of interface: switch interfaces, status
+ interfaces and adjustments interfaces. These interfaces are detailed below.</para>
<section>
<title>LFSCK switch interface</title>
<section>
<listitem><para>OI scrub magic id (an identifier unique to OI scrub).</para></listitem>
<listitem><para>OI files count.</para></listitem>
<listitem><para>Status: one of the status - <literal>init</literal>, <literal>scanning</literal>, <literal>completed</literal>, <literal>failed</literal>, <literal>stopped</literal>, <literal>paused</literal>, or <literal>crashed</literal>.</para></listitem>
- <listitem><para>Flags: including - <literal>recreated</literal> (OI file(s) is/are removed/recreated), <literal>inconsistent</literal> (restored from file-level backup), <literal>auto</literal> (triggered by non-UI mechanism), and <literal>upgrade</literal> (from Lustre 1.8 IGIF format.)</para></listitem>
+ <listitem><para>Flags: including - <literal>recreated</literal> (OI file(s) is/are removed/recreated),
+ <literal>inconsistent</literal> (restored from
+ file-level backup), <literal>auto</literal>
+ (triggered by non-UI mechanism), and
+ <literal>upgrade</literal> (from Lustre software
+ release 1.8 IGIF format.)</para></listitem>
<listitem><para>Parameters: OI scrub parameters, like <literal>failout</literal>.</para></listitem>
<listitem><para>Time Since Last Completed.</para></listitem>
<listitem><para>Time Since Latest Start.</para></listitem>
<listitem><para>LFSCK namespace magic.</para></listitem>
<listitem><para>LFSCK namespace version..</para></listitem>
<listitem><para>Status: one of the status - <literal>init</literal>, <literal>scanning-phase1</literal>, <literal>scanning-phase2</literal>, <literal>completed</literal>, <literal>failed</literal>, <literal>stopped</literal>, <literal>paused</literal>, or <literal>crashed</literal>.</para></listitem>
- <listitem><para>Flags: including - <literal>scanned-once</literal> (the first cycle scanning has been completed), <literal>inconsistent</literal> (one or more inconsistent FID-in-Dirent or LinkEA entries have been discovered), <literal>upgrade</literal> (from Lustre 1.8 IGIF format.)</para></listitem>
+ <listitem><para>Flags: including - <literal>scanned-once</literal> (the first cycle scanning has been
+ completed), <literal>inconsistent</literal> (one
+ or more inconsistent FID-in-Dirent or LinkEA
+ entries have been discovered),
+ <literal>upgrade</literal> (from Lustre software
+ release 1.8 IGIF format.)</para></listitem>
<listitem><para>Parameters: including <literal>dryrun</literal> and <literal>failout</literal>.</para></listitem>
<listitem><para>Time Since Last Completed.</para></listitem>
<listitem><para>Time Since Latest Start.</para></listitem>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="understandingfailover">
- <title xml:id="understandingfailover.title">Understanding Failover in Lustre</title>
- <para>This chapter describes failover in a Lustre system. It includes:</para>
+ <title xml:id="understandingfailover.title">Understanding Failover in a Lustre File System</title>
+ <para>This chapter describes failover in a Lustre file system. It includes:</para>
<itemizedlist>
<listitem>
<para>
<para>Availability is accomplished by replicating hardware and/or software so that when a primary server fails or is unavailable, a standby server can be switched into its place to run applications and associated resources. This process, called <emphasis role="italic">failover</emphasis>, should be automatic and, in most cases, completely application-transparent.</para>
<para>A failover hardware setup requires a pair of servers with a shared resource (typically a physical storage device, which may be based on SAN, NAS, hardware RAID, SCSI or FC technology). The method of sharing storage should be essentially transparent at the device level; the same physical logical unit number (LUN) should be visible from both servers. To ensure high availability at the physical storage level, we encourage the use of RAID arrays to protect against drive-level failures.</para>
<note>
- <para>Lustre does not provide redundancy for data; it depends exclusively on redundancy of backing storage devices. The backing OST storage should be RAID 5 or, preferably, RAID 6 storage. MDT storage should be RAID 1 or RAID 0+1.</para>
+ <para>The Lustre software does not provide redundancy for data; it depends exclusively on
+ redundancy of backing storage devices. The backing OST storage should be RAID 5 or,
+ preferably, RAID 6 storage. MDT storage should be RAID 1 or RAID 0+1.</para>
</note>
<section remap="h3">
<title><indexterm><primary>failover</primary><secondary>capabilities</secondary></indexterm>Failover Capabilities</title>
<para><emphasis role="bold">Resource management</emphasis> - Starts and stops the Lustre resources as a part of failover, maintains the cluster state, and carries out other resource management tasks.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Health monitoring</emphasis> - Verifies the availability of hardware and network resources and responds to health indications provided by Lustre.</para>
+ <para><emphasis role="bold">Health monitoring</emphasis> - Verifies the availability of
+ hardware and network resources and responds to health indications provided by the Lustre
+ software.</para>
</listitem>
</itemizedlist>
- <para>These capabilities can be provided by a variety of software and/or hardware solutions. For more information about using power management software or hardware and high availability (HA) software with Lustre, see <xref linkend="configuringfailover"/>.</para>
- <para>HA software is responsible for detecting failure of the primary Lustre server node and controlling the failover. Lustre works with any HA software that includes resource (I/O) fencing. For proper resource fencing, the HA software must be able to completely power off the failed server or disconnect it from the shared storage device. If two active nodes have access to the same storage device, data may be severely corrupted.</para>
+ <para>These capabilities can be provided by a variety of software and/or hardware solutions.
+ For more information about using power management software or hardware and high availability
+ (HA) software with the Lustre software, see <xref linkend="configuringfailover"/>.</para>
+ <para>HA software is responsible for detecting failure of the primary Lustre server node and
+ controlling the failover. The Lustre software works with any HA software that includes
+ resource (I/O) fencing. For proper resource fencing, the HA software must be able to
+ completely power off the failed server or disconnect it from the shared storage device. If
+ two active nodes have access to the same storage device, data may be severely
+ corrupted.</para>
</section>
<section remap="h3">
<title><indexterm><primary>failover</primary><secondary>configuration</secondary></indexterm>Types of Failover Configurations</title>
<para><emphasis role="bold">Active/active</emphasis> pair - In this configuration, both nodes are active, each providing a subset of resources. In case of a failure, the second node takes over resources from the failed node.</para>
</listitem>
</itemizedlist>
- <para>Before Lustre 2.4 MDSs are configured as an active/passive pair, while OSSs are deployed in an active/active configuration that provides redundancy without extra overhead. Often the standby MDS is the active MDS for another Lustre file system or the MGS, so no nodes are idle in the cluster.</para>
- <para condition='l24'>Lustre 2.4 introduces metadata targets for individual sub-directories. Active-active failover configurations are available for MDSs that serve MDTs on shared storage.</para>
+ <para>Before Lustre software release 2.4, MDSs are configured as an active/passive pair, while
+ OSSs are deployed in an active/active configuration that provides redundancy without extra
+ overhead. Often the standby MDS is the active MDS for another Lustre file system or the MGS,
+ so no nodes are idle in the cluster.</para>
+ <para condition="l24">Lustre software release 2.4 introduces metadata targets for individual
+ sub-directories. Active-active failover configurations are available for MDSs that serve
+ MDTs on shared storage.</para>
</section>
</section>
<section xml:id="dbdoclet.50540653_97944">
- <title><indexterm><primary>failover</primary><secondary>and Lustre</secondary></indexterm>Failover Functionality in Lustre</title>
- <para>The failover functionality provided in Lustre can be used for the following failover scenario. When a client attempts to do I/O to a failed Lustre target, it continues to try until it receives an answer from any of the configured failover nodes for the Lustre target. A user-space application does not detect anything unusual, except that the I/O may take longer to complete.</para>
- <para>Lustre failover requires two nodes configured as a failover pair, which must share one or more storage devices. Lustre can be configured to provide MDT or OST failover.</para>
+ <title><indexterm>
+ <primary>failover</primary>
+ <secondary>and Lustre</secondary>
+ </indexterm>Failover Functionality in a Lustre File System</title>
+ <para>The failover functionality provided in the Lustre software can be used for the following
+ failover scenario. When a client attempts to do I/O to a failed Lustre target, it continues to
+ try until it receives an answer from any of the configured failover nodes for the Lustre
+ target. A user-space application does not detect anything unusual, except that the I/O may
+ take longer to complete.</para>
+ <para>Lustre failover requires two nodes configured as a failover pair, which must share one or
+ more storage devices. A Lustre file system can be configured to provide MDT or OST
+ failover.</para>
<itemizedlist>
<listitem>
<para>For MDT failover, two MDSs are configured to serve the same MDT. Only one MDS node can serve an MDT at a time.</para>
- <para condition='l24'>Lustre 2.4 allows multiple MDTs. By placing two or more MDT partitions on storage shared by two MDSs, one MDS can fail and the remaining MDS can begin serving the unserved MDT. This is described as an active/active failover pair.</para>
+ <para condition="l24">Lustre software release 2.4 allows multiple MDTs. By placing two or
+ more MDT partitions on storage shared by two MDSs, one MDS can fail and the remaining MDS
+ can begin serving the unserved MDT. This is described as an active/active failover
+ pair.</para>
</listitem>
<listitem>
<para>For OST failover, multiple OSS nodes are configured to be able to serve the same OST. However, only one OSS node can serve the OST at a time. An OST can be moved between OSS nodes that have access to the same storage device using <literal>umount/mount</literal> commands.</para>
</listitem>
</itemizedlist>
- <para>To add a failover partner to a Lustre configuration, the <literal>--failnode</literal> or <literal>--servicenode</literal> option is used. This can be done at creation time (using <literal>mkfs.lustre</literal>) or later when the Lustre system is active (using <literal>tunefs.lustre</literal>). For explanations of these utilities, see <xref linkend="dbdoclet.50438219_75432"/> and <xref linkend="dbdoclet.50438219_39574"/>.</para>
+ <para>To add a failover partner to a Lustre file system configuration, the
+ <literal>--failnode</literal> or <literal>--servicenode</literal> option is used. This can
+ be done at creation time (using <literal>mkfs.lustre</literal>) or later when the Lustre file
+ system is active (using <literal>tunefs.lustre</literal>). For explanations of these
+ utilities, see <xref linkend="dbdoclet.50438219_75432"/> and <xref
+ linkend="dbdoclet.50438219_39574"/>.</para>
<para>Lustre failover capability can be used to upgrade the Lustre software between successive minor versions without cluster downtime. For more information, see <xref linkend="upgradinglustre"/>.</para>
<para>For information about configuring failover, see <xref linkend="configuringfailover"/>.</para>
<note>
- <para>Failover functionality in Lustre is provided only at the file system level. In a complete failover solution, failover functionality for system-level components, such as node failure detection or power control, must be provided by a third-party tool.</para>
+ <para>Failover functionality is provided only at the file system level in the Lustre
+ software. In a complete failover solution, failover functionality for system-level
+ components, such as node failure detection or power control, must be provided by a
+ third-party tool.</para>
</note>
<caution>
- <para>OST failover functionality does not protect against corruption caused by a disk failure. If the storage media (i.e., physical disk) used for an OST fails, Lustre cannot recover it. We strongly recommend that some form of RAID be used for OSTs. Lustre functionality assumes that the storage is reliable, so it adds no extra reliability features.</para>
+ <para>OST failover functionality does not protect against corruption caused by a disk failure.
+ If the storage media (i.e., physical disk) used for an OST fails, the Lustre software does
+ not provide a means to recover it. We strongly recommend that some form of RAID be used for
+ OSTs. Failover functionality provided in the Lustre software assumes that the storage is
+ reliable, so no extra reliability features are included.</para>
</caution>
<section remap="h3">
<title><indexterm><primary>failover</primary><secondary>MDT</secondary></indexterm>MDT Failover Configuration (Active/Passive)</title>
</section>
<section xml:id='dbdoclet.mdtactiveactive' condition='l24'>
<title><indexterm><primary>failover</primary><secondary>MDT</secondary></indexterm>MDT Failover Configuration (Active/Active)</title>
- <para>Multiple MDTs became available with the advent of Lustre 2.4. MDTs can be setup as an active/active failover configuration. A failover cluster is built from two MDSs as shown in <xref linkend="understandingfailover.fig.configmdts"/>.</para>
+ <para>Multiple MDTs became available with the advent of Lustre software release 2.4. MDTs can
+ be setup as an active/active failover configuration. A failover cluster is built from two
+ MDSs as shown in <xref linkend="understandingfailover.fig.configmdts"/>.</para>
<figure>
<title xml:id="understandingfailover.fig.configmdts"> Lustre failover configuration for a active/active MDTs </title>
<mediaobject>
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0"
xml:lang="en-US" xml:id="understandinglustre">
<title xml:id="understandinglustre.title">Understanding Lustre Architecture</title>
- <para>This chapter describes the Lustre architecture and features of Lustre. It includes the
- following sections:</para>
+ <para>This chapter describes the Lustre architecture and features of the Lustre file system. It
+ includes the following sections:</para>
<itemizedlist>
<listitem>
<para>
coherency.</para>
<para>A Lustre file system is currently not particularly well suited for
"peer-to-peer" usage models where clients and servers are running on the same node,
- each sharing a small amount of storage, due to the lack of Lustre-level data replication. In
- such uses, if one client/server fails, then the data stored on that node will not be
- accessible until the node is restarted.</para>
+ each sharing a small amount of storage, due to the lack of data replication at the Lustre
+ software level. In such uses, if one client/server fails, then the data stored on that node
+ will not be accessible until the node is restarted.</para>
<section remap="h3">
<title><indexterm>
<primary>Lustre</primary>
performance characteristics of a Lustre file system and some test results in production
systems.</para>
<table frame="all">
- <title xml:id="understandinglustre.tab1">Lustre Scalability and Performance</title>
+ <title xml:id="understandinglustre.tab1">Lustre File System Scalability and
+ Performance</title>
<tgroup cols="3">
<colspec colname="c1" colwidth="1*"/>
<colspec colname="c2" colwidth="2*"/>
<para>
<emphasis>MDS count:</emphasis></para>
<para> 1 primary + 1 backup</para>
- <para condition="l24">Since Lustre release 2.4: up to 4096 MDSs and up to 4096
- MDTs.</para>
+ <para condition="l24"><emphasis role="italic">Since Lustre software release 2.4:
+ </emphasis></para>
+ <para condition="l24">Up to 4096 MDSs and up to 4096 MDTs</para>
</entry>
<entry>
<para>
</tbody>
</tgroup>
</table>
- <para>Other Lustre features are:</para>
+ <para>Other Lustre software features are:</para>
<itemizedlist>
<listitem>
<para><emphasis role="bold">Performance-enhanced ext4 file system:</emphasis> The Lustre
<listitem>
<para><emphasis role="bold">High-availability:</emphasis> The Lustre file system supports
active/active failover using shared storage partitions for OSS targets (OSTs). Lustre
- Release 2.3 and earlier releases offer active/passive failover using a shared storage
- partition for the MDS target (MDT).</para>
- <para condition="l24">With Lustre Release 2.4 or later servers and clients it is possible
- to configure active/active failover of multiple MDTs. This allows application
+ software release 2.3 and earlier releases offer active/passive failover using a shared
+ storage partition for the MDS target (MDT).</para>
+ <para condition="l24">With Lustre software release 2.4 or later servers and clients it is
+ possible to configure active/active failover of multiple MDTs. This allows application
transparent recovery. The Lustre file system can work with a variety of high
availability (HA) managers to allow automated failover and has no single point of
failure (NSPF). Multiple mount protection (MMP) provides integrated protection from
</indexterm>Lustre Components</title>
<para>An installation of the Lustre software includes a management server (MGS) and one or more
Lustre file systems interconnected with Lustre networking (LNET).</para>
- <para>A basic configuration of Lustre components is shown in <xref
+ <para>A basic configuration of Lustre file system components is shown in <xref
linkend="understandinglustre.fig.cluster"/>.</para>
<figure>
- <title xml:id="understandinglustre.fig.cluster">Lustre components in a basic cluster </title>
+ <title xml:id="understandinglustre.fig.cluster">Lustre file system components in a basic
+ cluster </title>
<mediaobject>
<imageobject>
<imagedata scalefit="1" width="100%" fileref="./figures/Basic_Cluster.png"/>
</imageobject>
<textobject>
- <phrase> Lustre components in a basic cluster </phrase>
+ <phrase> Lustre file system components in a basic cluster </phrase>
</textobject>
</mediaobject>
</figure>
or more local MDTs.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Metadata Target (MDT</emphasis> ) - For Lustre Release 2.3 and
- earlier, each file system has one MDT. The MDT stores metadata (such as filenames,
- directories, permissions and file layout) on storage attached to an MDS. Each file
- system has one MDT. An MDT on a shared storage target can be available to multiple MDSs,
- although only one can access it at a time. If an active MDS fails, a standby MDS can
- serve the MDT and make it available to clients. This is referred to as MDS
+ <para><emphasis role="bold">Metadata Target (MDT</emphasis> ) - For Lustre software
+ release 2.3 and earlier, each file system has one MDT. The MDT stores metadata (such as
+ filenames, directories, permissions and file layout) on storage attached to an MDS. Each
+ file system has one MDT. An MDT on a shared storage target can be available to multiple
+ MDSs, although only one can access it at a time. If an active MDS fails, a standby MDS
+ can serve the MDT and make it available to clients. This is referred to as MDS
failover.</para>
- <para condition="l24">Since Lustre Release 2.4, multiple MDTs are supported. Each file
- system has at least one MDT. An MDT on a shared storage target can be available via
+ <para condition="l24">Since Lustre software release 2.4, multiple MDTs are supported. Each
+ file system has at least one MDT. An MDT on a shared storage target can be available via
multiple MDSs, although only one MDS can export the MDT to the clients at one time. Two
MDS machines share storage for two or more MDTs. After the failure of one MDS, the
remaining MDS begins serving the MDT(s) of the failed MDS.</para>
<title xml:id="understandinglustre.tab.storagerequire"><indexterm>
<primary>Lustre</primary>
<secondary>requirements</secondary>
- </indexterm>Storage and hardware requirements for Lustre components</title>
+ </indexterm>Storage and hardware requirements for Lustre file system components</title>
<tgroup cols="3">
<colspec colname="c1" colwidth="1*"/>
<colspec colname="c2" colwidth="3*"/>
<primary>Lustre</primary>
<secondary>cluster</secondary>
</indexterm>Lustre Cluster</title>
- <para>At scale, the Lustre cluster can include hundreds of OSSs and thousands of clients (see
- <xref linkend="understandinglustre.fig.lustrescale"/>). More than one type of network can
- be used in a Lustre cluster. Shared storage between OSSs enables failover capability. For
- more details about OSS failover, see <xref linkend="understandingfailover"/>.</para>
+ <para>At scale, a Lustre file system cluster can include hundreds of OSSs and thousands of
+ clients (see <xref linkend="understandinglustre.fig.lustrescale"/>). More than one type of
+ network can be used in a Lustre cluster. Shared storage between OSSs enables failover
+ capability. For more details about OSS failover, see <xref linkend="understandingfailover"
+ />.</para>
<figure>
<title xml:id="understandinglustre.fig.lustrescale"><indexterm>
<primary>Lustre</primary>
<imagedata scalefit="1" width="100%" fileref="./figures/Scaled_Cluster.png"/>
</imageobject>
<textobject>
- <phrase> Lustre cluster at scale </phrase>
+ <phrase> Lustre file system cluster at scale </phrase>
</textobject>
</mediaobject>
</figure>
<indexterm>
<primary>Lustre</primary>
<secondary>I/O</secondary>
- </indexterm> Lustre Storage and I/O</title>
- <para>In Lustre release 2.0, Lustre file identifiers (FIDs) were introduced to replace UNIX
- inode numbers for identifying files or objects. A FID is a 128-bit identifier that contains a
- unique 64-bit sequence number, a 32-bit object ID (OID), and a 32-bit version number. The
- sequence number is unique across all Lustre targets in a file system (OSTs and MDTs). This
- change enabled future support for multiple MDTs (introduced in Lustre release 2.3) and ZFS
- (introduced in Lustre release 2.4).</para>
- <para>Also introduced in 2.0 is a feature call <emphasis role="italic">FID-in-dirent</emphasis>
- (also known as <emphasis role="italic">dirdata</emphasis>) in which the FID is stored as part
- of the name of the file in the parent directory. This feature significantly improves
- performance for <literal>ls</literal> command executions by reducing disk I/O. The
- FID-in-dirent is generated at the time the file is created.</para>
+ </indexterm> Lustre File System Storage and I/O</title>
+ <para>In Lustre software release 2.0, Lustre file identifiers (FIDs) were introduced to replace
+ UNIX inode numbers for identifying files or objects. A FID is a 128-bit identifier that
+ contains a unique 64-bit sequence number, a 32-bit object ID (OID), and a 32-bit version
+ number. The sequence number is unique across all Lustre targets in a file system (OSTs and
+ MDTs). This change enabled future support for multiple MDTs (introduced in Lustre software
+ release 2.3) and ZFS (introduced in Lustre software release 2.4).</para>
+ <para>Also introduced in release 2.0 is a feature call <emphasis role="italic"
+ >FID-in-dirent</emphasis> (also known as <emphasis role="italic">dirdata</emphasis>) in
+ which the FID is stored as part of the name of the file in the parent directory. This feature
+ significantly improves performance for <literal>ls</literal> command executions by reducing
+ disk I/O. The FID-in-dirent is generated at the time the file is created.</para>
<note>
- <para>The FID-in-dirent feature is not compatible with the Lustre release 1.8 format.
- Therefore, when an upgrade from Lustre release 1.8 to a Lustre release 2.x is performed, the
- FID-in-dirent feature is not automatically enabled. For upgrades from Lustre release 1.8 to
- Lustre releases 2.0 through 2.3, FID-in-dirent can be enabled manually but only takes effect
- for new files. </para>
- <para>For more information about upgrading from Lustre release 1.8 and enabling FID-in-dirent
- for existing files, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ <para>The FID-in-dirent feature is not compatible with the Lustre software release 1.8 format.
+ Therefore, when an upgrade from Lustre software release 1.8 to a Lustre software release 2.x
+ is performed, the FID-in-dirent feature is not automatically enabled. For upgrades from
+ Lustre software release 1.8 to Lustre software releases 2.0 through 2.3, FID-in-dirent can
+ be enabled manually but only takes effect for new files. </para>
+ <para>For more information about upgrading from Lustre software release 1.8 and enabling
+ FID-in-dirent for existing files, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
linkend="upgradinglustre"/>Chapter 16 “Upgrading a Lustre File System”.</para>
</note>
- <para condition="l24">The LFSCK 1.5 file system administration tool released with Lustre release
- 2.4 provides functionality that enables FID-in-dirent for existing files. It includes the
- following functionality:<itemizedlist>
+ <para condition="l24">The LFSCK 1.5 file system administration tool released with Lustre
+ software release 2.4 provides functionality that enables FID-in-dirent for existing files. It
+ includes the following functionality:<itemizedlist>
<listitem>
<para>Generates IGIF mode FIDs for existing release 1.8 files.</para>
</listitem>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="upgradinglustre">
<title xml:id="upgradinglustre.title">Upgrading a Lustre File System</title>
<para>This chapter describes interoperability between Lustre software releases. It also provides
- procedures for upgrading from a Lustre 1.8 release to a Lustre 2.x release, from a Lustre 2.x
- release to a more recent Lustre 2.x release (major release upgrade), and from a Lustre 2.X.y
- release to 2.X.y (minor release upgrade). It includes the following sections:</para>
+ procedures for upgrading from Lustre software release 1.8 to Lustre softeware release 2.x ,
+ from a Lustre software release 2.x to a more recent Lustre software release 2.x (major release
+ upgrade), and from a a Lustre software release 2.x.y to a more recent Lustre software release
+ 2.x.y (minor release upgrade). It includes the following sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438205_82079"/></para>
</indexterm><indexterm>
<primary>upgrading</primary>
</indexterm>Release Interoperability and Upgrade Requirements</title>
- <para><emphasis role="italic"><emphasis role="bold">Lustre software version 2.x release (major)
+ <para><emphasis role="italic"><emphasis role="bold">Lustre software release 2.x (major)
upgrade:</emphasis></emphasis><itemizedlist>
<listitem>
<para>All servers must be upgraded at the same time, while some or all clients may be
upgraded.</para>
</listitem>
<listitem>
- <para>All servers must be be upgraded to a Linux kernel supported by the Lustre software. See the Linux Test Matrix at <xref
- xmlns:xlink="http://www.w3.org/1999/xlink" linkend="LustreTestMatrixTable"/> for a
- list of tested Lustre distributions.</para>
+ <para>All servers must be be upgraded to a Linux kernel supported by the Lustre software.
+ See the Linux Test Matrix at <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="LustreTestMatrixTable"/> for a list of tested Lustre distributions.</para>
</listitem>
<listitem>
- <para>Clients to be upgraded to the Lustre 2.4 release or higher must be running a compatible Linux distribution. See the Linux Test Matrix at <xref
+ <para>Clients to be upgraded to the Lustre software release 2.4 or higher must be running
+ a compatible Linux distribution. See the Linux Test Matrix at <xref
xmlns:xlink="http://www.w3.org/1999/xlink" linkend="LustreTestMatrixTable"/> for a
list of tested Linux distributions.</para>
</listitem>
</itemizedlist></para>
- <para><emphasis role="italic"><emphasis role="bold">Lustre 2.X.y release (minor)
- upgrade:</emphasis></emphasis></para>
+ <para><emphasis role="italic"><emphasis role="bold">Lustre software release 2.x.y release
+ (minor) upgrade:</emphasis></emphasis></para>
<itemizedlist>
<listitem>
<para>All servers must be upgraded at the same time, while some or all clients may be
</indexterm><indexterm>
<primary>wide striping</primary>
<secondary>large_xattr</secondary>
- </indexterm>Upgrading to Lustre Release 2.x (Major Release)</title>
- <para> The procedure for upgrading a Lustre release 2.x file system to a more recent 2.x release
- of the Lustre software is described in this section. </para>
+ </indexterm>Upgrading to Lustre Software Release 2.x (Major Release)</title>
+ <para> The procedure for upgrading from a Lustre software release 2.x to a more recent 2.x
+ release of the Lustre software is described in this section. </para>
<note>
- <para>This procedure can also be used to upgrade Lustre release 1.8.6-wc1 or later to any
- Lustre release 2.x. To upgrade other versions of 1.8.x, contact your support
- provider.</para>
+ <para>This procedure can also be used to upgrade Lustre software release 1.8.6-wc1 or later to
+ any Lustre software release 2.x. To upgrade other versions of Lustre software release 1.8.x,
+ contact your support provider.</para>
</note>
<note>
- <para condition="l22">In Lustre release 2.2, a feature has been added that allows striping
- across up to 2000 OSTs. By default, this "wide striping" feature is disabled. It is
+ <para condition="l22">In Lustre software release 2.2, a feature has been added that allows
+ striping across up to 2000 OSTs. By default, this "wide striping" feature is disabled. It is
activated by setting the <literal>large_xattr</literal> option on the MDT using either
<literal>mkfs.lustre</literal> or <literal>tune2fs</literal>. For example after upgrading
- an existing file system to Lustre release 2.2 or later, wide striping can be enabled by
- running the following command on the MDT device before mounting
+ an existing file system to Lustre software release 2.2 or later, wide striping can be
+ enabled by running the following command on the MDT device before mounting
it:<screen>tune2fs -O large_xattr</screen>Once the wide striping feature is enabled and in
use on the MDT, it is not possible to directly downgrade the MDT file system to an earlier
version of the Lustre software that does not support wide striping. To disable wide striping:<orderedlist>
option:<screen>lfs find ${mountpoint} --stripe-count=+160</screen></para>
</note>
<note condition="l24">
- <para>In Lustre release 2.4, a new feature allows using multiple MDTs, which can each serve
- one or more remote sub-directories in the file system. The <literal>root</literal> directory
- is always located on MDT0. </para>
- <para>Note that clients running a release prior to the Lustre 2.4 release can only see the
- namespace hosted by MDT0 and will return an IO error if an attempt is made to access a
- directory on another MDT.</para>
+ <para>In Lustre software release 2.4, a new feature allows using multiple MDTs, which can each
+ serve one or more remote sub-directories in the file system. The <literal>root</literal>
+ directory is always located on MDT0. </para>
+ <para>Note that clients running a release prior to the Lustre software release 2.4 can only
+ see the namespace hosted by MDT0 and will return an IO error if an attempt is made to access
+ a directory on another MDT.</para>
</note>
- <para>To upgrade a Lustre release 2.x file system to a more recent major release, complete these
+ <para>To upgrade a Lustre software release 2.x to a more recent major release, complete these
steps:</para>
<orderedlist>
<listitem>
</orderedlist>
</listitem>
<listitem>
- <para>(Optional) For upgrades to Lustre release 2.2 or higher, to enable wide striping on an
- existing MDT, run the following command on the MDT
+ <para>(Optional) For upgrades to Lustre software release 2.2 or higher, to enable wide
+ striping on an existing MDT, run the following command on the MDT
:<screen>mdt# tune2fs -O large_xattr <replaceable>device</replaceable></screen></para>
<para>For more information about wide striping, see <xref
xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_syy_gcl_qk"/>.</para>
</listitem>
<listitem>
- <para>(Optional) For upgrades to Lustre release 2.4 or higher, to format an additional MDT,
- complete these steps:<orderedlist numeration="loweralpha">
+ <para>(Optional) For upgrades to Lustre software release 2.4 or higher, to format an
+ additional MDT, complete these steps:<orderedlist numeration="loweralpha">
<listitem>
<para>Determine the index used for the first MDT (each MDT must have unique index).
Enter:<screen>client$ lctl dl | grep mdc
</listitem>
<listitem>
<para>(Optional) If you are upgrading to Lustre software release 2.3 or higher from Lustre
- software version 2.2 or earlier and want to enable the quota feature, complete these
+ software release 2.2 or earlier and want to enable the quota feature, complete these
steps: <orderedlist numeration="loweralpha">
<listitem>
<para>Before setting up the file system, enter on both the MDS and
</orderedlist></para>
</listitem>
<listitem>
- <para>(Optional) If you are upgrading from Lustre release 1.8, you must manually enable the
- FID-in-dirent feature. On the MDS,
+ <para>(Optional) If you are upgrading from Lustre software release 1.8, you must manually
+ enable the FID-in-dirent feature. On the MDS,
enter:<screen>tune2fs –O dirdata /dev/<replaceable>mdtdev</replaceable></screen></para>
<warning>
<para>This step is not reversible. Do not complete this step until you are sure you will
not be downgrading the Lustre software.</para>
</warning>
<para>This step only enables FID-in-dirent for newly created files. If you are upgrading to
- Lustre release 2.4, you can use LFSCK 1.5 to enable FID-in-dirent for existing files. For
- more information about FID-in-dirent and related functionalities in LFSCK 1.5, see <xref
- xmlns:xlink="http://www.w3.org/1999/xlink" linkend="understandinglustre.storageio"
- />.</para>
+ Lustre software release 2.4, you can use LFSCK 1.5 to enable FID-in-dirent for existing
+ files. For more information about FID-in-dirent and related functionalities in LFSCK 1.5,
+ see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="understandinglustre.storageio"/>.</para>
</listitem>
<listitem>
<para>Start the Lustre file system by starting the components in the order shown in the
<title><indexterm>
<primary>upgrading</primary>
<secondary>2.X.y to 2.X.y (minor release)</secondary>
- </indexterm>Upgrading to a Lustre Release 2.X.y (Minor Release)</title>
- <para>Rolling upgrades are supported for upgrading from any Lustre release 2.X.y to a more
- recent 2.X.y release. This allows the Lustre file system to continue to run while individual
- servers (or their failover partners) and clients are upgraded one at a time. The procedure for
- upgrading a Lustre release 2.X.y file system to a more recent minor release is described in
- this section.</para>
- <para>To upgrade Lustre release 2.X.y to a more recent minor release, complete these
+ </indexterm>Upgrading to Lustre Software Release 2.x.y (Minor Release)</title>
+ <para>Rolling upgrades are supported for upgrading from any Lustre software release 2.x.y to a
+ more recent Lustre software release 2.X.y. This allows the Lustre file system to continue to
+ run while individual servers (or their failover partners) and clients are upgraded one at a
+ time. The procedure for upgrading a Lustre software release 2.x.y to a more recent minor
+ release is described in this section.</para>
+ <para>To upgrade Lustre software release 2.x.y to a more recent minor release, complete these
steps:</para>
<orderedlist>
<listitem>
</entry>
<entry>
<para>File was last accessed N*24 hours ago. (There is no guarantee that <literal>atime</literal> is kept coherent across the cluster.)</para>
- <para>OSTs store a transient <literal>atime</literal> that is updated when clients do read requests. Permanent <literal>atime</literal> is written to the MDS when the file is closed. However, on-disk atime is only updated if it is more than 60 seconds old (<literal>/proc/fs/lustre/mds/*/max_atime_diff</literal>). Lustre considers the latest <literal>atime</literal> from all OSTs. If a <literal>setattr</literal> is set by user, then it is updated on both the MDS and OST, allowing the <literal>atime</literal> to go backward.</para>
+ <para>OSTs store a transient <literal>atime</literal> that is updated when clients
+ do read requests. Permanent <literal>atime</literal> is written to the MDS when
+ the file is closed. However, on-disk atime is only updated if it is more than 60
+ seconds old (<literal>/proc/fs/lustre/mds/*/max_atime_diff</literal>). The Lustre
+ software considers the latest <literal>atime</literal> from all OSTs. If a
+ <literal>setattr</literal> is set by user, then it is updated on both the MDS
+ and OST, allowing the <literal>atime</literal> to go backward.</para>
</entry>
</row>
<row>
<para> <literal>osts [path]</literal></para>
</entry>
<entry>
- <para>Lists all OSTs for the file system. If a path located on a Lustre-mounted file system is specified, then only OSTs belonging to this file system are displayed.</para>
+ <para>Lists all OSTs for the file system. If a path located on a mounted Lustre file
+ system is specified, then only OSTs belonging to this file system are
+ displayed.</para>
</entry>
</row>
<row>
<para> <literal>getname [path...]</literal></para>
</entry>
<entry>
- <para>List each Lustre instance associated with each Lustre mount point.
- If no path is specified, all Lustre mounts points are interrogated. If a list of paths is provided, the instance of each path is provided. If the path is not a Lustre instance 'No such device' is returned.</para>
+ <para>List each Lustre file system instance associated with each Lustre mount
+ point. If no path is specified, all Lustre mount points are interrogated. If a
+ list of paths is provided, the instance of each path is provided. If the path is
+ not a Lustre instance 'No such device' is returned.</para>
</entry>
</row>
<row>
<entry>
<para>Lists striping information for a given filename or directory. By default, the stripe count, stripe size and offset are returned.</para>
<para>If you only want specific striping information, then the options of <literal>--count</literal>,<literal>--size</literal>,<literal>--index</literal> or <literal>--offset</literal> plus various combinations of these options can be used to retrieve specific information.</para>
- <para>If the <literal>--raw</literal> option is specified, the stripe information is printed without substituting the filesystem's default values for unspecified fields. If the striping EA is not set, 0, 0, and -1 will be printed for the stripe count, size, and offset respectively.</para>
+ <para>If the <literal>--raw</literal> option is specified, the stripe information is
+ printed without substituting the file system default values for unspecified
+ fields. If the striping EA is not set, 0, 0, and -1 will be printed for the stripe
+ count, size, and offset respectively.</para>
<para condition='l24'>The <literal>-M</literal> prints the index of the MDT for a given directory. See <xref linkend='dbdoclet.rmremotedir'/>.</para>
</entry>
</row>
</entry>
<entry>
<para>Sets file system quotas for users or groups. Limits can be specified with <literal>--{block|inode}-{softlimit|hardlimit}</literal> or their short equivalents <literal>-b</literal>, <literal>-B</literal>, <literal>-i</literal>, <literal>-I</literal>. Users can set 1, 2, 3 or 4 limits.<footnote>
- <para>The old <literal>setquota</literal> interface is supported, but it may be removed in a future Lustre release.</para>
+ <para>The old <literal>setquota</literal> interface is supported, but it may be
+ removed in a future Lustre software release.</para>
</footnote>Also, limits can be specified with special suffixes, -b, -k, -m, -g, -t, and -p to indicate units of 1, 2^10, 2^20, 2^30, 2^40 and 2^50, respectively. By default, the block limits unit is 1 kilobyte (1,024), and block limits are always kilobyte-grained (even if specified in bytes). See <xref linkend="dbdoclet.50438206_11903"/>.</para>
</entry>
</row>
<screen>$ lfs setstripe -d /mnt/lustre/dir</screen>
<para>Lists the detailed object allocation of a given file.</para>
<screen>$ lfs getstripe -v /mnt/lustre/file1</screen>
- <para>List all the mounted lustre filesystems and corresponding Lustre instances.</para>
+ <para>List all the mounted Lustre file systems and corresponding Lustre instances.</para>
<screen>$ lfs getname</screen>
<para>Efficiently lists all files in a given directory and its subdirectories.</para>
<screen>$ lfs find /mnt/lustre</screen>
<screen>$ lfs df --pool <replaceable>filesystem</replaceable>[.<replaceable>pool</replaceable>] | <replaceable>pathname</replaceable></screen>
<para>List quotas of user 'bob'.</para>
<screen>$ lfs quota -u bob /mnt/lustre</screen>
- <para>Show grace times for user quotas on /mnt/lustre.</para>
+ <para>Show grace times for user quotas on <literal>/mnt/lustre</literal>.</para>
<screen>$ lfs quota -t -u /mnt/lustre</screen>
<para>Changes file owner and group.</para>
<screen>$ lfs quotachown -i /mnt/lustre</screen>
<title><indexterm><primary>lfsck</primary></indexterm>
<literal>lfsck</literal>
</title>
- <para><literal>lfsck</literal> ensures that objects are not referenced by multiple MDS files, that there are no orphan objects on the OSTs (objects that do not have any file on the MDS which references them), and that all of the objects referenced by the MDS exist. Under normal circumstances, Lustre maintains such coherency by distributed logging mechanisms, but under exceptional circumstances that may fail (e.g. disk failure, file system corruption leading to e2fsck repair). To avoid lengthy downtime, you can also run <literal>lfsck</literal> once Lustre is already started.</para>
+ <para><literal>lfsck</literal> ensures that objects are not referenced by multiple MDS files,
+ that there are no orphan objects on the OSTs (objects that do not have any file on the MDS
+ which references them), and that all of the objects referenced by the MDS exist. Under normal
+ circumstances, the Lustre software maintains such coherency by distributed logging mechanisms,
+ but under exceptional circumstances that may fail (e.g. disk failure, file system corruption
+ leading to e2fsck repair). To avoid lengthy downtime, you can also run
+ <literal>lfsck</literal> once the Lustre file system is already started.</para>
<para>The <literal>e2fsck</literal> utility is run on each of the local MDS and OST device file systems and verifies that the underlying <literal>ldiskfs</literal> is consistent. After <literal>e2fsck</literal> is run, <literal>lfsck</literal> does distributed coherency checking for the Lustre file system. In most cases, <literal>e2fsck</literal> is sufficient to repair any file system issues and <literal>lfsck</literal> is not required.</para>
<section remap="h5">
<title>Synopsis</title>
<title>Description</title>
<para>The <literal>filefrag</literal> utility reports the extent of fragmentation in a given file. Initially, <literal>filefrag</literal> attempts to obtain extent information using <literal>FIEMAP ioctl</literal>, which is efficient and fast. If <literal>FIEMAP</literal> is not supported, then <literal>filefrag</literal> uses <literal>FIBMAP</literal>.</para>
<note>
- <para>Lustre only supports <literal>FIEMAP ioctl</literal>. <literal>FIBMAP ioctl</literal> is not supported.</para>
+ <para>The Lustre software only supports <literal>FIEMAP ioctl</literal>. <literal>FIBMAP
+ ioctl</literal> is not supported.</para>
</note>
<para>In default mode <footnote>
<para>The default mode is faster than the verbose/extent mode.</para>
- </footnote>, <literal>filefrag</literal> returns the number of physically discontiguous extents in the file. In extent or verbose mode, each extent is printed with details. For Lustre, the extents are printed in device offset order, not logical offset order.</para>
+ </footnote>, <literal>filefrag</literal> returns the number of physically discontiguous
+ extents in the file. In extent or verbose mode, each extent is printed with details. For a
+ Lustre file system, the extents are printed in device offset order, not logical offset
+ order.</para>
</section>
<section remap="h5">
<title>Options</title>
<para> <literal>-b</literal></para>
</entry>
<entry>
- <para>Uses the 1024-byte blocksize for the output. By default, this blocksize is used by Lustre, since OSTs may use different block sizes.</para>
+ <para>Uses the 1024-byte blocksize for the output. By default, this blocksize is
+ used by the Lustre file system, since OSTs may use different block sizes.</para>
</entry>
</row>
<row>
<title><indexterm><primary>mount</primary></indexterm>
<literal>mount</literal>
</title>
- <para>Lustre uses the standard <literal>mount(8)</literal> Linux command. When mounting a Lustre file system, mount(8) executes the <literal>/sbin/mount.lustre</literal> command to complete the mount. The mount command supports these Lustre-specific options:</para>
+ <para>The standard <literal>mount(8)</literal> Linux command is used to mount a Lustre file
+ system. When mounting a Lustre file system, mount(8) executes the
+ <literal>/sbin/mount.lustre</literal> command to complete the mount. The mount command
+ supports these options specific to a Lustre file system:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<part version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink">
<title>Reference</title>
<partintro>
- <para>Part VI includes reference information on Lustre user utilities, configuration files and module parameters, programming interfaces, system configuration utilities, and system limits. You will find information in this section about:</para>
+ <para>Part VI includes reference information about Lustre file system user utilities,
+ configuration files and module parameters, programming interfaces, system configuration
+ utilities, and system limits. You will find information in this section about:</para>
<itemizedlist>
<listitem>
<?xml version="1.0" encoding="UTF-8"?>
<part version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink">
- <title>Troubleshooting Lustre</title>
+ <title>Troubleshooting a Lustre File System</title>
<partintro>
<para>Part V provides information about troubleshooting a Lustre file system. You will find information in this section about:</para>
<?xml version='1.0' encoding='UTF-8'?>
<book xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xmlns:xi="http://www.w3.org/2001/XInclude" version="5.0" xml:lang="en-US">
<info>
- <title>Lustre<superscript>*</superscript> File System Release 2.x</title>
+ <title>Lustre<superscript>*</superscript> Software Release 2.x</title>
<subtitle>Operations Manual</subtitle>
<copyright>
<year>2010</year>
git://git.whamcloud.com / doc / manual.git / commitdiff