<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="backupandrestore">
- <info>
- <title xml:id="backupandrestore.title">Backing Up and Restoring a File System</title>
- </info>
+ <title xml:id="backupandrestore.title">Backing Up and Restoring a File System</title>
<para>Lustre provides backups at the file system-level, device-level and file-level. This chapter describes how to backup and restore on Lustre, and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438207_56395">
- <title>17.1 Backing up a File System</title>
+
+
+ <title>
+ <indexterm><primary>backup</primary></indexterm>
+ <indexterm><primary>restoring</primary><see>backup</see></indexterm>
+ <indexterm><primary>LVM</primary><see>backup</see></indexterm>
+ <indexterm><primary>rsync</primary><see>backup</see></indexterm>
+
+ Backing up a File System</title>
<para>Backing up a complete file system gives you full control over the files to back up, and allows restoration of individual files as needed. File system-level backups are also the easiest to integrate into existing backup solutions.</para>
<para>File system backups are performed from a Lustre client (or many clients working parallel in different directories) rather than on individual server nodes; this is no different than backing up any other file system.</para>
<para>However, due to the large size of most Lustre file systems, it is not always possible to get a complete backup. We recommend that you back up subsets of a file system. This includes subdirectories of the entire file system, filesets for a single user, files incremented by date, and so on.</para>
<note>
<para>In order to allow Lustre to scale the filesystem namespace for future applications, Lustre 2.x internally uses a 128-bit file identifier for all files. To interface with user applications, Lustre presents 64-bit inode numbers for the <literal>stat()</literal>, <literal>fstat()</literal>, and <literal>readdir()</literal> system calls on 64-bit applications, and 32-bit inode numbers to 32-bit applications.</para>
- <para> Some 32-bit applications accessing Lustre filesystems (on both 32-bit and 64-bit CPUs) may experience problems with the <literal>stat()</literal>, <literal>fstat()</literal> or<literal> readdir()</literal> system calls under certain circumstances, though the Lustre client should return 32-bit inode numbers to these applications.</para>
- <para> In particular, if the Lustre filesystem is exported from a 64-bit client via NFS to a 32-bit client, the Linux NFS server will export 64-bit inode numbers to applications running on the NFS client. If the 32-bit applications are not compiled with Large File Support (LFS), then they return <literal>EOVERFLOW</literal> errors when accessing the Lustre files. To avoid this problem, Linux NFS clients can use the kernel command-line option "<literal>nfs.enable_ino64=0</literal>" in order to force the NFS client to export 32-bit inode numbers to the client.</para>
+ <para>Some 32-bit applications accessing Lustre filesystems (on both 32-bit and 64-bit CPUs) may experience problems with the <literal>stat()</literal>, <literal>fstat()</literal> or<literal> readdir()</literal> system calls under certain circumstances, though the Lustre client should return 32-bit inode numbers to these applications.</para>
+ <para>In particular, if the Lustre filesystem is exported from a 64-bit client via NFS to a 32-bit client, the Linux NFS server will export 64-bit inode numbers to applications running on the NFS client. If the 32-bit applications are not compiled with Large File Support (LFS), then they return <literal>EOVERFLOW</literal> errors when accessing the Lustre files. To avoid this problem, Linux NFS clients can use the kernel command-line option "<literal>nfs.enable_ino64=0</literal>" in order to force the NFS client to export 32-bit inode numbers to the client.</para>
<para><emphasis role="bold">Workaround</emphasis>: We very strongly recommend that backups using <literal>tar(1)</literal> and other utilities that depend on the inode number to uniquely identify an inode to be run on 64-bit clients. The 128-bit Lustre file identifiers cannot be uniquely mapped to a 32-bit inode number, and as a result these utilities may operate incorrectly on 32-bit clients.</para>
</note>
<section remap="h3">
- <title>17.1.1 Lustre_rsync</title>
+ <title><indexterm><primary>backup</primary><secondary>rsync</secondary></indexterm>Lustre_rsync</title>
<para>The <literal>lustre_rsync </literal>feature keeps the entire file system in sync on a backup by replicating the file system's changes to a second file system (the second file system need not be a Lustre file system, but it must be sufficiently large). <literal>lustre_rsync </literal>uses Lustre changelogs to efficiently synchronize the file systems without having to scan (directory walk) the Lustre file system. This efficiency is critically important for large file systems, and distinguishes the Lustre <literal>lustre_rsync</literal> feature from other replication/backup solutions.</para>
<section remap="h4">
- <title>17.1.1.1 Using Lustre_rsync</title>
+ <title><indexterm><primary>backup</primary><secondary>rsync</secondary><tertiary>using</tertiary></indexterm>Using Lustre_rsync</title>
<para>The <literal>lustre_rsync</literal> feature works by periodically running <literal>lustre_rsync</literal>, a userspace program used to synchronize changes in the Lustre file system onto the target file system. The <literal>lustre_rsync</literal> utility keeps a status file, which enables it to be safely interrupted and restarted without losing synchronization between the file systems.</para>
<para>The first time that <literal>lustre_rsync</literal> is run, the user must specify a set of parameters for the program to use. These parameters are described in the following table and in <xref linkend="dbdoclet.50438219_63667"/>. On subsequent runs, these parameters are stored in the the status file, and only the name of the status file needs to be passed to <literal>lustre_rsync</literal>.</para>
<para>Before using <literal>lustre_rsync</literal>:</para>
<itemizedlist>
<listitem>
- <para> Register the changelog user. For details, see the <xref linkend="systemconfigurationutilities"/> (<literal>changelog_register</literal>) parameter in the <xref linkend="systemconfigurationutilities"/> (<literal>lctl</literal>).</para>
+ <para>Register the changelog user. For details, see the <xref linkend="systemconfigurationutilities"/> (<literal>changelog_register</literal>) parameter in the <xref linkend="systemconfigurationutilities"/> (<literal>lctl</literal>).</para>
</listitem>
</itemizedlist>
<para>- AND -</para>
<para> <literal>--user=<user id></literal></para>
</entry>
<entry>
- <para> The changelog user ID for the specified MDT. To use <literal>lustre_rsync</literal>, the changelog user must be registered. For details, see the <literal>changelog_register</literal> parameter in <xref linkend="systemconfigurationutilities"/> (<literal>lctl</literal>). This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statusl</literal>) is not specified.</para>
+ <para>The changelog user ID for the specified MDT. To use <literal>lustre_rsync</literal>, the changelog user must be registered. For details, see the <literal>changelog_register</literal> parameter in <xref linkend="systemconfigurationutilities"/> (<literal>lctl</literal>). This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statusl</literal>) is not specified.</para>
</entry>
</row>
<row>
</row>
<row>
<entry>
- <literal>
- --xattr <yes|no>
- </literal>
+ <literal> --xattr <yes|no> </literal>
</entry>
<entry>
<para>Specifies whether extended attributes (<literal>xattrs</literal>) are synchronized or not. The default is to synchronize extended attributes.</para>
<para> <literal>--verbose</literal></para>
</entry>
<entry>
- <para> Produces verbose output.</para>
+ <para>Produces verbose output.</para>
</entry>
</row>
<row>
<para> <literal>--dry-run</literal></para>
</entry>
<entry>
- <para> Shows the output of <literal>lustre_rsync</literal> commands (<literal>copy</literal>, <literal>mkdir</literal>, etc.) on the target file system without actually executing them.</para>
+ <para>Shows the output of <literal>lustre_rsync</literal> commands (<literal>copy</literal>, <literal>mkdir</literal>, etc.) on the target file system without actually executing them.</para>
</entry>
</row>
<row>
<para> <literal>--abort-on-err</literal></para>
</entry>
<entry>
- <para> Stops processing the <literal>lustre_rsync</literal> operation if an error occurs. The default is to continue the operation.</para>
+ <para>Stops processing the <literal>lustre_rsync</literal> operation if an error occurs. The default is to continue the operation.</para>
</entry>
</row>
</tbody>
</informaltable>
</section>
<section remap="h4">
- <title>17.1.1.2 lustre_rsync Examples</title>
+ <title><indexterm><primary>backup</primary><secondary>rsync</secondary><tertiary>examples</tertiary></indexterm><literal>lustre_rsync</literal> Examples</title>
<para>Sample <literal>lustre_rsync</literal> commands are listed below.</para>
<para>Register a changelog user for an MDT (e.g. <literal>lustre-MDT0000</literal>).</para>
<screen># lctl --device lustre-MDT0000 changelog_register lustre-MDT0000 Registered\
</section>
</section>
<section xml:id="dbdoclet.50438207_71633">
- <title>17.2 Backing Up and Restoring an MDS or OST (Device Level)</title>
+ <title><indexterm><primary>backup</primary><secondary>MDS/OST device level</secondary></indexterm>Backing Up and Restoring an MDS or OST (Device Level)</title>
<para>In some cases, it is useful to do a full device-level backup of an individual device (MDT or OST), before replacing hardware, performing maintenance, etc. Doing full device-level backups ensures that all of the data and configuration files is preserved in the original state and is the easiest method of doing a backup. For the MDT file system, it may also be the fastest way to perform the backup and restore, since it can do large streaming read and write operations at the maximum bandwidth of the underlying devices.</para>
<note>
<para>Keeping an updated full backup of the MDT is especially important because a permanent failure of the MDT file system renders the much larger amount of data in all the OSTs largely inaccessible and unusable.</para>
<para>Even in the face of hardware errors, the <literal>ldiskfs</literal> file system is very robust and it may be possible to recover the file system data after running <literal>e2fsck -f</literal> on the new device.</para>
</section>
<section xml:id="dbdoclet.50438207_21638">
- <title>17.3 Making a File-Level Backup of an OST File System</title>
+ <title><indexterm><primary>backup</primary><secondary>OST file system</secondary></indexterm>Making a File-Level Backup of an OST File System</title>
<para>This procedure provides another way to backup or migrate the data of an OST at the file level, so that the unused space of the OST does not need to be backed up. Backing up a single OST device is not necessarily the best way to perform backups of the Lustre file system, since the files stored in the backup are not usable without metadata stored on the MDT. However, it is the preferred method for migration of OST devices, especially when it is desirable to reformat the underlying file system with different configuration options or to reduce fragmentation.</para>
<note>
<para>In Lustre 2.0 and 2.1 the only correct way to perform an MDT backup and restore is to do a device-level backup as is described in this section. The ability to do MDT file-level backups is not functional in these releases because of the inability to restore the Object Index (OI) file correctly (see bug 22741 for details).</para>
<para><emphasis role="bold">Unmount the file system.</emphasis></para>
<screen>[oss]# umount /mnt/ost</screen>
<note>
- <para>When restoring an OST backup on a different node as part of an OST migration, you also have to change server NIDs and use the --writeconf command to re-generate the configuration logs. See <xref linkend="lustremaintenance"/> (Changing a Server NID).</para>
+ <para>When restoring an OST backup on a different node as part of an OST migration, you also have to change server NIDs and use the <literal>--writeconf</literal> command to re-generate the configuration logs. See <xref linkend="lustremaintenance"/> (Changing a Server NID).</para>
</note>
</listitem>
</orderedlist>
</section>
<section xml:id="dbdoclet.50438207_22325">
- <title>17.4 Restoring a File-Level Backup</title>
+ <title><indexterm><primary>backup</primary><secondary>restoring file system backup</secondary></indexterm>Restoring a File-Level Backup</title>
<para>To restore data from a file-level backup, you need to format the device, restore the file data and then restore the EA data.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Format the new device.</emphasis></para>
+ <para>Format the new device.</para>
<screen>[oss]# mkfs.lustre --ost --index {<emphasis>OST index</emphasis>} {<emphasis>other options</emphasis>} newdev}</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Mount the file system.</emphasis></para>
+ <para>Mount the file system.</para>
<screen>[oss]# mount -t ldiskfs {<emphasis>newdev</emphasis>} /mnt/ost</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Change to the new file system mount point.</emphasis></para>
+ <para>Change to the new file system mount point.</para>
<screen>[oss]# cd /mnt/ost</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Restore the file system backup.</emphasis></para>
+ <para>Restore the file system backup.</para>
<screen>[oss]# tar xzvpf {<emphasis>backup file</emphasis>} --sparse</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Restore the file system extended attributes.</emphasis></para>
+ <para>Restore the file system extended attributes.</para>
<screen>[oss]# setfattr --restore=ea-${date}.bak</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Verify that the extended attributes were restored.</emphasis></para>
+ <para>Verify that the extended attributes were restored.</para>
<screen>[oss]# getfattr -d -m ".*" -e hex O/0/d0/100992 trusted.fid= \
0x0d822200000000004a8a73e500000000808a0100000000000000000000000000</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Change directory out of the file system.</emphasis></para>
+ <para>Change directory out of the file system.</para>
<screen>[oss]# cd -</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Unmount the new file system.</emphasis></para>
+ <para>Unmount the new file system.</para>
<screen>[oss]# umount /mnt/ost</screen>
</listitem>
</orderedlist>
<para>If the file system was used between the time the backup was made and when it was restored, then the <literal>lfsck</literal> tool (part of Lustre <literal>e2fsprogs</literal>) can optionally be run to ensure the file system is coherent. If all of the device file systems were backed up at the same time after the entire Lustre file system was stopped, this is not necessary. In either case, the file system should be immediately usable even if <literal>lfsck</literal> is not run, though there may be I/O errors reading from files that are present on the MDT but not the OSTs, and files that were created after the MDT backup will not be accessible/visible.</para>
</section>
<section xml:id="dbdoclet.50438207_31553">
- <title>17.5 Using LVM Snapshots with Lustre</title>
+ <title><indexterm><primary>backup</primary><secondary>using LVM</secondary></indexterm>Using LVM Snapshots with Lustre</title>
<para>If you want to perform disk-based backups (because, for example, access to the backup system needs to be as fast as to the primary Lustre file system), you can use the Linux LVM snapshot tool to maintain multiple, incremental file system backups.</para>
<para>Because LVM snapshots cost CPU cycles as new files are written, taking snapshots of the main Lustre file system will probably result in unacceptable performance losses. You should create a new, backup Lustre file system and periodically (e.g., nightly) back up new/changed files to it. Periodic snapshots can be taken of this backup file system to create a series of "full" backups.</para>
<note>
<para>Creating an LVM snapshot is not as reliable as making a separate backup, because the LVM snapshot shares the same disks as the primary MDT device, and depends on the primary MDT device for much of its data. If the primary MDT device becomes corrupted, this may result in the snapshot being corrupted.</para>
</note>
<section remap="h3">
- <title>17.5.1 Creating an LVM-based Backup File System</title>
+ <title><indexterm><primary>backup</primary><secondary>using LVM</secondary><tertiary>creating</tertiary></indexterm>Creating an LVM-based Backup File System</title>
<para>Use this procedure to create a backup Lustre file system for use with the LVM snapshot mechanism.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Create LVM volumes for the MDT and OSTs.</emphasis></para>
+ <para>Create LVM volumes for the MDT and OSTs.</para>
<para>Create LVM devices for your MDT and OST targets. Make sure not to use the entire disk for the targets; save some room for the snapshots. The snapshots start out as 0 size, but grow as you make changes to the current file system. If you expect to change 20% of the file system between backups, the most recent snapshot will be 20% of the target size, the next older one will be 40%, etc. Here is an example:</para>
<screen>cfs21:~# pvcreate /dev/sda1
Physical volume "/dev/sda1" successfully created
ACTIVE '/dev/volgroup/OST0' [200.00 MB] inherit</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Format the LVM volumes as Lustre targets.</emphasis></para>
+ <para>Format the LVM volumes as Lustre targets.</para>
<para>In this example, the backup file system is called 'main' and designates the current, most up-to-date backup.</para>
<screen>cfs21:~# mkfs.lustre --mdt --fsname=main /dev/volgroup/MDT
No management node specified, adding MGS to this MDT.
</orderedlist>
</section>
<section remap="h3">
- <title>17.5.2 Backing up New/Changed Files to the Backup File System</title>
+ <title><indexterm><primary>backup</primary><secondary>new/changed files</secondary></indexterm>Backing up New/Changed Files to the Backup File System</title>
<para>At periodic intervals e.g., nightly, back up new and changed files to the LVM-based backup file system.</para>
<screen>cfs21:~# cp /etc/passwd /mnt/main
fstab passwd</screen>
</section>
<section remap="h3">
- <title>17.5.3 Creating Snapshot Volumes</title>
+ <title><indexterm><primary>backup</primary><secondary>using LVM</secondary><tertiary>createing snapshots</tertiary></indexterm>Creating Snapshot Volumes</title>
<para>Whenever you want to make a "checkpoint" of the main Lustre file system, create LVM snapshots of all target MDT and OSTs in the LVM-based backup file system. You must decide the maximum size of a snapshot ahead of time, although you can dynamically change this later. The size of a daily snapshot is dependent on the amount of data changed daily in the main Lustre file system. It is likely that a two-day old snapshot will be twice as big as a one-day old snapshot.</para>
<para>You can create as many snapshots as you have room for in the volume group. If necessary, you can dynamically add disks to the volume group.</para>
<para>The snapshots of the target MDT and OSTs should be taken at the same point in time. Make sure that the cronjob updating the backup file system is not running, since that is the only thing writing to the disks. Here is an example:</para>
fstab passwd termcap</screen>
</section>
<section remap="h3">
- <title>17.5.4 Restoring the File System From a Snapshot</title>
+ <title><indexterm><primary>backup</primary><secondary>using LVM</secondary><tertiary>restoring</tertiary></indexterm>Restoring the File System From a Snapshot</title>
<para>Use this procedure to restore the file system from an LVM snapshot.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Rename the LVM snapshot.</emphasis></para>
+ <para>Rename the LVM snapshot.</para>
<para>Rename the file system snapshot from "main" to "back" so you can mount it without unmounting "main". This is recommended, but not required. Use the <literal>--reformat</literal> flag to <literal>tunefs.lustre</literal> to force the name change. For example:</para>
<screen>cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/volgroup/M\
DTb1
cfs21:~# umount /mnt/ostback</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Mount the file system from the LVM snapshot.</emphasis></para>
+ <para>Mount the file system from the LVM snapshot.</para>
<para>For example:</para>
<screen>cfs21:~# mount -t lustre /dev/volgroup/MDTb1 /mnt/mdtback \
cfs21:~# mount -t lustre cfs21:/back /mnt/back</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Note the old directory contents, as of the snapshot time.</emphasis></para>
+ <para>Note the old directory contents, as of the snapshot time.</para>
<para>For example:</para>
<screen>cfs21:~/cfs/b1_5/lustre/utils# ls /mnt/back
fstab passwds</screen>
</orderedlist>
</section>
<section remap="h3">
- <title>17.5.5 Deleting Old Snapshots</title>
+ <title><indexterm><primary>backup</primary><secondary>using LVM</secondary><tertiary>deleting</tertiary></indexterm>Deleting Old Snapshots</title>
<para>To reclaim disk space, you can erase old snapshots as your backup policy dictates. Run:</para>
<screen>lvremove /dev/volgroup/MDTb1</screen>
</section>
<section remap="h3">
- <title>17.5.6 Changing Snapshot Volume Size</title>
+ <title><indexterm><primary>backup</primary><secondary>using LVM</secondary><tertiary>resizing</tertiary></indexterm>Changing Snapshot Volume Size</title>
<para>You can also extend or shrink snapshot volumes if you find your daily deltas are smaller or larger than expected. Run:</para>
<screen>lvextend -L10G /dev/volgroup/MDTb1</screen>
<note>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="benchmarkingtests">
- <info>
- <title xml:id="benchmarkingtests.title">Benchmarking Lustre Performance (Lustre I/O Kit)</title>
- </info>
+ <title xml:id="benchmarkingtests.title">Benchmarking Lustre Performance (Lustre I/O Kit)</title>
<para>This chapter describes the Lustre I/O kit, a collection of I/O benchmarking tools for a Lustre cluster, and PIOS, a parallel I/O simulator for Linux and Solaris. It includes:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438212_44437">
- <title>24.1 Using Lustre I/O Kit Tools</title>
+ <title>
+ <indexterm><primary>benchmarking</primary><secondary>with Lustre I/O Kit</secondary></indexterm>
+ <indexterm><primary>profiling</primary><see>benchmarking</see></indexterm>
+ <indexterm><primary>tuning</primary><see>benchmarking</see></indexterm>
+ <indexterm><primary>performance</primary><see>benchmarking</see></indexterm>
+
+ Using Lustre I/O Kit Tools</title>
<para>The tools in the Lustre I/O Kit are used to benchmark Lustre hardware and validate that it is working as expected before you install the Lustre software. It can also be used to to validate the performance of the various hardware and software layers in the cluster and also to find and troubleshoot I/O issues.</para>
<para>Typically, performance is measured starting with single raw devices and then proceeding to groups of devices. Once raw performance has been established, other software layers are then added incrementally and tested.</para>
<section remap="h3">
- <title>24.1.1 Contents of the Lustre I/O Kit</title>
+ <title>Contents of the Lustre I/O Kit</title>
<para>The I/O kit contains three tests, each of which tests a progressively higher layer in the Lustre stack:</para>
<itemizedlist>
<listitem>
<para>A utility <literal>stats-collect</literal> is also provided to collect application profiling information from Lustre clients and servers. See <xref linkend="dbdoclet.50438212_58201"/> for more information.</para>
</section>
<section remap="h3">
- <title>24.1.2 Preparing to Use the Lustre I/O Kit</title>
+ <title>Preparing to Use the Lustre I/O Kit</title>
<para>The following prerequisites must be met to use the tests in the Lustre I/O kit:</para>
<itemizedlist>
<listitem>
</section>
</section>
<section xml:id="dbdoclet.50438212_51053">
- <title>24.2 Testing I/O Performance of Raw Hardware (<literal>sgpdd_survey</literal>)</title>
+ <title><indexterm><primary>benchmarking</primary><secondary>raw hardware with sgpdd_survey</secondary></indexterm>Testing I/O Performance of Raw Hardware (<literal>sgpdd_survey</literal>)</title>
<para>The <literal>sgpdd_survey</literal> tool is used to test bare metal I/O performance of the raw hardware, while bypassing as much of the kernel as possible. This survey may be used to characterize the performance of a SCSI device by simulating an OST serving multiple stripe files. The data gathered by this survey can help set expectations for the performance of a Lustre OST using this device.</para>
<para>The script uses <literal>sgp_dd</literal> to carry out raw sequential disk I/O. It runs with variable numbers of <literal>sgp_dd</literal> threads to show how performance varies with different request queue depths.</para>
<para>The script spawns variable numbers of <literal>sgp_dd</literal> instances, each reading or writing a separate area of the disk to demonstrate performance variance within a number of concurrent stripe files.</para>
<para>If you need to create raw devices to use the <literal>sgpdd_survey</literal> tool, note that raw device 0 cannot be used due to a bug in certain versions of the "raw" utility (including that shipped with RHEL4U4.)</para>
</note>
<section remap="h3">
- <title>24.2.1 Tuning Linux Storage Devices</title>
+ <title><indexterm><primary>benchmarking</primary><secondary>tuning storage</secondary></indexterm>Tuning Linux Storage Devices</title>
<para>To get large I/O transfers (1 MB) to disk, it may be necessary to tune several kernel parameters as specified:</para>
<screen>/sys/block/sdN/queue/max_sectors_kb = 4096
/sys/block/sdN/queue/max_phys_segments = 256
/sys/block/sdN/queue/scheduler</screen>
</section>
<section remap="h3">
- <title>24.2.2 Running sgpdd_survey</title>
+ <title>Running sgpdd_survey</title>
<para>The <literal>sgpdd_survey</literal> script must be customized for the particular device being tested and for the location where the script saves its working and result files (by specifying the <literal>${rslt}</literal> variable). Customization variables are described at the beginning of the script.</para>
<para>When the <literal>sgpdd_survey</literal> script runs, it creates a number of working files and a pair of result files. The names of all the files created start with the prefixdefined in the variable <literal>${rslt}</literal>. (The default value is <literal>/tmp</literal>.) The files include:</para>
<itemizedlist>
<screen>${rslt}_<emphasis><date/time></emphasis>.summary</screen>
</listitem>
<listitem>
- <para> Temporary (tmp) files</para>
+ <para>Temporary (tmp) files</para>
<screen>${rslt}_<emphasis><date/time></emphasis>_*
</screen>
</listitem>
<listitem>
- <para> Collected tmp files for post-mortem</para>
+ <para>Collected tmp files for post-mortem</para>
<screen>${rslt}_<emphasis><date/time></emphasis>.detail
</screen>
</listitem>
</section>
</section>
<section xml:id="dbdoclet.50438212_26516">
- <title>24.3 Testing OST Performance (<literal>obdfilter_survey</literal>)</title>
+ <title><indexterm><primary>benchmarking</primary><secondary>OST performance</secondary></indexterm>Testing OST Performance (<literal>obdfilter_survey</literal>)</title>
<para>The <literal>obdfilter_survey</literal> script generates sequential I/O from varying numbers of threads and objects (files) to simulate the I/O patterns of a Lustre client.</para>
<para>The <literal>obdfilter_survey</literal> script can be run directly on the OSS node to measure the OST storage performance without any intervening network, or it can be run remotely on a Lustre client to measure the OST performance including network overhead.</para>
<para>The <literal>obdfilter_survey</literal> is used to characterize the performance of the following:</para>
</listitem>
<listitem>
<para><emphasis role="bold">Remote file system over the network</emphasis> - In this mode the <literal>obdfilter_survey</literal> script generates I/O from a Lustre client to a remote OSS to write the data to the file system.</para>
- <para>To run the test against all the local OSCs, pass the parameter <literal>case=netdisk</literal> to the script. Alternately you can pass the target= parameter with one or more OSC devices (e.g., <literal>lustre-OST0000-osc-ffff88007754bc00</literal>) against which the tests are to be run.</para>
- <para>For more details, see <xref linkend="dbdoclet.50438212_62662"/>.</para>
+
<para>To run the test against all the local OSCs, pass the parameter <literal>case=netdisk</literal> to the script. Alternately you can pass the target= parameter with one or more OSC devices (e.g., <literal>lustre-OST0000-osc-ffff88007754bc00</literal>) against which the tests are to be run.</para>
+
<para>For more details, see <xref linkend="dbdoclet.50438212_62662"/>.</para>
</listitem>
</itemizedlist>
<caution>
<note>
<para>The <literal>obdfilter_survey</literal> script must be customized, depending on the components under test and where the script's working files should be kept. Customization variables are described at the beginning of the <literal>obdfilter_survey</literal> script. In particular, pay attention to the listed maximum values listed for each parameter in the script.</para>
</note>
- <section xml:id='dbdoclet.50438212_59319'>
- <title>24.3.1 Testing Local Disk Performance</title>
+ <section xml:id="dbdoclet.50438212_59319">
+ <title><indexterm><primary>benchmarking</primary><secondary>local disk</secondary></indexterm>Testing Local Disk Performance</title>
<para>The <literal>obdfilter_survey</literal> script can be run automatically or manually against a local disk. This script profiles the overall throughput of storage hardware, including the file system and RAID layers managing the storage, by sending workloads to the OSTs that vary in thread count, object count, and I/O size.</para>
<para>When the <literal>obdfilter_survey</literal> script is run, it provides information about the performance abilities of the storage hardware and shows the saturation points.</para>
<para>The <literal>plot-obdfilter</literal> script generates from the output of the <literal>obdfilter_survey</literal> a CSV file and parameters for importing into a spreadsheet or gnuplot to visualize the data.</para>
<para><emphasis role="bold">To perform an automatic run:</emphasis></para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Start the Lustre OSTs.</emphasis></para>
+ <para>Start the Lustre OSTs.</para>
<para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Verify that the obdecho module is loaded. Run:</emphasis></para>
+ <para>Verify that the obdecho module is loaded. Run:</para>
<screen>modprobe obdecho</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Run the <literal>obdfilter_survey</literal> script with the parameter <literal>case=disk</literal>.</emphasis></para>
+ <para>Run the <literal>obdfilter_survey</literal> script with the parameter <literal>case=disk</literal>.</para>
<para>For example, to run a local test with up to two objects (nobjhi), up to two threads (thrhi), and 1024 MB transfer size (size):</para>
<screen>$ nobjhi=2 thrhi=2 size=1024 case=disk sh obdfilter-survey</screen>
</listitem>
<para><emphasis role="italic">To perform a manual run:</emphasis></para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Start the Lustre OSTs.</emphasis></para>
+ <para>Start the Lustre OSTs.</para>
<para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Verify that the <literal>obdecho</literal> module is loaded. Run:</emphasis></para>
+ <para>Verify that the <literal>obdecho</literal> module is loaded. Run:</para>
<screen>modprobe obdecho</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Determine the OST names.</emphasis></para>
+ <para>Determine the OST names.</para>
<para>On the OSS nodes to be tested, run the lctldl command. The OST device names are listed in the fourth column of the output. For example:</para>
<screen>$ lctl dl |grep obdfilter
0 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 1159
...</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">List all OSTs you want to test.</emphasis></para>
+ <para>List all OSTs you want to test.</para>
<para>Use the <literal>target=parameter</literal> to list the OSTs separated by spaces. List the individual OSTs by name using the format <emphasis>
<literal><fsname>-<OSTnumber></literal>
</emphasis> (for example, lustre-OST0001). You do not have to specify an MDS or LOV.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Run the <literal>obdfilter_survey</literal> script with the <literal>target=parameter</literal>.</emphasis></para>
+ <para>Run the <literal>obdfilter_survey</literal> script with the <literal>target=parameter</literal>.</para>
<para>For example, to run a local test with up to two objects (<literal>nobjhi</literal>), up to two threads (<literal>thrhi</literal>), and 1024 Mb (size) transfer size:</para>
<screen>$ nobjhi=2 thrhi=2 size=1024 targets='lustre-OST0001 \
lustre-OST0002' sh obdfilter-survey</screen>
</listitem>
</orderedlist>
</section>
- <section xml:id='dbdoclet.50438212_36037'>
- <title>24.3.2 Testing Network Performance</title>
+ <section xml:id="dbdoclet.50438212_36037">
+ <title><indexterm><primary>benchmarking</primary><secondary>network</secondary></indexterm>Testing Network Performance</title>
<para>The <literal>obdfilter_survey</literal> script can only be run automatically against a network; no manual test is provided.</para>
<para>To run the network test, a specific Lustre setup is needed. Make sure that these configuration requirements have been met.</para>
<para><emphasis role="bold">To perform an automatic run:</emphasis></para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Start the Lustre OSTs.</emphasis></para>
+ <para>Start the Lustre OSTs.</para>
<para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Verify that the <literal>obdecho</literal> module is loaded. Run:</emphasis></para>
+ <para>Verify that the <literal>obdecho</literal> module is loaded. Run:</para>
<screen>modprobe obdecho</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Start <literal>lctl</literal> and check the device list, which must be empty. Run</emphasis>:</para>
+ <para>Start <literal>lctl</literal> and check the device list, which must be empty. Run:</para>
<screen>lctl dl</screen>
</listitem>
<listitem>
sh odbfilter-survey</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">On the server side, view the statistics at:</emphasis></para>
+ <para>On the server side, view the statistics at:</para>
<screen>/proc/fs/lustre/obdecho/<emphasis><echo_srv></emphasis>/stats</screen>
<para>where <emphasis>
<literal><echo_srv></literal>
</listitem>
</orderedlist>
</section>
- <section xml:id='dbdoclet.50438212_62662'>
- <title>24.3.3 Testing Remote Disk Performance</title>
+ <section xml:id="dbdoclet.50438212_62662">
+ <title><indexterm><primary>benchmarking</primary><secondary>remote disk</secondary></indexterm>Testing Remote Disk Performance</title>
<para>The <literal>obdfilter_survey</literal> script can be run automatically or manually against a network disk. To run the network disk test, start with a standard Lustre configuration. No special setup is needed.</para>
<para><emphasis role="bold">To perform an automatic run:</emphasis></para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Start the Lustre OSTs.</emphasis></para>
+ <para>Start the Lustre OSTs.</para>
<para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Verify that the <literal>obdecho</literal> module is loaded. Run:</emphasis></para>
+ <para>Verify that the <literal>obdecho</literal> module is loaded. Run:</para>
<screen>modprobe obdecho</screen>
</listitem>
<listitem>
<para><emphasis role="bold">To perform a manual run:</emphasis></para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Start the Lustre OSTs.</emphasis></para>
+ <para>Start the Lustre OSTs.</para>
<para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Verify that the <literal>obdecho</literal> module is loaded. Run:</emphasis></para>
+ <para>Verify that the <literal>obdecho</literal> module is loaded. Run:</para>
<para>modprobe obdecho</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Determine the OSC names.</emphasis></para>
+ <para>Determine the OSC names.</para>
<para>On the OSS nodes to be tested, run the <literal>lctl dl</literal> command. The OSC device names are listed in the fourth column of the output. For example:</para>
<screen>$ lctl dl |grep obdfilter
3 UP osc lustre-OST0000-osc-ffff88007754bc00 \
</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">List all OSCs you want to test.</emphasis></para>
+ <para>List all OSCs you want to test.</para>
<para>Use the <literal>target=parameter</literal> to list the OSCs separated by spaces. List the individual OSCs by name seperated by spaces using the format <literal><replaceable><fsname>-<OST_name></replaceable>-osc-<replaceable><OSC_number></replaceable></literal> (for example, <literal>lustre-OST0000-osc-ffff88007754bc00</literal>). You <replaceable role="bold">do not have to specify an MDS or LOV.</replaceable></para>
</listitem>
<listitem>
</orderedlist>
</section>
<section remap="h3">
- <title>24.3.4 Output Files</title>
+ <title>Output Files</title>
<para>When the <literal>obdfilter_survey</literal> script runs, it creates a number of working files and a pair of result files. All files start with the prefix defined in the variable <literal>${rslt}</literal>.</para>
<informaltable frame="all">
<tgroup cols="2">
<para>The <literal>obdfilter_survey</literal> script may not clean up properly if it is aborted or if it encounters an unrecoverable error. In this case, a manual cleanup may be required, possibly including killing any running instances of <literal>lctl</literal> (local or remote), removing <literal>echo_client</literal> instances created by the script and unloading <literal>obdecho</literal>.</para>
</note>
<section remap="h4">
- <title>24.3.4.1 Script Output</title>
+ <title>Script Output</title>
<para>The <literal>.summary</literal> file and <literal>stdout</literal> of the <literal>obdfilter_survey</literal> script contain lines like:</para>
<screen>ost 8 sz 67108864K rsz 1024 obj 8 thr 8 write 613.54 [ 64.00, 82.00]
</screen>
</note>
</section>
<section remap="h4">
- <title>24.3.4.2 Visualizing Results</title>
+ <title>Visualizing Results</title>
<para>It is useful to import the <literal>obdfilter_survey</literal> script summary data (it is fixed width) into Excel (or any graphing package) and graph the bandwidth versus the number of threads for varying numbers of concurrent regions. This shows how the OSS performs for a given number of concurrently-accessed objects (files) with varying numbers of I/Os in flight.</para>
<para>It is also useful to monitor and record average disk I/O sizes during each test using the 'disk io size' histogram in the file <literal>/proc/fs/lustre/obdfilter/</literal> (see <xref linkend="dbdoclet.50438271_55057"/> for details). These numbers help identify problems in the system when full-sized I/Os are not submitted to the underlying disk. This may be caused by problems in the device driver or Linux block layer.</para>
<screen> */brw_stats</screen>
</section>
</section>
<section xml:id="dbdoclet.50438212_85136">
- <title>24.4 Testing OST I/O Performance (<literal>ost_survey</literal>)</title>
+ <title><indexterm><primary>benchmarking</primary><secondary>OST I/O</secondary></indexterm>Testing OST I/O Performance (<literal>ost_survey</literal>)</title>
<para>The <literal>ost_survey</literal> tool is a shell script that uses <literal>lfs setstripe</literal> to perform I/O against a single OST. The script writes a file (currently using <literal>dd</literal>) to each OST in the Lustre file system, and compares read and write speeds. The <literal>ost_survey</literal> tool is used to detect anomalies between otherwise identical disk subsystems.</para>
<note>
<para>We have frequently discovered wide performance variations across all LUNs in a cluster. This may be caused by faulty disks, RAID parity reconstruction during the test, or faulty network hardware.</para>
</screen>
</section>
<section xml:id="dbdoclet.50438212_58201">
- <title>24.5 Collecting Application Profiling Information (<literal>stats-collect</literal>)</title>
+ <title><indexterm><primary>benchmarking</primary><secondary>application profiling</secondary></indexterm>Collecting Application Profiling Information (<literal>stats-collect</literal>)</title>
<para>The <literal>stats-collect</literal> utility contains the following scripts used to collect application profiling information from Lustre clients and servers:</para>
<itemizedlist>
<listitem>
<para>The <literal>stats-collect</literal> utility requires:</para>
<itemizedlist>
<listitem>
- <para> Lustre to be installed and set up on your cluster</para>
+ <para>Lustre to be installed and set up on your cluster</para>
</listitem>
<listitem>
- <para> SSH and SCP access to these nodes without requiring a password</para>
+ <para>SSH and SCP access to these nodes without requiring a password</para>
</listitem>
</itemizedlist>
<section remap="h3">
- <title>24.5.1 Using <literal>stats-collect</literal></title>
+ <title>Using <literal>stats-collect</literal></title>
<para>The stats-collect utility is configured by including profiling configuration variables in the config.sh script. Each configuration variable takes the following form, where 0 indicates statistics are to be collected only when the script starts and stops and <emphasis>n</emphasis> indicates the interval in seconds at which statistics are to be collected:</para>
<screen><emphasis><statistic></emphasis>_INTERVAL=<emphasis>[</emphasis>0<emphasis>|n]</emphasis></screen>
<para>Statistics that can be collected include:</para>
<para>Begin collecting statistics on each node specified in the config.sh script.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Starting the collect profile daemon on each node by entering:</emphasis></para>
+ <para>Starting the collect profile daemon on each node by entering:</para>
<screen>sh gather_stats_everywhere.sh config.sh start
</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Run the test.</emphasis></para>
+ <para>Run the test.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Stop collecting statistics on each node, clean up the temporary file, and create a profiling tarball.</emphasis></para>
+ <para>Stop collecting statistics on each node, clean up the temporary file, and create a profiling tarball.</para>
<para>Enter:</para>
<screen>sh gather_stats_everywhere.sh config.sh stop <emphasis><log_name.tgz></emphasis></screen>
<para>When <emphasis>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configurationfilesmoduleparameters">
- <info>
<title xml:id="configurationfilesmoduleparameters.title">Configuration Files and Module Parameters</title>
- </info>
<para>This section describes configuration files and module parameters and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438293_15350">
- <title>35.1 Introduction</title>
+ <title>
+ <indexterm><primary>configuring</primary></indexterm>
+ <indexterm><primary>LNET</primary><see>configuring</see></indexterm>
+
+
+ Introduction</title>
<para>LNET network hardware and routing are now configured via module parameters. Parameters should be specified in the <literal>/etc/modprobe.conf </literal>file, for example:</para>
<screen>alias lustre llite
options lnet networks=tcp0,elan0</screen>
<para>For the following parameters, default option settings are shown in parenthesis. Changes to parameters marked with a W affect running systems. (Unmarked parameters can only be set when LNET loads for the first time.) Changes to parameters marked with <literal>Wc</literal> only have effect when connections are established (existing connections are not affected by these changes.)</para>
</section>
<section xml:id="dbdoclet.50438293_78010">
- <title>35.2 Module Options</title>
+ <title>
+ <indexterm><primary>configuring</primary><secondary>module options</secondary></indexterm>
+
+ Module Options</title>
<itemizedlist>
<listitem>
<para>With routed or other multi-network configurations, use <literal>ip2nets</literal> rather than networks, so all nodes can use the same configuration.</para>
</listitem>
</itemizedlist>
<section remap="h3">
- <title>35.2.1 LNET Options</title>
+ <title><indexterm><primary>configuring</primary><secondary>LNET options</secondary></indexterm>
+LNET Options</title>
<para>This section describes LNET options.</para>
<section remap="h4">
- <title>35.2.1.1 Network Topology</title>
+ <title><indexterm><primary>configuring</primary><secondary>network topology</secondary></indexterm>
+Network Topology</title>
<para>Network topology module parameters determine which networks a node should join, whether it should route between these networks, and how it communicates with non-local networks.</para>
<para>Here is a list of various networks and the supported software stacks:</para>
<informaltable frame="all">
elan 198.128.88.98@tcp # TCP NID of router '</screen>
</section>
<section remap="h4">
- <title>35.2.1.2 networks ("tcp")</title>
+ <title><indexterm><primary>configuring</primary><secondary>network</secondary><tertiary>tcp</tertiary></indexterm>
+networks ("tcp")</title>
<para>This is an alternative to "<literal>ip2nets</literal>" which can be used to specify the networks to be instantiated explicitly. The syntax is a simple comma separated list of <literal><net-spec></literal>s (see above). The default is only used if neither 'ip2nets' nor 'networks' is specified.</para>
</section>
<section remap="h4">
- <title>35.2.1.3 routes ("")</title>
+ <title><indexterm><primary>configuring</primary><secondary>network</secondary><tertiary>routes</tertiary></indexterm>
+routes ("")</title>
<para>This is a string that lists networks and the NIDs of routers that forward to them.</para>
<para>It has the following syntax (<literal><w></literal> is one or more whitespace characters):</para>
<screen><routes> :== <route>{ ; <route> }
<para>If the target network string contains no expansions, then the hopcount defaults to 1 and may be omitted (that is, the remote network is adjacent). In practice, this is true for most multi-network configurations. It is an error to specify an inconsistent hop count for a given target network. This is why an explicit hopcount is required if the target network string specifies more than one network.</para>
</section>
<section remap="h4">
- <title>35.2.1.4 forwarding ("")</title>
+ <title><indexterm><primary>configuring</primary><secondary>network</secondary><tertiary>forwarding</tertiary></indexterm>
+forwarding ("")</title>
<para>This is a string that can be set either to "<literal>enabled</literal>" or "<literal>disabled</literal>" for explicit control of whether this node should act as a router, forwarding communications between all local networks.</para>
<para>A standalone router can be started by simply starting LNET ('<literal>modprobe ptlrpc</literal>') with appropriate network topology options.</para>
<informaltable frame="all">
</section>
</section>
<section remap="h3">
- <title>35.2.2 <literal>SOCKLND</literal> Kernel TCP/IP LND</title>
+ <title><indexterm><primary>configuring</primary><secondary>network</secondary><tertiary>SOCKLND</tertiary></indexterm>
+<literal>SOCKLND</literal> Kernel TCP/IP LND</title>
<para>The <literal>SOCKLND</literal> kernel TCP/IP LND (<literal>socklnd</literal>) is connection-based and uses the acceptor to establish communications via sockets with its peers.</para>
<para>It supports multiple instances and load balances dynamically over multiple interfaces. If no interfaces are specified by the <literal>ip2nets</literal> or networks module parameter, all non-loopback IP interfaces are used. The address-within-network is determined by the address of the first IP interface an instance of the <literal>socklnd</literal> encounters.</para>
<para>Consider a node on the 'edge' of an InfiniBand network, with a low-bandwidth management Ethernet (<literal>eth0</literal>), IP over IB configured (<literal>ipoib0</literal>), and a pair of GigE NICs (<literal>eth1</literal>,<literal>eth2</literal>) providing off-cluster connectivity. This node should be configured with '<literal>networks=vib,tcp(eth1,eth2)</literal>' to ensure that the <literal>socklnd</literal> ignores the management Ethernet and IPoIB.</para>
</informaltable>
</section>
<section remap="h3">
- <title>35.2.3 Portals LND (Linux)</title>
+ <title><indexterm><primary>configuring</primary><secondary>portals</secondary></indexterm>Portals LND (Linux)</title>
<para>The Portals LND Linux (<literal>ptllnd</literal>) can be used as a interface layer to communicate with Sandia Portals networking devices. This version is intended to work on Cray XT3 Linux nodes that use Cray Portals as a network transport.</para>
<para><emphasis role="bold">Message Buffers</emphasis></para>
<para>When <literal>ptllnd</literal> starts up, it allocates and posts sufficient message buffers to allow all expected peers (set by concurrent_peers) to send one unsolicited message. The first message that a peer actually sends is (so-called) "<literal>HELLO</literal>" message, used to negotiate how much additional buffering to setup (typically 8 messages). If 10000 peers actually exist, then enough buffers are posted for 80000 messages.</para>
</informaltable>
</section>
<section remap="h3">
- <title>35.2.4 MX LND</title>
+ <title><indexterm><primary>configuring</primary><secondary>MX LND</secondary></indexterm>MX LND</title>
<para><literal>MXLND</literal> supports a number of load-time parameters using Linux's module parameter system. The following variables are available:</para>
<informaltable frame="all">
<tgroup cols="2">
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringfailover">
- <info>
- <title xml:id="configuringfailover.title">Configuring Lustre Failover</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringfailover">
+ <title xml:id="configuringfailover.title">Configuring Lustre Failover</title>
<para>This chapter describes how to configure Lustre failover using the Heartbeat cluster infrastructure daemon. It includes:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<note>
- <para>
- Using Lustre Failover is optional
+ <para>Using Lustre Failover is optional
</para>
</note>
<section xml:id="dbdoclet.50438188_82389">
- <title>11.1 Creating a Failover Environment</title>
+ <title><indexterm><primary>High availability</primary><see>failover</see></indexterm><indexterm><primary>failover</primary></indexterm>Creating a Failover Environment</title>
<para>Lustre provides failover mechanisms only at the file system level. No failover functionality is provided for system-level components, such as node failure detection or power control, as would typically be provided in a complete failover solution. Additional tools are also needed to provide resource fencing, control and monitoring.</para>
<section remap="h3">
- <title>11.1.1 Power Management Software</title>
+ <title><indexterm><primary>failover</primary><secondary>power management</secondary></indexterm>Power Management Software</title>
<para>Lustre failover requires power control and management capability to verify that a failed node is shut down before I/O is directed to the failover node. This avoids double-mounting the two nodes, and the risk of unrecoverable data corruption. A variety of power management tools will work, but two packages that are commonly used with Lustre are STONITH and PowerMan.</para>
<para>Shoot The Other Node In The HEAD (STONITH), is a set of power management tools provided with the Linux-HA package. STONITH has native support for many power control devices and is extensible. It uses expect scripts to automate control.</para>
<para>PowerMan, available from the Lawrence Livermore National Laboratory (LLNL), is used to control remote power control (RPC) devices from a central location. PowerMan provides native support for several RPC varieties and expect-like configuration simplifies the addition of new devices.</para>
<para><link xl:href="https://computing.llnl.gov/linux/powerman.html">https://computing.llnl.gov/linux/powerman.html</link></para>
</section>
<section remap="h3">
- <title>11.1.2 Power Equipment</title>
+ <title><indexterm><primary>failover</primary><secondary>power equipment</secondary></indexterm>Power Equipment</title>
<para>Lustre failover also requires the use of RPC devices, which come in different configurations. Lustre server nodes may be equipped with some kind of service processor that allows remote power control. If a Lustre server node is not equipped with a service processor, then a multi-port, Ethernet-addressable RPC may be used as an alternative. For recommended products, refer to the list of supported RPC devices on the PowerMan website.</para>
<para><link xl:href="https://computing.llnl.gov/linux/powerman.html">https://computing.llnl.gov/linux/powerman.html</link></para>
</section>
</section>
<section xml:id="dbdoclet.50438188_92688">
- <title>11.2 Setting up High-Availability (HA) Software with Lustre</title>
+ <title><indexterm><primary>failover</primary><secondary>setup</secondary></indexterm>Setting up High-Availability (HA) Software with Lustre</title>
<para>Lustre must be combined with high-availability (HA) software to enable a complete Lustre failover solution. Lustre can be used with several HA packages including:</para>
<itemizedlist>
<listitem>
<para><emphasis>Red Hat Cluster Manager</emphasis> - For more information about setting up Lustre failover with Red Hat Cluster Manager, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Red_Hat_Cluster_Manager_with_Lustre">Using Red Hat Cluster Manager with Lustre</link>.</para>
</listitem>
<listitem>
- <para><emphasis>Pacemaker</emphasis> - For more information about setting up Lustre failover with Pacemaker, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre">Using Pacemaker with Lustre</link>.<anchor xml:id="dbdoclet.50438188_61775" xreflabel=""/></para>
+ <para><emphasis>Pacemaker</emphasis> - For more information about setting up Lustre failover with Pacemaker, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre">Using Pacemaker with Lustre</link>.</para>
</listitem>
</itemizedlist>
</section>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlnet">
- <info>
- <title xml:id="configuringlnet.title">Configuring Lustre Networking (LNET)</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlnet">
+ <title xml:id="configuringlnet.title">Configuring Lustre Networking (LNET)</title>
<para>This chapter describes how to configure Lustre Networking (LNET). It includes the following sections:</para>
<itemizedlist>
<listitem>
</itemizedlist>
<note>
<para>Configuring LNET is optional.</para>
- <para> LNET will, by default, use the first TCP/IP interface it discovers on a system (<literal>eth0</literal>). If this network configuration is sufficient, you do not need to configure LNET. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
+ <para>LNET will, by default, use the first TCP/IP interface it discovers on a system (<literal>eth0</literal>). If this network configuration is sufficient, you do not need to configure LNET. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
</note>
<section xml:id="dbdoclet.50438216_33148">
- <title>9.1 Overview of LNET Module Parameters</title>
+ <title><indexterm><primary>LNET</primary></indexterm>
+
+ Overview of LNET Module Parameters</title>
<para>LNET kernel module (lnet) parameters specify how LNET is to be configured to work with Lustre, including which NICs will be configured to work with Lustre and the routing to be used with Lustre.</para>
<para>Parameters for lnet are specified in the <literal>modprobe.conf</literal> or <literal>modules.conf</literal> file (depending on your Linux distribution) in one or more entries with the syntax:</para>
- <screen>options lnet <<emphasis>parameter</emphasis>>=<<emphasis>parameter value</emphasis>>
-</screen>
+ <screen>options lnet <<emphasis>parameter</emphasis>>=<<emphasis>parameter value</emphasis>></screen>
<para>To specify the network interfaces that are to be used for Lustre, set either the networks parameter or the <literal>ip2nets</literal> parameter (only one of these parameters can be used at a time):</para>
<itemizedlist>
<listitem>
<para>We recommend that you use 'dotted-quad' notation for IP addresses rather than host names to make it easier to read debug logs and debug configurations with multiple interfaces.</para>
</note>
<section remap="h3">
- <title>9.1.1 Using a Lustre Network Identifier (NID) to Identify a Node</title>
+ <title><indexterm><primary>LNET</primary><secondary>using NID</secondary></indexterm>Using a Lustre Network Identifier (NID) to Identify a Node</title>
<para>A Lustre network identifier (NID) is used to uniquely identify a Lustre network endpoint by node ID and network type. The format of the NID is:</para>
<screen><<emphasis>network</emphasis><emphasis>id</emphasis>>@<<emphasis>network</emphasis><emphasis>type</emphasis>></screen>
<para>Examples are:</para>
<para>The first entry above identifes a TCP/IP node, while the second entry identifies an InfiniBand node.</para>
<para>When a mount command is run on a client, the client uses the NID of the MDS to retrieve configuration information. If an MDS has more than one NID, the client should use the appropriate NID for its local network.</para>
<para>To determine the appropriate NID to specify in the mount command, use the <literal>lctl</literal> command. To display MDS NIDs, run on the MDS :</para>
- <screen>lctl list_nids
-</screen>
+ <screen>lctl list_nids</screen>
<para>To determine if a client can reach the MDS using a particular NID, run on the client:</para>
- <screen>lctl which_nid <<emphasis>MDS NID</emphasis>><anchor xml:id="dbdoclet.50438216_14112" xreflabel=""/></screen>
+ <screen>lctl which_nid <<emphasis>MDS NID</emphasis>></screen>
</section>
</section>
<section xml:id="dbdoclet.50438216_46279">
- <title>9.2 Setting the LNET Module networks Parameter</title>
+ <title><indexterm><primary>LNET</primary><secondary>module parameters</secondary></indexterm>Setting the LNET Module networks Parameter</title>
<para>If a node has more than one network interface, you'll typically want to dedicate a specific interface to Lustre. You can do this by including an entry in the <literal>modprobe.conf</literal> file on the node that sets the LNET module <literal>networks</literal> parameter:</para>
- <screen>options lnet networks=<<emphasis>comma-separated list of networks</emphasis>>
-</screen>
+ <screen>options lnet networks=<<emphasis>comma-separated list of networks</emphasis>></screen>
<para>This example specifies that a Lustre node will use a TCP/IP interface and an InfiniBand interface:</para>
- <screen>options lnet networks=tcp0(eth0),o2ib(ib0)
-</screen>
+ <screen>options lnet networks=tcp0(eth0),o2ib(ib0)</screen>
<para>This example specifies that the Lustre node will use the TCP/IP interface <literal>eth1</literal>:</para>
- <screen>options lnet networks=tcp0(eth1)
-</screen>
+ <screen>options lnet networks=tcp0(eth1)</screen>
<para>Depending on the network design, it may be necessary to specify explicit interfaces. To explicitly specify that interface <literal>eth2</literal> be used for network <literal>tcp0</literal> and <literal>eth3</literal> be used for <literal>tcp1</literal> , use this entry:</para>
- <screen>options lnet networks=tcp0(eth2),tcp1(eth3)
-</screen>
+ <screen>options lnet networks=tcp0(eth2),tcp1(eth3)</screen>
<para>When more than one interface is available during the network setup, Lustre chooses the best route based on the hop count. Once the network connection is established, Lustre expects the network to stay connected. In a Lustre network, connections do not fail over to another interface, even if multiple interfaces are available on the same node.</para>
<note>
<para>LNET lines in <literal>modprobe.conf</literal> are only used by the local node to determine what to call its interfaces. They are not used for routing decisions.</para>
</note>
<section remap="h3">
- <title>9.2.1 <anchor xml:id="dbdoclet.50438216_74334" xreflabel=""/>Multihome Server Example</title>
+ <title><indexterm><primary>configuing</primary><secondary>multihome</secondary></indexterm>Multihome Server Example</title>
<para>If a server with multiple IP addresses (multihome server) is connected to a Lustre network, certain configuration setting are required. An example illustrating these setting consists of a network with the following nodes:</para>
<itemizedlist>
<listitem>
<para> On each server, <literal>svr1</literal> and <literal>svr2</literal>, include the following line in the modprobe.conf file:</para>
</listitem>
</itemizedlist>
- <screen>options lnet networks=tcp0(eth0),tcp1(eth1),o2ib
-</screen>
+ <screen>options lnet networks=tcp0(eth0),tcp1(eth1),o2ib</screen>
<itemizedlist>
<listitem>
<para> For TCP-only clients, the first available non-loopback IP interface is used for <literal>tcp0</literal>. Thus, TCP clients with only one interface do not need to have options defined in the <literal>modprobe.conf</literal> file.</para>
<para> On the InfiniBand clients, include the following line in the <literal>modprobe.conf</literal> file:</para>
</listitem>
</itemizedlist>
- <screen>options lnet networks=o2ib
-</screen>
+ <screen>options lnet networks=o2ib</screen>
<note>
<para>By default, Lustre ignores the loopback (<literal>lo0</literal>) interface. Lustre does not ignore IP addresses aliased to the loopback. If you alias IP addresses to the loopback interface, you must specify all Lustre networks using the LNET networks parameter.</para>
</note>
</section>
</section>
<section xml:id="dbdoclet.50438216_31414">
- <title>9.3 Setting the LNET Module ip2nets Parameter</title>
+ <title><indexterm><primary>LNET</primary><secondary>ip2nets</secondary></indexterm>Setting the LNET Module ip2nets Parameter</title>
<para>The <literal>ip2nets</literal> option is typically used when a single, universal <literal>modprobe.conf</literal> file is run on all servers and clients. Each node identifies the locally available networks based on the listed IP address patterns that match the node's local IP addresses.</para>
<para>Note that the IP address patterns listed in the <literal>ip2nets</literal> option are <emphasis>only</emphasis> used to identify the networks that an individual node should instantiate. They are <emphasis>not</emphasis> used by LNET for any other communications purpose.</para>
<para>For the example below, the nodes in the network have these IP addresses:</para>
</itemizedlist>
<para>The following entry is placed in the <literal>modprobe.conf</literal> file on each server and client:</para>
<screen>options lnet 'ip2nets="tcp0(eth0) 192.168.0.[2,4]; \
-tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"'
-</screen>
+tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"'</screen>
<para>Each entry in <literal>ip2nets</literal> is referred to as a 'rule'.</para>
<para>The order of LNET entries is important when configuring servers. If a server node can be reached using more than one network, the first network specified in <literal>modprobe.conf</literal> will be used.</para>
<para>Because <literal>svr1</literal> and <literal>svr2</literal> match the first rule, LNET uses <literal>eth0</literal> for <literal>tcp0</literal> on those machines. (Although <literal>svr1</literal> and <literal>svr2</literal> also match the second rule, the first matching rule for a particular network is used).</para>
<para>The <literal>[2-8/2]</literal> format indicates a range of 2-8 stepped by 2; that is 2,4,6,8. Thus, the clients at <literal>132.6.3.5</literal> will not find a matching o2ib network.</para>
</section>
<section xml:id="dbdoclet.50438216_71227">
- <title>9.4 Setting the LNET Module routes Parameter</title>
+ <title><indexterm><primary>LNET</primary><secondary>routes</secondary></indexterm>Setting the LNET Module routes Parameter</title>
<para>The LNET module routes parameter is used to identify routers in a Lustre configuration. These parameters are set in <literal>modprob.conf</literal> on each Lustre node.</para>
<para>The LNET routes parameter specifies a colon-separated list of router definitions. Each route is defined as a network number, followed by a list of routers:</para>
- <screen>routes=<<emphasis>net type</emphasis>> <<emphasis>router NID(s)</emphasis>>
-</screen>
+ <screen>routes=<<emphasis>net type</emphasis>> <<emphasis>router NID(s)</emphasis>></screen>
<para>This example specifies bi-directional routing in which TCP clients can reach Lustre resources on the IB networks and IB servers can access the TCP networks:</para>
<screen>options lnet 'ip2nets="tcp0 192.168.0.*; \
o2ib0(ib0) 132.6.1.[1-128]"' 'routes="tcp0 132.6.1.[1-8]@o2ib0; \
- o2ib0 192.16.8.0.[1-8]@tcp0"'
-</screen>
+ o2ib0 192.16.8.0.[1-8]@tcp0"'</screen>
<para>All LNET routers that bridge two networks are equivalent. They are not configured as primary or secondary, and the load is balanced across all available routers.</para>
<para>The number of LNET routers is not limited. Enough routers should be used to handle the required file serving bandwidth plus a 25 percent margin for headroom.</para>
<section remap="h3">
- <title>9.4.1 <anchor xml:id="dbdoclet.50438216_34481" xreflabel=""/><anchor xml:id="dbdoclet.50438216_56435" xreflabel=""/>Routing Example</title>
+ <title><indexterm><primary>LNET</primary><secondary>routing example</secondary></indexterm>Routing Example</title>
<para>On the clients, place the following entry in the modprobe.conf file</para>
- <screen>lnet networks="tcp" routes="o2ib0 192.168.0.[1-8]@tcp0"
-</screen>
+ <screen>lnet networks="tcp" routes="o2ib0 192.168.0.[1-8]@tcp0"</screen>
<para>On the router nodes, use:</para>
- <screen>lnet networks="tcp o2ib" forwarding=enabled
-</screen>
+ <screen>lnet networks="tcp o2ib" forwarding=enabled </screen>
<para>On the MDS, use the reverse as shown below:</para>
- <screen>lnet networks="o2ib0" rountes="tcp0 132.6.1.[1-8]@o2ib0"
-</screen>
+ <screen>lnet networks="o2ib0" rountes="tcp0 132.6.1.[1-8]@o2ib0" </screen>
<para>To start the routers, run:</para>
<screen>modprobe lnet
-lctl network configure
-</screen>
+lctl network configure</screen>
</section>
</section>
<section xml:id="dbdoclet.50438216_10523">
- <title>9.5 Testing the LNET Configuration</title>
+ <title><indexterm><primary>LNET</primary><secondary>testing</secondary></indexterm>Testing the LNET Configuration</title>
<para>After configuring Lustre Networking, it is highly recommended that you test your LNET configuration using the LNET Self-Test provided with the Lustre software. For more information about using LNET Self-Test, see <xref linkend="lnetselftest"/>.</para>
</section>
<section xml:id="dbdoclet.50438216_35668">
- <title>9.6 Configuring the Router Checker</title>
+ <title><indexterm><primary>LNET</primary><secondary>route checker</secondary></indexterm>Configuring the Router Checker</title>
<para>In a Lustre configuration in which different types of networks, such as a TCP/IP network and an Infiniband network, are connected by routers, a router checker can be run on the clients and servers in the routed configuration to monitor the status of the routers. In a multi-hop routing configuration, router checkers can be configured on routers to monitor the health of their next-hop routers.</para>
<para>A router checker is configured by setting lnet parameters in modprobe.conf by including an entry in this form:</para>
- <screen>options lnet <<emphasis>router checker parameter</emphasis>>=<<emphasis>parameter value</emphasis>>
-</screen>
+ <screen>options lnet <<emphasis>router checker parameter</emphasis>>=<<emphasis>parameter value</emphasis>></screen>
<para>The router checker parameters are:</para>
<itemizedlist>
<listitem>
<para><literal>live_router_check_interval</literal> - Specifies a time interval in seconds after which the router checker will ping the live routers. The default value is 0, meaning no checking is done. To set the value to 60, enter:</para>
</listitem>
</itemizedlist>
- <screen>options lnet live_router_check_interval=60
-</screen>
+ <screen>options lnet live_router_check_interval=60</screen>
<itemizedlist>
<listitem>
<para><literal>dead_router_check_interval</literal> - Specifies a time interval in seconds after which the router checker will check for dead routers. The default value is 0, meaning no checking is done. To set the value to 60, enter:</para>
</listitem>
</itemizedlist>
- <screen>options lnet dead_router_check_interval=60
-</screen>
+ <screen>options lnet dead_router_check_interval=60</screen>
<itemizedlist>
<listitem>
<para>auto_down - Enables/disables (1/0) the automatic marking of router state as up or down. The default value is 1. To disable router marking, enter:</para>
</listitem>
</itemizedlist>
- <screen>options lnet auto_down=0
-</screen>
+ <screen>options lnet auto_down=0</screen>
<itemizedlist>
<listitem>
<para><literal>router_ping_timeout</literal> - Specifies a timeout for the router checker when it checks live or dead routers. The router checker sends a ping message to each dead or live router once every dead_router_check_interval or live_router_check_interval respectively. The default value is 50. To set the value to 60, enter:</para>
</listitem>
</itemizedlist>
- <screen>options lnet router_ping_timeout=60
-</screen>
+ <screen>options lnet router_ping_timeout=60</screen>
<note>
<para>The router_ping_timeout is consistent with the default LND timeouts. You may have to increase it on very large clusters if the LND timeout is also increased. For larger clusters, we suggest increasing the check interval.</para>
</note>
<para>check_routers_before_use - Specifies that routers are to be checked before use. Set to off by default. If this parameter is set to on, the dead_router_check_interval parameter must be given a positive integer value.</para>
</listitem>
</itemizedlist>
- <screen>options lnet check_routers_before_use=on
-</screen>
+ <screen>options lnet check_routers_before_use=on</screen>
<para>The router checker obtains the following information from each router:</para>
<itemizedlist>
<listitem>
<para>If 100 packets have been sent successfully through a router, the sent-packets counter for that router will have a value of 100.</para>
</section>
<section xml:id="dbdoclet.50438216_15200">
- <title>9.7 Best Practices for LNET Options</title>
+ <title><indexterm><primary>LNET</primary><secondary>best practice</secondary></indexterm>Best Practices for LNET Options</title>
<para>For the <literal>networks</literal>, <literal>ip2nets</literal>, and <literal>routes</literal> options, follow these best practices to avoid configuration errors.</para>
<section remap="h5">
- <title>Escaping commas with quotes</title>
+ <title><indexterm><primary>LNET</primary><secondary>escaping commas with quotes</secondary></indexterm>Escaping commas with quotes</title>
<para>Depending on the Linux distribution, commas may need to be escaped using single or double quotes. In the extreme case, the <literal>options</literal> entry would look like this:</para>
<para><screen>options lnet'networks="tcp0,elan0"' 'routes="tcp [2,10]@elan0"'</screen></para>
<para>Added quotes may confuse some distributions. Messages such as the following may indicate an issue related to added quotes:</para>
<para>A <literal>'Refusing connection - no matching NID'</literal> message generally points to an error in the LNET module configuration.</para>
</section>
<section remap="h5">
- <title>Including comments</title>
+ <title><indexterm><primary>LNET</primary><secondary>comments</secondary></indexterm>Including comments</title>
<para><emphasis>Place the semicolon terminating a comment immediately after the comment.</emphasis> LNET silently ignores everything between the <literal>#</literal> character at the beginning of the comment and the next semicolon.</para>
<para>In this <emphasis>incorrect</emphasis> example, LNET silently ignores <literal>pt11 192.168.0.[92,96]</literal>, resulting in these nodes not being properly initialized. No error message is generated.</para>
- <screen>options lnet ip2nets=
-# comment with semicolon BEFORE comment \
-pt11 192.168.0.[92,96];
-</screen>
+ <screen>options lnet ip2nets="pt10 192.168.0.[89,93]; # comment with semicolon BEFORE comment \
+pt11 192.168.0.[92,96];</screen>
<para>This <emphasis role="italic">correct</emphasis> example shows the required syntax:
</para>
<para><screen>options lnet ip2nets="pt10 192.168.0.[89,93] \
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlustre">
- <info>
- <title xml:id="configuringlustre.title">Configuring Lustre</title>
- </info>
+ <title xml:id="configuringlustre.title">Configuring Lustre</title>
<para>This chapter shows how to configure a simple Lustre system comprised of a combined MGS/MDT, an OST and a client. It includes:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438267_50692">
- <title>10.1 Configuring a Simple Lustre File System</title>
+ <title>
+ <indexterm><primary>Lustre</primary><secondary>configuring</secondary></indexterm>
+ Configuring a Simple Lustre File System</title>
<para>A Lustre system can be set up in a variety of configurations by using the administrative utilities provided with Lustre. The procedure below shows how to to configure a simple Lustre file system consisting of a combined MGS/MDS, one OSS with two OSTs, and a client. For an overview of the entire Lustre installation procedure, see <xref linkend="installoverview"/>.</para>
<para>This configuration procedure assumes you have completed the following:</para>
<itemizedlist>
<para><emphasis>Set up network interface bonding on Ethernet interfaces.</emphasis> For information about setting up network interface bonding, see <xref linkend="settingupbonding"/>.</para>
</listitem>
<listitem>
- <para><emphasis>Set</emphasis>lnet<emphasis>module parameters to specify how Lustre Networking (LNET) is to be configured to work with Lustre and test the LNET configuration.</emphasis> LNET will, by default, use the first TCP/IP interface it discovers on a system. If this network configuration is sufficient, you do not need to configure LNET. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
+ <para><emphasis>Set</emphasis> lnet <emphasis>module parameters to specify how Lustre Networking (LNET) is to be configured to work with Lustre and test the LNET configuration.</emphasis> LNET will, by default, use the first TCP/IP interface it discovers on a system. If this network configuration is sufficient, you do not need to configure LNET. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
</listitem>
</itemizedlist>
<para>For information about configuring LNET, see <xref linkend="configuringlnet"/>. For information about testing LNET, see <xref linkend="lnetselftest"/>.</para>
</listitem>
</itemizedlist>
<note>
- <para>
-The <literal>sgpdd_survey</literal> script overwrites the device being tested so it must be run before the OSTs are configured.</para>
+ <para>The <literal>sgpdd_survey</literal> script overwrites the device being tested so it must be run before the OSTs are configured.</para>
</note>
<para>To configure a simple Lustre file system, complete these steps:</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Create</emphasis> a combined MGS/MDT file system on a block device. On the MDS node, run:</para>
- <screen>mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgs --mdt <<emphasis>block device name</emphasis>>
-</screen>
+ <para>Create a combined MGS/MDT file system on a block device. On the MDS node, run:</para>
+ <screen>mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgs --mdt <<emphasis>block device name</emphasis>></screen>
<para>The default file system name (<literal>fsname</literal>) is <literal>lustre</literal>.</para>
<note>
<para>If you plan to generate multiple file systems, the MGS should be created separately on its own dedicated block device, by running:</para>
- <para> mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgs <<emphasis>block device name</emphasis>></para>
+ <screen>mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgs <<emphasis>block device name</emphasis>></screen>
</note>
</listitem>
<listitem>
- <para><emphasis role="bold">Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</emphasis></para>
- <screen>mount -t lustre <<emphasis>block device name</emphasis>> <<emphasis>mount point</emphasis>>
-</screen>
+ <para>Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</para>
+ <screen>mount -t lustre <<emphasis>block device name</emphasis>> <<emphasis>mount point</emphasis>></screen>
<note>
<para>If you have created and MGS and an MDT on separate block devices, mount them both.</para>
</note>
</listitem>
<listitem xml:id="dbdoclet.50438267_pgfId-1290915">
- <para><emphasis role="bold">Create the OST. On the OSS node, run:</emphasis></para>
- <screen>mkfs.lustre --ost --fsname=<<emphasis>fsname</emphasis>> --mgsnode=<<emphasis>NID</emphasis>> <<emphasis>block device name</emphasis>>
-</screen>
+ <para>Create the OST. On the OSS node, run:</para>
+ <screen>mkfs.lustre --ost --fsname=<<emphasis>fsname</emphasis>> --mgsnode=<<emphasis>NID</emphasis>> <<emphasis>block device name</emphasis>></screen>
<para>When you create an OST, you are formatting a <literal>ldiskfs</literal> file system on a block storage device like you would with any local file system.</para>
<para>You can have as many OSTs per OSS as the hardware or drivers allow. For more information about storage and memory requirements for a Lustre file system, see <xref linkend="settinguplustresystem"/>.</para>
<para>You can only configure one OST per block device. You should create an OST that uses the raw block device and does not use partitioning.</para>
<para>If you are using block devices that are accessible from multiple OSS nodes, ensure that you mount the OSTs from only one OSS node at at time. It is strongly recommended that multiple-mount protection be enabled for such devices to prevent serious data corruption. For more information about multiple-mount protection, see <xref linkend="managingfailover"/>.</para>
<note>
- <para>
-Lustre currently supports block devices up to 16 TB on OEL 5/RHEL 5 (up to 8 TB on other distributions). If the device size is only slightly larger that 16 TB, it is recommended that you limit the file system size to 16 TB at format time. If the size is significantly larger than 16 TB, you should reconfigure the storage into devices smaller than 16 TB. We recommend that you not place partitions on top of RAID 5/6 block devices due to negative impacts on performance.</para>
+ <para>Lustre currently supports block devices up to 16 TB on OEL 5/RHEL 5 (up to 8 TB on other distributions). If the device size is only slightly larger that 16 TB, it is recommended that you limit the file system size to 16 TB at format time. If the size is significantly larger than 16 TB, you should reconfigure the storage into devices smaller than 16 TB. We recommend that you not place partitions on top of RAID 5/6 block devices due to negative impacts on performance.</para>
</note>
</listitem>
<listitem xml:id="dbdoclet.50438267_pgfId-1293955">
- <para><emphasis role="bold">Mount the OST. On the OSS node where the OST was created, run:</emphasis></para>
+ <para>Mount the OST. On the OSS node where the OST was created, run:</para>
<screen>mount -t lustre <emphasis><block device name> <mount point></emphasis></screen>
<note>
<para>
</note>
</listitem>
<listitem xml:id="dbdoclet.50438267_pgfId-1290934">
- <para><emphasis role="bold">Mount the Lustre file system on the client. On the client node, run:</emphasis></para>
+ <para>Mount the Lustre file system on the client. On the client node, run:</para>
<screen>mount -t lustre <<emphasis>MGS node</emphasis>>:/<<emphasis>fsname</emphasis>> <<emphasis>mount point</emphasis>>
</screen>
<note>
- <para>
- To create additional clients, repeat <xref linkend="dbdoclet.50438267_pgfId-1290934"/>Step 5.</para>
+ <para>To create additional clients, repeat <xref linkend="dbdoclet.50438267_pgfId-1290934"/>Step 5.</para>
</note>
</listitem>
<listitem>
<para><note>
- <para>
-If you have a problem mounting the file system, check the syslogs on the client and all the servers for errors and also check the network settings. A common issue with newly-installed systems is that hosts.deny or firewall rules may prevent connections on port 988.</para>
+ <para>If you have a problem mounting the file system, check the syslogs on the client and all the servers for errors and also check the network settings. A common issue with newly-installed systems is that hosts.deny or firewall rules may prevent connections on port 988.</para>
</note><emphasis role="bold">Verify that the file system started and is working correctly. Do this by running the <literal>lfs</literal><literal> df</literal>, <literal>dd</literal> and <literal>ls</literal> commands on the client node.</emphasis></para>
</listitem>
<listitem>
- <para><emphasis role="bold"><emphasis><emphasis role="italic">(Optional)</emphasis> Run benchmarking to</emphasis>ols to validate the performance of hardware and software layers in the cluster. Available tools include:</emphasis></para>
+ <para><emphasis role="italic">(Optional)</emphasis> Run benchmarking tools to validate the performance of hardware and software layers in the cluster. Available tools include:</para>
<itemizedlist>
<listitem>
<para><literal>obdfilter_survey</literal> - Characterizes the storage performance of a Lustre file system. For details, see <xref linkend="dbdoclet.50438212_26516"/>.</para>
</listitem>
</orderedlist>
<section remap="h3">
- <title>10.1.1 Simple Lustre <anchor xml:id="dbdoclet.50438267_marker-1290955" xreflabel=""/>Configuration Example</title>
+ <title>
+ <indexterm><primary>Lustre</primary><secondary>configuring</secondary><tertiary>simple example</tertiary></indexterm>
+ Simple Lustre Configuration Example</title>
<para>To see the steps in a simple Lustre configuration, follow this example in which a combined MGS/MDT and two OSTs are created. Three block devices are used, one for the combined MGS/MDS node and one for each OSS node. Common parameters used in the example are listed below, along with individual node parameters.</para>
<informaltable frame="all">
<tgroup cols="4">
<para> <literal>10.2.0.1@tcp0</literal></para>
</entry>
<entry>
- <para> Node for the combined MGS/MDS</para>
+ <para>Node for the combined MGS/MDS</para>
</entry>
</row>
<row>
<para><literal> temp</literal></para>
</entry>
<entry>
- <para> Name of the Lustre file system</para>
+ <para>Name of the Lustre file system</para>
</entry>
</row>
<row>
<para> <literal>TCP/IP</literal></para>
</entry>
<entry>
- <para> Network type used for Lustre file system temp</para>
+ <para>Network type used for Lustre file system temp</para>
</entry>
</row>
</tbody>
<para> <literal>mdt1</literal></para>
</entry>
<entry>
- <para> MDS in Lustre file system temp</para>
+ <para>MDS in Lustre file system temp</para>
</entry>
</row>
<row>
<para> <literal>/dev/sdb</literal></para>
</entry>
<entry>
- <para> Block device for the combined MGS/MDS node</para>
+ <para>Block device for the combined MGS/MDS node</para>
</entry>
</row>
<row>
<para> <literal>/mnt/mdt</literal></para>
</entry>
<entry>
- <para> Mount point for the <literal>mdt1</literal> block device (<literal>/dev/sdb</literal>) on the MGS/MDS node</para>
+ <para>Mount point for the <literal>mdt1</literal> block device (<literal>/dev/sdb</literal>) on the MGS/MDS node</para>
</entry>
</row>
<row>
</row>
<row>
<entry>
- <para>  </para>
+ <para> </para>
</entry>
<entry>
<para> <emphasis role="bold">OSS node</emphasis></para>
</row>
<row>
<entry>
- <para>  </para>
+ <para> </para>
</entry>
<entry>
<para> <emphasis role="bold">OST</emphasis></para>
</entry>
</row>
<row>
- <entry>
- <para>  </para>
- </entry>
+ <entry/>
<entry>
<para> <emphasis role="bold">block device</emphasis></para>
</entry>
</row>
<row>
<entry>
- <para>  </para>
+ <para> </para>
</entry>
<entry>
<para> <emphasis role="bold">mount point</emphasis></para>
</row>
<row>
<entry>
- <para>  </para>
+ <para> </para>
</entry>
<entry>
<para> <emphasis role="bold">client node</emphasis></para>
</row>
<row>
<entry>
- <para>  </para>
+ <para> </para>
</entry>
<entry>
<para> <emphasis role="bold">mount point</emphasis></para>
<para>For this example, complete the steps below:</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Create a combined MGS/MDT file system on the block device. On the MDS node, run:</emphasis></para>
- <screen>[root@mds /]# mkfs.lustre --fsname=temp --mgs --mdt /dev/sdb
-</screen>
+ <para>Create a combined MGS/MDT file system on the block device. On the MDS node, run:</para>
+ <screen>[root@mds /]# mkfs.lustre --fsname=temp --mgs --mdt /dev/sdb</screen>
<para>This command generates this output:</para>
<screen> Permanent disk data:
Target: temp-MDTffff
options -i 4096 -I 512 -q -O dir_index,uninit_groups -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-MDTffff -i 4096 -I 512 -q -O
dir_index,uninit_groups -F /dev/sdb
-Writing CONFIGS/mountdata
-</screen>
+Writing CONFIGS/mountdata </screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</emphasis></para>
- <screen>[root@mds /]# mount -t lustre /dev/sdb /mnt/mdt
-</screen>
+ <para>Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</para>
+ <screen>[root@mds /]# mount -t lustre /dev/sdb /mnt/mdt</screen>
<para>This command generates this output:</para>
<screen>Lustre: temp-MDT0000: new disk, initializing
Lustre: 3009:0:(lproc_mds.c:262:lprocfs_wr_group_upcall()) temp-MDT0000: gr\
oup upcall set to /usr/sbin/l_getgroups
Lustre: temp-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups
-Lustre: Server temp-MDT0000 on device /dev/sdb has started
-</screen>
+Lustre: Server temp-MDT0000 on device /dev/sdb has started </screen>
</listitem>
<listitem xml:id="dbdoclet.50438267_pgfId-1291170">
- <para><emphasis role="bold">Create and mount <literal>ost1</literal>.</emphasis></para>
+ <para>Create and mount <literal>ost1</literal>.</para>
<para>In this example, the OSTs (<literal>ost1</literal> and <literal>ost2</literal>) are being created on different OSSs (<literal>oss1</literal> and <literal>oss2</literal> respectively).</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Create <literal>ost1</literal>. On <literal>oss1</literal> node, run:</emphasis></para>
- <screen>[root@oss1 /]# mkfs.lustre --ost --fsname=temp --mgsnode=10.2.0.1@tcp0 /dev\
-/sdc
-</screen>
+ <para>Create <literal>ost1</literal>. On <literal>oss1</literal> node, run:</para>
+ <screen>[root@oss1 /]# mkfs.lustre --ost --fsname=temp --mgsnode=10.2.0.1@tcp0 /dev/sdc</screen>
<para>The command generates this output:</para>
<screen> Permanent disk data:
Target: temp-OSTffff
options -I 256 -q -O dir_index,uninit_groups -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-OSTffff -I 256 -q -O
dir_index,uninit_groups -F /dev/sdc
-Writing CONFIGS/mountdata
-</screen>
+Writing CONFIGS/mountdata </screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Mount ost1 on the OSS on which it was created. On <literal>oss1</literal> node, run:</emphasis></para>
- <screen>root@oss1 /] mount -t lustre /dev/sdc /mnt/ost1
-</screen>
+ <para>Mount ost1 on the OSS on which it was created. On <literal>oss1</literal> node, run:</para>
+ <screen>root@oss1 /] mount -t lustre /dev/sdc /mnt/ost1 </screen>
<para>The command generates this output:</para>
<screen>LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: temp-OST0000: new disk, initializing
-Lustre: Server temp-OST0000 on device /dev/sdb has started
-</screen>
+Lustre: Server temp-OST0000 on device /dev/sdb has started</screen>
<para>Shortly afterwards, this output appears:</para>
<screen>Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0
-Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphans
-</screen>
+Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphans </screen>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Create and mount <literal>ost2</literal>.</emphasis></para>
+ <para>Create and mount <literal>ost2</literal>.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Create ost2. On <literal>oss2</literal> node, run:</emphasis></para>
+ <para>Create ost2. On <literal>oss2</literal> node, run:</para>
<screen>[root@oss2 /]# mkfs.lustre --ost --fsname=temp --mgsnode=10.2.0.1@tcp0 /dev\
-/sdd
-</screen>
+/sdd</screen>
<para>The command generates this output:</para>
<screen> Permanent disk data:
Target: temp-OSTffff
options -I 256 -q -O dir_index,uninit_groups -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-OSTffff -I 256 -q -O
dir_index,uninit_groups -F /dev/sdc
-Writing CONFIGS/mountdata
-</screen>
+Writing CONFIGS/mountdata </screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Mount ost2 on the OSS on which it was created. On <literal>oss2</literal> node, run:</emphasis></para>
- <screen>root@oss2 /] mount -t lustre /dev/sdd /mnt/ost2
-</screen>
+ <para>Mount ost2 on the OSS on which it was created. On <literal>oss2</literal> node, run:</para>
+ <screen>root@oss2 /] mount -t lustre /dev/sdd /mnt/ost2 </screen>
<para>The command generates this output:</para>
<screen>LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: temp-OST0000: new disk, initializing
-Lustre: Server temp-OST0000 on device /dev/sdb has started
-</screen>
+Lustre: Server temp-OST0000 on device /dev/sdb has started</screen>
<para>Shortly afterwards, this output appears:</para>
<screen>Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0
-Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphans
-</screen>
+Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphans </screen>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Mount the Lustre file system on the client. On the client node, run:</emphasis></para>
- <screen>root@client1 /] mount -t lustre 10.2.0.1@tcp0:/temp /lustre
-</screen>
+ <para>Mount the Lustre file system on the client. On the client node, run:</para>
+ <screen>root@client1 /] mount -t lustre 10.2.0.1@tcp0:/temp /lustre </screen>
<para>This command generates this output:</para>
- <screen>Lustre: Client temp-client has started
-</screen>
+ <screen>Lustre: Client temp-client has started</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Verify that the file system started and is working by running the <literal>df</literal>, <literal>dd</literal> and <literal>ls</literal> commands on the client node.</emphasis></para>
+ <para>Verify that the file system started and is working by running the <literal>df</literal>, <literal>dd</literal> and <literal>ls</literal> commands on the client node.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Run the <literal>lfs df -h</literal> command:</emphasis></para>
- <screen>[root@client1 /] lfs df -h
-</screen>
+ <para>Run the <literal>lfs df -h</literal> command:</para>
+ <screen>[root@client1 /] lfs df -h </screen>
<para>The <literal>lfs df -h</literal> command lists space usage per OST and the MDT in human-readable format. This command generates output similar to this:</para>
<screen>UUID bytes Used Available \
Use% Mounted on
temp-OST0001_UUID 800.0G 400.0M 799.6G 0% \
/lustre[OST:1]
filesystem summary: 1.6T 800.0M 1.6T \
-0% /lustre
-
-</screen>
+0% /lustre</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Run the <literal>lfs df -ih</literal> command.</emphasis></para>
- <screen>[root@client1 /] lfs df -ih
-</screen>
+ <para>Run the <literal>lfs df -ih</literal> command.</para>
+ <screen>[root@client1 /] lfs df -ih</screen>
<para>The <literal>lfs df -ih</literal> command lists inode usage per OST and the MDT. This command generates output similar to this:</para>
<screen>UUID Inodes IUsed IFree IUse% Mounted\
on
temp-MDT0000_UUID 2.5M 32 2.5M 0% /lustre[MDT:0]
temp-OST0000_UUID 5.5M 54 5.5M 0% /lustre[OST:0]
temp-OST0001_UUID 5.5M 54 5.5M 0% /lustre[OST:1]
-filesystem summary: 2.5M 32 2.5M 0% /lustre
-
-</screen>
+filesystem summary: 2.5M 32 2.5M 0% /lustre</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Run the <literal>dd</literal> command:</emphasis></para>
+ <para>Run the <literal>dd</literal> command:</para>
<screen>[root@client1 /] cd /lustre
-[root@client1 /lustre] dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2
-</screen>
+[root@client1 /lustre] dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2</screen>
<para>The <literal>dd</literal> command verifies write functionality by creating a file containing all zeros (<literal>0</literal>s). In this command, an 8 MB file is created. This command generates output similar to this:</para>
<screen>2+0 records in
2+0 records out
-8388608 bytes (8.4 MB) copied, 0.159628 seconds, 52.6 MB/s
-</screen>
+8388608 bytes (8.4 MB) copied, 0.159628 seconds, 52.6 MB/s</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Run the <literal>ls</literal> command:</emphasis></para>
- <screen>[root@client1 /lustre] ls -lsah
-</screen>
+ <para>Run the <literal>ls</literal> command:</para>
+ <screen>[root@client1 /lustre] ls -lsah</screen>
<para>The <literal>ls -lsah</literal> command lists files and directories in the current working directory. This command generates output similar to this:</para>
<screen>total 8.0M
4.0K drwxr-xr-x 2 root root 4.0K Oct 16 15:27 .
</section>
</section>
<section xml:id="dbdoclet.50438267_76752">
- <title>10.2 Additional Configuration Options</title>
+ <title>
+ <indexterm><primary>Lustre</primary><secondary>configuring</secondary><tertiary>additional options</tertiary></indexterm>
+ Additional Configuration Options</title>
<para>This section describes how to scale the Lustre file system or make configuration changes using the Lustre configuration utilities.</para>
<section remap="h3">
- <title>10.2.1 Scaling the <anchor xml:id="dbdoclet.50438267_marker-1292440" xreflabel=""/>Lustre File System</title>
+ <title>
+ <indexterm><primary>Lustre</primary><secondary>configuring</secondary><tertiary>for scale</tertiary></indexterm>
+ Scaling the Lustre File System</title>
<para>A Lustre file system can be scaled by adding OSTs or clients. For instructions on creating additional OSTs repeat Step <xref linkend="dbdoclet.50438267_pgfId-1291170"/> and Step <xref linkend="dbdoclet.50438267_pgfId-1293955"/> above. For mounting additional clients, repeat Step <xref linkend="dbdoclet.50438267_pgfId-1290934"/> for each client.</para>
</section>
<section remap="h3">
- <title>10.2.2 <anchor xml:id="dbdoclet.50438267_50212" xreflabel=""/>Changing Striping Defaults</title>
+ <title>
+ <indexterm><primary>Lustre</primary><secondary>configuring</secondary><tertiary>striping</tertiary></indexterm>
+ Changing Striping Defaults</title>
<para>The default settings for the file layout stripe pattern are shown in <xref linkend="configuringlustre.tab.stripe"/>.</para>
<table frame="none" xml:id="configuringlustre.tab.stripe">
<title>Default stripe pattern</title>
<para>Use the <literal>lfs setstripe</literal> command described in <xref linkend="managingstripingfreespace"/> to change the file layout configuration.</para>
</section>
<section remap="h3">
- <title>10.2.3 Using the Lustre Configuration Utilities</title>
+ <title>
+ <indexterm><primary>Lustre</primary><secondary>configuring</secondary><tertiary>utilities</tertiary></indexterm>
+ Using the Lustre Configuration Utilities</title>
<para>If additional configuration is necessary, several configuration utilities are available:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<para>For examples using these utilities, see the topic <xref linkend="systemconfigurationutilities"/></para>
- <para>The lfs utility is usful for configuring and querying a variety of options related to files. For more information, see <xref linkend="userutilities"/>.</para>
+ <para>The <literal>lfs</literal> utility is usful for configuring and querying a variety of options related to files. For more information, see <xref linkend="userutilities"/>.</para>
<note>
- <para>
-Some sample scripts are included in the directory where Lustre is installed. If you have installed the Lustre source code, the scripts are located in the <literal>lustre/tests</literal> sub-directory. These scripts enable quick setup of some simple standard Lustre configurations.</para>
+ <para>Some sample scripts are included in the directory where Lustre is installed. If you have installed the Lustre source code, the scripts are located in the <literal>lustre/tests</literal> sub-directory. These scripts enable quick setup of some simple standard Lustre configurations.</para>
</note>
</section>
</section>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringquotas">
- <info>
- <title xml:id="configuringquotas.title">Configuring and Managing Quotas</title>
- </info>
+ <title xml:id="configuringquotas.title">Configuring and Managing Quotas</title>
<para>This chapter describes how to configure quotas and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438217_54945">
- <title>21.1 Working with Quotas</title>
+ <title>
+ <indexterm><primary>Quotas</primary><secondary>configuring</secondary></indexterm>
+ Working with Quotas</title>
<para>Quotas allow a system administrator to limit the amount of disk space a user or group can use in a directory. Quotas are set by root, and can be specified for individual users and/or groups. Before a file is written to a partition where quotas are set, the quota of the creator's group is checked. If a quota exists, then the file size counts towards the group's quota. If no quota exists, then the owner's user quota is checked before the file is written. Similarly, inode usage for specific functions can be controlled if a user over-uses the allocated space.</para>
<para>Lustre quota enforcement differs from standard Linux quota enforcement in several ways:</para>
<itemizedlist>
</caution>
</section>
<section xml:id="dbdoclet.50438217_31982">
- <title>21.2 Enabling Disk Quotas</title>
+ <title><indexterm><primary>Quotas</primary><secondary>enabling disk</secondary></indexterm>Enabling Disk Quotas</title>
<para>Use this procedure to enable (configure) disk quotas in Lustre.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">
-If you have re-complied your Linux kernel, be sure that <literal>CONFIG_QUOTA</literal> and <literal>CONFIG_QUOTACTL</literal> are enabled. Also, verify that <literal>CONFIG_QFMT_V1</literal> and/or <literal>CONFIG_QFMT_V2</literal> are enabled.</emphasis></para>
-
+ <para>If you have re-complied your Linux kernel, be sure that <literal>CONFIG_QUOTA</literal> and <literal>CONFIG_QUOTACTL</literal> are enabled. Also, verify that <literal>CONFIG_QFMT_V1</literal> and/or <literal>CONFIG_QFMT_V2</literal> are enabled.</para>
<para>Quota is enabled in all Linux 2.6 kernels supplied for Lustre.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Start the server.</emphasis></para>
+ <para>Start the server.</para>
</listitem>
<listitem>
-<para>
- <emphasis role="bold">
- Mount the Lustre file system on the client and verify that the <literal>lquota</literal> module has loaded properly by using the <literal>lsmod</literal> command.
- </emphasis>
- </para> <screen>$ lsmod
+ Mount the Lustre file system on the client and verify that the <literal>lquota</literal> module has loaded properly by using the <literal>lsmod</literal> command.
+
+ </para>
+ <screen>$ lsmod
[root@oss161 ~]# lsmod
Module Size Used by
obdfilter 220532 1
<para>If you are using <literal>mkfs.lustre --param mdt.quota_type=ug</literal> or <literal>tunefs.lustre --param ost.quota_type=ug</literal>, be sure to run the command on all OSTs and the MDT. Otherwise, abnormal results may occur.</para>
</caution>
<section remap="h4">
- <title>21.2.0.1 Administrative and Operational Quotas</title>
+ <title><indexterm><primary>Quotas</primary><secondary>administrating</secondary></indexterm>Administrative and Operational Quotas</title>
<para>Lustre has two kinds of quota files:</para>
<itemizedlist>
<listitem>
</section>
</section>
<section xml:id="dbdoclet.50438217_49939">
- <title>21.3 Creating Quota Files and Quota Administration</title>
+ <title><indexterm><primary>Quotas</primary><secondary>creating</secondary></indexterm>Creating Quota Files and Quota Administration</title>
<para>Once each quota-enabled file system is remounted, it is capable of working with disk quotas. However, the file system is not yet ready to support quotas. If <literal>umount</literal> has been done regularly, run the <literal>lfs</literal> command with the <literal>quotaon</literal> option. If <literal>umount</literal> has not been done, perform these steps:</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Take Lustre ''offline''.</emphasis></para>
+ <para>Take Lustre ''offline''.</para>
<para>That is, verify that no write operations (append, write, truncate, create or delete) are being performed (preparing to run <literal>lfs quotacheck</literal>). Operations that do not change Lustre files (such as read or mount) are okay to run.</para>
<caution>
<para>When <literal>lfs quotacheck</literal> is run, Lustre must NOT be performing any write operations. Failure to follow this caution may cause the statistic information of quota to be inaccurate. For example, the number of blocks used by OSTs for users or groups will be inaccurate, which can cause unexpected quota problems.</para>
</caution>
</listitem>
<listitem>
- <para> <emphasis role="bold">Run the <emphasis role="bold">
- <literal>lfs</literal>
- </emphasis> command with the <emphasis role="bold">
- <literal>quotacheck</literal>
- </emphasis> option:</emphasis></para>
+ <para> Run the <emphasis role="bold">
+ <literal>lfs</literal>
+ </emphasis> command with the <emphasis role="bold">
+ <literal>quotacheck</literal>
+ </emphasis> option:</para>
<screen># lfs quotacheck -ug /mnt/lustre</screen>
<para>By default, quota is turned on after <literal>quotacheck</literal> completes. Available options are:</para>
<itemizedlist>
- 0 - 0 -</screen>
</section>
<section xml:id="dbdoclet.50438217_15106">
- <title>21.4 Quota Allocation</title>
+ <title><indexterm><primary>Quotas</primary><secondary>allocating</secondary></indexterm>Quota Allocation</title>
<para>In Lustre, quota must be properly allocated or users may experience unnecessary failures. The file system block quota is divided up among the OSTs within the file system. Each OST requests an allocation which is increased up to the quota limit. The quota allocation is then <emphasis role="italic">quantized</emphasis> to reduce the number of quota-related request traffic. By default, Lustre supports both user and group quotas to limit disk usage and file counts.</para>
<para>The quota system in Lustre is completely compatible with the quota systems used on other file systems. The Lustre quota system distributes quotas from the quota master. Generally, the MDS is the quota master for both inodes and blocks. All OSTs and the MDS are quota slaves to the OSS nodes. To reduce quota requests and get reasonably accurate quota distribution, the transfer quota unit (qunit) between quota master and quota slaves is changed dynamically by the lquota module. The default minimum value of qunit is 1 MB for blocks and 2 for inodes. The proc entries to set these values are: <literal>/proc/fs/lustre/mds/lustre-MDT*/quota_least_bunit</literal> and <literal>/proc/fs/lustre/mds/lustre-MDT*/quota_least_iunit</literal>. The default maximum value of <literal>qunit</literal> is 128 MB for blocks and 5120 for inodes. The proc entries to set these values are <literal>quota_bunit_sz</literal> and <literal>quota_iunit_sz</literal> in the MDT and OSTs.</para>
<note>
<para>It is not critical to know a file system's total inode count. Instead, you should know (accurately), the free inode count and the used inode count for a file system. Lustre manipulates the total inode count in order to accurately report the other two values.</para>
<para>The values set for the MDS must match the values set on the OSTs.</para>
<para>The <literal>quota_bunit_sz</literal> parameter displays bytes, however <literal>lfs setquota</literal> uses KBs. The <literal>quota_bunit_sz</literal> parameter must be a multiple of 1024. A proper minimum KB size for <literal>lfs setquota</literal> can be calculated as:</para>
- <informalexample><para>
+ <informalexample>
+ <para>
<emphasis role="bold">Size in KBs = minimum_quota_bunit_sz * (number of OSTS + 1) = 1024 * (number of OSTs +1)</emphasis>
- </para></informalexample>
+ </para>
+ </informalexample>
<para>We add one (1) to the number of OSTs as the MDS also consumes KBs. As inodes are only consumed on the MDS, the minimum inode size for <literal>lfs setquota</literal> is equal to <literal>quota_iunit_sz</literal>.</para>
<note>
<para>Setting the quota below this limit may prevent the user from all file creation.</para>
</section>
</section>
<section xml:id="dbdoclet.50438217_27895">
- <title>21.5 Known Issues with Quotas</title>
+ <title><indexterm><primary>Quotas</primary><secondary>known issues</secondary></indexterm>Known Issues with Quotas</title>
<para>Using quotas in Lustre can be complex and there are several known issues.</para>
<section remap="h3">
- <title>21.5.1 Granted Cache and Quota Limits</title>
+ <title>Granted Cache and Quota Limits</title>
<para>In Lustre, granted cache does not respect quota limits. In this situation, OSTs grant cache to Lustre client to accelerate I/O. Granting cache causes writes to be successful in OSTs, even if they exceed the quota limits, and will overwrite them.</para>
<para>The sequence is:</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">A user writes files to Lustre.</emphasis></para>
+ <para>A user writes files to Lustre.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">If the Lustre client has enough granted cache, then it returns 'success' to users and arranges the writes to the OSTs.</emphasis></para>
+ <para>If the Lustre client has enough granted cache, then it returns 'success' to users and arranges the writes to the OSTs.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Because Lustre clients have delivered success to users, the OSTs cannot fail these writes.</emphasis></para>
+ <para>Because Lustre clients have delivered success to users, the OSTs cannot fail these writes.</para>
</listitem>
</orderedlist>
<para>Because of granted cache, writes always overwrite quota limitations. For example, if you set a 400 GB quota on user A and use IOR to write for user A from a bundle of clients, you will write much more data than 400 GB, and cause an out-of-quota error (<literal>-EDQUOT</literal>).</para>
<note>
<para>The effect of granted cache on quota limits can be mitigated, but not eradicated. Reduce the <literal>max_dirty_buffer</literal> in the clients (can be set from 0 to 512). To set <literal>max_dirty_buffer</literal> to 0:</para>
- <para> * In releases after Lustre 1.6.5, <literal>lctl set_param osc.*.max_dirty_mb=0</literal>.</para>
- <para> * In releases before Lustre 1.6.5, <literal>proc/fs/lustre/osc/*/max_dirty_mb; do echo 512 > $O</literal></para>
+ <itemizedlist>
+ <listitem>
+ <para>In releases after Lustre 1.6.5, <literal>lctl set_param osc.*.max_dirty_mb=0</literal>.</para>
+ </listitem>
+ <listitem>
+ <para>In releases before Lustre 1.6.5, <literal>proc/fs/lustre/osc/*/max_dirty_mb; do echo 512 > $O</literal></para>
+ </listitem>
+ </itemizedlist>
</note>
</section>
<section remap="h3">
- <title>21.5.2 Quota Limits</title>
+ <title><indexterm><primary>Quotas</primary><secondary>limits</secondary></indexterm>Quota Limits</title>
<para>Available quota limits depend on the Lustre version you are using.</para>
<itemizedlist>
<listitem>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438217_66360">
- <title>21.5.3 Quota File Formats</title>
+ <title><indexterm><primary>Quotas</primary><secondary>file formats</secondary></indexterm>Quota File Formats</title>
<para>Lustre 1.6.5 introduced the v2 file format for administrative quotas, with 64-bit limits that support large-limits handling. The old quota file format (v1), with 32-bit limits, is also supported. Lustre 1.6.6 introduced the v2 file format for operational quotas. A few notes regarding the current quota file formats:</para>
<para>Lustre 1.6.5 and later use <literal>mdt.quota_type</literal> to force a specific administrative quota version (v2 or v1).</para>
<itemizedlist>
</section>
</section>
<section xml:id="dbdoclet.50438217_20772">
- <title>21.6 Lustre Quota Statistics</title>
+ <title><indexterm><primary>Quotas</primary><secondary>statistics</secondary></indexterm>Lustre Quota Statistics</title>
<para>Lustre includes statistics that monitor quota activity, such as the kinds of quota RPCs sent during a specific period, the average time to complete the RPCs, etc. These statistics are useful to measure performance of a Lustre file system.</para>
<para>Each quota statistic consists of a quota event and <literal>min_time</literal>, <literal>max_time</literal> and <literal>sum_time</literal> values for the event.</para>
<informaltable frame="all">
</tgroup>
</informaltable>
<section remap="h3">
- <title>21.6.1 Interpreting Quota Statistics</title>
+ <title>Interpreting Quota Statistics</title>
<para>Quota statistics are an important measure of a Lustre file system's performance. Interpreting these statistics correctly can help you diagnose problems with quotas, and may indicate adjustments to improve system performance.</para>
<para>For example, if you run this command on the OSTs:</para>
<screen>cat /proc/fs/lustre/lquota/lustre-OST0000/stats</screen>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringstorage">
- <info>
- <title xml:id="configuringstorage.title">Configuring Storage on a Lustre File System</title>
- </info>
+ <title xml:id="configuringstorage.title">Configuring Storage on a Lustre File System</title>
<para>This chapter describes best practices for storage selection and file system options to optimize perforance on RAID, and includes the following sections:</para>
<itemizedlist>
<listitem>
<para><emphasis role="bold">It is strongly recommended that hardware RAID be used with Lustre.</emphasis> Lustre currently does not support any redundancy at the file system level and RAID is required to protect agains disk failure.</para>
</note>
<section xml:id="dbdoclet.50438208_60972">
- <title>6.1 Selecting Storage for the MDT and OSTs</title>
+ <title>
+ <indexterm><primary>storage</primary><secondary>configuring</secondary></indexterm>
+ Selecting Storage for the MDT and OSTs</title>
<para>The Lustre architecture allows the use of any kind of block device as backend storage. The characteristics of such devices, particularly in the case of failures, vary significantly and have an impact on configuration choices.</para>
<para>This section describes issues and recommendations regarding backend storage.</para>
<section remap="h3">
- <title>6.1.1 Metadata Target (MDT)</title>
+ <title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>MDT</tertiary></indexterm>Metadata Target (MDT)</title>
<para>I/O on the MDT is typically mostly reads and writes of small amounts of data. For this reason, we recommend that you use RAID 1 for MDT storage. If you require more capacity for an MDT than one disk provides, we recommend RAID 1 + 0 or RAID 10.</para>
</section>
<section remap="h3">
- <title>6.1.2 Object Storage Server (OST)</title>
+ <title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>OST</tertiary></indexterm>Object Storage Server (OST)</title>
<para>A quick calculation makes it clear that without further redundancy, RAID 6 is required for large clusters and RAID 5 is not acceptable:</para>
<blockquote>
<para>For a 2 PB file system (2,000 disks of 1 TB capacity) assume the mean time to failure (MTTF) of a disk is about 1,000 days. This means that the expected failure rate is 2000/1000 = 2 disks per day. Repair time at 10% of disk bandwidth is 1000 GB at 10MB/sec = 100,000 sec, or about 1 day.</para>
</section>
</section>
<section xml:id="dbdoclet.50438208_23285">
- <title>6.2 Reliability Best Practices</title>
+ <title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>for best practice</tertiary></indexterm>Reliability Best Practices</title>
<para>RAID monitoring software is recommended to quickly detect faulty disks and allow them to be replaced to avoid double failures and data loss. Hot spare disks are recommended so that rebuilds happen without delays.</para>
<para>Backups of the metadata file systems are recommended. For details, see <xref linkend="backupandrestore"/>.</para>
</section>
<section xml:id="dbdoclet.50438208_40705">
- <title>6.3 Performance Tradeoffs</title>
+ <title><indexterm><primary>storage</primary><secondary>performance tradeoffs</secondary></indexterm>Performance Tradeoffs</title>
<para>A writeback cache can dramatically increase write performance on many types of RAID arrays if the writes are not done at full stripe width. Unfortunately, unless the RAID array has battery-backed cache (a feature only found in some higher-priced hardware RAID arrays), interrupting the power to the array may result in out-of-sequence writes or corruption of RAID parity and future data loss.</para>
<para>If writeback cache is enabled, a file system check is required after the array loses power. Data may also be lost because of this.</para>
<para>Therefore, we recommend against the use of writeback cache when data integrity is critical. You should carefully consider whether the benefits of using writeback cache outweigh the risks.</para>
</section>
<section xml:id="dbdoclet.50438208_51921">
- <title>6.4 Formatting Options for RAID Devices</title>
+ <title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>RAID options</tertiary></indexterm>Formatting Options for RAID Devices</title>
<para>When formatting a file system on a RAID device, it is beneficial to ensure that I/O requests are aligned with the underlying RAID geometry. This ensures that the Lustre RPCs do not generate unnecessary disk operations which may reduce performance dramatically. Use the <literal>--mkfsoptions</literal> parameter to specify additional parameters when formatting the OST or MDT.</para>
<para>For RAID 5, RAID 6, or RAID 1+0 storage, specifying the following option to the <literal>--mkfsoptions</literal> parameter option improves the layout of the file system metadata, ensuring that no single disk contains all of the allocation bitmaps:</para>
- <screen>-E stride = <chunk_blocks>
-</screen>
+ <screen>-E stride = <chunk_blocks> </screen>
<para>The <literal><chunk_blocks></literal> variable is in units of 4096-byte blocks and represents the amount of contiguous data written to a single disk before moving to the next disk. This is alternately referred to as the RAID stripe size. This is applicable to both MDT and OST file systems.</para>
<para>For more information on how to override the defaults while formatting MDT or OST file systems, see <xref linkend="dbdoclet.50438256_84701"/>.</para>
<section remap="h3">
- <title>6.4.1 Computing file system parameters for mkfs</title>
+ <title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>for mkfs</tertiary></indexterm>Computing file system parameters for mkfs</title>
<para>For best results, use RAID 5 with 5 or 9 disks or RAID 6 with 6 or 10 disks, each on a different controller. The stripe width is the optimal minimum I/O size. Ideally, the RAID configuration should allow 1 MB Lustre RPCs to fit evenly on a single RAID stripe without an expensive read-modify-write cycle. Use this formula to determine the <emphasis>
<literal><stripe_width></literal>
</emphasis>, where <emphasis>
<literal><stripe_width_blocks></literal>
</emphasis> parameter is only relevant for RAID 5 or RAID 6, and is not needed for RAID 1 plus 0.</para>
<para>Run <literal>--reformat</literal> on the file system device (<literal>/dev/sdc</literal>), specifying the RAID geometry to the underlying ldiskfs file system, where:</para>
- <screen>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, stripe_width=<emphasis><stripe_width_blocks></emphasis>"
-</screen>
-
<informalexample><para>A RAID 6 configuration with 6 disks has 4 data and 2 parity disks. The <emphasis>
+ <screen>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, stripe_width=<emphasis><stripe_width_blocks></emphasis>"</screen>
+ <informalexample>
+
<para>A RAID 6 configuration with 6 disks has 4 data and 2 parity disks. The <emphasis>
<literal><chunk_blocks></literal>
-</emphasis> <= 1024KB/4 = 256KB.</para></informalexample>
+ </emphasis> <= 1024KB/4 = 256KB.</para>
+ </informalexample>
<para>Because the number of data disks is equal to the power of 2, the stripe width is equal to 1 MB.</para>
- <screen>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, stripe_width=<emphasis><stripe_width_blocks></emphasis>"...
-</screen>
+ <screen>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, stripe_width=<emphasis><stripe_width_blocks></emphasis>"...</screen>
</section>
<section remap="h3">
- <title>6.4.2 Choosing Parameters for an External <anchor xml:id="dbdoclet.50438208_marker-1289927" xreflabel=""/>Journal</title>
+ <title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>external journal</tertiary></indexterm>Choosing Parameters for an External Journal</title>
<para>If you have configured a RAID array and use it directly as an OST, it contains both data and metadata. For better performance, we recommend putting the OST journal on a separate device, by creating a small RAID 1 array and using it as an external journal for the OST.</para>
<para>Lustre's default journal size is 400 MB. A journal size of up to 1 GB has shown increased performance but diminishing returns are seen for larger journals. Additionally, a copy of the journal is kept in RAM. Therefore, make sure you have enough memory available to hold copies of all the journals.</para>
<para>The file system journal options are specified to mkfs.luster using the <literal>--mkfsoptions</literal> parameter. For example:</para>
- <screen>--mkfsoptions "<other options> -j -J device=/dev/mdJ"
-</screen>
+ <screen>--mkfsoptions "<other options> -j -J device=/dev/mdJ" </screen>
<para>To create an external journal, perform these steps for each OST on the OSS:</para>
<orderedlist>
<listitem>
</section>
</section>
<section xml:id="dbdoclet.50438208_88516">
- <title>6.5 Connecting a SAN to a Lustre File System</title>
+ <title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>SAN</tertiary></indexterm>Connecting a SAN to a Lustre File System</title>
<para>Depending on your cluster size and workload, you may want to connect a SAN to a Lustre file system. Before making this connection, consider the following:</para>
<itemizedlist>
<listitem>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="installoverview">
- <info>
- <title xml:id="installoverview.title">Installation Overview</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="installoverview">
+ <title xml:id="installoverview.title">Installation Overview</title>
<para>This chapter provides on overview of the procedures required to set up, install and configure a Lustre file system.</para>
<note>
<para>
If you are new to Lustre, you may find it helpful to refer to <xref linkend="part.intro"/> for a description of the Lustre architecture, file system components and terminology before proceeding with the installation procedure.</para>
</note>
<section remap="h2">
- <title>4.1 Steps to Installing Lustre</title>
+ <title>
+ <indexterm><primary>installing</primary></indexterm>
+ Steps to Installing Lustre</title>
<para>To set up Lustre file system hardware and install and configure the Lustre software, refer the the chapters below in the order listed:</para>
<orderedlist>
<listitem>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="installinglustre">
- <info>
- <title xml:id="installinglustre.title">Installing the Lustre Software</title>
- </info>
+ <title xml:id="installinglustre.title">Installing the Lustre Software</title>
<para>This chapter describes how to install the Lustre software. It includes:</para>
<itemizedlist>
<listitem>
</itemizedlist>
<para>For hardware/system requirements, see <xref linkend="settinguplustresystem"/>.</para>
<section xml:id="dbdoclet.50438261_99193">
- <title>8.1 Preparing to Install the Lustre Software</title>
+ <title>
+ <indexterm><primary>installing</primary><secondary>preparation</secondary></indexterm>
+ Preparing to Install the Lustre Software</title>
<para>If you are using a supported Linux distribution and architecture, you can install Lustre from downloaded packages (RPMs). For a list of supported configurations, see the <link xl:href="http://wiki.whamcloud.com/">Lustre 2.0</link> topic on the Lustre wiki.</para>
<para>A list of supported Lustre LNET drivers can be found in <xref linkend="understandinglustrenetworking"/>.</para>
<para>If you are not using a supported configuration, you can install Lustre directly from the source code. For more information on this installation method, see <xref linkend="installinglustrefromsourcecode"/>.</para>
- <section remap="h3">
- <title>8.1.1 <anchor xml:id="dbdoclet.50438261_37079" xreflabel=""/>Required Software</title>
+ <section xml:id="dbdoclet.50438261_37079">
+ <title><indexterm><primary>installing</primary><secondary>requirements</secondary></indexterm>
+Required Software</title>
<para>To install Lustre, the following are required:</para>
<itemizedlist>
<listitem>
<para><emphasis>(On Linux servers only)</emphasis> Linux kernel patched with Lustre-specific patches for your platform and architecture. A Linux patched kernel can be installed on a client if it is desireable to use the same kernel on all nodes, but this is not required.</para>
</listitem>
<listitem>
- <para> Lustre modules compiled for the Linux kernel (see <xref linkend="installinglustre.tab.req"/> for which packages are required for servers and clients in your configuration).</para>
+ <para>Lustre modules compiled for the Linux kernel (see <xref linkend="installinglustre.tab.req"/> for which packages are required for servers and clients in your configuration).</para>
</listitem>
<listitem>
- <para> Lustre utilities required for configuring Lustre (see <xref linkend="installinglustre.tab.req"/> for which packages are required for servers and clients in your configuration).</para>
+ <para>Lustre utilities required for configuring Lustre (see <xref linkend="installinglustre.tab.req"/> for which packages are required for servers and clients in your configuration).</para>
</listitem>
<listitem>
<para><emphasis>(On Linux servers only) </emphasis><literal>e2fsprogs</literal> package containing Lustre-specific tools (<literal>e2fsck</literal> and <literal>lfsck</literal>) used to repair a backing file system. This replaces the existing <literal>e2fsprogs</literal> package but provides complete <literal>e2fsprogs</literal> functionality</para>
<thead>
<row>
<entry nameend="c2" namest="c1">
- <para><emphasis role="bold"><anchor xml:id="dbdoclet.50438261_97803" xreflabel=""/>Lustre Package</emphasis></para>
+ <para><emphasis role="bold">Lustre Package</emphasis></para>
</entry>
<entry>
<para><emphasis role="bold">Description</emphasis></para>
<para>In all supported Lustre installations, a patched kernel must be run on each server, including the the MGS, the MDS, and all OSSs. Running a patched kernel on a Lustre client is only required if the client will be used for multiple purposes, such as running as both a client and an OST or if you want to use the same kernel on all nodes.</para>
<para>Lustre RPM packages are available on the <link xl:href="http://build.whamcloud.com">Whamcloud download site</link>. They must be installed in the order described in the <xref linkend="dbdoclet.50438261_41900"/>.</para>
<section remap="h4">
- <title>8.1.1.1 Network-specific Kernel Modules and Libraries</title>
+ <title><indexterm><primary>installing</primary><secondary>network</secondary></indexterm>
+Network-specific Kernel Modules and Libraries</title>
<para>Network-specific kernel modules and libraries may be needed such as the Lustre-specific OFED package required for an InfiniBand network interconnect.</para>
</section>
<section remap="h4">
- <title>8.1.1.2 Lustre-Specific Tools <anchor xml:id="dbdoclet.50438261_marker-1292890" xreflabel=""/>and Utilities</title>
+ <title><indexterm><primary>installing</primary><secondary>tools and utilities</secondary></indexterm>
+Lustre-Specific Tools and Utilities</title>
<para>Several third-party utilities are must be installed on servers:</para>
<itemizedlist>
<listitem>
- <para><emphasis role="bold">e2fsprogs</emphasis><anchor xml:id="dbdoclet.50438261_marker-1292893" xreflabel=""/> : Lustre requires a recent version of e2fsprogs that understands extents. Use <literal>e2fsprogs-1.41-90-wc1</literal> or later, available at:</para>
+ <para><emphasis role="bold">e2fsprogs</emphasis> : Lustre requires a recent version of e2fsprogs that understands extents. Use <literal>e2fsprogs-1.41-90-wc1</literal> or later, available at:</para>
<para><link xl:href="http://build.whamcloud.com/">http://build.whamcloud.com/</link></para>
<para>A quilt patchset of all changes to the vanilla e2fsprogs is available in <literal>e2fsprogs-{version}-patches.tgz</literal>.</para>
<note>
</note>
</listitem>
<listitem>
- <para><emphasis role="bold">Perl</emphasis><anchor xml:id="dbdoclet.50438261_marker-1292899" xreflabel=""/> - Various userspace utilities are written in Perl. Any recent version of Perl will work with Lustre.</para>
+ <para><emphasis role="bold">Perl</emphasis> - Various userspace utilities are written in Perl. Any recent version of Perl will work with Lustre.</para>
</listitem>
</itemizedlist>
</section>
<section remap="h4">
- <title>8.1.1.3 <anchor xml:id="dbdoclet.50438261_29838" xreflabel=""/>(Optional) <anchor xml:id="dbdoclet.50438261_marker-1292902" xreflabel=""/>High-Availability Software</title>
+ <title><indexterm><primary>installing</primary><secondary>high-availability</secondary></indexterm>
+(Optional) High-Availability Software</title>
<para>If you plan to enable failover server functionality with Lustre (either on an OSS or the MDS), you must add high-availability (HA) software to your cluster software. For more information, see <xref linkend="dbdoclet.50438188_92688"/>.</para>
</section>
<section remap="h4">
- <title>8.1.1.4 (Optional) Debugging Tools and Other Optional Packages</title>
+ <title>(Optional) Debugging Tools and Other Optional Packages</title>
<para>A variety of optional packages are provided on the <emphasis>
<link xl:href="http://downloads.whamcloud.com">Whamcloud download site</link>
</emphasis>. These include debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages.</para>
</emphasis> on the Lustre wiki.</para>
</section>
</section>
- <section remap="h3">
- <title>8.1.2 <anchor xml:id="dbdoclet.50438261_19503" xreflabel=""/>Environmental <anchor xml:id="dbdoclet.50438261_marker-1292914" xreflabel=""/>Requirements</title>
+ <section xml:id="dbdoclet.50438261_19503">
+ <title><indexterm><primary>installing</primary><secondary>environment</secondary></indexterm>
+Environmental Requirements</title>
<para>Make sure the following environmental requirements are met before installing Lustre:</para>
<itemizedlist>
<listitem>
</section>
</section>
<section xml:id="dbdoclet.50438261_41900">
- <title>8.2 Lustre Installation Procedure</title>
+ <title>Lustre Installation Procedure</title>
<caution>
<para>Before installing Lustre, back up ALL data. Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured or administered properly.</para>
</caution>
<para>Use this procedure to install Lustre from RPMs.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Verify that all Lustre installation requirements have been met.</emphasis></para>
+ <para>Verify that all Lustre installation requirements have been met.</para>
<para>For more information on these prerequisites, see:</para>
<itemizedlist>
<listitem>
<para> Hardware requirements in <xref linkend="settinguplustresystem"/></para>
</listitem>
<listitem>
- <para> Software and environmental requirements in <xref linkend="installinglustre"/></para>
+ <para>Software and environmental requirements in <xref linkend="installinglustre"/></para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Download the Lustre RPMs.</emphasis></para>
+ <para>Download the Lustre RPMs.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">On the <link xl:href="http://build.whamcloud.com/">Lustre download site</link>, select your platform.</emphasis></para>
+ <para>On the <link xl:href="http://build.whamcloud.com/">Lustre download site</link>, select your platform.</para>
<para>The files required to install Lustre (kernels, modules and utilities RPMs) are listed for the selected platform.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Download the required files.</emphasis></para>
+ <para>Download the required files.</para>
</listitem>
</orderedlist>
</listitem>
<listitem xml:id="dbdoclet.50438261_88296">
- <para><emphasis role="bold">Install the Lustre packages on the servers.</emphasis></para>
+ <para>Install the Lustre packages on the servers.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Refer to <xref linkend="installinglustre.tab.req"/> to determine which Lustre packages are to be installed on servers for your platform and architecture.</emphasis></para>
+ <para>Refer to <xref linkend="installinglustre.tab.req"/> to determine which Lustre packages are to be installed on servers for your platform and architecture.</para>
<para>Some Lustre packages are installed on the Lustre servers (MGS, MDS, and OSSs). Others are installed on Lustre clients.</para>
<para>Lustre packages must be installed in the order specified in the following steps.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Install the kernel, modules and ldiskfs packages.</emphasis></para>
+ <para>Install the kernel, modules and <literal>ldiskfs</literal> packages.</para>
<para>Use the <literal>rpm -ivh</literal> command to install the kernel, module and ldiskfs packages.</para>
<note>
- <para>It is not recommended that you use the rpm -Uvh command to install a kernel, because this may leave you with an unbootable system if the new kernel doesn't work for some reason.</para>
+ <para>It is not recommended that you use the <literal>rpm -Uvh</literal> command to install a kernel, because this may leave you with an unbootable system if the new kernel doesn't work for some reason.</para>
</note>
<para>For example, the command in the following example would install required packages on a server with Infiniband networking</para>
<screen>$ rpm -ivh kernel-<ver>_lustre-<ver> kernel-ib-<ver> \
</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Update the bootloader (<literal>grub.conf</literal> or <literal>lilo.conf</literal>) configuration file as needed.</emphasis></para>
+ <para>Update the bootloader (<literal>grub.conf</literal> or <literal>lilo.conf</literal>) configuration file as needed.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Verify that the bootloader configuration file has been updated with an entry for the patched kernel.</emphasis></para>
+ <para>Verify that the bootloader configuration file has been updated with an entry for the patched kernel.</para>
<para>Before you can boot to a new distribution or kernel, there must be an entry for it in the bootloader configuration file. Often this is added automatically when the kernel RPM is installed.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below:</emphasis></para>
+ <para>Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below:</para>
<screen>selinux=0
</screen>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Install the utilities/userspace packages.</emphasis></para>
+ <para>Install the utilities/userspace packages.</para>
<para>Use the <literal>rpm -ivh</literal> command to install the utilities packages. For example:</para>
<screen>$ rpm -ivh lustre-<ver>
</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Install the e2fsprogs package.</emphasis></para>
+ <para>Install the <literal>e2fsprogs</literal> package.</para>
<para>Use the <literal>rpm -ivh</literal> command to install the <literal>e2fsprogs</literal> package. For example:</para>
<screen>$ rpm -ivh e2fsprogs-<ver>
</screen>
<para>If <literal>e2fsprogs</literal> is already installed on your Linux system, install the Lustre-specific <literal>e2fsprogs</literal> version by using <literal>rpm -Uvh</literal> to upgrade the existing <literal>e2fsprogs</literal> package. For example:</para>
<screen>$ rpm -Uvh e2fsprogs-<ver> </screen>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry>
- <para><emphasis role="bold">Note -</emphasis>The <literal>rpm</literal> command options <literal>--force</literal> or <literal>--nodeps</literal> should not be used to install or update the Lustre-specific <literal>e2fsprogs</literal> package. If errors are reported, file a bug (for instructions see the topic <emphasis>
- <link xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</link>
- </emphasis> on the Lustre wiki).</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note>
+ <para>The <literal>rpm</literal> command options <literal>--force</literal> or <literal>--nodeps</literal> should not be used to install or update the Lustre-specific <literal>e2fsprogs</literal> package. If errors are reported, file a bug (for instructions see the topic <emphasis>
+ <link xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</link>
+ </emphasis> on the Lustre wiki).</para>
+ </note>
</listitem>
<listitem>
- <para><emphasis role="bold"><emphasis role="italic">
- <emphasis role="italic">(Optional)</emphasis>
- </emphasis> To add optional packages to your Lustre file system, install them now.</emphasis></para>
+ <para><emphasis role="italic">
+ <emphasis role="italic">(Optional)</emphasis>
+ </emphasis> To add optional packages to your Lustre file system, install them now.</para>
<para>Optional packages include file system creation and repair tools, debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages. A complete list of optional packages for your platform is provided on the <link xl:href="http://wiki.whamcloud.com">Whamcloud wiki</link>.</para>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Install the Lustre packages on the clients.</emphasis></para>
+ <para>Install the Lustre packages on the clients.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Refer to <xref linkend="installinglustre.tab.req"/> to determine which Lustre packages are to be installed on clients for your platform and architecture.</emphasis></para>
+ <para>Refer to <xref linkend="installinglustre.tab.req"/> to determine which Lustre packages are to be installed on clients for your platform and architecture.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Install the module packages for clients.</emphasis></para>
+ <para>Install the module packages for clients.</para>
<para>Use the <literal>rpm -ivh</literal> command to install the lustre-client and <literal>lustre-client-modules-<ver></literal> packages. For example:</para>
- <screen>$ rpm -ivh lustre-client-modules-<ver> kernel-ib-<ver>
-</screen>
+ <screen>$ rpm -ivh lustre-client-modules-<ver> kernel-ib-<ver></screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Install the utilities/userspace packages for clients.</emphasis></para>
+ <para>Install the utilities/userspace packages for clients.</para>
<para>Use the <literal>rpm -ivh</literal> command to install the utilities packages. For example:</para>
- <screen>$ rpm -ivh lustre-client
-</screen>
+ <screen>$ rpm -ivh lustre-client</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Update the bootloader (grub.conf or lilo.conf) configuration file as needed.</emphasis></para>
+ <para>Update the bootloader (grub.conf or lilo.conf) configuration file as needed.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Verify that the bootloader configuration file has been updated with an entry for the patched kernel.</emphasis></para>
+ <para>Verify that the bootloader configuration file has been updated with an entry for the patched kernel.</para>
<para>Before you can boot to a new distribution or kernel, there must be an entry for it in the bootloader configuration file. Often this is added automatically when the kernel RPM is installed.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below:</emphasis></para>
- <screen>selinux=0
-</screen>
+ <para>Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below:</para>
+ <screen>selinux=0 </screen>
</listitem>
</orderedlist>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Reboot the patched clients and the servers.</emphasis></para>
+ <para>Reboot the patched clients and the servers.</para>
<orderedlist>
<listitem>
<para>If you applied the patched kernel to any clients, reboot them.</para>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="installinglustrefromsourcecode">
- <info>
- <title xml:id="installinglustrefromsourcecode.title">Installing Lustre from Source Code</title>
- </info>
+ <title xml:id="installinglustrefromsourcecode.title">Installing Lustre from Source Code</title>
<para>If you need to build a customized Lustre server kernel or are using a Linux kernel that has not been tested with the version of Lustre you are installing, you may need to build and install Lustre from source code. This chapter describes:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438210_69313">
- <title>29.1 Overview and Prerequisites</title>
+ <title>
+ <indexterm><primary>installing</primary><secondary>from source</secondary></indexterm>
+ Overview and Prerequisites</title>
<para>Lustre can be installed from either pre-built binary packages (RPMs) or freely-available source code. Installing from the package release is recommended unless you need to customize the Lustre server kernel or will be using an Linux kernel that has not been tested with Lustre. For a list of supported Linux distributions and architectures, see the topic <link xl:href="http://wiki.lustre.org/index.php/Lustre_2.0">Lustre_2.0</link> on the Lustre wiki. The procedure for installing Lustre from RPMs is describe in <xref linkend="installinglustre"/>.</para>
<para>To install Lustre from source code, the following are required:</para>
<itemizedlist>
</listitem>
</itemizedlist>
<note>
-
<para>When using third-party network hardware with Lustre, the third-party modules (typically, the drivers) must be linked against the Linux kernel. The LNET modules in Lustre also need these references. To meet these requirements, a specific process must be followed to install and recompile Lustre. See <xref linkend="dbdoclet.50438210_27248"/>, for an example showing how to install Lustre 1.6.6 using the Myricom MX 1.2.7 driver. The same process can be used for other third-party network stacks.</para>
+ <para>When using third-party network hardware with Lustre, the third-party modules (typically, the drivers) must be linked against the Linux kernel. The LNET modules in Lustre also need these references. To meet these requirements, a specific process must be followed to install and recompile Lustre. See <xref linkend="dbdoclet.50438210_27248"/>, for an example showing how to install Lustre 1.6.6 using the Myricom MX 1.2.7 driver. The same process can be used for other third-party network stacks.</para>
</note>
</section>
<section xml:id="dbdoclet.50438210_65411">
- <title>29.2 Patching the Kernel</title>
+ <title><indexterm><primary>installing</primary><secondary>from source</secondary><tertiary>patching the kernel</tertiary></indexterm>Patching the Kernel</title>
<para>If you are using non-standard hardware, plan to apply a Lustre patch, or have another reason not to use packaged Lustre binaries, you have to apply several Lustre patches to the core kernel and run the Lustre configure script against the kernel.</para>
<section remap="h3">
- <title>29.2.1 Introducing the <literal>quilt</literal> Utility</title>
- <para>To simplify the process of applying Lustre patches to the kernel, we recommend that you use the <literal>qu</literal>ilt utility.</para>
- <para><literal>quil</literal>t manages a stack of patches on a single source tree. A series file lists the patch files and the order in which they are applied. Patches are applied, incrementally, on the base tree and all preceding patches. You can:</para>
+ <title><indexterm><primary>quilt</primary></indexterm>Introducing the <literal>quilt</literal> Utility</title>
+ <para>To simplify the process of applying Lustre patches to the kernel, we recommend that you use the <literal>quilt</literal> utility.</para>
+ <para><literal>quilt</literal> manages a stack of patches on a single source tree. A series file lists the patch files and the order in which they are applied. Patches are applied, incrementally, on the base tree and all preceding patches. You can:</para>
<itemizedlist>
<listitem>
<para>Apply patches from the stack (<literal>quilt push</literal>)</para>
<para>For additional information on using Quilt, including its commands, see <link xl:href="http://www.suse.de/~agruen/quilt.pdf">Introduction to Quilt</link> and the <link xl:href="http://linux.die.net/man/1/quilt"><literal>quilt(1</literal>) man page.</link></para>
</section>
<section remap="h3">
- <title>29.2.2 Get the Lustre Source and Unpatched Kernel</title>
+ <title>Get the Lustre Source and Unpatched Kernel</title>
<para>The Lustre Engineering Team has targeted several Linux kernels for use with Lustre servers (MDS/OSS) and provides a series of patches for each one. The Lustre patches are maintained in the kernel_patch directory bundled with the Lustre source code.</para>
<caution>
<para>Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured and administered correctly. Before installing Lustre, be cautious and back up ALL data.</para>
<para>For more information on these prerequisites, see:</para>
<itemizedlist>
<listitem>
-
<para> Hardware requirements in <xref linkend="settinguplustresystem"/>.</para>
+ <para> Hardware requirements in <xref linkend="settinguplustresystem"/>.</para>
</listitem>
<listitem>
-
<para> Software and environmental requirements in <xref linkend="dbdoclet.50438261_99193"/></para>
+ <para> Software and environmental requirements in <xref linkend="dbdoclet.50438261_99193"/></para>
</listitem>
</itemizedlist>
</listitem>
</orderedlist>
</section>
<section remap="h3">
- <title>29.2.3 Patch the Kernel</title>
+ <title>Patch the Kernel</title>
<para>This procedure describes how to use Quilt to apply the Lustre patches to the kernel. To illustrate the steps in this procedure, a RHEL 5 kernel is patched for Lustre 1.6.5.1.</para>
<orderedlist>
<listitem>
</section>
</section>
<section xml:id="dbdoclet.50438210_47529">
- <title>29.3 Creating and Installing the Lustre Packages</title>
+ <title>
+ <indexterm><primary>installing</primary><secondary>from source</secondary><tertiary>packaging</tertiary></indexterm>
+
+ Creating and Installing the Lustre Packages</title>
<para>After patching the kernel, configure it to work with Lustre, create the Lustre packages (RPMs) and install them.</para>
<orderedlist>
<listitem>
lustre-source-1.6.5.1-\2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i686.rpm</screen>
<note>
-
<para>If the steps to create the RPMs fail, contact Lustre Support by reporting a bug. See <xref linkend="dbdoclet.50438198_30989"/>.</para>
+ <para>If the steps to create the RPMs fail, contact Lustre Support by reporting a bug. See <xref linkend="dbdoclet.50438198_30989"/>.</para>
</note>
<note>
<para>Several features and packages are available that extend the core functionality of Lustre. These features/packages can be enabled at the build time by issuing appropriate arguments to the configure command. For a list of these features and packages, run <literal>./configure -help</literal> in the Lustre source tree. The configs/ directory of the kernel source contains the config files matching each the kernel version. Copy one to <literal>.config</literal> at the root of the kernel tree.</para>
<para>Example result:</para>
<screen>kernel-2.6.95.0.3.EL_lustre.1.6.5.1custom-1.i686.rpm</screen>
<note>
-
<para>Step <xref linkend="dbdoclet.50438210_41207"/> is only valid for RedHat and SuSE kernels. If you are using a stock Linux kernel, you need to get a script to create the kernel RPM.</para>
+ <para>Step <xref linkend="dbdoclet.50438210_41207"/> is only valid for RedHat and SuSE kernels. If you are using a stock Linux kernel, you need to get a script to create the kernel RPM.</para>
</note>
</listitem>
<listitem>
</orderedlist>
</section>
<section xml:id="dbdoclet.50438210_27248">
- <title>29.4 Installing Lustre with a Third-Party Network Stack</title>
+ <title>
+ <indexterm><primary>installing</primary><secondary>from source</secondary><tertiary>3rd-party network</tertiary></indexterm>
+ Installing Lustre with a Third-Party Network Stack</title>
<para>When using third-party network hardware, you must follow a specific process to install and recompile Lustre. This section provides an installation example, describing how to install Lustre 1.6.6 while using the Myricom MX 1.2.7 driver. The same process is used for other third-party network stacks, by replacing MX-specific references in <xref linkend="dbdoclet.50438210_12366"/> with the stack-specific build and using the proper <literal>--with</literal> option when configuring the Lustre source code.</para>
<orderedlist>
<listitem xml:id="dbdoclet.50438210_12366">
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lnetselftest">
- <info>
- <title xml:id="lnetselftest.title">Testing Lustre Network Performance (LNET Self-Test)</title>
- </info>
+ <title xml:id="lnetselftest.title">Testing Lustre Network Performance (LNET Self-Test)</title>
<para><anchor xml:id="dbdoclet.50438223_36273" xreflabel=""/>This chapter describes the LNET self-test, which is used by site administrators to confirm that Lustre Networking (LNET) has been properly installed and configured, and that underlying network software and hardware are performing according to expectations. The chapter includes:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438223_91742">
- <title>23.1 LNET Self-Test Overview</title>
+ <title><indexterm><primary>LNET</primary><secondary>self-test</secondary></indexterm>
+LNET Self-Test Overview</title>
<para>LNET self-test is a kernel module that runs over LNET and the Lustre network drivers (LNDs. It is designed to:</para>
<itemizedlist>
<listitem>
- <para> Test the connection ability of the Lustre network</para>
+ <para>Test the connection ability of the Lustre network</para>
</listitem>
<listitem>
- <para> Run regression tests of the Lustre network</para>
+ <para>Run regression tests of the Lustre network</para>
</listitem>
<listitem>
- <para> Test performance of the Lustre network</para>
+ <para>Test performance of the Lustre network</para>
</listitem>
</itemizedlist>
<para>After you have obtained performance results for your Lustre network, refer to <xref linkend="lustretuning"/> for information about parameters that can be used to tune LNET for optimum performance.</para>
<para>Test nodes can be in either kernel or userspace. A console user can invite a kernel test node to join the test session by running lstadd_groupNID, but the console user cannot actively add a userspace test node to the test-session. However, the console user can passively accept a test node to the test session while the test node is running lstclient to connect to the console.</para>
</note>
<section remap="h3">
- <title>23.1.1 Prerequisites</title>
+ <title>Prerequisites</title>
<para>To run LNET self-test, these modules must be loaded:</para>
<itemizedlist>
<listitem>
</section>
</section>
<section xml:id="dbdoclet.50438223_48138">
- <title>23.2 Using LNET Self-Test</title>
+ <title>Using LNET Self-Test</title>
<para>This section describes how to create and run an LNET self-test. The examples shown are for a test that simulates the traffic pattern of a set of Lustre servers on a TCP network accessed by Lustre clients on an InfiniBand network connected via LNET routers. In this example, half the clients are reading and half the clients are writing.</para>
<section remap="h3">
- <title>23.2.1 Creating a Session</title>
+ <title>Creating a Session</title>
<para>A <emphasis>session</emphasis> is a set of processes that run on a test node. Only one session can be run at a time on a test node to ensure that the session has exclusive use of the node. The console node is used to create, change or destroy a session (new_session, end_session, show_session). For more about session parameters, see <xref linkend="dbdoclet.50438223_91247"/>.</para>
<para>Almost all operations should be performed within the context of a session. From the console node, a user can only operate nodes in his own session. If a session ends, the session context in all test nodes is stopped.</para>
<para>The following commands set the LST_SESSION environment variable to identify the session on the console node and create a session called read_write:</para>
lst new_session read_write</screen>
</section>
<section remap="h3">
- <title>23.2.2 Setting Up Groups</title>
+ <title>Setting Up Groups</title>
<para>A <emphasis>group</emphasis> is a named collection of nodes. Any number of groups can exist in a single LNET self-test session. Group membership is not restricted in that a node can be included in any number of groups.</para>
<para>Each node in a group has a rank, determined by the order in which it was added to the group. The rank is used to establish test traffic patterns.</para>
<para>A user can only control nodes in his/her session. To allocate nodes to the session, the user needs to add nodes to a group (of the session). All nodes in a group can be referenced by the group name. A node can be allocated to multiple groups of a session.</para>
<para>A console user can associate kernel space test nodes with the session by running lst <literal>add_group</literal> <literal>NIDs</literal>, but a userspace test node cannot be actively added to the session. However, the console user can passively "accept" a test node to associate with a test session while the test node running <literal>lstclient</literal> connects to the console node, i.e: <literal>lstclient --sesid CONSOLE_NID --group NAME</literal>).</para>
</note>
</section>
- <section xml:id='dbdoclet.50438223_42848'>
- <title>23.2.3 Defining and Running the Tests</title>
+ <section xml:id="dbdoclet.50438223_42848">
+ <title>Defining and Running the Tests</title>
<para>A <emphasis>test</emphasis> generates a network load between two groups of nodes, a source group identified using the <literal>--from</literal> parameter and a target group identified using the <literal>--to</literal> parameter. When a test is running, each node in the <literal>--from<replaceable><group></replaceable></literal> simulates a client by sending requests to nodes in the <literal>--to<replaceable><group></replaceable></literal>, which are simulating a set of servers, and then receives responses in return. This activity is designed to mimic Lustre RPC traffic.</para>
<para>A <emphasis>batch</emphasis> is a collection of tests that are started and stopped together and run in parallel. A test must always be run as part of a batch, even if it is just a single test. Users can only run or stop a test batch, not individual tests.</para>
<para>Tests in a batch are non-destructive to the file system, and can be run in a normal Lustre environment (provided the performance impact is acceptable).</para>
<para>The traffic pattern and test intensity is determined by several properties such as test type, distribution of test nodes, concurrency of test, and RDMA operation type. For more details, see <xref linkend="dbdoclet.50438223_36860"/>.</para>
</section>
<section remap="h3">
- <title>23.2.4 Sample Script</title>
+ <title>Sample Script</title>
<para>This sample LNET self-test script simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an InfiniBand network (connected via LNET routers). In this example, half the clients are reading and half the clients are writing.</para>
<para>Run this script on the console node:</para>
<screen>#!/bin/bash
</section>
</section>
<section xml:id="dbdoclet.50438223_27277">
- <title>23.3 LNET Self-Test Command Reference</title>
+ <title>LNET Self-Test Command Reference</title>
<para>The LNET self-test (<literal>lst</literal>) utility is used to issue LNET self-test commands. The <literal>lst</literal> utility takes a number of command line arguments. The first argument is the command name and subsequent arguments are command-specific.</para>
<section xml:id="dbdoclet.50438223_91247">
- <title>23.3.1 Session Commands</title>
+ <title>Session Commands</title>
<para>This section describes <literal>lst</literal> session commands.</para>
<para><emphasis role="bold">
<literal>LST_SESSION</literal>
<screen>$ lst show_session</screen>
</section>
<section remap="h3">
- <title>23.3.2 Group Commands</title>
+ <title>Group Commands</title>
<para>This section describes <literal>lst</literal> group commands.</para>
<para><literal>
<replaceable role="bold">add_group</replaceable>
<tbody>
<row>
<entry>
- <para>
- <literal>
- --refresh
- </literal>
+ <para>
+ <literal> --refresh </literal>
</para>
</entry>
<entry nameend="c3" namest="c2">
</row>
<row>
<entry>
- <para>
- <literal>
- --active
- </literal>
+ <para>
+ <literal> --active </literal>
</para>
</entry>
<entry>
</row>
<row>
<entry>
- <para>
- <literal>
- --busy
- </literal>
+ <para>
+ <literal> --busy </literal>
</para>
</entry>
<entry>
</row>
<row>
<entry>
- <literal>
- --down
- </literal>
+ <literal> --down </literal>
</entry>
<entry>
<para> Lists the down nodes.</para>
</row>
<row>
<entry>
- <literal>
- --unknown
- </literal>
+ <literal> --unknown </literal>
</entry>
<entry>
<para> Lists unknown nodes.</para>
</row>
<row>
<entry>
- <literal>
- --all
- </literal>
+ <literal> --all </literal>
</entry>
<entry>
<para> Lists all nodes.</para>
<tbody>
<row>
<entry>
- <literal>
- --sesid<replaceable><NID></replaceable>
- </literal>
+ <literal> --sesid<replaceable><NID></replaceable></literal>
</entry>
<entry>
<para> The first console's NID.</para>
</row>
<row>
<entry>
- <literal>
- --group<replaceable><name></replaceable>
- </literal>
+ <literal> --group<replaceable><name></replaceable></literal>
</entry>
<entry>
<para> The test group to join.</para>
</row>
<row>
<entry>
- <literal>
- --server_mode
- </literal>
+ <literal> --server_mode </literal>
</entry>
<entry>
<para> When included, forces LNET to behave as a server, such as starting an acceptor if the underlying NID needs it or using privileged ports. Only root is allowed to use the <literal>--server_mode</literal> option.</para>
<screen>Client1 $ lstclient --sesid 192.168.1.52@tcp |--group clients --server_mode</screen>
</section>
<section xml:id="dbdoclet.50438223_36860">
- <title>23.3.3 Batch and Test Commands</title>
+ <title>Batch and Test Commands</title>
<para>This section describes <literal>lst</literal> batch and test commands.</para>
<para><literal>
<replaceable role="bold">add_batch NAME</replaceable>
<screen>$ lst add_batch bulkperf</screen>
<para>Creates a batch test called <literal>bulkperf</literal>.</para>
<screen>
- <replaceable role="bold">add_test --batch <replaceable><<replaceable role="italic">batchname</replaceable>></replaceable> [--loop<#>] <replaceable role="bold">[--concurrency</replaceable><replaceable><#></replaceable><replaceable role="bold">] [--distribute</replaceable><replaceable><#:#></replaceable><replaceable role="bold">]</replaceable> \
---from<replaceable> <<replaceable role="italic">group</replaceable>></replaceable> --to <replaceable><<replaceable role="italic">group</replaceable>></replaceable> {brw|ping} <test options></replaceable>
+ <replaceable role="bold">add_test --batch <replaceable><<replaceable role="italic">batchname</replaceable>></replaceable> [--loop<#>] <replaceable role="bold">[--concurrency</replaceable><replaceable><#></replaceable><replaceable role="bold">] [--distribute</replaceable><replaceable><#:#></replaceable><replaceable role="bold">]</replaceable> \ --from<replaceable> <<replaceable role="italic">group</replaceable>></replaceable> --to <replaceable><<replaceable role="italic">group</replaceable>></replaceable> {brw|ping} <test options></replaceable>
</screen>
<para>Adds a test to a batch. The parameters are described below.</para>
<informaltable frame="all">
</row>
<row>
<entry>
- <literal>
- --loop<replaceable><#></replaceable>
- </literal>
+ <literal> --loop<replaceable><#></replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
<para>Number of times to run the test.</para>
</row>
<row>
<entry>
- <literal>
- --concurrency<replaceable><#></replaceable>
- </literal>
+ <literal> --concurrency<replaceable><#></replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
<para>The number of requests that are active at one time.</para>
</row>
<row>
<entry>
- <literal>
- --distribute<replaceable><#:#></replaceable>
- </literal>
+ <literal> --distribute<replaceable><#:#></replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
<para>Determines the ratio of client nodes to server nodes for the specified test. This allows you to specify a wide range of topologies, including one-to-one and all-to-all. Distribution divides the source group into subsets, which are paired with equivalent subsets from the target group so only nodes in matching subsets communicate.</para>
</row>
<row>
<entry>
- <literal>
- --from<replaceable><group></replaceable>
- </literal>
+ <literal> --from<replaceable><group></replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
<para>The source group (test client).</para>
</row>
<row>
<entry>
- <literal>
- --to<replaceable><group></replaceable>
- </literal>
+ <literal> --to<replaceable><group></replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
<para>The target group (test server).</para>
<literal>ping</literal>
</entry>
<entry nameend="c3" namest="c2">
-
<para>Sends a small request message, resulting in a small reply message. For more details, see <xref linkend="dbdoclet.50438223_42848"/></para>
+ <para>Sends a small request message, resulting in a small reply message. For more details, see <xref linkend="dbdoclet.50438223_42848"/></para>
</entry>
</row>
<row>
<literal>brw</literal>
</entry>
<entry nameend="c3" namest="c2">
-
<para>Sends a small request message followed by a bulk data transfer, resulting in a small reply message. <xref linkend="dbdoclet.50438223_42848"/>. Options are:</para>
+ <para>Sends a small request message followed by a bulk data transfer, resulting in a small reply message. <xref linkend="dbdoclet.50438223_42848"/>. Options are:</para>
</entry>
</row>
<row>
<para> </para>
</entry>
<entry>
- <literal>
- read | write
- </literal>
+ <literal> read | write </literal>
</entry>
<entry>
<para>Read or write. The default is read.</para>
</entry>
</row>
<row>
+ <entry/>
<entry>
- </entry>
- <entry>
- <literal>
- size=<#>| <#>K | <#>M
- </literal>
+ <literal> size=<#>| <#>K | <#>M </literal>
</entry>
<entry>
<para>I/O size in bytes, KB or MB (i.e., <literal>size=1024</literal>, <literal>size=4</literal>K, <literal>size=1M</literal>). The default is 4K bytes.</para>
</entry>
</row>
<row>
+ <entry/>
<entry>
- </entry>
- <entry>
- <literal>
- check=full|simple
- </literal>
+ <literal> check=full|simple </literal>
</entry>
<entry>
<para>A data validation check (checksum of data). The default is that no check is done.</para>
--distribute 4:2 (C1,C2,C3,C4->S1,S2), (C5, C6->S3, S1)
--distribute 6:3 (C1,C2,C3,C4,C5,C6->S1,S2,S3)</screen>
<para>The setting <literal>--distribute 1:1</literal> is the default setting where each source node communicates with one target node.</para>
- <para>When the setting <literal>--distribute 1:<replaceable><n></replaceable></literal> (where <literal><replaceable><n></replaceable></literal> is the size of the target group) is used, each source node communicates with every node in the target group.</para>
+ <para>When the setting <literal>--distribute 1:<replaceable><n></replaceable></literal> (where <literal>
+ <replaceable><n></replaceable>
+ </literal> is the size of the target group) is used, each source node communicates with every node in the target group.</para>
<para>Note that if there are more source nodes than target nodes, some source nodes may share the same target nodes. Also, if there are more target nodes than source nodes, some higher-ranked target nodes will be idle.</para>
<para><emphasis role="bold">Example showing a <literal>brw</literal> test:</emphasis></para>
<screen>$ lst add_group clients 192.168.1.[10-17]@tcp
<tbody>
<row>
<entry>
- <literal>
- --test<replaceable><index></replaceable>
- </literal>
+ <literal> --test<replaceable><index></replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
<para>Lists tests in a batch. If no option is used, all tests in the batch are listed. IIf one of these options are used, only specified tests in the batch are listed:</para>
<para> </para>
</entry>
<entry>
- <literal>
- server | client
- </literal>
+ <literal> server | client </literal>
</entry>
<entry>
<para>Lists client and server nodes in a batch test.</para>
<replaceable role="bold">] [--loop</replaceable>
<replaceable><#></replaceable>
<replaceable role="bold">] [--delay</replaceable>
- <replaceable><<replaceable role="italic">seconds</replaceable>></replaceable>
+ <replaceable><<replaceable role="italic">seconds</replaceable>></replaceable>
<replaceable role="bold">] [--all]</replaceable>
</literal>
</emphasis></para>
<tbody>
<row>
<entry>
- <literal>
- --test<replaceable><index></replaceable>
- </literal>
+ <literal> --test<replaceable><index></replaceable></literal>
</entry>
<entry>
<para> Only queries the specified test. The test index starts from 1.</para>
</row>
<row>
<entry>
- <literal>
- --timeout<replaceable><seconds></replaceable>
- </literal>
+ <literal> --timeout<replaceable><seconds></replaceable></literal>
</entry>
<entry>
<para> The timeout value to wait for RPC. The default is 5 seconds.</para>
</row>
<row>
<entry>
- <literal>
- --loop<replaceable><#></replaceable>
- </literal>
+ <literal> --loop<replaceable><#></replaceable></literal>
</entry>
<entry>
<para> The loop count of the query.</para>
</row>
<row>
<entry>
- <literal>
- --delay<replaceable><seconds></replaceable>
- </literal>
+ <literal> --delay<replaceable><seconds></replaceable></literal>
</entry>
<entry>
<para> The interval of each query. The default is 5 seconds.</para>
</row>
<row>
<entry>
- <literal>
- --all
- </literal>
+ <literal> --all </literal>
</entry>
<entry>
<para> The list status of all nodes in a batch or a test.</para>
Batch is idle</screen>
</section>
<section remap="h3">
- <title>23.3.4 Other Commands</title>
+ <title>Other Commands</title>
<para>This section describes other <literal>lst</literal> commands.</para>
<para><literal>
<replaceable role="bold">
<tbody>
<row>
<entry>
- <literal>
- --session
- </literal>
+ <literal> --session </literal>
</entry>
<entry>
<para>Pings all nodes in the current session.</para>
</row>
<row>
<entry>
- <literal>
- --group<replaceable><name></replaceable>
- </literal>
+ <literal> --group<replaceable><name></replaceable></literal>
</entry>
<entry>
<para>Pings all nodes in a specified group.</para>
</row>
<row>
<entry>
- <literal>
- --nodes<replaceable><NIDs></replaceable>
- </literal>
+ <literal> --nodes<replaceable><NIDs></replaceable></literal>
</entry>
<entry>
<para>Pings all specified nodes.</para>
</row>
<row>
<entry>
- <literal>
- --batch<replaceable><name></replaceable>
- </literal>
+ <literal> --batch<replaceable><name></replaceable></literal>
</entry>
<entry>
<para>Pings all client nodes in a batch.</para>
</row>
<row>
<entry>
- <literal>
- --server
- </literal>
+ <literal> --server </literal>
</entry>
<entry>
<para>Sends RPC to all server nodes instead of client nodes. This option is only used with <literal>--batch<replaceable><name></replaceable></literal>.</para>
</row>
<row>
<entry>
- <literal>
- --timeout<replaceable><seconds></replaceable>
- </literal>
+ <literal> --timeout<replaceable><seconds></replaceable></literal>
</entry>
<entry>
<para>The RPC timeout value.</para>
<tbody>
<row>
<entry>
- <literal>
- --bw
- </literal>
+ <literal> --bw </literal>
</entry>
<entry>
<para>Displays the bandwidth of the specified group/nodes.</para>
</row>
<row>
<entry>
- <literal>
- --rate
- </literal>
+ <literal> --rate </literal>
</entry>
<entry>
<para>Displays the rate of RPCs of the specified group/nodes.</para>
</row>
<row>
<entry>
- <literal>
- --read
- </literal>
+ <literal> --read </literal>
</entry>
<entry>
<para>Displays the read statistics of the specified group/nodes.</para>
</row>
<row>
<entry>
- <literal>
- --write
- </literal>
+ <literal> --write </literal>
</entry>
<entry>
<para>Displays the write statistics of the specified group/nodes.</para>
</row>
<row>
<entry>
- <literal>
- --max
- </literal>
+ <literal> --max </literal>
</entry>
<entry>
<para>Displays the maximum value of the statistics.</para>
</row>
<row>
<entry>
- <literal>
- --min
- </literal>
+ <literal> --min </literal>
</entry>
<entry>
<para>Displays the minimum value of the statistics.</para>
</row>
<row>
<entry>
- <literal>
- --avg
- </literal>
+ <literal> --avg </literal>
</entry>
<entry>
<para>Displays the average of the statistics.</para>
</row>
<row>
<entry>
- <literal>
- --timeout<replaceable><seconds></replaceable>
- </literal>
+ <literal> --timeout<replaceable><seconds></replaceable></literal>
</entry>
<entry>
<para>The timeout of the statistics RPC. The default is 5 seconds.</para>
</row>
<row>
<entry>
- <literal>
- --delay<replaceable><seconds></replaceable>
- </literal>
+ <literal> --delay<replaceable><seconds></replaceable></literal>
</entry>
<entry>
<para>The interval of the statistics (in seconds).</para>
<tbody>
<row>
<entry>
- <literal>
- --session
- </literal>
+ <literal> --session </literal>
</entry>
<entry>
<para>Lists errors in the current test session. With this option, historical RPC errors are not listed.</para>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustredebugging">
- <info>
- <title xml:id="lustredebugging.title">Lustre Debugging</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustredebugging">
+ <title xml:id="lustredebugging.title">Lustre Debugging</title>
<para>This chapter describes tips and information to debug Lustre, and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438274_15874">
- <title>28.1 Diagnostic and Debugging Tools</title>
+ <title><indexterm><primary>debugging</primary></indexterm>
+Diagnostic and Debugging Tools</title>
<para>A variety of diagnostic and analysis tools are available to debug issues with the Lustre software. Some of these are provided in Linux distributions, while others have been developed and are made available by the Lustre project.</para>
<section remap="h3">
- <title>28.1.1 Lustre Debugging Tools</title>
+ <title><indexterm><primary>debugging</primary><secondary>tools</secondary></indexterm>
+Lustre Debugging Tools</title>
<para>The following in-kernel debug mechanisms are incorporated into the Lustre software:</para>
<itemizedlist>
<listitem>
</emphasis> - This tool is used with the debug_kernel option to manually dump the Lustre debugging log or post-process debugging logs that are dumped automatically. For more information about the lctl tool, see <xref linkend="dbdoclet.50438274_62472"/> and <xref linkend="dbdoclet.50438219_38274"/>.</para>
</listitem>
<listitem>
-
<para><replaceable role="bold">Lustre subsystem asserts</replaceable> - A panic-style assertion (LBUG) in the kernel causes Lustre to dump the debug log to the file <literal>/tmp/lustre-log.<replaceable><timestamp></replaceable></literal> where it can be retrieved after a reboot. For more information, see <xref linkend="dbdoclet.50438198_40669"/>.</para>
+ <para><replaceable role="bold">Lustre subsystem asserts</replaceable> - A panic-style assertion (LBUG) in the kernel causes Lustre to dump the debug log to the file <literal>/tmp/lustre-log.<replaceable><timestamp></replaceable></literal> where it can be retrieved after a reboot. For more information, see <xref linkend="dbdoclet.50438198_40669"/>.</para>
</listitem>
<listitem>
<para>
- <literal><replaceable>lfs</replaceable></literal>
+ <literal>
+ <replaceable>lfs</replaceable>
+ </literal>
- This utility provides access to the extended attributes (EAs) of a Lustre file (along with other information). For more inforamtion about lfs, see <xref linkend="dbdoclet.50438206_94597"/>.</para>
</listitem>
</itemizedlist>
</section>
<section remap="h3">
- <title>28.1.2 External Debugging Tools</title>
+ <title><indexterm><primary>debugging</primary><secondary>external tools</secondary></indexterm>External Debugging Tools</title>
<para>The tools described in this section are provided in the Linux kernel or are available at an external website. For information about using some of these tools for Lustre debugging, see <xref linkend="dbdoclet.50438274_23607"/> and <xref linkend="dbdoclet.50438274_80443"/>.</para>
<section remap="h4">
- <title>28.1.2.1 Tools for Administrators and Developers</title>
+ <title><indexterm><primary>debugging</primary><secondary>admin tools</secondary></indexterm>Tools for Administrators and Developers</title>
<para>Some general debugging tools provided as a part of the standard Linux distro are:</para>
<itemizedlist>
<listitem>
</itemizedlist>
</section>
<section remap="h4">
- <title>28.1.2.2 Tools for Developers</title>
+ <title><indexterm><primary>debugging</primary><secondary>developer tools</secondary></indexterm>Tools for Developers</title>
<para>The tools described in this section may be useful for debugging Lustre in a development environment.</para>
<para>Of general interest is:</para>
<itemizedlist>
</section>
</section>
<section xml:id="dbdoclet.50438274_23607">
- <title>28.2 Lustre Debugging Procedures</title>
+ <title><indexterm><primary>debugging</primary><secondary>procedure</secondary></indexterm>Lustre Debugging Procedures</title>
<para>The procedures below may be useful to administrators or developers debugging a Lustre files system.</para>
<section remap="h3">
- <title>28.2.1 Understanding the Lustre Debug Messaging Format</title>
+ <title><indexterm><primary>debugging</primary><secondary>message format</secondary></indexterm>Understanding the Lustre Debug Messaging Format</title>
<para>Lustre debug messages are categorized by originating sybsystem, message type, and locaton in the source code. For a list of subsystems and message types, see <xref linkend="dbdoclet.50438274_57603"/>.</para>
<note>
<para>For a current list of subsystems and debug message types, see <literal>lnet/include/libcfs/libcfs.h</literal> in the Lustre tree</para>
</note>
<para>The elements of a Lustre debug message are described in <xref linkend="dbdoclet.50438274_57177"/> Format of Lustre Debug Messages.</para>
<section xml:id="dbdoclet.50438274_57603">
- <title>28.2.1.1 Lustre Debug Messages</title>
+ <title>Lustre Debug Messages</title>
<para>Each Lustre debug message has the tag of the subsystem it originated in, the message type, and the location in the source code. The subsystems and debug types used in Lustre are as follows:</para>
<itemizedlist>
<listitem>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438274_57177">
- <title>28.2.1.2 Format of Lustre Debug Messages</title>
+ <title>Format of Lustre Debug Messages</title>
<para>Lustre uses the <literal>CDEBUG</literal> and <literal>CERROR</literal> macros to print the debug or error messages. To print the message, the <literal>CDEBUG</literal> macro uses <literal>portals_debug_msg</literal> (<literal>portals/linux/oslib/debug.c</literal>). The message format is described below, along with an example.</para>
<informaltable frame="all">
<tgroup cols="2">
</informaltable>
</section>
<section remap="h4">
- <title>28.2.1.3 Lustre Debug Messages Buffer</title>
+ <title>Lustre Debug Messages Buffer</title>
<para>Lustre debug messages are maintained in a buffer, with the maximum buffer size specified (in MBs) by the <literal>debug_mb</literal> parameter (<literal>/proc/sys/lnet/debug_mb</literal>). The buffer is circular, so debug messages are kept until the allocated buffer limit is reached, and then the first messages are overwritten.</para>
</section>
</section>
- <section xml:id='dbdoclet.50438274_62472'>
- <title>28.2.2 Using the lctl Tool to View Debug Messages</title>
+ <section xml:id="dbdoclet.50438274_62472">
+ <title><indexterm><primary>debugging</primary><secondary>using lctl</secondary></indexterm>Using the lctl Tool to View Debug Messages</title>
<para>The <literal>lctl</literal> tool allows debug messages to be filtered based on subsystems and message types to extract information useful for troubleshooting from a kernel debug log. For a command reference, see <xref linkend="dbdoclet.50438219_38274"/>.</para>
<para>You can use <literal>lctl</literal> to:</para>
<itemizedlist>
<para>Debug messages displayed with <literal>lctl</literal> are also subject to the kernel debug masks; the filters are additive.</para>
</note>
<section remap="h4">
- <title>28.2.2.1 Sample <literal>lctl</literal> Run</title>
+ <title><indexterm><primary>debugging</primary><secondary>lctl example</secondary></indexterm>Sample <literal>lctl</literal> Run</title>
<para>Below is a sample run using the <literal>lctl</literal> command.</para>
<screen>bash-2.04# ./lctl
lctl > debug_kernel /tmp/lustre_logs/log_all
</section>
</section>
<section remap="h3">
- <title>28.2.3 Dumping the Buffer to a File (<literal>debug_daemon</literal>)</title>
+ <title>Dumping the Buffer to a File (<literal>debug_daemon</literal>)</title>
<para>The <literal>debug_daemon</literal> option is used by <literal>lctl</literal> to control the dumping of the <literal>debug_kernel</literal> buffer to a user-specified file. This functionality uses a kernel thread on top of <literal>debug_kernel</literal>, which works in parallel with the <literal>debug_daemon</literal> command.</para>
<para>The <literal>debug_daemon</literal> is highly dependent on file system write speed. File system write operations may not be fast enough to flush out all of the <literal>debug_buffer</literal> if the Lustre file system is under heavy system load and continues to <literal>CDEBUG</literal> to the <literal>debug_buffer</literal>. The <literal>debug_daemon</literal> will write the message <literal>DEBUG MARKER:</literal> Trace buffer full into the <literal>debug_buffer</literal> to indicate the <literal>debug_buffer</literal> contents are overlapping before the <literal>debug_daemon</literal> flushes data to a file.</para>
<para>Users can use <literal>lctl control</literal> to start or stop the Lustre daemon from dumping the <literal>debug_buffe</literal>r to a file. Users can also temporarily hold daemon from dumping the file. Use of the <literal>debug_daemon</literal> sub-command to <literal>lctl</literal> can provide the same function.</para>
<section remap="h4">
- <title>28.2.3.1 <literal>lctl debug_daemon</literal> Commands</title>
+ <title><literal>lctl debug_daemon</literal> Commands</title>
<para>This section describes <literal>lctl debug_daemon</literal> commands.</para>
<para>To initiate the <literal>debug_daemon</literal> to start dumping <literal>debug_buffer</literal> into a file., enter</para>
<screen>$ lctl debug_daemon start [{file} {megabytes}]</screen>
</section>
</section>
<section remap="h3">
- <title>28.2.4 Controlling Information Written to the Kernel Debug Log</title>
+ <title><indexterm><primary>debugging</primary><secondary>kernel debug log</secondary></indexterm>Controlling Information Written to the Kernel Debug Log</title>
<para>Masks are provided in <literal>/proc/sys/lnet/subsystem_debug</literal> and <literal>/proc/sys/lnet/debug</literal> to be used with the systctl command to determine what information is to be written to the debug log. The subsystem_debug mask determines the information written to the log based on the subsystem (such as iobdfilter, net, portals, or OSC). The debug mask controls information based on debug type (such as info, error, trace, or alloc).</para>
<para>To turn off Lustre debugging completely:</para>
<screen>sysctl -w lnet.debug=0 </screen>
<para>The various options available to print to kernel debug logs are listed in <literal>lnet/include/libcfs/libcfs.h</literal></para>
</section>
<section remap="h3">
- <title>28.2.5 Troubleshooting with <literal>strace</literal></title>
+ <title><indexterm><primary>debugging</primary><secondary>using strace</secondary></indexterm>Troubleshooting with <literal>strace</literal></title>
<para>The <literal>strace</literal> utility provided with the Linux distribution enables system calls to be traced by intercepting all the system calls made by a process and recording the system call name, arguments, and return values.</para>
<para>To invoke <literal>strace</literal> on a program, enter:</para>
<screen>$ strace <emphasis><program> <args></emphasis> </screen>
<screen>$ strace -o /r/tmp/vi.strace </screen>
</section>
<section remap="h3">
- <title>28.2.6 Looking at Disk Content</title>
+ <title><indexterm><primary>debugging</primary><secondary>disk contents</secondary></indexterm>Looking at Disk Content</title>
<para>In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The <literal>lfs</literal> tool can be used to obtain this information for a given file using the <literal>getstripe</literal> subcommand. Use a corresponding <literal>lfs setstripe</literal> command to specify striping attributes for a new file or directory.</para>
<para>The <literal>lfs getstripe</literal> utility is written in C; it takes a Lustre filename as input and lists all the objects that form a part of this file. To obtain this information for the file <literal>/mnt/lustre/frog</literal> in Lustre file system, run:</para>
<screen>$ lfs getstripe /mnt/lustre/frog
debugfs: quit</screen>
</section>
<section remap="h3">
- <title>28.2.7 Finding the Lustre UUID of an OST</title>
+ <title>Finding the Lustre UUID of an OST</title>
<para>To determine the Lustre UUID of an obdfilter disk (for example, if you mix up the cables on your OST devices or the SCSI bus numbering suddenly changes and the SCSI devices get new names), use <literal>debugfs</literal> to get the <literal>last_rcvd</literal> file.</para>
</section>
<section remap="h3">
- <title>28.2.8 Printing Debug Messages to the Console</title>
+ <title>Printing Debug Messages to the Console</title>
<para>To dump debug messages to the console (<literal>/var/log/messages</literal>), set the corresponding debug mask in the <literal>printk</literal> flag:</para>
<screen>sysctl -w lnet.printk=-1 </screen>
<para>This slows down the system dramatically. It is also possible to selectively enable or disable this capability for particular flags using:</para>
<para>It is possible to disable warning, error, and console messages, though it is strongly recommended to have something like <literal>lctl debug_daemon</literal> running to capture this data to a local file system for debugging purposes.</para>
</section>
<section remap="h3">
- <title>28.2.9 Tracing Lock Traffic</title>
+ <title>Tracing Lock Traffic</title>
<para>Lustre has a specific debug type category for tracing lock traffic. Use:</para>
<screen>lctl> filter all_types
lctl> show dlmtrace
</section>
</section>
<section xml:id="dbdoclet.50438274_80443">
- <title>28.3 Lustre Debugging for Developers</title>
+ <title><indexterm><primary>debugging</primary><secondary>developers tools</secondary></indexterm>Lustre Debugging for Developers</title>
<para>The procedures in this section may be useful to developers debugging Lustre code.</para>
<section remap="h3">
- <title>28.3.1 Adding Debugging to the Lustre Source Code</title>
+ <title>Adding Debugging to the Lustre Source Code</title>
<para>The debugging infrastructure provides a number of macros that can be used in Lustre source code to aid in debugging or reporting serious errors.</para>
<para>To use these macros, you will need to set the <literal>DEBUG_SUBSYSTEM</literal> variable at the top of the file as shown below:</para>
<screen>#define DEBUG_SUBSYSTEM S_PORTALS</screen>
<row>
<entry>
<para><literal>
- <replaceable role="bold">OBD_FAIL_RAND</replaceable>
+ <replaceable role="bold">OBD_FAIL_RAND</replaceable>
</literal></para>
</entry>
<entry>
</informaltable>
</section>
<section remap="h3">
- <title>28.3.2 Accessing a <literal>ptlrpc</literal> Request History</title>
+ <title>Accessing a <literal>ptlrpc</literal> Request History</title>
<para>Each service maintains a request history, which can be useful for first occurrence troubleshooting.</para>
<para><literal>ptlrpc</literal> is an RPC protocol layered on LNET that deals with stateful servers and has semantics and built-in support for recovery.</para>
<para>A <literal>prlrpc</literal> request history works as follows:</para>
<para>Request history is accessed and controlled using the following /proc files under the service directory:</para>
<itemizedlist>
<listitem>
- <para>
- <literal>
- req_buffer_history_len
- </literal>
+ <para><literal>req_buffer_history_len </literal>
</para>
<para>Number of request buffers currently in the history</para>
</listitem>
</itemizedlist>
<itemizedlist>
<listitem>
- <para>
- <literal>
- req_buffer_history_max
- </literal>
+ <para><literal>req_buffer_history_max </literal>
</para>
<para>Maximum number of request buffers to keep</para>
</listitem>
</itemizedlist>
<itemizedlist>
<listitem>
- <para>
- <literal>
- req_history
- </literal>
+ <para><literal>req_history </literal>
</para>
<para>The request history</para>
</listitem>
<row>
<entry>
<para>
- <literal><replaceable>xid</replaceable></literal>
+ <literal>
+ <replaceable>xid</replaceable>
+ </literal>
</para>
</entry>
<entry>
</informaltable>
</section>
<section remap="h3">
- <title>28.3.3 Finding Memory Leaks Using <literal>leak_finder.pl</literal></title>
+ <title><indexterm><primary>debugging</primary><secondary>memory leaks</secondary></indexterm>Finding Memory Leaks Using <literal>leak_finder.pl</literal></title>
<para>Memory leaks can occur in code when memory has been allocated and then not freed once it is no longer required. The <literal>leak_finder.pl</literal> program provides a way to find memory leaks.</para>
<para>Before running this program, you must turn on debugging to collect all <literal>malloc</literal> and free entries. Run:</para>
<screen>sysctl -w lnet.debug=+malloc </screen>
<para>Then complete the following steps:</para>
<orderedlist>
<listitem>
-
<para>Dump the log into a user-specified log file using lctl (see <xref linkend="dbdoclet.50438274_62472"/>).</para>
+ <para>Dump the log into a user-specified log file using lctl (see <xref linkend="dbdoclet.50438274_62472"/>).</para>
</listitem>
<listitem>
<para>Run the leak finder on the newly-created log dump:</para>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremaintenance">
- <info>
- <title xml:id="lustremaintenance.title">Lustre Maintenance</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremaintenance">
+ <title xml:id="lustremaintenance.title">Lustre Maintenance</title>
<para>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre maintenance tasks:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438199_42877">
- <title>14.1 Working with Inactive OSTs</title>
+ <title>
+ <indexterm><primary>maintance</primary></indexterm>
+ <indexterm><primary>maintance</primary><secondary>inactive OSTs</secondary></indexterm>
+ Working with Inactive OSTs</title>
<para>To mount a client or an MDT with one or more inactive OSTs, run commands similar to this:</para>
<screen>client> mount -o exclude=testfs-OST0000 -t lustre uml1:/testfs\ /mnt/testfs
client> cat /proc/fs/lustre/lov/testfs-clilov-*/target_obd</screen>
<para>To activate an inactive OST on a live client or MDT, use the <literal>lctl activate</literal> command on the OSC device. For example:</para>
- <screen>lctl --device 7 activate
- </screen>
+ <screen>lctl --device 7 activate</screen>
<note>
- <para>
- A colon-separated list can also be specified. For example, <literal>exclude=testfs-OST0000:testfs-OST0001</literal>.</para>
+ <para>A colon-separated list can also be specified. For example, <literal>exclude=testfs-OST0000:testfs-OST0001</literal>.</para>
</note>
<section xml:id="dbdoclet.50438199_15240">
- <title>14.2 Finding Nodes in the Lustre File System</title>
+ <title><indexterm><primary>maintance</primary><secondary>finding nodes</secondary></indexterm>
+Finding Nodes in the Lustre File System</title>
<para>There may be situations in which you need to find all nodes in your Lustre file system or get the names of all OSTs.</para>
<para>To get a list of all Lustre nodes, run this command on the MGS:</para>
<screen># cat /proc/fs/lustre/mgs/MGS/live/*</screen>
<note>
- <para>
- This command must be run on the MGS.
+ <para>This command must be run on the MGS.
</para>
</note>
<para>In this example, file system lustre has three nodes, <literal>lustre-MDT0000</literal>, <literal>lustre-OST0000</literal>, and <literal>lustre-OST0001</literal>.</para>
<para>To get the names of all OSTs, run this command on the MDS:</para>
<screen># cat /proc/fs/lustre/lov/<fsname>-mdtlov/target_obd </screen>
<note>
- <para>
- This command must be run on the MDS.
+ <para>This command must be run on the MDS.
</para>
</note>
<para>In this example, there are two OSTs, lustre-OST0000 and lustre-OST0001, which are both active.</para>
1: lustre-OST0001_UUID ACTIVE </screen>
</section>
<section xml:id="dbdoclet.50438199_26070">
- <title>14.3 Mounting a Server Without Lustre Service</title>
+ <title><indexterm><primary>maintance</primary><secondary>mounting a server</secondary></indexterm>
+Mounting a Server Without Lustre Service</title>
<para>If you are using a combined MGS/MDT, but you only want to start the MGS and not the MDT, run this command:</para>
<screen>mount -t lustre <MDT partition> -o nosvc <mount point></screen>
<para>The <literal><MDT partition></literal> variable is the combined MGS/MDT.</para>
- <para>In this example, the combined MGS/MDT is <literal>testfs-MDT0000</literal> and the mount point is <literal>/mnt/test/md</literal>t.</para>
+ <para>In this example, the combined MGS/MDT is <literal>testfs-MDT0000</literal> and the mount point is <literal>/mnt/test/mdt</literal>.</para>
<screen>$ mount -t lustre -L testfs-MDT0000 -o nosvc /mnt/test/mdt</screen>
</section>
<section xml:id="dbdoclet.50438199_54623">
- <title>14.4 Regenerating Lustre Configuration Logs</title>
+ <title><indexterm><primary>maintance</primary><secondary>regenerating config logs</secondary></indexterm>
+Regenerating Lustre Configuration Logs</title>
<para>If the Lustre system's configuration logs are in a state where the file system cannot be started, use the <literal>writeconf</literal> command to erase them. After the <literal>writeconf</literal> command is run and the servers restart, the configuration logs are re-generated and stored on the MGS (as in a new file system).</para>
<para>You should only use the <literal>writeconf</literal> command if:</para>
<itemizedlist>
<listitem>
- <para> The configuration logs are in a state where the file system cannot start</para>
+ <para>The configuration logs are in a state where the file system cannot start</para>
</listitem>
<listitem>
- <para> A server NID is being changed</para>
+ <para>A server NID is being changed</para>
</listitem>
</itemizedlist>
<para>The <literal>writeconf</literal> command is destructive to some configuration items (i.e., OST pools information and items set via <literal>conf_param</literal>), and should be used with caution. To avoid problems:</para>
<itemizedlist>
<listitem>
- <para> Shut down the file system before running the <literal>writeconf</literal> command</para>
+ <para>Shut down the file system before running the <literal>writeconf</literal> command</para>
</listitem>
<listitem>
- <para> Run the <literal>writeconf</literal> command on all servers (MDT first, then OSTs)</para>
+ <para>Run the <literal>writeconf</literal> command on all servers (MDT first, then OSTs)</para>
</listitem>
<listitem>
- <para> Start the file system in this order:</para>
+ <para>Start the file system in this order:</para>
<itemizedlist>
<listitem>
- <para> MGS (or the combined MGS/MDT)</para>
+ <para>MGS (or the combined MGS/MDT)</para>
</listitem>
<listitem>
- <para> MDT</para>
+ <para>MDT</para>
</listitem>
<listitem>
- <para> OSTs</para>
+ <para>OSTs</para>
</listitem>
<listitem>
- <para> Lustre clients</para>
+ <para>Lustre clients</para>
</listitem>
</itemizedlist>
</listitem>
</itemizedlist>
<caution>
- <para>
- The OST pools feature enables a group of OSTs to be named for file striping purposes. If you use OST pools, be aware that running the <literal>writeconf</literal> command erases <emphasis role="bold">all</emphasis> pools information (as well as any other parameters set via <literal>lctl conf_param</literal>). We recommend that the pools definitions (and <literal>conf_param</literal> settings) be executed via a script, so they can be reproduced easily after a <literal>writeconf</literal> is performed.</para>
+ <para>The OST pools feature enables a group of OSTs to be named for file striping purposes. If you use OST pools, be aware that running the <literal>writeconf</literal> command erases <emphasis role="bold">all</emphasis> pools information (as well as any other parameters set via <literal>lctl conf_param</literal>). We recommend that the pools definitions (and <literal>conf_param</literal> settings) be executed via a script, so they can be reproduced easily after a <literal>writeconf</literal> is performed.</para>
</caution>
<para>To regenerate Lustre's system configuration logs:</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Shut down the file system in this order.</emphasis></para>
+ <para>Shut down the file system in this order.</para>
<orderedlist>
<listitem>
<para>Unmount the clients.</para>
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Make sure the the MDT and OST devices are available.</emphasis></para>
+ <para>Make sure the the MDT and OST devices are available.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Run the <literal>writeconf</literal> command on all servers.</emphasis></para>
+ <para>Run the <literal>writeconf</literal> command on all servers.</para>
<para>Run writeconf on the MDT first, and then the OSTs.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">On the MDT, run:</emphasis></para>
+ <para>On the MDT, run:</para>
<screen><mdt node>$ tunefs.lustre --writeconf <device></screen>
</listitem>
<listitem>
-<para>
- <emphasis role="bold">
- On each OST, run:
- </emphasis>
+ <para>
+ On each OST, run:
+
<screen><ost node>$ tunefs.lustre --writeconf <device></screen>
</para>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Restart the file system in this order.</emphasis></para>
+ <para>Restart the file system in this order.</para>
<orderedlist>
<listitem>
<para>Mount the MGS (or the combined MGS/MDT).</para>
<para>After the <literal>writeconf</literal> command is run, the configuration logs are re-generated as servers restart.</para>
</section>
<section xml:id="dbdoclet.50438199_31353">
- <title>14.5 Changing a Server NID</title>
+ <title><indexterm><primary>maintance</primary><secondary>changing a NID</secondary></indexterm>
+Changing a Server NID</title>
<para>If you need to change the NID on the MDT or an OST, run the <literal>writeconf</literal> command to erase Lustre configuration information (including server NIDs), and then re-generate the system configuration using updated server NIDs.</para>
<para>Change a server NID in these situations:</para>
<itemizedlist>
<para>To change a server NID:</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Update the LNET configuration in the <literal>/etc/modprobe.conf</literal> file so the list of server NIDs (<literal>lctl list_nids</literal>) is correct.</emphasis></para>
+ <para>Update the LNET configuration in the <literal>/etc/modprobe.conf</literal> file so the list of server NIDs (<literal>lctl list_nids</literal>) is correct.</para>
<para>The <literal>lctl list_nids</literal> command indicates which network(s) are configured to work with Lustre.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Shut down the file system in this order.</emphasis></para>
+ <para>Shut down the file system in this order.</para>
<orderedlist>
<listitem>
<para>Unmount the clients.</para>
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Run the writeconf command on all servers.</emphasis></para>
+ <para>Run the writeconf command on all servers.</para>
<para>Run writeconf on the MDT first, and then the OSTs.</para>
<orderedlist>
<listitem>
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Restart the file system in this order.</emphasis></para>
+ <para>Restart the file system in this order.</para>
<orderedlist>
<listitem>
<para>Mount the MGS (or the combined MGS/MDT).</para>
<para>After the <literal>writeconf</literal> command is run, the configuration logs are re-generated as servers restart, and server NIDs in the updated <literal>list_nids</literal> file are used.</para>
</section>
<section xml:id="dbdoclet.50438199_22527">
- <title>14.6 Adding a New OST to a Lustre File System</title>
+ <title><indexterm><primary>maintance</primary><secondary>adding a OST</secondary></indexterm>
+Adding a New OST to a Lustre File System</title>
<para>To add an OST to existing Lustre file system:</para>
<orderedlist>
<listitem>
- <para> <emphasis role="bold">Add a new OST by passing on the following commands, run:</emphasis></para>
+ <para> Add a new OST by passing on the following commands, run:</para>
<screen>$ mkfs.lustre --fsname=spfs --ost --mgsnode=mds16@tcp0 /dev/sda
$ mkdir -p /mnt/test/ost0
$ mount -t lustre /dev/sda /mnt/test/ost0 </screen>
</listitem>
<listitem>
- <para> <emphasis role="bold">Migrate the data (possibly).</emphasis></para>
+ <para> Migrate the data (possibly).</para>
<para>The file system is quite unbalanced when new empty OSTs are added. New file creations are automatically balanced. If this is a scratch file system or files are pruned at a regular interval, then no further work may be needed.</para>
<para>New files being created will preferentially be placed on the empty OST. As old files are deleted, they will release space on the old OST.</para>
<para>Files existing prior to the expansion can optionally be rebalanced with an in-place copy, which can be done with a simple script. The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
<para>For example, to rebalance all files within <literal>/mnt/lustre/dir</literal>, enter:</para>
<screen>lfs_migrate /mnt/lustre/file</screen>
<para>To migrate files within the <literal>/test</literal> filesystem on <literal>OST0004</literal> that are larger than 4GB in size, enter:</para>
- <screen>lfs find /test -obd test-OST0004 -size +4G | lfs_migrate -y
-</screen>
+ <screen>lfs find /test -obd test-OST0004 -size +4G | lfs_migrate -y</screen>
<para>See <xref linkend="dbdoclet.50438206_42260"/> for more details.</para>
</listitem>
</orderedlist>
</section>
<section xml:id="dbdoclet.50438199_14978">
- <title>14.7 Removing and Restoring OSTs</title>
+ <title><indexterm><primary>maintance</primary><secondary>restoring a OST</secondary></indexterm>
+ <indexterm><primary>maintance</primary><secondary>removing a OST</secondary></indexterm>
+Removing and Restoring OSTs</title>
<para>OSTs can be removed from and restored to a Lustre file system. Currently in Lustre, removing an OST really means that the OST is 'deactivated' in the file system, not permanently removed. A removed OST still appears in the file system; do not create a new OST with the same name.</para>
<para>You may want to remove (deactivate) an OST and prevent new files from being written to it in several situations:</para>
<itemizedlist>
<listitem>
- <para> Hard drive has failed and a RAID resync/rebuild is underway</para>
+ <para>Hard drive has failed and a RAID resync/rebuild is underway</para>
</listitem>
<listitem>
- <para> OST is nearing its space capacity</para>
+ <para>OST is nearing its space capacity</para>
</listitem>
</itemizedlist>
<section remap="h3">
- <title>14.7.1 Removing an OST from the File System</title>
+ <title><indexterm><primary>maintance</primary><secondary>removing a OST</secondary></indexterm>
+ Removing an OST from the File System</title>
<para>OSTs can be removed from a Lustre file system. Currently in Lustre, removing an OST actually means that the OST is 'deactivated' from the file system, not permanently removed. A removed OST still appears in the device listing; you should not normally create a new OST with the same name.</para>
<para>You may want to deactivate an OST and prevent new files from being written to it in several situations:</para>
<itemizedlist>
<listitem>
- <para> OST is nearing its space capacity</para>
+ <para>OST is nearing its space capacity</para>
</listitem>
<listitem>
- <para> Hard drive has failed and a RAID resync/rebuild is underway</para>
+ <para>Hard drive has failed and a RAID resync/rebuild is underway</para>
</listitem>
<listitem>
- <para> OST storage has failed permanently</para>
+ <para>OST storage has failed permanently</para>
</listitem>
</itemizedlist>
<para>When removing an OST, remember that the MDT does not communicate directly with OSTs. Rather, each OST has a corresponding OSC which communicates with the MDT. It is necessary to determine the device number of the OSC that corresponds to the OST. Then, you use this device number to deactivate the OSC on the MDT.</para>
<para>To remove an OST from the file system:</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">For the OST to be removed, determine the device number of the corresponding OSC on the MDT.</emphasis></para>
+ <para>For the OST to be removed, determine the device number of the corresponding OSC on the MDT.</para>
<orderedlist>
<listitem>
<para>List all OSCs on the node, along with their device numbers. Run:</para>
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Temporarily deactivate the OSC on the MDT. On the MDT, run:</emphasis> </para>
+ <para>Temporarily deactivate the OSC on the MDT. On the MDT, run: </para>
<screen>$ mdt> lctl --device >devno< deactivate</screen>
- <para>
-For example, based on the command output in Step 1, to deactivate device 13 (the MDT’s OSC for <literal>OST-0000</literal>), the command would be:
+ <para>For example, based on the command output in Step 1, to deactivate device 13 (the MDT’s OSC for <literal>OST-0000</literal>), the command would be:
</para>
<screen>$ mdt> lctl --device 13 deactivate</screen>
- <para>
-This marks the OST as inactive on the MDS, so no new objects are assigned to the OST. This does not prevent use of existing objects for reads or writes.
+ <para>This marks the OST as inactive on the MDS, so no new objects are assigned to the OST. This does not prevent use of existing objects for reads or writes.
</para>
<note>
<para>Do not deactivate the OST on the clients. Do so causes errors (EIOs), and the copy out to fail. </para>
</caution>
</listitem>
<listitem>
- <para>
-<emphasis role="bold">Discover all files that have objects residing on the deactivated OST. </emphasis></para>
+ <para>Discover all files that have objects residing on the deactivated OST. </para>
<para>Depending on whether the deactivated OST is available or not, the data from that OST may be migrated to other OSTs, or may need to be restored from backup.
</para>
<orderedlist>
<listitem>
- <para>
-If the OST is still online and available, find all files with objects on the deactivated OST, and copy them to other OSTs in the file system to:
+ <para>If the OST is still online and available, find all files with objects on the deactivated OST, and copy them to other OSTs in the file system to:
</para>
<screen>[client]# lfs find --obd <OST UUID> <mount_point> | lfs_migrate -y</screen>
</listitem>
- <listitem> <para>If the OST is no longer available, delete the files on that OST and restore them from backup: <screen>[client]# lfs find --obd <OST UUID> -print0 <mount_point> | \
- tee /tmp/files_to_restore | xargs -0 -n 1 unlink</screen> The list of files that need to be restored from backup is stored in <literal>/tmp/files_to_restore</literal>. Restoring these files is beyond the scope of this document. </para></listitem>
+ <listitem>
+ <para>If the OST is no longer available, delete the files on that OST and restore them from backup: <screen>[client]# lfs find --obd <OST UUID> -print0 <mount_point> | \
+ tee /tmp/files_to_restore | xargs -0 -n 1 unlink</screen>The list of files that need to be restored from backup is stored in <literal>/tmp/files_to_restore</literal>. Restoring these files is beyond the scope of this document. </para>
+ </listitem>
</orderedlist>
</listitem>
- <listitem> <para><emphasis role="bold">Deactivate the OST.</emphasis></para> <orderedlist>
- <listitem> <para>To temporarily disable the deactivated OST, enter: <screen>[client]# lctl set_param osc.<fsname>-<OST name>-*.active=0</screen>If there is expected to be a replacement OST in some short time (a few days), the OST can temporarily be deactivated on the clients: <note>
- <para>This setting is only temporary and will be reset if the clients or MDS are rebooted. It needs to be run on all clients.</para>
- </note></para></listitem>
- <listitem><para>To permanently disable the deactivated OST, enter: <screen>[mgs]# lctl conf_param {OST name}.osc.active=0</screen></para></listitem>
- </orderedlist><para>If there is not expected to be a replacement for this OST in the near future, permanently deactivate the OST on all clients and the MDS: </para><note>
+ <listitem>
+ <para>Deactivate the OST.</para>
+ <orderedlist>
+ <listitem>
+ <para>To temporarily disable the deactivated OST, enter: <screen>[client]# lctl set_param osc.<fsname>-<OST name>-*.active=0</screen>If there is expected to be a replacement OST in some short time (a few days), the OST can temporarily be deactivated on the clients: <note>
+ <para>This setting is only temporary and will be reset if the clients or MDS are rebooted. It needs to be run on all clients.</para>
+ </note></para>
+ </listitem>
+ <listitem>
+ <para>To permanently disable the deactivated OST, enter: <screen>[mgs]# lctl conf_param {OST name}.osc.active=0</screen></para>
+ </listitem>
+ </orderedlist>
+ <para>If there is not expected to be a replacement for this OST in the near future, permanently deactivate the OST on all clients and the MDS: </para>
+ <note>
<para>A removed OST still appears in the file system; do not create a new OST with the same name.</para>
- </note></listitem>
+ </note>
+ </listitem>
</orderedlist>
</section>
- <section remap="h3"><title>14.7.2 Backing Up OST Configuration Files</title><para>If the OST device is still accessible, then the Lustre configuration files on the OST should be backed up and saved for future use in order to avoid difficulties when a replacement OST is returned to service. These files rarely change, so they can and should be backed up while the OST is functional and accessible. If the deactivated OST is still available to mount (i.e. has not permanently failed or is unmountable due to severe corruption), an effort should be made to preserve these files. </para><orderedlist>
+ <section remap="h3">
+ <title><indexterm><primary>maintance</primary><secondary>backing up OST config</secondary></indexterm>
+<indexterm><primary>backup</primary><secondary>OST config</secondary></indexterm>
+Backing Up OST Configuration Files</title>
+ <para>If the OST device is still accessible, then the Lustre configuration files on the OST should be backed up and saved for future use in order to avoid difficulties when a replacement OST is returned to service. These files rarely change, so they can and should be backed up while the OST is functional and accessible. If the deactivated OST is still available to mount (i.e. has not permanently failed or is unmountable due to severe corruption), an effort should be made to preserve these files. </para>
+ <orderedlist>
<listitem>
- <para>
- <emphasis role="bold">Mount the OST filesystem.</emphasis>
+ <para>Mount the OST filesystem.
<screen>[oss]# mkdir -p /mnt/ost
[oss]# mount -t ldiskfs {ostdev} /mnt/ost</screen>
</para>
</listitem>
<listitem>
- <para>
- <emphasis role="bold">Back up the OST configuration files.</emphasis>
+ <para>Back up the OST configuration files.
<screen>[oss]# tar cvf {ostname}.tar -C /mnt/ost last_rcvd \
CONFIGS/ O/0/LAST_ID</screen>
</para>
</listitem>
<listitem>
- <emphasis role="bold">Unmount the OST filesystem.</emphasis>
+ <para>
+ Unmount the OST filesystem.
<screen>[oss]# umount /mnt/ost</screen>
</para>
</listitem>
- </orderedlist></section>
- <section><title>14.7.3 Restoring OST Configuration Files</title>
-
- <para>
- If the original OST is still available, it is best to follow the OST backup and restore procedure given in either <xref linkend='dbdoclet.50438207_71633'/>, or <xref linkend='dbdoclet.50438207_21638'/> and <xref linkend='dbdoclet.50438207_22325'/>.
+ </orderedlist>
+ </section>
+ <section>
+ <title><indexterm><primary>maintance</primary><secondary>restoring OST config</secondary></indexterm>
+<indexterm><primary>backup</primary><secondary>restoring OST config</secondary></indexterm>
+Restoring OST Configuration Files</title>
+ <para>If the original OST is still available, it is best to follow the OST backup and restore procedure given in either <xref linkend="dbdoclet.50438207_71633"/>, or <xref linkend="dbdoclet.50438207_21638"/> and <xref linkend="dbdoclet.50438207_22325"/>.
</para>
-
<para>To replace an OST that was removed from service due to corruption or hardware failure, the file system needs to be formatted for Lustre, and the Lustre configuration should be restored, if available.
</para>
-
<para>If the OST configuration files were not backed up, due to the OST file system being completely inaccessible, it is still possible to replace the failed OST with a new one at the same OST index. </para>
- <orderedlist>
+ <para>If the OST configuration files were not backed up, due to the OST file system being completely inaccessible, it is still possible to replace the failed OST with a new one at the same OST index. </para>
+ <orderedlist>
<listitem>
<para>
- <emphasis role="bold">Format the OST file system. </emphasis>
+ Format the OST file system.
<screen>[oss]# mkfs.lustre --ost --index {OST index} {other options} \
{newdev}</screen>
</para>
</listitem>
<listitem>
<para>
- <emphasis role="bold">Mount the OST filesystem. </emphasis>
+ Mount the OST filesystem.
<screen>[oss]# mkdir /mnt/ost
[oss]# mount -t ldiskfs {newdev} /mnt/ost</screen>
</para>
</listitem>
<listitem>
- <para>
- <emphasis role="bold">Restore the OST configuration files, if available. </emphasis>
- <screen>[oss]# tar xvf {ostname}.tar -C /mnt/ost</screen>
- </para>
+ <para>Restore the OST configuration files, if available.
+ <screen>[oss]# tar xvf {ostname}.tar -C /mnt/ost</screen></para>
</listitem>
<listitem>
- <para>
- <emphasis role="bold">Recreate the OST configuration files, if unavailable.</emphasis>
+ <para>Recreate the OST configuration files, if unavailable.
</para>
- <para>Follow the procedure in <xref linkend='dbdoclet.50438198_69657'/> to recreate the LAST_ID file for this OST index. The <literal>last_rcvd</literal> file will be recreated when the OST is first mounted using the default parameters, which are normally correct for all file systems.
+ <para>Follow the procedure in <xref linkend="dbdoclet.50438198_69657"/> to recreate the LAST_ID file for this OST index. The <literal>last_rcvd</literal> file will be recreated when the OST is first mounted using the default parameters, which are normally correct for all file systems.
The <literal>CONFIGS/mountdata</literal> file is created by <literal>mkfs.lustre</literal> at format time, but has flags set that request it to register itself with the MGS. It is possible to copy these flags from another working OST (which should be the same):
-<screen>[oss2]# debugfs -c -R "dump CONFIGS/mountdata /tmp/ldd" \
-{other_osdev}
+<screen>[oss2]# debugfs -c -R "dump CONFIGS/mountdata /tmp/ldd" {other_osdev}
[oss2]# scp /tmp/ldd oss:/tmp/ldd
-[oss]# dd if=/tmp/ldd of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 \
-seek=5 skip=5</screen></para>
- </listitem>
+[oss]# dd if=/tmp/ldd of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 skip=5</screen></para>
+ </listitem>
<listitem>
<para>
- <emphasis role="bold">Unmount the OST filesystem.</emphasis>
+ Unmount the OST filesystem.
<screen>[oss]# umount /mnt/ost</screen>
</para>
</listitem>
- </orderedlist></section>
- <section><title>14.7.4 Returning a Deactivated OST to Service</title><para> If the OST was permanently deactivated, it needs to be reactivated in the MGS configuration. <screen>[mgs]# lctl conf_param {OST name}.osc.active=1</screen> If the OST was temporarily deactivated, it needs to be reactivated on the MDS and clients. <screen>[mds]# lctl --device <devno> activate
- [client]# lctl set_param osc.<fsname>-<OST name>-*.active=0</screen></para></section>
+ </orderedlist>
+ </section>
+ <section>
+ <title><indexterm><primary>maintance</primary><secondary>reintroducing an OSTs</secondary></indexterm>
+Returning a Deactivated OST to Service</title>
+ <para> If the OST was permanently deactivated, it needs to be reactivated in the MGS configuration. <screen>[mgs]# lctl conf_param {OST name}.osc.active=1</screen> If the OST was temporarily deactivated, it needs to be reactivated on the MDS and clients. <screen>[mds]# lctl --device <devno> activate
+ [client]# lctl set_param osc.<fsname>-<OST name>-*.active=0</screen></para>
+ </section>
</section>
- <section xml:id="dbdoclet.50438199_77819"><title>14.8 Aborting Recovery</title><para>You can abort recovery with either the <literal>lctl</literal> utility or by mounting the target with the <literal>abort_recov</literal> option (<literal>mount -o abort_recov</literal>). When starting a target, run: <screen>$ mount -t lustre -L <MDT name> -o abort_recov <mount_point></screen> Note - The recovery process is blocked until all OSTs are available. </para></section>
- <section xml:id="dbdoclet.50438199_12607"><title>14.9 Determining Which Machine is Serving an OST </title> <para>In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the NID that is serving a specific OST, run one of the following commands on a client (you do not need to be a root user): <screen>client$ lctl get_param osc.${fsname}-${OSTname}*.ost_conn_uuid</screen> For example: <screen>client$ lctl get_param osc.*-OST0000*.ost_conn_uuid
-osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen> - OR - <screen>client$ lctl get_param osc.*.ost_conn_uuid
+ <section xml:id="dbdoclet.50438199_77819">
+ <title><indexterm><primary>maintance</primary><secondary>aborting recovery</secondary></indexterm>
+ <indexterm><primary>backup</primary><secondary>aborting recovery</secondary></indexterm>
+Aborting Recovery</title>
+ <para>You can abort recovery with either the <literal>lctl</literal> utility or by mounting the target with the <literal>abort_recov</literal> option (<literal>mount -o abort_recov</literal>). When starting a target, run: <screen>$ mount -t lustre -L <MDT name> -o abort_recov <mount_point></screen></para>
+ <note>
+ <para>The recovery process is blocked until all OSTs are available. </para>
+ </note>
+ </section>
+ <section xml:id="dbdoclet.50438199_12607">
+ <title><indexterm><primary>maintance</primary><secondary>identifying OST host</secondary></indexterm>
+Determining Which Machine is Serving an OST </title>
+ <para>In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the NID that is serving a specific OST, run one of the following commands on a client (you do not need to be a root user): <screen>client$ lctl get_param osc.${fsname}-${OSTname}*.ost_conn_uuid</screen>For example: <screen>client$ lctl get_param osc.*-OST0000*.ost_conn_uuid
+osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen>- OR - <screen>client$ lctl get_param osc.*.ost_conn_uuid
osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
osc.lustre-OST0001-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
osc.lustre-OST0002-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
osc.lustre-OST0003-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-osc.lustre-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen></para></section>
- <section xml:id="dbdoclet.50438199_62333"><title>14.10 Changing the Address of a Failover Node</title><para>To change the address of a failover node (e.g, to use node X instead of node Y), run this command on the OSS/OST partition: <screen>tunefs.lustre --erase-params --failnode=<NID> <device> </screen></para></section>
+osc.lustre-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen></para>
+ </section>
+ <section xml:id="dbdoclet.50438199_62333">
+ <title><indexterm><primary>maintance</primary><secondary>changing failover node address</secondary></indexterm>
+Changing the Address of a Failover Node</title>
+ <para>To change the address of a failover node (e.g, to use node X instead of node Y), run this command on the OSS/OST partition: <screen>tunefs.lustre --erase-params --failnode=<NID> <device> </screen></para>
+ </section>
</section>
</chapter>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremonitoring">
- <info>
- <title xml:id="lustremonitoring.title">Lustre Monitoring</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremonitoring">
+ <title xml:id="lustremonitoring.title">Lustre Monitoring</title>
<para>This chapter provides information on monitoring Lustre and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438273_18711">
- <title>12.1 Lustre <anchor xml:id="dbdoclet.50438273_marker-1297383" xreflabel=""/>Changelogs</title>
+ <title><indexterm><primary>change logs</primary><see>monitoring</see></indexterm>
+<indexterm><primary>monitoring</primary></indexterm>
+<indexterm><primary>monitoring</primary><secondary>change logs</secondary></indexterm>
+
+Lustre Changelogs</title>
<para>The changelogs feature records events that change the file system namespace or file metadata. Changes such as file creation, deletion, renaming, attribute changes, etc. are recorded with the target and parent file identifiers (FIDs), the name of the target, and a timestamp. These records can be used for a variety of purposes:</para>
<itemizedlist>
<listitem>
- <para> Capture recent changes to feed into an archiving system.</para>
+ <para>Capture recent changes to feed into an archiving system.</para>
</listitem>
<listitem>
- <para> Use changelog entries to exactly replicate changes in a file system mirror.</para>
+ <para>Use changelog entries to exactly replicate changes in a file system mirror.</para>
</listitem>
<listitem>
- <para> Set up "watch scripts" that take action on certain events or directories.</para>
+ <para>Set up "watch scripts" that take action on certain events or directories.</para>
</listitem>
<listitem>
- <para> Maintain a rough audit trail (file/directory changes with timestamps, but no user information).</para>
+ <para>Maintain a rough audit trail (file/directory changes with timestamps, but no user information).</para>
</listitem>
</itemizedlist>
<para>Changelogs record types are:</para>
</informaltable>
<para>FID-to-full-pathname and pathname-to-FID functions are also included to map target and parent FIDs into the file system namespace.</para>
<section remap="h3">
- <title>12.1.1 Working with Changelogs</title>
+ <title><indexterm><primary>monitoring</primary><secondary>change logs</secondary></indexterm>
+Working with Changelogs</title>
<para>Several commands are available to work with changelogs.</para>
<section remap="h5">
- <title>lctl changelog_register</title>
+ <title>
+ <literal>lctl changelog_register</literal>
+ </title>
<para>Because changelog records take up space on the MDT, the system administration must register changelog users. The registrants specify which records they are "done with", and the system purges up to the greatest common record.</para>
<para>To register a new changelog user, run:</para>
<screen>lctl --device <mdt_device> changelog_register
<para>Changelog entries are not purged beyond a registered user's set point (see <literal>lfs changelog_clear</literal>).</para>
</section>
<section remap="h5">
- <title>lfs changelog</title>
+ <title>
+ <literal>lfs changelog</literal>
+ </title>
<para>To display the metadata changes on an MDT (the changelog records), run:</para>
- <screen>lfs changelog <MDT name> [startrec [endrec]]
-</screen>
+ <screen>lfs changelog <MDT name> [startrec [endrec]] </screen>
<para>It is optional whether to specify the start and end records.</para>
<para>These are sample changelog records:</para>
<screen>2 02MKDIR 4298396676 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0]\
4 06UNLNK 4298404466 0x0 t=[0x200000405:0x15fa:0x0] p=[0x200000405:0x15f9:0\
x0] chloe.jpg
5 07RMDIR 4298405394 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0]\
- pics
-</screen>
+ pics </screen>
</section>
<section remap="h5">
- <title>lfs changelog_clear</title>
+ <title>
+ <literal>lfs changelog_clear</literal>
+ </title>
<para>To clear old changelog records for a specific user (records that the user no longer needs), run:</para>
- <screen>lfs changelog_clear <MDT name> <user ID> <endrec>
-</screen>
+ <screen>lfs changelog_clear <MDT name> <user ID> <endrec></screen>
<para>The <literal>changelog_clear</literal> command indicates that changelog records previous to <endrec> are no longer of interest to a particular user <user ID>, potentially allowing the MDT to free up disk space. An <literal><endrec></literal> value of 0 indicates the current last record. To run <literal>changelog_clear</literal>, the changelog user must be registered on the MDT node using <literal>lctl</literal>.</para>
<para>When all changelog users are done with records < X, the records are deleted.</para>
</section>
<section remap="h5">
- <title>lctl changelog_deregister</title>
+ <title>
+ <literal>lctl changelog_deregister</literal>
+ </title>
<para>To deregister (unregister) a changelog user, run:</para>
<screen>lctl --device <mdt_device> changelog_deregister <user ID> </screen>
<para> <literal>changelog_deregister cl1</literal> effectively does a <literal>changelog_clear cl1 0</literal> as it deregisters.</para>
</section>
</section>
<section remap="h3">
- <title>12.1.2 Changelog Examples</title>
+ <title>Changelog Examples</title>
<para>This section provides examples of different changelog commands.</para>
<section remap="h5">
<title>Registering a Changelog User</title>
<para>To register a new changelog user for a device (<literal>lustre-MDT0000</literal>):</para>
<screen># lctl --device lustre-MDT0000 changelog_register
-lustre-MDT0000: Registered changelog userid 'cl1'
-</screen>
+lustre-MDT0000: Registered changelog userid 'cl1'</screen>
</section>
<section remap="h5">
<title>Displaying Changelog Records</title>
b4:0xca2c7dde:0x0] mydir
3 14SATTR 19:10:27.329356533 2010.03.24 0x0 t=[0x200000420:0x3:0x0]
4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
-0000420:0x3:0x0] hosts
-</screen>
+0000420:0x3:0x0] hosts </screen>
<para>Changelog records include this information:</para>
<screen>rec#
operation_type(numerical/text)
flags
t=target_FID
p=parent_FID
-target_name
-</screen>
+target_name</screen>
<para>Displayed in this format:</para>
<screen>rec# operation_type(numerical/text) timestamp datestamp flags t=target_FID \
-p=parent_FID target_name
-</screen>
+p=parent_FID target_name</screen>
<para>For example:</para>
<screen>4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
-0000420:0x3:0x0] hosts
-</screen>
+0000420:0x3:0x0] hosts</screen>
</section>
<section remap="h5">
<title>Clearing Changelog Records</title>
<para>To notify a device that a specific user (<literal>cl1</literal>) no longer needs records (up to and including 3):</para>
- <screen>$ lfs changelog_clear lustre-MDT0000 cl1 3
-</screen>
+ <screen>$ lfs changelog_clear lustre-MDT0000 cl1 3</screen>
<para>To confirm that the <literal>changelog_clear</literal> operation was successful, run <literal>lfs changelog</literal>; only records after id-3 are listed:</para>
<screen>$ lfs changelog lustre-MDT0000
4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
-0000420:0x3:0x0] hosts
-</screen>
+0000420:0x3:0x0] hosts</screen>
</section>
<section remap="h5">
<title>Deregistering a Changelog User</title>
<para>To deregister a changelog user (<literal>cl1</literal>) for a specific device (<literal>lustre-MDT0000</literal>):</para>
<screen># lctl --device lustre-MDT0000 changelog_deregister cl1
-lustre-MDT0000: Deregistered changelog user 'cl1'
-</screen>
+lustre-MDT0000: Deregistered changelog user 'cl1'</screen>
<para>The deregistration operation clears all changelog records for the specified user (<literal>cl1</literal>).</para>
<screen>$ lfs changelog lustre-MDT0000
5 00MARK 19:13:40.858292517 2010.03.24 0x0 t=[0x40001:0x0:0x0] p=[0:0x0:0x\
0] mdd_obd-lustre-MDT0000-0
</screen>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry>
- <para><emphasis role="bold">Note -</emphasis>MARK records typically indicate changelog recording status changes.</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note>
+ <para>MARK records typically indicate changelog recording status changes.</para>
+ </note>
</section>
<section remap="h5">
<title>Displaying the Changelog Index and Registered Users</title>
$ cp /etc/hosts /mnt/lustre/mydir/foo/file
$ ln /mnt/lustre/mydir/foo/file /mnt/lustre/mydir/myhardlink
</screen>
- <para> Only item types that are in the mask show up in the changelog.</para>
+ <para>Only item types that are in the mask show up in the changelog.</para>
<screen>$ lfs changelog lustre-MDT0000
9 03HLINK 19:19:35.171867477 2010.03.24 0x0 t=[0x200000420:0x6:0x0] p=[0x20\
0000420:0x3:0x0] myhardlink
</section>
</section>
<section xml:id="dbdoclet.50438273_81684">
- <title>12.2 Lustre <anchor xml:id="dbdoclet.50438273_marker-1297386" xreflabel=""/>Monitoring Tool</title>
+ <title><indexterm><primary>monitoring</primary><secondary>Lustre Monitoring Tool</secondary></indexterm>
+Lustre Monitoring Tool</title>
<para>The Lustre Monitoring Tool (LMT) is a Python-based, distributed system developed and maintained by Lawrence Livermore National Lab (LLNL)). It provides a ''top'' like display of activity on server-side nodes (MDS, OSS and portals routers) on one or more Lustre file systems. It does not provide support for monitoring clients. For more information on LMT, including the setup procedure, see:</para>
<para><link xl:href="http://code.google.com/p/lmt/">http://code.google.com/p/lmt/</link></para>
<para>LMT questions can be directed to:</para>
<para><link xl:href="mailto:lmt-discuss@googlegroups.com">lmt-discuss@googlegroups.com</link></para>
</section>
<section xml:id="dbdoclet.50438273_80593">
- <title>12.3 Collect<anchor xml:id="dbdoclet.50438273_marker-1297391" xreflabel=""/>L</title>
- <para>CollectL is another tool that can be used to monitor Lustre. You can run CollectL on a Lustre system that has any combination of MDSs, OSTs and clients. The collected data can be written to a file for continuous logging and played back at a later time. It can also be converted to a format suitable for plotting.</para>
- <para>For more information about CollectL, see:</para>
+ <title>
+ <literal>CollectL</literal>
+ </title>
+ <para><literal>CollectL</literal> is another tool that can be used to monitor Lustre. You can run <literal>CollectL</literal> on a Lustre system that has any combination of MDSs, OSTs and clients. The collected data can be written to a file for continuous logging and played back at a later time. It can also be converted to a format suitable for plotting.</para>
+ <para>For more information about <literal>CollectL</literal>, see:</para>
<para><link xl:href="http://collectl.sourceforge.net">http://collectl.sourceforge.net</link></para>
<para>Lustre-specific documentation is also available. See:</para>
<para><link xl:href="http://collectl.sourceforge.net/Tutorial-Lustre.html">http://collectl.sourceforge.net/Tutorial-Lustre.html</link></para>
</section>
<section xml:id="dbdoclet.50438273_44185">
- <title>12.4 Other Monitoring Options</title>
+ <title><indexterm><primary>monitoring</primary><secondary>additional tools</secondary></indexterm>
+Other Monitoring Options</title>
<para>A variety of standard tools are available publically.</para>
- <para>Another option is to script a simple monitoring solution that looks at various reports from ipconfig, as well as the procfs files generated by Lustre.</para>
- <para><anchor xml:id="dbdoclet.50438273_67514" xreflabel=""/> </para>
+ <para>Another option is to script a simple monitoring solution that looks at various reports from <literal>ipconfig</literal>, as well as the <literal>procfs</literal> files generated by Lustre.</para>
</section>
</chapter>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustreoperations">
- <info>
- <title xml:id="lustreoperations.title">Lustre Operations</title>
- </info>
+ <title xml:id="lustreoperations.title">Lustre Operations</title>
<para>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre administration tasks:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438194_42877">
- <title>13.1 Mounting by Label</title>
+ <title><indexterm><primary>operations</primary></indexterm>
+<indexterm><primary>operations</primary><secondary>mounting by label</secondary></indexterm>
+Mounting by Label</title>
<para>The file system name is limited to 8 characters. We have encoded the file system and target information in the disk label, so you can mount by label. This allows system administrators to move disks around without worrying about issues such as SCSI disk reordering or getting the <literal>/dev/device</literal> wrong for a shared target. Soon, file system naming will be made as fail-safe as possible. Currently, Linux disk labels are limited to 16 characters. To identify the target within the file system, 8 characters are reserved, leaving 8 characters for the file system name:</para>
- <para><fsname>-MDT0000 or <fsname>-OST0a19</para>
+ <screen><fsname>-MDT0000 or <fsname>-OST0a19</screen>
<para>To mount by label, use this command:</para>
- <screen>$ mount -t lustre -L <file system label> <mount point>
-</screen>
+ <screen>$ mount -t lustre -L <file system label> <mount point></screen>
<para>This is an example of mount-by-label:</para>
- <screen>$ mount -t lustre -L testfs-MDT0000 /mnt/mdt
-</screen>
+ <screen>$ mount -t lustre -L testfs-MDT0000 /mnt/mdt</screen>
<caution>
<para>Mount-by-label should NOT be used in a multi-path environment.</para>
</caution>
<para>Although the file system name is internally limited to 8 characters, you can mount the clients at any mount point, so file system users are not subjected to short names. Here is an example:</para>
- <screen>mount -t lustre uml1@tcp0:/shortfs /mnt/<long-file_system-name>
-</screen>
+ <screen>mount -t lustre uml1@tcp0:/shortfs /mnt/<long-file_system-name></screen>
</section>
<section xml:id="dbdoclet.50438194_24122">
- <title>13.2 Starting <anchor xml:id="dbdoclet.50438194_marker-1305696" xreflabel=""/>Lustre</title>
+ <title><indexterm><primary>operations</primary><secondary>starting</secondary></indexterm>Starting Lustre</title>
<para>The startup order of Lustre components depends on whether you have a combined MGS/MDT or these components are separate.</para>
<itemizedlist>
<listitem>
- <para> If you have a combined MGS/MDT, the recommended startup order is OSTs, then the MGS/MDT, and then clients.</para>
- </listitem>
- <listitem>
- <para> </para>
+ <para>If you have a combined MGS/MDT, the recommended startup order is OSTs, then the MGS/MDT, and then clients.</para>
</listitem>
<listitem>
<para> If the MGS and MDT are separate, the recommended startup order is: MGS, then OSTs, then the MDT, and then clients.</para>
</listitem>
- <listitem>
- <para> </para>
- </listitem>
</itemizedlist>
<note>
<para>If an OST is added to a Lustre file system with a combined MGS/MDT, then the startup order changes slightly; the MGS must be started first because the OST needs to write its configuration data to it. In this scenario, the startup order is MGS/MDT, then OSTs, then the clients.</para>
</note>
</section>
<section xml:id="dbdoclet.50438194_84876">
- <title>13.3 Mounting a <anchor xml:id="dbdoclet.50438194_marker-1298863" xreflabel=""/>Server</title>
+ <title><indexterm><primary>operations</primary><secondary>mounting</secondary></indexterm>Mounting a Server</title>
<para>Starting a Lustre server is straightforward and only involves the mount command. Lustre servers can be added to <literal>/etc/fstab</literal>:</para>
- <screen>mount -t lustre
-</screen>
+ <screen>mount -t lustre</screen>
<para>The mount command generates output similar to this:</para>
<screen>/dev/sda1 on /mnt/test/mdt type lustre (rw)
/dev/sda2 on /mnt/test/ost0 type lustre (rw)
-192.168.0.21@tcp:/testfs on /mnt/testfs type lustre (rw)
-</screen>
+192.168.0.21@tcp:/testfs on /mnt/testfs type lustre (rw)</screen>
<para>In this example, the MDT, an OST (ost0) and file system (testfs) are mounted.</para>
<screen>LABEL=testfs-MDT0000 /mnt/test/mdt lustre defaults,_netdev,noauto 0 0
-LABEL=testfs-OST0000 /mnt/test/ost0 lustre defaults,_netdev,noauto 0 0
-</screen>
+LABEL=testfs-OST0000 /mnt/test/ost0 lustre defaults,_netdev,noauto 0 0</screen>
<para>In general, it is wise to specify noauto and let your high-availability (HA) package manage when to mount the device. If you are not using failover, make sure that networking has been started before mounting a Lustre server. RedHat, SuSE, Debian (and perhaps others) use the <literal>_netdev</literal> flag to ensure that these disks are mounted after the network is up.</para>
<para>We are mounting by disk label here--the label of a device can be read with <literal>e2label</literal>. The label of a newly-formatted Lustre server ends in <literal>FFFF</literal>, meaning that it has yet to be assigned. The assignment takes place when the server is first started, and the disk label is updated.</para>
<caution>
</caution>
</section>
<section xml:id="dbdoclet.50438194_69255">
- <title>13.4 Unmounting a<anchor xml:id="dbdoclet.50438194_marker-1298879" xreflabel=""/> Server</title>
+ <title><indexterm><primary>operations</primary><secondary>unmounting</secondary></indexterm>Unmounting a Server</title>
<para>To stop a Lustre server, use the <literal>umount <mount point></literal> command.</para>
<para>For example, to stop <literal>ost0</literal> on mount point <literal>/mnt/test</literal>, run:</para>
- <screen>$ umount /mnt/test
-</screen>
+ <screen>$ umount /mnt/test</screen>
<para>Gracefully stopping a server with the <literal>umount</literal> command preserves the state of the connected clients. The next time the server is started, it waits for clients to reconnect, and then goes through the recovery procedure.</para>
<para>If the force (<literal>-f</literal>) flag is used, then the server evicts all clients and stops WITHOUT recovery. Upon restart, the server does not wait for recovery. Any currently connected clients receive I/O errors until they reconnect.</para>
<note>
</note>
</section>
<section xml:id="dbdoclet.50438194_57420">
- <title>13.5 Specifying Fail<anchor xml:id="dbdoclet.50438194_marker-1298926" xreflabel=""/>out/Failover Mode for OSTs</title>
+ <title><indexterm><primary>operations</primary><secondary>failover</secondary></indexterm>Specifying Failout/Failover Mode for OSTs</title>
<para>Lustre uses two modes, failout and failover, to handle an OST that has become unreachable because it fails, is taken off the network, is unmounted, etc.</para>
<itemizedlist>
<listitem>
<para> In <emphasis>failout</emphasis> mode, Lustre clients immediately receive errors (EIOs) after a timeout, instead of waiting for the OST to recover.</para>
</listitem>
<listitem>
- <para> </para>
- </listitem>
- <listitem>
<para> In <emphasis>failover</emphasis> mode, Lustre clients wait for the OST to recover.</para>
</listitem>
- <listitem>
- <para> </para>
- </listitem>
</itemizedlist>
<para>By default, the Lustre file system uses failover mode for OSTs. To specify failout mode instead, run this command:</para>
- <screen>$ mkfs.lustre --fsname=<fsname> --ost --mgsnode=<MGS node NID> --param="failover\
-.mode=failout" <block device name>
-</screen>
+ <screen>$ mkfs.lustre --fsname=<fsname> --ost --mgsnode=<MGS node NID> --param="failover.mode=failout" <block device name></screen>
<para>In this example, failout mode is specified for the OSTs on MGS <literal>uml1</literal>, file system <literal>testfs</literal>.</para>
- <screen>$ mkfs.lustre --fsname=testfs --ost --mgsnode=uml1 --param="failover.mode=fa\
-ilout" /dev/sdb
-</screen>
+ <screen>$ mkfs.lustre --fsname=testfs --ost --mgsnode=uml1 --param="failover.mode=failout" /dev/sdb </screen>
<caution>
<para>Before running this command, unmount all OSTs that will be affected by the change in the failover/failout mode.</para>
</caution>
</note>
</section>
<section xml:id="dbdoclet.50438194_54138">
- <title>13.6 Handling <anchor xml:id="dbdoclet.50438194_marker-1307136" xreflabel=""/>Degraded OST RAID Arrays</title>
+ <title><indexterm><primary>operations</primary><secondary>degraded OST RAID</secondary></indexterm>Handling Degraded OST RAID Arrays</title>
<para>Lustre includes functionality that notifies Lustre if an external RAID array has degraded performance (resulting in reduced overall file system performance), either because a disk has failed and not been replaced, or because a disk was replaced and is undergoing a rebuild. To avoid a global performance slowdown due to a degraded OST, the MDS can avoid the OST for new object allocation if it is notified of the degraded state.</para>
<para>A parameter for each OST, called <literal>degraded</literal>, specifies whether the OST is running in degraded mode or not.</para>
<para>To mark the OST as degraded, use:</para>
<para>It is recommended that this be implemented by an automated script that monitors the status of individual RAID devices.</para>
</section>
<section xml:id="dbdoclet.50438194_88063">
- <title>13.7 Running Multiple<anchor xml:id="dbdoclet.50438194_marker-1298939" xreflabel=""/> Lustre File Systems</title>
+ <title><indexterm><primary>operations</primary><secondary>multiple file systems</secondary></indexterm>Running Multiple Lustre File Systems</title>
<para>There may be situations in which you want to run multiple file systems. This is doable, as long as you follow specific naming conventions.</para>
<para>By default, the <literal>mkfs.lustre</literal> command creates a file system named <literal>lustre</literal>. To specify a different file system name (limited to 8 characters), run this command:</para>
<para><screen>mkfs.lustre --fsname=<new file system name></screen></para>
<para>The MDT, OSTs and clients in the new file system must share the same name (prepended to the device name). For example, for a new file system named <literal>foo</literal>, the MDT and two OSTs would be named <literal>foo-MDT0000</literal>, <literal>foo-OST0000</literal>, and <literal>foo-OST0001</literal>.</para>
</note>
<para>To mount a client on the file system, run:</para>
- <screen>mount -t lustre mgsnode:/<new fsname> <mountpoint>
-</screen>
+ <screen>mount -t lustre mgsnode:/<new fsname> <mountpoint></screen>
<para>For example, to mount a client on file system foo at mount point /mnt/lustre1, run:</para>
- <screen>mount -t lustre mgsnode:/foo /mnt/lustre1
-</screen>
+ <screen>mount -t lustre mgsnode:/foo /mnt/lustre1</screen>
<note>
<para>If a client(s) will be mounted on several file systems, add the following line to <literal>/etc/xattr.conf</literal> file to avoid problems when files are moved between the file systems: <literal>lustre.* skip</literal></para>
</note>
ossbarnode# mkfs.lustre --fsname=bar --ost --mgsnode=mgsnode@tcp0 /mnt/lust\
re1
ossbarnode# mkfs.lustre --fsname=bar --ost --mgsnode=mgsnode@tcp0 /mnt/lust\
-re2
-</screen>
+re2</screen>
<para>To mount a client on file system foo at mount point <literal>/mnt/lustre1</literal>, run:</para>
- <screen>mount -t lustre mgsnode@tcp0:/foo /mnt/lustre1
-</screen>
+ <screen>mount -t lustre mgsnode@tcp0:/foo /mnt/lustre1</screen>
<para>To mount a client on file system bar at mount point <literal>/mnt/lustre2</literal>, run:</para>
- <screen>mount -t lustre mgsnode@tcp0:/bar /mnt/lustre2
-</screen>
+ <screen>mount -t lustre mgsnode@tcp0:/bar /mnt/lustre2</screen>
</section>
<section xml:id="dbdoclet.50438194_88980">
- <title>13.8 Setting <anchor xml:id="dbdoclet.50438194_marker-1302467" xreflabel=""/>and Retrieving Lustre Parameters</title>
+ <title><indexterm><primary>operations</primary><secondary>parameters</secondary></indexterm>Setting and Retrieving Lustre Parameters</title>
<para>Several options are available for setting parameters in Lustre:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438194_17237">
- <title>13.8.1 Setting Parameters with <literal>mkfs.lustre</literal></title>
+ <title>Setting Parameters with <literal>mkfs.lustre</literal></title>
<para>When the file system is created, parameters can simply be added as a <literal>--param</literal> option to the <literal>mkfs.lustre</literal> command. For example:</para>
- <screen>$ mkfs.lustre --mdt --param="sys.timeout=50" /dev/sda
-</screen>
+ <screen>$ mkfs.lustre --mdt --param="sys.timeout=50" /dev/sda</screen>
<para>For more details about creating a file system,see <xref linkend="configuringlustre"/>. For more details about <literal>mkfs.lustre</literal>, see <xref linkend="systemconfigurationutilities"/>.</para>
</section>
<section xml:id="dbdoclet.50438194_55253">
- <title>13.8.2 Setting Parameters with <literal>tunefs.lustre</literal></title>
+ <title>Setting Parameters with <literal>tunefs.lustre</literal></title>
<para>If a server (OSS or MDS) is stopped, parameters can be added using the <literal>--param</literal> option to the <literal>tunefs.lustre</literal> command. For example:</para>
- <screen>$ tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda
-</screen>
+ <screen>$ tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda</screen>
<para>With <literal>tunefs.lustre</literal>, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old <literal>tunefs.lustre</literal> parameters and just use newly-specified parameters, run:</para>
<screen>$ tunefs.lustre --erase-params --param=<new parameters> </screen>
<para>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <literal><obd|fsname>.<obdtype>.<proc_file_name>=<value></literal>. For example:</para>
- <screen>$ tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1
-</screen>
- <para>For more details about tunefs.lustre, see <link xl:href="SystemConfigurationUtilities.html#50438219_66186">Chapter 36: System Configuration Utilities</link>.</para>
+ <screen>$ tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1</screen>
+ <para>For more details about <literal>tunefs.lustre</literal>, see <xref linkend="systemconfigurationutilities"/>.</para>
</section>
<section xml:id="dbdoclet.50438194_51490">
- <title>13.8.3 Setting Parameters with <literal>lctl</literal></title>
+ <title>Setting Parameters with <literal>lctl</literal></title>
<para>When the file system is running, the <literal>lctl</literal> command can be used to set parameters (temporary or permanent) and report current parameter values. Temporary parameters are active as long as the server or client is not shut down. Permanent parameters live through server and client reboots.</para>
<note>
<para>The lctl list_param command enables users to list all parameters that can be set. See <xref linkend="dbdoclet.50438194_88217"/>.</para>
</note>
<para>For more details about the <literal>lctl</literal> command, see the examples in the sections below and <xref linkend="systemconfigurationutilities"/>.</para>
<section remap="h4">
- <title>13.8.3.1 Setting Temporary Parameters</title>
+ <title>Setting Temporary Parameters</title>
<para>Use <literal>lctl set_param</literal> to set temporary parameters on the node where it is run. These parameters map to items in <literal>/proc/{fs,sys}/{lnet,lustre}</literal>. The <literal>lctl set_param</literal> command uses this syntax:</para>
- <screen>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value>
-</screen>
+ <screen>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value></screen>
<para>For example:</para>
<screen># lctl set_param osc.*.max_dirty_mb=1024
osc.myth-OST0000-osc.max_dirty_mb=32
osc.myth-OST0001-osc.max_dirty_mb=32
osc.myth-OST0002-osc.max_dirty_mb=32
osc.myth-OST0003-osc.max_dirty_mb=32
-osc.myth-OST0004-osc.max_dirty_mb=32
-</screen>
+osc.myth-OST0004-osc.max_dirty_mb=32</screen>
</section>
- <section remap="h4">
- <title>13.8.3.2 <anchor xml:id="dbdoclet.50438194_64195" xreflabel=""/>Setting Permanent Parameters</title>
+ <section xml:id="dbdoclet.50438194_64195">
+ <title>Setting Permanent Parameters</title>
<para>Use the <literal>lctl conf_param</literal> command to set permanent parameters. In general, the <literal>lctl conf_param</literal> command can be used to specify any parameter settable in a <literal>/proc/fs/lustre</literal> file, with its own OBD device. The <literal>lctl conf_param</literal> command uses this syntax (same as the <literal>mkfs.lustre</literal> and <literal>tunefs.lustre</literal> commands):</para>
- <screen><obd|fsname>.<obdtype>.<proc_file_name>=<value>)
-</screen>
+ <screen><obd|fsname>.<obdtype>.<proc_file_name>=<value>) </screen>
<para>Here are a few examples of <literal>lctl conf_param</literal> commands:</para>
<screen>$ mgs> lctl conf_param testfs-MDT0000.sys.timeout=40
$ lctl conf_param testfs-MDT0000.mdt.group_upcall=NONE
$ lctl conf_param testfs-MDT0000.lov.stripesize=2M
$ lctl conf_param testfs-OST0000.osc.max_dirty_mb=29.15
$ lctl conf_param testfs-OST0000.ost.client_cache_seconds=15
-$ lctl conf_param testfs.sys.timeout=40
-</screen>
+$ lctl conf_param testfs.sys.timeout=40 </screen>
<caution>
<para>Parameters specified with the <literal>lctl conf_param</literal> command are set permanently in the file system's configuration file on the MGS.</para>
</caution>
</section>
- <section remap="h4">
- <title>13.8.3.3 <anchor xml:id="dbdoclet.50438194_88217" xreflabel=""/>Listing Parameters</title>
+ <section xml:id="dbdoclet.50438194_88217">
+ <title>Listing Parameters</title>
<para>To list Lustre or LNET parameters that are available to set, use the <literal>lctl list_param</literal> command. For example:</para>
- <screen>lctl list_param [-FR] <obdtype>.<obdname>
-</screen>
+ <screen>lctl list_param [-FR] <obdtype>.<obdname></screen>
<para>The following arguments are available for the <literal>lctl list_param</literal> command.</para>
<para><literal>-F</literal> Add '<literal>/</literal>', '<literal>@</literal>' or '<literal>=</literal>' for directories, symlinks and writeable files, respectively</para>
<para><literal>-R</literal> Recursively lists all parameters under the specified path</para>
<para>For example:</para>
- <screen>$ lctl list_param obdfilter.lustre-OST0000
-</screen>
+ <screen>$ lctl list_param obdfilter.lustre-OST0000 </screen>
</section>
<section xml:id="dbdoclet.50438194_63247">
- <title>13.8.3.4 Reporting Current Parameter Values</title>
+ <title>Reporting Current Parameter Values</title>
<para>To report current Lustre parameter values, use the <literal>lctl get_param</literal> command with this syntax:</para>
- <screen>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name>
-</screen>
+ <screen>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name></screen>
<para>This example reports data on RPC service times.</para>
<screen>$ lctl get_param -n ost.*.ost_io.timeouts
-service : cur 1 worst 30 (at 1257150393, 85d23h58m54s ago) 1 1 1 1
-</screen>
+service : cur 1 worst 30 (at 1257150393, 85d23h58m54s ago) 1 1 1 1 </screen>
<para>This example reports the number of inodes available on each OST.</para>
<screen># lctl get_param osc.*.filesfree
osc.myth-OST0000-osc-ffff88006dd20000.filesfree=217623
osc.myth-OST0001-osc-ffff88006dd20000.filesfree=5075042
osc.myth-OST0002-osc-ffff88006dd20000.filesfree=3762034
osc.myth-OST0003-osc-ffff88006dd20000.filesfree=91052
-osc.myth-OST0004-osc-ffff88006dd20000.filesfree=129651<anchor xml:id="dbdoclet.50438194_88030" xreflabel=""/><anchor xml:id="dbdoclet.50438194_54623" xreflabel=""/></screen>
+osc.myth-OST0004-osc-ffff88006dd20000.filesfree=129651</screen>
</section>
</section>
</section>
<section xml:id="dbdoclet.50438194_41817">
- <title>13.9 <anchor xml:id="dbdoclet.50438194_42379" xreflabel=""/><anchor xml:id="dbdoclet.50438194_50129" xreflabel=""/>Specifying NIDs and Failover</title>
+ <title><indexterm><primary>operations</primary><secondary>failover</secondary></indexterm>Specifying NIDs and Failover</title>
<para>If a node has multiple network interfaces, it may have multiple NIDs. When a node is specified, all of its NIDs must be listed, delimited by commas (<literal>,</literal>) so other nodes can choose the NID that is appropriate for their network interfaces. When failover nodes are specified, they are delimited by a colon (<literal>:</literal>) or by repeating a keyword (<literal>--mgsnode=</literal> or <literal>--failnode=</literal>). To obtain all NIDs from a node (while LNET is running), run:</para>
- <screen>lctl list_nids
-</screen>
+ <screen>lctl list_nids</screen>
<para>This displays the server's NIDs (networks configured to work with Lustre).</para>
<para>This example has a combined MGS/MDT failover pair on uml1 and uml2, and a OST failover pair on uml3 and uml4. There are corresponding Elan addresses on uml1 and uml2.</para>
<screen>uml1> mkfs.lustre --fsname=testfs --mdt --mgs --failnode=uml2,2@elan /dev/sda1
client> mount -t lustre uml1,1@elan:uml2,2@elan:/testfs /mnt/testfs
uml1> umount /mnt/mdt
uml2> mount -t lustre /dev/sda1 /mnt/test/mdt
-uml2> cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status
-</screen>
+uml2> cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status</screen>
<para>Where multiple NIDs are specified, comma-separation (for example, <literal>uml2,2@elan</literal>) means that the two NIDs refer to the same host, and that Lustre needs to choose the "best" one for communication. Colon-separation (for example, <literal>uml1:uml2</literal>) means that the two NIDs refer to two different hosts, and should be treated as failover locations (Lustre tries the first one, and if that fails, it tries the second one.)</para>
<note>
<para>If you have an MGS or MDT configured for failover, perform these steps:</para>
<orderedlist>
<listitem>
- <para>On the OST, list the NIDs of all MGS nodes at mkfs time.</para>
+ <para>On the OST, list the NIDs of all MGS nodes at <literal>mkfs</literal> time.</para>
<screen>OST# mkfs.lustre --fsname sunfs --ost --mgsnode=10.0.0.1 \
--mgsnode=10.0.0.2 /dev/{device}</screen>
</listitem>
</note>
</section>
<section xml:id="dbdoclet.50438194_70905">
- <title>13.10 Erasing a File System</title>
+ <title><indexterm><primary>operations</primary><secondary>erasing a file system</secondary></indexterm>Erasing a File System</title>
<para>If you want to erase a file system, run this command on your targets:</para>
- <screen>$ "mkfs.lustre -reformat"
-</screen>
+ <screen>$ "mkfs.lustre -reformat"</screen>
<para>If you are using a separate MGS and want to keep other file systems defined on that MGS, then set the <literal>writeconf</literal> flag on the MDT for that file system. The <literal>writeconf</literal> flag causes the configuration logs to be erased; they are regenerated the next time the servers start.</para>
<para>To set the <literal>writeconf</literal> flag on the MDT:</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Unmount all clients/servers using this file system, run:</emphasis></para>
- <screen>$ umount /mnt/lustre
-</screen>
+ <para>Unmount all clients/servers using this file system, run:</para>
+ <screen>$ umount /mnt/lustre</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Erase the file system and, presumably, replace it with another file system, run:</emphasis></para>
- <screen>$ mkfs.lustre -reformat --fsname spfs --mdt --mgs /dev/sda
-</screen>
+ <para>Erase the file system and, presumably, replace it with another file system, run:</para>
+ <screen>$ mkfs.lustre -reformat --fsname spfs --mdt --mgs /dev/sda</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">If you have a separate MGS (that you do not want to reformat), then add the "writeconf" flag to <literal>mkfs.lustre</literal> on the MDT, run:</emphasis></para>
- <screen>$ mkfs.lustre --reformat --writeconf -fsname spfs --mdt \ --mgs /dev/sda
-</screen>
+ <para>If you have a separate MGS (that you do not want to reformat), then add the "writeconf" flag to <literal>mkfs.lustre</literal> on the MDT, run:</para>
+ <screen>$ mkfs.lustre --reformat --writeconf -fsname spfs --mdt \ --mgs /dev/sda</screen>
</listitem>
</orderedlist>
<note>
</note>
</section>
<section xml:id="dbdoclet.50438194_16954">
- <title>13.11 Reclaiming Reserved Disk Space</title>
+ <title><indexterm><primary>operations</primary><secondary>reclaiming space</secondary></indexterm>Reclaiming Reserved Disk Space</title>
<para>All current Lustre installations run the ldiskfs file system internally on service nodes. By default, ldiskfs reserves 5% of the disk space for the root user. In order to reclaim this space, run the following command on your OSSs:</para>
- <screen>tune2fs [-m reserved_blocks_percent] [device]
-</screen>
+ <screen>tune2fs [-m reserved_blocks_percent] [device]</screen>
<para>You do not need to shut down Lustre before running this command or restart it afterwards.</para>
</section>
<section xml:id="dbdoclet.50438194_69998">
- <title>13.12 Replacing an Existing OST or MDS</title>
+ <title><indexterm><primary>operations</primary><secondary>replacing a OST or MDS</secondary></indexterm>Replacing an Existing OST or MDS</title>
<para>To copy the contents of an existing OST to a new OST (or an old MDS to a new MDS), use one of these methods:</para>
<itemizedlist>
<listitem>
<para>Connect the old OST disk and new OST disk to a single machine, mount both, and use rsync to copy all data between the OST file systems.</para>
- </listitem>
- </itemizedlist>
- <para>For example:</para>
- <screen>mount -t ldiskfs /dev/old /mnt/ost_old
+ <para>For example:</para>
+ <screen>mount -t ldiskfs /dev/old /mnt/ost_old
mount -t ldiskfs /dev/new /mnt/ost_new
rsync -aSv /mnt/ost_old/ /mnt/ost_new
-# note trailing slash on ost_old/
-</screen>
+# note trailing slash on ost_old/</screen>
+ </listitem>
+ </itemizedlist>
<itemizedlist>
<listitem>
- <para> If you are unable to connect both sets of disk to the same computer, use <literal>rsync</literal> to copy over the network using <literal>rsh</literal> (or <literal>ssh</literal> with <literal>-e ssh</literal>):</para>
+ <para>If you are unable to connect both sets of disk to the same computer, use <literal>rsync</literal> to copy over the network using <literal>rsh</literal> (or <literal>ssh</literal> with <literal>-e ssh</literal>):</para>
+ <screen>rsync -aSvz /mnt/ost_old/ new_ost_node:/mnt/ost_new</screen>
</listitem>
</itemizedlist>
- <screen>rsync -aSvz /mnt/ost_old/ new_ost_node:/mnt/ost_new
-</screen>
<itemizedlist>
<listitem>
- <para> Use the same procedure for the MDS, with one additional step:</para>
+ <para>Use the same procedure for the MDS, with one additional step:</para>
+ <screen>cd /mnt/mds_old; getfattr -R -e base64 -d . > /tmp/mdsea; \<copy all MDS file\
+s as above>; cd /mnt/mds_new; setfattr \--restore=/tmp/mdsea</screen>
</listitem>
</itemizedlist>
- <screen>cd /mnt/mds_old; getfattr -R -e base64 -d . > /tmp/mdsea; \<copy all MDS file\
-s as above>; cd /mnt/mds_new; setfattr \--restore=/tmp/mdsea
-</screen>
</section>
<section xml:id="dbdoclet.50438194_30872">
- <title>13.13 Identifying To Which Lustre File an OST Object Belongs</title>
+ <title><indexterm><primary>operations</primary><secondary>identifying OSTs</secondary></indexterm>Identifying To Which Lustre File an OST Object Belongs</title>
<para>Use this procedure to identify the file containing a given object on a given OST.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">On the OST (as root), run <literal>debugfs</literal> to display the file identifier (<literal>FID</literal>) of the file associated with the object.</emphasis></para>
+ <para>On the OST (as root), run <literal>debugfs</literal> to display the file identifier (<literal>FID</literal>) of the file associated with the object.</para>
<para>For example, if the object is <literal>34976</literal> on <literal>/dev/lustre/ost_test2</literal>, the debug command is:</para>
- <screen># debugfs -c -R "stat /O/0/d$((34976 %32))/34976" /dev/lustre/ost_test2
-</screen>
+ <screen># debugfs -c -R "stat /O/0/d$((34976 %32))/34976" /dev/lustre/ost_test2 </screen>
<para>The command output is:</para>
<screen>debugfs 1.41.5.sun2 (23-Apr-2009)
/dev/lustre/ost_test2: catastrophic mode - not reading inode or group bitma\
00 00 00 00 00 00 00 00 00 " (32)
BLOCKS:
(0-63):47968-48031
-TOTAL: 64
-</screen>
+TOTAL: 64</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Note the FID's EA and apply it to the <literal>osd_inode_id</literal> mapping.</emphasis></para>
+ <para>Note the FID's EA and apply it to the <literal>osd_inode_id</literal> mapping.</para>
<para>In this example, the FID's EA is:</para>
<screen>e2001100000000002543c18700000000a0880000000000000000000000000000
struct osd_inode_id {
__u64 oii_ino; /* inode number */
__u32 oii_gen; /* inode generation */
__u32 oii_pad; /* alignment padding */
-};
-</screen>
+};</screen>
<para>After swapping, you get an inode number of <literal>0x001100e2</literal> and generation of <literal>0</literal>.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">On the MDT (as root), use <literal>debugfs</literal> to find the file associated with the inode.</emphasis></para>
- <screen># debugfs -c -R "ncheck 0x001100e2" /dev/lustre/mdt_test
-</screen>
+ <para>On the MDT (as root), use <literal>debugfs</literal> to find the file associated with the inode.</para>
+ <screen># debugfs -c -R "ncheck 0x001100e2" /dev/lustre/mdt_test </screen>
<para>Here is the command output:</para>
<screen>debugfs 1.41.5.sun2 (23-Apr-2009)
/dev/lustre/mdt_test: catastrophic mode - not reading inode or group bitmap\
s
Inode Pathname
-1114338 /ROOT/brian-laptop-guest/clients/client11/~dmtmp/PWRPNT/ZD16.BMP
-</screen>
+1114338 /ROOT/brian-laptop-guest/clients/client11/~dmtmp/PWRPNT/ZD16.BMP</screen>
</listitem>
</orderedlist>
<para>The command lists the inode and pathname associated with the object.</para>
<para><literal>Debugfs</literal>' ''ncheck'' is a brute-force search that may take a long time to complete.</para>
</note>
<note>
- <para>To find the Lustre file from a disk LBA, follow the steps listed in the document at this URL: <emphasis><link xl:href='http://smartmontools.sourceforge.net/badblockhowto.html'>http://smartmontools.sourceforge.net/badblockhowto.html</link>. </emphasis> Then, follow the steps above to resolve the Lustre filename.</para>
+ <para>To find the Lustre file from a disk LBA, follow the steps listed in the document at this URL: <emphasis><link xl:href="http://smartmontools.sourceforge.net/badblockhowto.html">http://smartmontools.sourceforge.net/badblockhowto.html</link>. </emphasis> Then, follow the steps above to resolve the Lustre filename.</para>
</note>
</section>
</chapter>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustreproc">
- <info>
- <title xml:id="lustreproc.title">LustreProc</title>
- </info>
+ <title xml:id="lustreproc.title">LustreProc</title>
<para>The <literal>/proc</literal> file system acts as an interface to internal data structures in the kernel. The <literal>/proc</literal> variables can be used to control aspects of Lustre performance and provide information.</para>
<para>This chapter describes Lustre /proc entries and includes the following sections:</para>
<itemizedlist>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438271_90999">
- <title>31.1 Proc Entries for Lustre</title>
+ <title><indexterm><primary>proc</primary></indexterm>Proc Entries for Lustre</title>
<para>This section describes <literal>/proc</literal> entries for Lustre.</para>
<section remap="h3">
- <title>31.1.1 Locating Lustre File Systems and Servers</title>
+ <title>Locating Lustre File Systems and Servers</title>
<para>Use the proc files on the MGS to locate the following:</para>
<itemizedlist>
<listitem>
lustre-MDT0000</screen>
</section>
<section remap="h3">
- <title>31.1.2 Lustre Timeouts</title>
+ <title><indexterm><primary>proc</primary><secondary>timeouts</secondary></indexterm>Lustre Timeouts</title>
<para>Lustre uses two types of timeouts.</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<para>Specific Lustre timeouts are described below.</para>
- <para><literal>
- /proc/sys/lustre/timeout
- </literal></para>
+ <para><literal> /proc/sys/lustre/timeout </literal></para>
<para>This is the time period that a client waits for a server to complete an RPC (default is 100s). Servers wait half of this time for a normal client RPC to complete and a quarter of this time for a single bulk request (read or write of up to 1 MB) to complete. The client pings recoverable targets (MDS and OSTs) at one quarter of the timeout, and the server waits one and a half times the timeout before evicting a client for being "stale."</para>
<note>
<para>Lustre sends periodic 'PING' messages to servers with which it had no communication for a specified period of time. Any network activity on the file system that triggers network traffic toward servers also works as a health check.</para>
</note>
- <para><literal>
- /proc/sys/lustre/ldlm_timeout
- </literal></para>
+ <para><literal> /proc/sys/lustre/ldlm_timeout </literal></para>
<para>This is the time period for which a server will wait for a client to reply to an initial AST (lock cancellation request) where default is 20s for an OST and 6s for an MDS. If the client replies to the AST, the server will give it a normal timeout (half of the client timeout) to flush any dirty data and release the lock.</para>
- <para><literal>
- /proc/sys/lustre/fail_loc
- </literal></para>
+ <para><literal> /proc/sys/lustre/fail_loc </literal></para>
<para>This is the internal debugging failure hook.</para>
<para>See <literal>lustre/include/linux/obd_support.h</literal> for the definitions of individual failure locations. The default value is 0 (zero).</para>
<screen>sysctl -w lustre.fail_loc=0x80000122 # drop a single reply</screen>
- <para><literal>
- /proc/sys/lustre/dump_on_timeout
- </literal></para>
+ <para><literal> /proc/sys/lustre/dump_on_timeout </literal></para>
<para>This triggers dumps of the Lustre debug log when timeouts occur. The default value is 0 (zero).</para>
- <para><literal>
- /proc/sys/lustre/dump_on_eviction
- </literal></para>
+ <para><literal> /proc/sys/lustre/dump_on_eviction </literal></para>
<para>This triggers dumps of the Lustre debug log when an eviction occurs. The default value is 0 (zero). By default, debug logs are dumped to the /tmp folder; this location can be changed via /proc.</para>
</section>
<section remap="h3">
- <title>31.1.3 Adaptive Timeouts</title>
+ <title><indexterm><primary>proc</primary><secondary>adaptive timeouts</secondary></indexterm>Adaptive Timeouts</title>
<para>Lustre offers an adaptive mechanism to set RPC timeouts. The adaptive timeouts feature (enabled, by default) causes servers to track actual RPC completion times, and to report estimated completion times for future RPCs back to clients. The clients use these estimates to set their future RPC timeout values. If server request processing slows down for any reason, the RPC completion estimates increase, and the clients allow more time for RPC completion.</para>
<para>If RPCs queued on the server approach their timeouts, then the server sends an early reply to the client, telling the client to allow more time. In this manner, clients avoid RPC timeouts and disconnect/reconnect cycles. Conversely, as a server speeds up, RPC timeout values decrease, allowing faster detection of non-responsive servers and faster attempts to reconnect to a server's failover partner.</para>
<para>In previous Lustre versions, the static obd_timeout (<literal>/proc/sys/lustre/timeout</literal>) value was used as the maximum completion time for all RPCs; this value also affected the client-server ping interval and initial recovery timer. Now, with adaptive timeouts, obd_timeout is only used for the ping interval and initial recovery estimate. When a client reconnects during recovery, the server uses the client's timeout value to reset the recovery wait period; i.e., the server learns how long the client had been willing to wait, and takes this into account when adjusting the recovery period.</para>
<section remap="h4">
- <title>31.1.3.1 Configuring Adaptive Timeouts</title>
+ <title><indexterm><primary>proc</primary><secondary>configuring adaptive timeouts</secondary></indexterm><indexterm><primary>configuring</primary><secondary>adaptive timeouts</secondary></indexterm>Configuring Adaptive Timeouts</title>
<para>One of the goals of adaptive timeouts is to relieve users from having to tune the <literal>obd_timeout</literal> value. In general, <literal>obd_timeout</literal> should no longer need to be changed. However, there are several parameters related to adaptive timeouts that users can set. In most situations, the default values should be used.</para>
<para>The following parameters can be set persistently system-wide using <literal>lctl conf_param</literal> on the MGS. For example, <literal>lctl conf_param work1.sys.at_max=1500</literal> sets the at_max value for all servers and clients using the work1 file system.</para>
<note>
<tbody>
<row>
<entry>
- <para> <literal>
- at_min
- </literal></para>
+ <para> <literal> at_min </literal></para>
</entry>
<entry>
<para>Sets the minimum adaptive timeout (in seconds). Default value is 0. The at_min parameter is the minimum processing time that a server will report. Clients base their timeouts on this value, but they do not use this value directly. If you experience cases in which, for unknown reasons, the adaptive timeout value is too short and clients time out their RPCs (usually due to temporary network outages), then you can increase the at_min value to compensate for this. Ideally, users should leave at_min set to its default.</para>
</row>
<row>
<entry>
- <para> <literal>
- at_max
- </literal></para>
+ <para> <literal> at_max </literal></para>
</entry>
<entry>
<para>Sets the maximum adaptive timeout (in seconds). The <literal>at_max</literal> parameter is an upper-limit on the service time estimate, and is used as a 'failsafe' in case of rogue/bad/buggy code that would lead to never-ending estimate increases. If at_max is reached, an RPC request is considered 'broken' and should time out.</para>
</row>
<row>
<entry>
- <para> <literal>
- at_history
- </literal></para>
+ <para> <literal> at_history </literal></para>
</entry>
<entry>
<para>Sets a time period (in seconds) within which adaptive timeouts remember the slowest event that occurred. Default value is 600.</para>
</row>
<row>
<entry>
- <para> <literal>
- at_early_margin
- </literal></para>
+ <para> <literal> at_early_margin </literal></para>
</entry>
<entry>
<para>Sets how far before the deadline Lustre sends an early reply. Default value is 5<footnote>
</row>
<row>
<entry>
- <para> <literal>
- at_extra
- </literal></para>
+ <para> <literal> at_extra </literal></para>
</entry>
<entry>
<para>Sets the incremental amount of time that a server asks for, with each early reply. The server does not know how much time the RPC will take, so it asks for a fixed value. Default value is 30<footnote>
</row>
<row>
<entry>
- <para> <literal>
- ldlm_enqueue_min
- </literal></para>
+ <para> <literal> ldlm_enqueue_min </literal></para>
</entry>
<entry>
<para> Sets the minimum lock enqueue time. Default value is 100. The <literal>ldlm_enqueue</literal> time is the maximum of the measured enqueue estimate (influenced by at_min and at_max parameters), multiplied by a weighting factor, and the <literal>ldlm_enqueue_min</literal> setting. LDLM lock enqueues were based on the <literal>obd_timeout</literal> value; now they have a dedicated minimum value. Lock enqueues increase as the measured enqueue times increase (similar to adaptive timeouts).</para>
</note>
</section>
<section remap="h4">
- <title>31.1.3.2 Interpreting Adaptive Timeouts Information</title>
+ <title><indexterm><primary>proc</primary><secondary>interpreting adaptive timeouts</secondary></indexterm>Interpreting Adaptive Timeouts Information</title>
<para>Adaptive timeouts information can be read from <literal>/proc/fs/lustre/*/timeouts</literal> files (for each service and client) or with the lctl command.</para>
<para>This is an example from the <literal>/proc/fs/lustre/*/timeouts</literal> files:</para>
<screen>cfs21:~# cat /proc/fs/lustre/ost/OSS/ost_io/timeouts</screen>
</section>
</section>
<section remap="h3">
- <title>31.1.4 LNET Information</title>
+ <title><indexterm><primary>proc</primary><secondary>LNET</secondary></indexterm><indexterm><primary>LNET</primary><secondary>proc</secondary></indexterm>LNET Information</title>
<para>This section describes<literal> /proc</literal> entries for LNET information.</para>
- <para><literal>
- /proc/sys/lnet/peers
- </literal></para>
+ <para><literal> /proc/sys/lnet/peers </literal></para>
<para>Shows all NIDs known to this node and also gives information on the queue state.</para>
<screen># cat /proc/sys/lnet/peers
nid refs state max rtr min tx min queue
<row>
<entry>
<para>
- <literal><replaceable>refs</replaceable></literal>
+ <literal>
+ <replaceable>refs</replaceable>
+ </literal>
</para>
</entry>
<entry>
<row>
<entry>
<para>
- <literal><replaceable>state</replaceable></literal>
+ <literal>
+ <replaceable>state</replaceable>
+ </literal>
</para>
</entry>
<entry>
</row>
<row>
<entry>
- <para> <literal>
- max
- </literal></para>
+ <para> <literal> max </literal></para>
</entry>
<entry>
<para>Maximum number of concurrent sends from this peer</para>
</row>
<row>
<entry>
- <para> <literal>
- rtr
- </literal></para>
+ <para> <literal> rtr </literal></para>
</entry>
<entry>
<para>Routing buffer credits.</para>
</row>
<row>
<entry>
- <para> <literal>
- min
- </literal></para>
+ <para> <literal> min </literal></para>
</entry>
<entry>
<para>Minimum routing buffer credits seen.</para>
</row>
<row>
<entry>
- <para> <literal>
- tx
- </literal></para>
+ <para> <literal> tx </literal></para>
</entry>
<entry>
<para>Send credits.</para>
</row>
<row>
<entry>
- <para> <literal>
- min
- </literal></para>
+ <para> <literal> min </literal></para>
</entry>
<entry>
<para>Minimum send credits seen.</para>
</row>
<row>
<entry>
- <para> <literal>
- queue
- </literal></para>
+ <para> <literal> queue </literal></para>
</entry>
<entry>
<para>Total bytes in active/queued sends.</para>
<para>If <literal>rtr/tx</literal> is less than max, there are operations in progress. The number of operations is equal to <literal>rtr</literal> or <literal>tx</literal> subtracted from max.</para>
<para>If <literal>rtr/tx</literal> is greater that max, there are operations blocking.</para>
<para>LNET also limits concurrent sends and router buffers allocated to a single peer so that no peer can occupy all these resources.</para>
- <para><literal>
- /proc/sys/lnet/nis
- </literal></para>
+ <para><literal> /proc/sys/lnet/nis </literal></para>
<screen># cat /proc/sys/lnet/nis
nid refs peer max tx min
0@lo 3 0 0 0 0
<tbody>
<row>
<entry>
- <para> <literal>
- nid
- </literal></para>
+ <para> <literal> nid </literal></para>
</entry>
<entry>
<para>Network interface</para>
</row>
<row>
<entry>
- <para> <literal>
- refs
- </literal></para>
+ <para> <literal> refs </literal></para>
</entry>
<entry>
<para>Internal reference counter</para>
</row>
<row>
<entry>
- <para> <literal>
- peer
- </literal></para>
+ <para> <literal> peer </literal></para>
</entry>
<entry>
<para>Number of peer-to-peer send credits on this NID. Credits are used to size buffer pools</para>
</row>
<row>
<entry>
- <para> <literal>
- max
- </literal></para>
+ <para> <literal> max </literal></para>
</entry>
<entry>
<para>Total number of send credits on this NID.</para>
</row>
<row>
<entry>
- <para> <literal>
- tx
- </literal></para>
+ <para> <literal> tx </literal></para>
</entry>
<entry>
<para>Current number of send credits available on this NID.</para>
</row>
<row>
<entry>
- <para> <literal>
- min
- </literal></para>
+ <para> <literal> min </literal></para>
</entry>
<entry>
<para>Lowest number of send credits available on this NID.</para>
</row>
<row>
<entry>
- <para> <literal>
- queue
- </literal></para>
+ <para> <literal> queue </literal></para>
</entry>
<entry>
<para>Total bytes in active/queued sends.</para>
</screen>
</section>
<section remap="h3">
- <title>31.1.5 Free Space Distribution</title>
+ <title><indexterm><primary>proc</primary><secondary>free space</secondary></indexterm>Free Space Distribution</title>
<para>Free-space stripe weighting, as set, gives a priority of "0" to free space (versus trying to place the stripes "widely" -- nicely distributed across OSSs and OSTs to maximize network balancing). To adjust this priority (as a percentage), use the <literal>qos_prio_free</literal> proc tunable:</para>
<screen>$ cat /proc/fs/lustre/lov/<fsname>-mdtlov/qos_prio_free</screen>
<para>Currently, the default is 90%. You can permanently set this value by running this command on the MGS:</para>
<para>Setting the priority to 100% means that OSS distribution does not count in the weighting, but the stripe assignment is still done via weighting. If OST 2 has twice as much free space as OST 1, it is twice as likely to be used, but it is NOT guaranteed to be used.</para>
<para>Also note that free-space stripe weighting does not activate until two OSTs are imbalanced by more than 20%. Until then, a faster round-robin stripe allocator is used. (The new round-robin order also maximizes network balancing.)</para>
<section remap="h4">
- <title>31.1.5.1 Managing Stripe Allocation</title>
+ <title><indexterm><primary>proc</primary><secondary>striping</secondary></indexterm>Managing Stripe Allocation</title>
<para>The MDS uses two methods to manage stripe allocation and determine which OSTs to use for file object storage:</para>
<itemizedlist>
<listitem>
</section>
</section>
<section xml:id="dbdoclet.50438271_78950">
- <title>31.2 Lustre I/O Tunables</title>
+ <title><indexterm><primary>proc</primary><secondary>I/O tunables</secondary></indexterm>Lustre I/O Tunables</title>
<para>The section describes I/O tunables.</para>
- <para><literal>
- /proc/fs/lustre/llite/<fsname>-<uid>/max_cache_mb
- </literal></para>
+ <para><literal> /proc/fs/lustre/llite/<fsname>-<uid>/max_cache_mb </literal></para>
<screen># cat /proc/fs/lustre/llite/lustre-ce63ca00/max_cached_mb 128</screen>
<para>This tunable is the maximum amount of inactive data cached by the client (default is 3/4 of RAM).</para>
<section remap="h3">
- <title>31.2.1 Client I/O RPC Stream Tunables</title>
+ <title><indexterm><primary>proc</primary><secondary>RPC tunables</secondary></indexterm>Client I/O RPC Stream Tunables</title>
<para>The Lustre engine always attempts to pack an optimal amount of data into each I/O RPC and attempts to keep a consistent number of issued RPCs in progress at a time. Lustre exposes several tuning variables to adjust behavior according to network conditions and cluster size. Each OSC has its own tree of these tunables. For example:</para>
<screen>$ ls -d /proc/fs/lustre/osc/OSC_client_ost1_MNT_client_2 /localhost
/proc/fs/lustre/osc/OSC_uml0_ost1_MNT_localhost
blocksizefilesfree max_dirty_mb ost_server_uuid stats</screen>
<para>... and so on.</para>
<para>RPC stream tunables are described below.</para>
- <para><literal>
- /proc/fs/lustre/osc/<object name>/max_dirty_mb
- </literal></para>
+ <para><literal> /proc/fs/lustre/osc/<object name>/max_dirty_mb </literal></para>
<para>This tunable controls how many MBs of dirty data can be written and queued up in the OSC. POSIX file writes that are cached contribute to this count. When the limit is reached, additional writes stall until previously-cached writes are written to the server. This may be changed by writing a single ASCII integer to the file. Only values between 0 and 512 are allowable. If 0 is given, no writes are cached. Performance suffers noticeably unless you use large writes (1 MB or more).</para>
- <para><literal>
- /proc/fs/lustre/osc/<object name>/cur_dirty_bytes
- </literal></para>
+ <para><literal> /proc/fs/lustre/osc/<object name>/cur_dirty_bytes </literal></para>
<para>This tunable is a read-only value that returns the current amount of bytes written and cached on this OSC.</para>
- <para><literal>
- /proc/fs/lustre/osc/<object name>/max_pages_per_rpc
- </literal></para>
+ <para><literal> /proc/fs/lustre/osc/<object name>/max_pages_per_rpc </literal></para>
<para>This tunable is the maximum number of pages that will undergo I/O in a single RPC to the OST. The minimum is a single page and the maximum for this setting is platform dependent (256 for i386/x86_64, possibly less for ia64/PPC with larger <literal>PAGE_SIZE</literal>), though generally amounts to a total of 1 MB in the RPC.</para>
- <para><literal>
- /proc/fs/lustre/osc/<object name>/max_rpcs_in_flight
- </literal></para>
+ <para><literal> /proc/fs/lustre/osc/<object name>/max_rpcs_in_flight </literal></para>
<para>This tunable is the maximum number of concurrent RPCs in flight from an OSC to its OST. If the OSC tries to initiate an RPC but finds that it already has the same number of RPCs outstanding, it will wait to issue further RPCs until some complete. The minimum setting is 1 and maximum setting is 32. If you are looking to improve small file I/O performance, increase the <literal>max_rpcs_in_flight</literal> value.</para>
<para>To maximize performance, the value for <literal>max_dirty_mb</literal> is recommended to be 4 * <literal>max_pages_per_rpc</literal> * <literal>max_rpcs_in_flight</literal>.</para>
<note>
<para>The
- <literal><replaceable><object name></replaceable></literal>
+ <literal>
+ <replaceable><object name></replaceable>
+ </literal>
varies depending on the specific Lustre configuration. For <literal><object name></literal> examples, refer to the sample command output.</para>
</note>
</section>
<section remap="h3">
- <title>31.2.2 Watching the Client RPC Stream</title>
+ <title><indexterm><primary>proc</primary><secondary>watching RPC</secondary></indexterm>Watching the Client RPC Stream</title>
<para>The same directory contains a <literal>rpc_stats</literal> file with a histogram showing the composition of previous RPCs. The histogram can be cleared by writing any value into the <literal>rpc_stats</literal> file.</para>
<screen># cat /proc/fs/lustre/osc/spfs-OST0000-osc-c45f9c00/rpc_stats
snapshot_time: 1174867307.156604 (secs.usecs)
</informaltable>
</section>
<section remap="h3">
- <title>31.2.3 Client Read-Write Offset Survey</title>
+ <title><indexterm><primary>proc</primary><secondary>read/write survey</secondary></indexterm>Client Read-Write Offset Survey</title>
<para>The offset_stats parameter maintains statistics for occurrences where a series of read or write calls from a process did not access the next sequential location. The offset field is reset to 0 (zero) whenever a different file is read/written.</para>
<para>Read/write offset statistics are off, by default. The statistics can be activated by writing anything into the <literal>offset_stats</literal> file.</para>
<para>Example:</para>
<tbody>
<row>
<entry>
- <para> <literal>
- R/W
- </literal></para>
+ <para> <literal> R/W </literal></para>
</entry>
<entry>
<para>Whether the non-sequential call was a read or write</para>
</row>
<row>
<entry>
- <para> <literal>
- PID
- </literal></para>
+ <para> <literal> PID </literal></para>
</entry>
<entry>
<para>Process ID which made the read/write call.</para>
</row>
<row>
<entry>
- <para> <literal>
- Range Start/Range End
- </literal></para>
+ <para> <literal> Range Start/Range End </literal></para>
</entry>
<entry>
<para>Range in which the read/write calls were sequential.</para>
</row>
<row>
<entry>
- <para> <literal>
- Smallest Extent
- </literal></para>
+ <para> <literal> Smallest Extent </literal></para>
</entry>
<entry>
<para>Smallest extent (single read/write) in the corresponding range.</para>
</row>
<row>
<entry>
- <para> <literal>
- Largest Extent
- </literal></para>
+ <para> <literal> Largest Extent </literal></para>
</entry>
<entry>
<para>Largest extent (single read/write) in the corresponding range.</para>
</row>
<row>
<entry>
- <para> <literal>
- Offset
- </literal></para>
+ <para> <literal> Offset </literal></para>
</entry>
<entry>
<para>Difference from the previous range end to the current range start.</para>
</informaltable>
</section>
<section remap="h3">
- <title>31.2.4 Client Read-Write Extents Survey</title>
+ <title><indexterm><primary>proc</primary><secondary>read/write survey</secondary></indexterm>Client Read-Write Extents Survey</title>
<para><emphasis role="bold">Client-Based I/O Extent Size Survey</emphasis></para>
<para>The <literal>rw_extent_stats</literal> histogram in the <literal>llite</literal> directory shows you the statistics for the sizes of the read-write I/O extents. This file does not maintain the per-process statistics.</para>
<para>Example:</para>
</screen>
</section>
<section xml:id="dbdoclet.50438271_55057">
- <title>31.2.5 Watching the OST Block I/O Stream</title>
+ <title><indexterm><primary>proc</primary><secondary>block I/O</secondary></indexterm>Watching the OST Block I/O Stream</title>
<para>Similarly, there is a <literal>brw_stats</literal> histogram in the obdfilter directory which shows you the statistics for number of I/O requests sent to the disk, their size and whether they are contiguous on the disk or not.</para>
<screen>cat /proc/fs/lustre/obdfilter/lustre-OST0000/brw_stats
snapshot_time: 1174875636.764630 (secs:usecs)
<tbody>
<row>
<entry>
- <para> <literal>
- pages per brw
- </literal></para>
+ <para> <literal> pages per brw </literal></para>
</entry>
<entry>
<para>Number of pages per RPC request, which should match aggregate client <literal>rpc_stats</literal>.</para>
</row>
<row>
<entry>
- <para> <literal>
- discont pages
- </literal></para>
+ <para> <literal> discont pages </literal></para>
</entry>
<entry>
<para>Number of discontinuities in the logical file offset of each page in a single RPC.</para>
</row>
<row>
<entry>
- <para> <literal>
- discont blocks
- </literal></para>
+ <para> <literal> discont blocks </literal></para>
</entry>
<entry>
<para>Number of discontinuities in the physical block allocation in the file system for a single RPC.</para>
</itemizedlist>
</section>
<section remap="h3">
- <title>31.2.6 Using File Readahead and Directory Statahead</title>
+ <title><indexterm><primary>proc</primary><secondary>readahead</secondary></indexterm>Using File Readahead and Directory Statahead</title>
<para>Lustre 1.6.5.1 introduced file readahead and directory statahead functionality that read data into memory in anticipation of a process actually requesting the data. File readahead functionality reads file content data into memory. Directory statahead functionality reads metadata into memory. When readahead and/or statahead work well, a data-consuming process finds that the information it needs is available when requested, and it is unnecessary to wait for network I/O.</para>
<section remap="h4">
- <title>31.2.6.1 Tuning File Readahead</title>
+ <title>Tuning File Readahead</title>
<para>File readahead is triggered when two or more sequential reads by an application fail to be satisfied by the Linux buffer cache. The size of the initial readahead is 1 MB. Additional readaheads grow linearly, and increment until the readahead cache on the client is full at 40 MB.</para>
- <para><literal>
- /proc/fs/lustre/llite/<fsname>-<uid>/max_read_ahead_mb
- </literal></para>
+ <para><literal> /proc/fs/lustre/llite/<fsname>-<uid>/max_read_ahead_mb </literal></para>
<para>This tunable controls the maximum amount of data readahead on a file. Files are read ahead in RPC-sized chunks (1 MB or the size of read() call, if larger) after the second sequential read on a file descriptor. Random reads are done at the size of the read() call only (no readahead). Reads to non-contiguous regions of the file reset the readahead algorithm, and readahead is not triggered again until there are sequential reads again. To disable readahead, set this tunable to 0. The default value is 40 MB.</para>
- <para><literal>
- /proc/fs/lustre/llite/<fsname>-<uid>/max_read_ahead_whole_mb
- </literal></para>
+ <para><literal> /proc/fs/lustre/llite/<fsname>-<uid>/max_read_ahead_whole_mb </literal></para>
<para>This tunable controls the maximum size of a file that is read in its entirety, regardless of the size of the <literal>read()</literal>.</para>
</section>
<section remap="h4">
- <title>31.2.6.2 Tuning Directory Statahead</title>
+ <title>Tuning Directory Statahead</title>
<para>When the <literal>ls -l</literal> process opens a directory, its process ID is recorded. When the first directory entry is ''stated'' with this recorded process ID, a statahead thread is triggered which stats ahead all of the directory entries, in order. The <literal>ls -l</literal> process can use the stated directory entries directly, improving performance.</para>
- <para><literal>
- /proc/fs/lustre/llite/*/statahead_max
- </literal></para>
+ <para><literal> /proc/fs/lustre/llite/*/statahead_max </literal></para>
<para>This tunable controls whether directory <literal>statahead</literal> is enabled and the maximum statahead count. By default, statahead is active.</para>
<para>To disable statahead, set this tunable to:</para>
<screen>echo 0 > /proc/fs/lustre/llite/*/statahead_max</screen>
<para>The maximum value of n is 8192.</para>
<para>
<literal>
- <replaceable role="bold">
- /proc/fs/lustre/llite/*/statahead_status
- </replaceable>
+ <replaceable role="bold"> /proc/fs/lustre/llite/*/statahead_status </replaceable>
</literal>
</para>
<para>This is a read-only interface that indicates the current statahead status.</para>
</section>
</section>
<section remap="h3">
- <title>31.2.7 OSS Read Cache</title>
+ <title><indexterm><primary>proc</primary><secondary>read cache</secondary></indexterm>OSS Read Cache</title>
<para>The OSS read cache feature provides read-only caching of data on an OSS. This functionality uses the regular Linux page cache to store the data. Just like caching from a regular filesytem in Linux, OSS read cache uses as much physical memory as is allocated.</para>
<para>OSS read cache improves Lustre performance in these situations:</para>
<itemizedlist>
</listitem>
</itemizedlist>
<section remap="h4">
- <title>31.2.7.1 Using OSS Read Cache</title>
+ <title>Using OSS Read Cache</title>
<para>OSS read cache is implemented on the OSS, and does not require any special support on the client side. Since OSS read cache uses the memory available in the Linux page cache, you should use I/O patterns to determine the appropriate amount of memory for the cache; if the data is mostly reads, then more cache is required than for writes.</para>
<para>OSS read cache is enabled, by default, and managed by the following tunables:</para>
<itemizedlist>
</section>
</section>
<section remap="h3">
- <title>31.2.8 OSS Asynchronous Journal Commit</title>
+ <title><indexterm><primary>proc</primary><secondary>OSS journal</secondary></indexterm>OSS Asynchronous Journal Commit</title>
<para>The OSS asynchronous journal commit feature synchronously writes data to disk without forcing a journal flush. This reduces the number of seeks and significantly improves performance on some hardware.</para>
<note>
<para>Asynchronous journal commit cannot work with O_DIRECT writes, a journal flush is still forced.</para>
<para>Similarly, when asynchronous journal commit is disabled, (<literal>sync_journal=1</literal>), <literal>sync_on_lock_cancel</literal> is enforced to never.</para>
</section>
<section remap="h3">
- <title>31.2.9 <literal>mballoc</literal> History</title>
- <para><literal>
- /proc/fs/ldiskfs/sda/mb_history
- </literal></para>
+ <title><indexterm><primary>proc</primary><secondary>mballoc history</secondary></indexterm><literal>mballoc</literal> History</title>
+ <para><literal> /proc/fs/ldiskfs/sda/mb_history </literal></para>
<para>Multi-Block-Allocate (<literal>mballoc</literal>), enables Lustre to ask <literal>ldiskfs</literal> to allocate multiple blocks with a single request to the block allocator. Typically, an <literal>ldiskfs</literal> file system allocates only one block per time. Each <literal>mballoc</literal>-enabled partition has this file. This is sample output:</para>
<screen>pid inode goal result found grps cr \ merge tail broken
2838 139267 17/12288/1 17/12288/1 1 0 0 \ M 1 8192
<para>Number of groups scanned (<literal>grps</literal> column) should be small. If it reaches a few dozen often, then either your disk file system is pretty fragmented or <literal>mballoc</literal> is doing something wrong in the group selection part.</para>
</section>
<section remap="h3">
- <title>31.2.10 <literal>mballoc3</literal> Tunables</title>
+ <title><indexterm><primary>proc</primary><secondary>mballoc3 tunables</secondary></indexterm><literal>mballoc3</literal> Tunables</title>
<para>Lustre version 1.6.1 and later includes <literal>mballoc3</literal>, which was built on top of <literal>mballoc2</literal>. By default, mballoc3 is enabled, and adds these features:</para>
<itemizedlist>
<listitem>
</row>
<row>
<entry>
- <para> <literal>
- max_to_scan
- </literal></para>
+ <para> <literal> max_to_scan </literal></para>
</entry>
<entry>
<para>Maximum number of free chunks that <literal>mballoc</literal> finds before a final decision to avoid livelock.</para>
</row>
<row>
<entry>
- <para> <literal>
- min_to_scan
- </literal></para>
+ <para> <literal> min_to_scan </literal></para>
</entry>
<entry>
<para>Minimum number of free chunks that <literal>mballoc</literal> finds before a final decision. This is useful for a very small request, to resist fragmentation of big free chunks.</para>
</row>
<row>
<entry>
- <para> <literal>
- order2_req
- </literal></para>
+ <para> <literal> order2_req </literal></para>
</entry>
<entry>
<para>For requests equal to 2^N (where N >= <literal>order2_req</literal>), a very fast search via buddy structures is used.</para>
</row>
<row>
<entry>
- <para> <literal>
- stream_req
- </literal></para>
+ <para> <literal> stream_req </literal></para>
</entry>
<entry>
<para>Requests smaller or equal to this value are packed together to form large write I/Os.</para>
<tbody>
<row>
<entry>
- <para> <literal>
- stats
- </literal></para>
+ <para> <literal> stats </literal></para>
</entry>
<entry>
<para>Enables/disables the collection of statistics. Collected statistics can be found in <literal>/proc/fs/ldiskfs2/<dev>/mb_history</literal>.</para>
</row>
<row>
<entry>
- <para> <literal>
- max_to_scan
- </literal></para>
+ <para> <literal> max_to_scan </literal></para>
</entry>
<entry>
<para>Maximum number of free chunks that <literal>mballoc</literal> finds before a final decision to avoid livelock.</para>
</row>
<row>
<entry>
- <para> <literal>
- min_to_scan
- </literal></para>
+ <para> <literal> min_to_scan </literal></para>
</entry>
<entry>
<para>Minimum number of free chunks that <literal>mballoc</literal> finds before a final decision. This is useful for a very small request, to resist fragmentation of big free chunks.</para>
</row>
<row>
<entry>
- <para> <literal>
- order2_req
- </literal></para>
+ <para> <literal> order2_req </literal></para>
</entry>
<entry>
<para>For requests equal to 2^N (where N >= <literal>order2_req</literal>), a very fast search via buddy structures is used.</para>
</row>
<row>
<entry>
- <para> <literal>
- small_req
- </literal></para>
+ <para> <literal> small_req </literal></para>
</entry>
<entry morerows="1">
<para>All requests are divided into 3 categories:</para>
</row>
<row>
<entry>
- <para> <literal>
- large_req
- </literal></para>
+ <para> <literal> large_req </literal></para>
</entry>
</row>
<row>
<entry>
- <para> <literal>
- prealloc_table
- </literal></para>
+ <para> <literal> prealloc_table </literal></para>
</entry>
<entry>
<para>The amount of space to preallocate depends on the current file size. The idea is that for small files we do not need 1 MB preallocations and for large files, 1 MB preallocations are not large enough; it is better to preallocate 4 MB.</para>
</row>
<row>
<entry>
- <para> <literal>
- group_prealloc
- </literal></para>
+ <para> <literal> group_prealloc </literal></para>
</entry>
<entry>
<para>The amount of space preallocated for small requests to be grouped.</para>
</informaltable>
</section>
<section remap="h3">
- <title>31.2.11 Locking</title>
- <para><literal>
- /proc/fs/lustre/ldlm/ldlm/namespaces/<OSC name|MDC name>/lru_size
- </literal></para>
+ <title><indexterm><primary>proc</primary><secondary>locking</secondary></indexterm>Locking</title>
+ <para><literal> /proc/fs/lustre/ldlm/ldlm/namespaces/<OSC name|MDC name>/lru_size </literal></para>
<para>The <literal>lru_size</literal> parameter is used to control the number of client-side locks in an LRU queue. LRU size is dynamic, based on load. This optimizes the number of locks available to nodes that have different workloads (e.g., login/build nodes vs. compute nodes vs. backup nodes).</para>
<para>The total number of locks available is a function of the server's RAM. The default limit is 50 locks/1 MB of RAM. If there is too much memory pressure, then the LRU size is shrunk. The number of locks on the server is limited to {number of OST/MDT on node} * {number of clients} * {client lru_size}.</para>
<itemizedlist>
<screen>$ lctl get_param ldlm.namespaces.*.pool.limit</screen>
</section>
<section xml:id="dbdoclet.50438271_87260">
- <title>31.2.12 Setting MDS and OSS Thread Counts</title>
+ <title><indexterm><primary>proc</primary><secondary>thread counts</secondary></indexterm>Setting MDS and OSS Thread Counts</title>
<para>MDS and OSS thread counts (minimum and maximum) can be set via the <literal>{min,max}_thread_count tunable</literal>. For each service, a new <literal>/proc/fs/lustre/{service}/*/thread_{min,max,started}</literal> entry is created. The tunable, <literal>{service}.thread_{min,max,started}</literal>, can be used to set the minimum and maximum thread counts or get the current number of running threads for the following services.</para>
<informaltable frame="all">
<tgroup cols="2">
</row>
<row>
<entry>
- <literal>
- mdt.MDS.mds
- </literal>
+ <literal> mdt.MDS.mds </literal>
</entry>
<entry>
<para>normal metadata ops</para>
</row>
<row>
<entry>
- <literal>
- mdt.MDS.mds_readpage
- </literal>
+ <literal> mdt.MDS.mds_readpage </literal>
</entry>
<entry>
<para>metadata readdir</para>
</row>
<row>
<entry>
- <literal>
- mdt.MDS.mds_setattr
- </literal>
+ <literal> mdt.MDS.mds_setattr </literal>
</entry>
<entry>
<para>metadata setattr</para>
</row>
<row>
<entry>
- <literal>
- ost.OSS.ost
- </literal>
+ <literal> ost.OSS.ost </literal>
</entry>
<entry>
<para>normal data</para>
</row>
<row>
<entry>
- <literal>
- ost.OSS.ost_io
- </literal>
+ <literal> ost.OSS.ost_io </literal>
</entry>
<entry>
<para>bulk data IO</para>
</row>
<row>
<entry>
- <literal>
- ost.OSS.ost_create
- </literal>
+ <literal> ost.OSS.ost_create </literal>
</entry>
<entry>
<para>OST object pre-creation service</para>
</row>
<row>
<entry>
- <literal>
- ldlm.services.ldlm_canceld
- </literal>
+ <literal> ldlm.services.ldlm_canceld </literal>
</entry>
<entry>
<para>DLM lock cancel</para>
</row>
<row>
<entry>
- <literal>
- ldlm.services.ldlm_cbd
- </literal>
+ <literal> ldlm.services.ldlm_cbd </literal>
</entry>
<entry>
<para>DLM lock grant</para>
</section>
</section>
<section xml:id="dbdoclet.50438271_83523">
- <title>31.3 Debug</title>
- <para><literal>
- /proc/sys/lnet/debug
- </literal></para>
+ <title><indexterm><primary>proc</primary><secondary>debug</secondary></indexterm>Debug</title>
+ <para><literal> /proc/sys/lnet/debug </literal></para>
<para>By default, Lustre generates a detailed log of all operations to aid in debugging. The level of debugging can affect the performance or speed you achieve with Lustre. Therefore, it is useful to reduce this overhead by turning down the debug level<footnote>
<para>This controls the level of Lustre debugging kept in the internal log buffer. It does not alter the level of debugging that goes to syslog.</para>
</footnote> to improve performance. Raise the debug level when you need to collect the logs for debugging problems. The debugging mask can be set with "symbolic names" instead of the numerical values that were used in prior releases. The new symbolic format is shown in the examples below.</para>
# echo "-warning" > /proc/sys/lnet/debug
# cat /proc/sys/lnet/debug
neterror ha</screen>
- <para><literal>
- /proc/sys/lnet/subsystem_debug
- </literal></para>
+ <para><literal> /proc/sys/lnet/subsystem_debug </literal></para>
<para>This controls the debug logs for subsystems (see <literal>S_*</literal> definitions).</para>
- <para><literal>
- /proc/sys/lnet/debug_path
- </literal></para>
+ <para><literal> /proc/sys/lnet/debug_path </literal></para>
<para>This indicates the location where debugging symbols should be stored for <literal>gdb</literal>. The default is set to <literal>/r/tmp/lustre-log-localhost.localdomain</literal>.</para>
<para>These values can also be set via <literal>sysctl -w lnet.debug={value}</literal></para>
<note>
<para>The above entries only exist when Lustre has already been loaded.</para>
</note>
- <para><literal>
- /proc/sys/lnet/panic_on_lbug
- </literal></para>
+ <para><literal> /proc/sys/lnet/panic_on_lbug </literal></para>
<para>This causes Lustre to call ''panic'' when it detects an internal problem (an <literal>LBUG</literal>); panic crashes the node. This is particularly useful when a kernel crash dump utility is configured. The crash dump is triggered when the internal inconsistency is detected by Lustre.</para>
- <para><literal>
- /proc/sys/lnet/upcall
- </literal></para>
+ <para><literal> /proc/sys/lnet/upcall </literal></para>
<para>This allows you to specify the path to the binary which will be invoked when an <literal>LBUG</literal> is encountered. This binary is called with four parameters. The first one is the string ''<literal>LBUG</literal>''. The second one is the file where the <literal>LBUG</literal> occurred. The third one is the function name. The fourth one is the line number in the file.</para>
<section remap="h3">
- <title>31.3.1 RPC Information for Other OBD Devices</title>
+ <title>RPC Information for Other OBD Devices</title>
<para>Some OBD devices maintain a count of the number of RPC events that they process. Sometimes these events are more specific to operations of the device, like llite, than actual raw RPC counts.</para>
<screen>$ find /proc/fs/lustre/ -name stats
/proc/fs/lustre/osc/lustre-OST0001-osc-ce63ca00/stats
/proc/fs/lustre/llite/lustre-ce63ca00/stats
</screen>
<section remap="h4">
- <title>31.3.1.1 Interpreting OST Statistics</title>
+ <title><indexterm><primary>proc</primary><secondary>statistics</secondary></indexterm>Interpreting OST Statistics</title>
<note>
<para>See also <xref linkend="dbdoclet.50438219_84890"/> (llobdstat) and <xref linkend="dbdoclet.50438273_80593"/> (CollectL).</para>
</note>
<tbody>
<row>
<entry>
- <para> <literal>
- Cur. Count
- </literal></para>
+ <para> <literal> Cur. Count </literal></para>
</entry>
<entry>
<para>Number of events of each type sent in the last interval (in this example, 10s)</para>
</row>
<row>
<entry>
- <para> <literal>
- Cur. Rate
- </literal></para>
+ <para> <literal> Cur. Rate </literal></para>
</entry>
<entry>
<para>Number of events per second in the last interval</para>
</row>
<row>
<entry>
- <para> <literal>
- #Events
- </literal></para>
+ <para> <literal> #Events </literal></para>
</entry>
<entry>
<para>Total number of such events since the system started</para>
</row>
<row>
<entry>
- <para> <literal>
- Unit
- </literal></para>
+ <para> <literal> Unit </literal></para>
</entry>
<entry>
<para>Unit of measurement for that statistic (microseconds, requests, buffers)</para>
</row>
<row>
<entry>
- <para> <literal>
- last
- </literal></para>
+ <para> <literal> last </literal></para>
</entry>
<entry>
<para>Average rate of these events (in units/event) for the last interval during which they arrived. For instance, in the above mentioned case of <literal>ost_destroy</literal> it took an average of 736 microseconds per destroy for the 400 object destroys in the previous 10 seconds.</para>
</row>
<row>
<entry>
- <para> <literal>
- min
- </literal></para>
+ <para> <literal> min </literal></para>
</entry>
<entry>
<para>Minimum rate (in units/events) since the service started</para>
</row>
<row>
<entry>
- <para> <literal>
- avg
- </literal></para>
+ <para> <literal> avg </literal></para>
</entry>
<entry>
<para>Average rate</para>
</row>
<row>
<entry>
- <para> <literal>
- max
- </literal></para>
+ <para> <literal> max </literal></para>
</entry>
<entry>
<para>Maximum rate</para>
</row>
<row>
<entry>
- <para> <literal>
- stddev
- </literal></para>
+ <para> <literal> stddev </literal></para>
</entry>
<entry>
<para>Standard deviation (not measured in all cases)</para>
<tbody>
<row>
<entry>
- <para> <literal>
- req_waittime
- </literal></para>
+ <para> <literal> req_waittime </literal></para>
</entry>
<entry>
<para>Amount of time a request waited in the queue before being handled by an available server thread.</para>
</row>
<row>
<entry>
- <para> <literal>
- req_qdepth
- </literal></para>
+ <para> <literal> req_qdepth </literal></para>
</entry>
<entry>
<para>Number of requests waiting to be handled in the queue for this service.</para>
</row>
<row>
<entry>
- <para> <literal>
- req_active
- </literal></para>
+ <para> <literal> req_active </literal></para>
</entry>
<entry>
<para>Number of requests currently being handled.</para>
</row>
<row>
<entry>
- <para> <literal>
- reqbuf_avail
- </literal></para>
+ <para> <literal> reqbuf_avail </literal></para>
</entry>
<entry>
<para>Number of unsolicited lnet request buffers for this service.</para>
<tbody>
<row>
<entry>
- <para> <literal>
- ldlm_enqueue
- </literal></para>
+ <para> <literal> ldlm_enqueue </literal></para>
</entry>
<entry>
<para>Time it takes to enqueue a lock (this includes file open on the MDS)</para>
</row>
<row>
<entry>
- <para> <literal>
- mds_reint
- </literal></para>
+ <para> <literal> mds_reint </literal></para>
</entry>
<entry>
<para>Time it takes to process an MDS modification record (includes create, <literal>mkdir</literal>, <literal>unlink</literal>, <literal>rename</literal> and <literal>setattr</literal>)</para>
</informaltable>
</section>
<section remap="h4">
- <title>31.3.1.2 Interpreting MDT Statistics</title>
+ <title><indexterm><primary>proc</primary><secondary>statistics</secondary></indexterm>Interpreting MDT Statistics</title>
<note>
<para>See also <xref linkend="dbdoclet.50438219_84890"/> (llobdstat) and <xref linkend="dbdoclet.50438273_80593"/> (CollectL).</para>
</note>
setattr 1 samples [reqs]
getattr 3 samples [reqs]
llog_init 6 samples [reqs]
-notify 16 samples [reqs]
-</screen>
+notify 16 samples [reqs]</screen>
</section>
</section>
</section>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustreprogramminginterfaces">
- <info>
- <title xml:id="lustreprogramminginterfaces.title">Lustre Programming Interfaces</title>
- </info>
+ <title xml:id="lustreprogramminginterfaces.title">Lustre Programming Interfaces</title>
<para>This chapter describes public programming interfaces to control various aspects of Lustre from userspace. These interfaces are generally not guaranteed to remain unchanged over time, although we will make an effort to notify the user community well in advance of major changes. This chapter includes the following section:</para>
<itemizedlist>
<listitem>
<para>Lustre programming interface man pages are found in the <literal>lustre/doc</literal> folder.</para>
</note>
<section xml:id="dbdoclet.50438291_32926">
- <title>33.1 User/Group Cache Upcall</title>
+ <title><indexterm><primary>programming</primary><secondary>upcall</secondary></indexterm>User/Group Cache Upcall</title>
<para>This section describes user and group upcall.</para>
<note>
<para>For information on a universal UID/GID, see <xref linkend="dbdoclet.50438261_19503"/>.</para>
</note>
<section remap="h3">
- <title>33.1.1 Name</title>
+ <title>Name</title>
<para>Use <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall</literal> to look up a given user's group membership.</para>
</section>
<section remap="h3">
- <title>33.1.2 Description</title>
+ <title>Description</title>
<para>The group upcall file contains the path to an executable that, when installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the <literal>mds_grp_downcall_data</literal> data structure (see <xref linkend="dbdoclet.50438291_33759"/>) and write it to the <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info</literal> pseudo-file.</para>
<para>For a sample upcall program, see <literal>lustre/utils/l_getgroups.c</literal> in the Lustre source distribution.</para>
<section remap="h4">
- <title>33.1.2.1 Primary and Secondary Groups</title>
+ <title>Primary and Secondary Groups</title>
<para>The mechanism for the primary/secondary group is as follows:</para>
<itemizedlist>
<listitem>
* they can be listed together, seperated by ',',
* when perm and noperm are in the same line (item), noperm is preferential,
* when they are in different lines (items), the latter is preferential,
-* '*' nid is as default perm, and is not preferential.*/
-</screen>
+* '*' nid is as default perm, and is not preferential.*/</screen>
<para>Currently, <literal>rmtacl/normtacl</literal> can be ignored (part of security functionality), and used for remote clients. The /usr/sbin/l_getidentity utility can parse <literal>/etc/lustre/perm.conf</literal> to obtain permission mask for specified UID.</para>
<itemizedlist>
<listitem>
</section>
</section>
<section remap="h3">
- <title>33.1.3 Parameters</title>
+ <title>Parameters</title>
<itemizedlist>
<listitem>
<para> Name of the MDS service</para>
</listitem>
</itemizedlist>
</section>
- <section remap="h3">
- <title>33.1.4 <anchor xml:id="dbdoclet.50438291_33759" xreflabel=""/>Data Structures</title>
+ <section xml:id="dbdoclet.50438291_33759">
+ <title>Data Structures</title>
<screen>struct identity_downcall_data {
__u32 idd_magic;
__u32 idd_err;
</section>
</section>
<section xml:id="dbdoclet.50438291_73963">
- <title>33.2 <literal>l_getgroups</literal><anchor xml:id="dbdoclet.50438291_marker-1294565" xreflabel=""/> Utility</title>
+ <title><indexterm><primary>programming</primary><secondary>l_getgroups</secondary></indexterm><literal>l_getgroups</literal> Utility</title>
<para>The <literal>l_getgroups</literal> utility handles Lustre user/group cache upcall.</para>
<section remap="h5">
<title>Synopsis</title>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustrerecovery">
- <info>
- <title xml:id="lustrerecovery.title">Lustre Recovery</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustrerecovery">
+ <title xml:id="lustrerecovery.title">Lustre Recovery</title>
<para>This chapter describes how recovery is implemented in Lustre and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438268_58047">
- <title>30.1 Recovery Overview</title>
+ <title>
+ <indexterm><primary>recovery</primary></indexterm>
+ <indexterm><primary>recovery</primary><secondary>VBR</secondary><see>version-based recovery</see></indexterm>
+ <indexterm><primary>recovery</primary><secondary>commit on share</secondary><see>ommit on share</see></indexterm>
+ <indexterm><primary>lustre</primary><secondary>recovery</secondary><see>recovery</see></indexterm>
+ Recovery Overview</title>
<para>Lustre's recovery feature is responsible for dealing with node or network failure and returning the cluster to a consistent, performant state. Because Lustre allows servers to perform asynchronous update operations to the on-disk file system (i.e., the server can reply without waiting for the update to synchronously commit to disk), the clients may have state in memory that is newer than what the server can recover from disk after a crash.</para>
<para>A handful of different types of failures can cause recovery to occur:</para>
<itemizedlist>
<para>Currently, all Lustre failure and recovery operations are based on the concept of connection failure; all imports or exports associated with a given connection are considered to fail if any of them fail.</para>
<para>For information on Lustre recovery, see <xref linkend="dbdoclet.50438268_65824"/>. For information on recovering from a corrupt file system, see <xref linkend="dbdoclet.50438268_83826"/>. For information on resolving orphaned objects, a common issue after recovery, see <xref linkend="dbdoclet.50438225_13916"/>.</para>
<section remap="h3">
- <title>30.1.1 Client Failure</title>
+ <title><indexterm><primary>recovery</primary><secondary>client failure</secondary></indexterm>Client Failure</title>
<para>Recovery from client failure in Lustre is based on lock revocation and other resources, so surviving clients can continue their work uninterrupted. If a client fails to timely respond to a blocking lock callback from the Distributed Lock Manager (DLM) or fails to communicate with the server in a long period of time (i.e., no pings), the client is forcibly removed from the cluster (evicted). This enables other clients to acquire locks blocked by the dead client's locks, and also frees resources (file handles, export data) associated with that client. Note that this scenario can be caused by a network partition, as well as an actual client node system failure. <xref linkend="dbdoclet.50438268_96876"/> describes this case in more detail.</para>
</section>
<section xml:id="dbdoclet.50438268_43796">
- <title>30.1.2 Client Eviction</title>
+ <title><indexterm><primary>recovery</primary><secondary>client eviction</secondary></indexterm>Client Eviction</title>
<para>If a client is not behaving properly from the server's point of view, it will be evicted. This ensures that the whole file system can continue to function in the presence of failed or misbehaving clients. An evicted client must invalidate all locks, which in turn, results in all cached inodes becoming invalidated and all cached data being flushed.</para>
<para>Reasons why a client might be evicted:</para>
<itemizedlist>
<listitem>
- <para> Failure to respond to a server request in a timely manner</para>
+ <para>Failure to respond to a server request in a timely manner</para>
<itemizedlist>
<listitem>
- <para> Blocking lock callback (i.e., client holds lock that another client/server wants)</para>
+ <para>Blocking lock callback (i.e., client holds lock that another client/server wants)</para>
</listitem>
<listitem>
- <para> Lock completion callback (i.e., client is granted lock previously held by another client)</para>
+ <para>Lock completion callback (i.e., client is granted lock previously held by another client)</para>
</listitem>
<listitem>
- <para> Lock glimpse callback (i.e., client is asked for size of object by another client)</para>
+ <para>Lock glimpse callback (i.e., client is asked for size of object by another client)</para>
</listitem>
<listitem>
- <para> Server shutdown notification (with simplified interoperability)</para>
+ <para>Server shutdown notification (with simplified interoperability)</para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
- <para> Failure to ping the server in a timely manner, unless the server is receiving no RPC traffic at all (which may indicate a network partition).</para>
+ <para>Failure to ping the server in a timely manner, unless the server is receiving no RPC traffic at all (which may indicate a network partition).</para>
</listitem>
</itemizedlist>
</section>
<section remap="h3">
- <title>30.1.3 MDS Failure (Failover)</title>
+ <title><indexterm><primary>recovery</primary><secondary>MDS failure</secondary></indexterm>MDS Failure (Failover)</title>
<para>Highly-available (HA) Lustre operation requires that the metadata server have a peer configured for failover, including the use of a shared storage device for the MDT backing file system. The actual mechanism for detecting peer failure, power off (STONITH) of the failed peer (to prevent it from continuing to modify the shared disk), and takeover of the Lustre MDS service on the backup node depends on external HA software such as Heartbeat. It is also possible to have MDS recovery with a single MDS node. In this case, recovery will take as long as is needed for the single MDS to be restarted.</para>
<para>When clients detect an MDS failure (either by timeouts of in-flight requests or idle-time ping messages), they connect to the new backup MDS and use the Metadata Replay protocol. Metadata Replay is responsible for ensuring that the backup MDS re-acquires state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk.</para>
<para>The reconnection to a new (or restarted) MDS is managed by the file system configuration loaded by the client when the file system is first mounted. If a failover MDS has been configured (using the <literal>--failnode=</literal> option to <literal>mkfs.lustre</literal> or <literal>tunefs.lustre</literal>), the client tries to reconnect to both the primary and backup MDS until one of them responds that the failed MDT is again available. At that point, the client begins recovery. For more information, see <xref linkend="dbdoclet.50438268_65824"/>.</para>
<para>Transaction numbers are used to ensure that operations are replayed in the order they were originally performed, so that they are guaranteed to succeed and present the same filesystem state as before the failure. In addition, clients inform the new server of their existing lock state (including locks that have not yet been granted). All metadata and lock replay must complete before new, non-recovery operations are permitted. In addition, only clients that were connected at the time of MDS failure are permitted to reconnect during the recovery window, to avoid the introduction of state changes that might conflict with what is being replayed by previously-connected clients.</para>
</section>
<section remap="h3">
- <title>30.1.4 OST Failure (Failover)</title>
+ <title><indexterm><primary>recovery</primary><secondary>OST failure</secondary></indexterm>OST Failure (Failover)</title>
<para>When an OST fails or has communication problems with the client, the default action is that the corresponding OSC enters recovery, and I/O requests going to that OST are blocked waiting for OST recovery or failover. It is possible to administratively mark the OSC as <emphasis>inactive</emphasis> on the client, in which case file operations that involve the failed OST will return an IO error (<literal>-EIO</literal>). Otherwise, the application waits until the OST has recovered or the client process is interrupted (e.g. ,with <emphasis>CTRL-C</emphasis>).</para>
<para>The MDS (via the LOV) detects that an OST is unavailable and skips it when assigning objects to new files. When the OST is restarted or re-establishes communication with the MDS, the MDS and OST automatically perform orphan recovery to destroy any objects that belong to files that were deleted while the OST was unavailable. For more information, see <xref linkend="troubleshootingrecovery"/> (Working with Orphaned Objects).</para>
<para>While the OSC to OST operation recovery protocol is the same as that between the MDC and MDT using the Metadata Replay protocol, typically the OST commits bulk write operations to disk synchronously and each reply indicates that the request is already committed and the data does not need to be saved for recovery. In some cases, the OST replies to the client before the operation is committed to disk (e.g. truncate, destroy, setattr, and I/O operations in very new versions of Lustre), and normal replay and resend handling is done, including resending of the bulk writes. In this case, the client keeps a copy of the data available in memory until the server indicates that the write has committed to disk.</para>
</note>
</section>
<section xml:id="dbdoclet.50438268_96876">
- <title>30.1.5 Network Partition</title>
+ <title><indexterm><primary>recovery</primary><secondary>network</secondary></indexterm>Network Partition</title>
<para>Network failures may be transient. To avoid invoking recovery, the client tries, initially, to re-send any timed out request to the server. If the resend also fails, the client tries to re-establish a connection to the server. Clients can detect harmless partition upon reconnect if the server has not had any reason to evict the client.</para>
<para>If a request was processed by the server, but the reply was dropped (i.e., did not arrive back at the client), the server must reconstruct the reply when the client resends the request, rather than performing the same request twice.</para>
</section>
<section remap="h3">
- <title>30.1.6 Failed Recovery</title>
+ <title><indexterm><primary>recovery</primary><secondary>failed recovery</secondary></indexterm>Failed Recovery</title>
<para>In the case of failed recovery, a client is evicted by the server and must reconnect after having flushed its saved state related to that server, as described in <xref linkend="dbdoclet.50438268_43796"/>, above. Failed recovery might occur for a number of reasons, including:</para>
<itemizedlist>
<listitem>
</section>
</section>
<section xml:id="dbdoclet.50438268_65824">
- <title>30.2 Metadata Replay</title>
+ <title><indexterm><primary>recovery</primary><secondary>metadata replay</secondary></indexterm>Metadata Replay</title>
<para>Highly available Lustre operation requires that the MDS have a peer configured for failover, including the use of a shared storage device for the MDS backing file system. When a client detects an MDS failure, it connects to the new MDS and uses the metadata replay protocol to replay its requests.</para>
<para>Metadata replay ensures that the failover MDS re-accumulates state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk.</para>
<section remap="h3">
- <title>30.2.1 XID Numbers</title>
+ <title>XID Numbers</title>
<para>Each request sent by the client contains an XID number, which is a client-unique, monotonically increasing 64-bit integer. The initial value of the XID is chosen so that it is highly unlikely that the same client node reconnecting to the same server after a reboot would have the same XID sequence. The XID is used by the client to order all of the requests that it sends, until such a time that the request is assigned a transaction number. The XID is also used in Reply Reconstruction to uniquely identify per-client requests at the server.</para>
</section>
<section remap="h3">
- <title>30.2.2 Transaction Numbers</title>
+ <title>Transaction Numbers</title>
<para>Each client request processed by the server that involves any state change (metadata update, file open, write, etc., depending on server type) is assigned a transaction number by the server that is a target-unique, monontonically increasing, server-wide 64-bit integer. The transaction number for each file system-modifying request is sent back to the client along with the reply to that client request. The transaction numbers allow the client and server to unambiguously order every modification to the file system in case recovery is needed.</para>
<para>Each reply sent to a client (regardless of request type) also contains the last committed transaction number that indicates the highest transaction number committed to the file system. The <literal>ldiskfs</literal> backing file system that Lustre uses enforces the requirement that any earlier disk operation will always be committed to disk before a later disk operation, so the last committed transaction number also reports that any requests with a lower transaction number have been committed to disk.</para>
</section>
<section remap="h3">
- <title>30.2.3 Replay and Resend</title>
+ <title>Replay and Resend</title>
<para>Lustre recovery can be separated into two distinct types of operations: <emphasis>replay</emphasis> and <emphasis>resend</emphasis>.</para>
<para><emphasis>Replay</emphasis> operations are those for which the client received a reply from the server that the operation had been successfully completed. These operations need to be redone in exactly the same manner after a server restart as had been reported before the server failed. Replay can only happen if the server failed; otherwise it will not have lost any state in memory.</para>
<para><emphasis>Resend</emphasis> operations are those for which the client never received a reply, so their final state is unknown to the client. The client sends unanswered requests to the server again in XID order, and again awaits a reply for each one. In some cases, resent requests have been handled and committed to disk by the server (possibly also having dependent operations committed), in which case, the server performs reply reconstruction for the lost reply. In other cases, the server did not receive the lost request at all and processing proceeds as with any normal request. These are what happen in the case of a network interruption. It is also possible that the server received the request, but was unable to reply or commit it to disk before failure.</para>
</section>
<section remap="h3">
- <title>30.2.4 Client Replay List</title>
+ <title>Client Replay List</title>
<para>All file system-modifying requests have the potential to be required for server state recovery (replay) in case of a server failure. Replies that have an assigned transaction number that is higher than the last committed transaction number received in any reply from each server are preserved for later replay in a per-server replay list. As each reply is received from the server, it is checked to see if it has a higher last committed transaction number than the previous highest last committed number. Most requests that now have a lower transaction number can safely be removed from the replay list. One exception to this rule is for open requests, which need to be saved for replay until the file is closed so that the MDS can properly reference count open-unlinked files.</para>
</section>
<section remap="h3">
- <title>30.2.5 Server Recovery</title>
+ <title>Server Recovery</title>
<para>A server enters recovery if it was not shut down cleanly. If, upon startup, if any client entries are in the <literal>last_rcvd</literal> file for any previously connected clients, the server enters recovery mode and waits for these previously-connected clients to reconnect and begin replaying or resending their requests. This allows the server to recreate state that was exposed to clients (a request that completed successfully) but was not committed to disk before failure.</para>
<para>In the absence of any client connection attempts, the server waits indefinitely for the clients to reconnect. This is intended to handle the case where the server has a network problem and clients are unable to reconnect and/or if the server needs to be restarted repeatedly to resolve some problem with hardware or software. Once the server detects client connection attempts - either new clients or previously-connected clients - a recovery timer starts and forces recovery to finish in a finite time regardless of whether the previously-connected clients are available or not.</para>
<para>If no client entries are present in the <literal>last_rcvd</literal> file, or if the administrator manually aborts recovery, the server does not wait for client reconnection and proceeds to allow all clients to connect.</para>
<para>If the client times out during replay, it attempts to reconnect. If the client is unable to reconnect, <literal>REPLAY</literal> fails and it returns to <literal>DISCON</literal> state. It is possible that clients will timeout frequently during <literal>REPLAY</literal>, so reconnection should not delay an already slow process more than necessary. We can mitigate this by increasing the timeout during replay.</para>
</section>
<section remap="h3">
- <title>30.2.6 Request Replay</title>
+ <title>Request Replay</title>
<para>If a client was previously connected, it gets a response from the server telling it that the server is in recovery and what the last committed transaction number on disk is. The client can then iterate through its replay list and use this last committed transaction number to prune any previously-committed requests. It replays any newer requests to the server in transaction number order, one at a time, waiting for a reply from the server before replaying the next request.</para>
<para>Open requests that are on the replay list may have a transaction number lower than the server's last committed transaction number. The server processes those open requests immediately. The server then processes replayed requests from all of the clients in transaction number order, starting at the last committed transaction number to ensure that the state is updated on disk in exactly the same manner as it was before the crash. As each replayed request is processed, the last committed transaction is incremented. If the server receives a replay request from a client that is higher than the current last committed transaction, that request is put aside until other clients provide the intervening transactions. In this manner, the server replays requests in the same sequence as they were previously executed on the server until either all clients are out of requests to replay or there is a gap in a sequence.</para>
</section>
<section remap="h3">
- <title>30.2.7 Gaps in the Replay Sequence</title>
+ <title>Gaps in the Replay Sequence</title>
<para>In some cases, a gap may occur in the reply sequence. This might be caused by lost replies, where the request was processed and committed to disk but the reply was not received by the client. It can also be caused by clients missing from recovery due to partial network failure or client death.</para>
<para>In the case where all clients have reconnected, but there is a gap in the replay sequence the only possibility is that some requests were processed by the server but the reply was lost. Since the client must still have these requests in its resend list, they are processed after recovery is finished.</para>
<para>In the case where all clients have not reconnected, it is likely that the failed clients had requests that will no longer be replayed. The VBR feature is used to determine if a request following a transaction gap is safe to be replayed. Each item in the file system (MDS inode or OST object) stores on disk the number of the last transaction in which it was modified. Each reply from the server contains the previous version number of the objects that it affects. During VBR replay, the server matches the previous version numbers in the resend request against the current version number. If the versions match, the request is the next one that affects the object and can be safely replayed. For more information, see <xref linkend="dbdoclet.50438268_80068"/>.</para>
</section>
<section remap="h3">
- <title>30.2.8 Lock Recovery</title>
+ <title><indexterm><primary>recovery</primary><secondary>locks</secondary></indexterm>Lock Recovery</title>
<para>If all requests were replayed successfully and all clients reconnected, clients then do lock replay locks -- that is, every client sends information about every lock it holds from this server and its state (whenever it was granted or not, what mode, what properties and so on), and then recovery completes successfully. Currently, Lustre does not do lock verification and just trusts clients to present an accurate lock state. This does not impart any security concerns since Lustre 1.x clients are trusted for other information (e.g. user ID) during normal operation also.</para>
<para>After all of the saved requests and locks have been replayed, the client sends an <literal>MDS_GETSTATUS</literal> request with last-replay flag set. The reply to that request is held back until all clients have completed replay (sent the same flagged getstatus request), so that clients don't send non-recovery requests before recovery is complete.</para>
</section>
<section remap="h3">
- <title>30.2.9 Request Resend</title>
+ <title>Request Resend</title>
<para>Once all of the previously-shared state has been recovered on the server (the target file system is up-to-date with client cache and the server has recreated locks representing the locks held by the client), the client can resend any requests that did not receive an earlier reply. This processing is done like normal request processing, and, in some cases, the server may do reply reconstruction.</para>
</section>
</section>
<section xml:id="dbdoclet.50438268_23736">
- <title>30.3 Reply Reconstruction</title>
+ <title>Reply Reconstruction</title>
<para>When a reply is dropped, the MDS needs to be able to reconstruct the reply when the original request is re-sent. This must be done without repeating any non-idempotent operations, while preserving the integrity of the locking system. In the event of MDS failover, the information used to reconstruct the reply must be serialized on the disk in transactions that are joined or nested with those operating on the disk.</para>
<section remap="h3">
- <title>30.3.1 Required State</title>
+ <title>Required State</title>
<para>For the majority of requests, it is sufficient for the server to store three pieces of data in the <literal>last_rcvd</literal> file:</para>
<itemizedlist>
<listitem>
<para>For open requests, the "disposition" of the open must also be stored.</para>
</section>
<section remap="h3">
- <title>30.3.2 Reconstruction of Open Replies</title>
+ <title>Reconstruction of Open Replies</title>
<para>An open reply consists of up to three pieces of information (in addition to the contents of the "request log"):</para>
<itemizedlist>
<listitem>
</section>
</section>
<section xml:id="dbdoclet.50438268_80068">
- <title>30.4 Version-based Recovery</title>
+ <title><indexterm><primary>Version-based recovery (VBR)</primary></indexterm>Version-based Recovery</title>
<para>The Version-based Recovery (VBR) feature improves Lustre reliability in cases where client requests (RPCs) fail to replay during recovery
<footnote>
<para>There are two scenarios under which client RPCs are not replayed: (1) Non-functioning or isolated clients do not reconnect, and they cannot replay their RPCs, causing a gap in the replay sequence. These clients get errors and are evicted. (2) Functioning clients connect, but they cannot replay some or all of their RPCs that occurred after the gap caused by the non-functioning/isolated clients. These clients get errors (caused by the failed clients). With VBR, these requests have a better chance to replay because the "gaps" are only related to specific files that the missing client(s) changed.</para>
<para>During normal operation, the server:</para>
<itemizedlist>
<listitem>
- <para> Updates the versions of all inodes involved in a given operation</para>
+ <para>Updates the versions of all inodes involved in a given operation</para>
</listitem>
<listitem>
- <para> Returns the old and new inode versions to the client with the reply</para>
+ <para>Returns the old and new inode versions to the client with the reply</para>
</listitem>
</itemizedlist>
<para>When the recovery mechanism is underway, VBR follows these steps:</para>
</orderedlist>
<para>VBR recovery is fully transparent to users. It may lead to slightly longer recovery times if the cluster loses several clients during server recovery.</para>
<section remap="h3">
- <title>30.4.1 VBR Messages</title>
+ <title><indexterm><primary>Version-based recovery (VBR)</primary><secondary>messages</secondary></indexterm>VBR Messages</title>
<para>The VBR feature is built into the Lustre recovery functionality. It cannot be disabled. These are some VBR messages that may be displayed:</para>
<screen>DEBUG_REQ(D_WARNING, req, "Version mismatch during replay\n");</screen>
<para>This message indicates why the client was evicted. No action is needed.</para>
<para>This message indicates why the recovery failed. No action is needed.</para>
</section>
<section remap="h3">
- <title>30.4.2 Tips for Using VBR</title>
+ <title><indexterm><primary>Version-based recovery (VBR)</primary><secondary>tips</secondary></indexterm>Tips for Using VBR</title>
<para>VBR will be successful for clients which do not share data with other client. Therefore, the strategy for reliable use of VBR is to store a client's data in its own directory, where possible. VBR can recover these clients, even if other clients are lost.</para>
</section>
</section>
<section xml:id="dbdoclet.50438268_83826">
- <title>30.5 Commit on Share</title>
+ <title><indexterm><primary>commit on share</primary></indexterm>Commit on Share</title>
<para>The commit-on-share (COS) feature makes Lustre recovery more reliable by preventing missing clients from causing cascading evictions of other clients. With COS enabled, if some Lustre clients miss the recovery window after a reboot or a server failure, the remaining clients are not evicted.</para>
<note>
<para>The commit-on-share feature is enabled, by default.</para>
</note>
<section remap="h3">
- <title>30.5.1 Working with Commit on Share</title>
+ <title><indexterm><primary>commit on share</primary><secondary>working with</secondary></indexterm>Working with Commit on Share</title>
<para>To illustrate how COS works, let's first look at the old recovery scenario. After a service restart, the MDS would boot and enter recovery mode. Clients began reconnecting and replaying their uncommitted transactions. Clients could replay transactions independently as long as their transactions did not depend on each other (one client's transactions did not depend on a different client's transactions). The MDS is able to determine whether one transaction is dependent on another transaction via the <xref linkend="dbdoclet.50438268_80068"/> feature.</para>
<para>If there was a dependency between client transactions (for example, creating and deleting the same file), and one or more clients did not reconnect in time, then some clients may have been evicted because their transactions depended on transactions from the missing clients. Evictions of those clients caused more clients to be evicted and so on, resulting in "cascading" client evictions.</para>
<para>COS addresses the problem of cascading evictions by eliminating dependent transactions between clients. It ensures that one transaction is committed to disk if another client performs a transaction dependent on the first one. With no dependent, uncommitted transactions to apply, the clients replay their requests independently without the risk of being evicted.</para>
</section>
<section remap="h3">
- <title>30.5.2 Tuning Commit On Share</title>
+ <title><indexterm><primary>commit on share</primary><secondary>tuning</secondary></indexterm>Tuning Commit On Share</title>
<para>Commit on Share can be enabled or disabled using the <literal>mdt.commit_on_sharing</literal> tunable (0/1). This tunable can be set when the MDS is created (<literal>mkfs.lustre</literal>) or when the Lustre file system is active, using the <literal>lctl set/get_param</literal> or <literal>lctl conf_param</literal> commands.</para>
<para>To set a default value for COS (disable/enable) when the file system is created, use:</para>
<screen>--param mdt.commit_on_sharing=0/1
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretroubleshooting">
- <info>
- <title xml:id="lustretroubleshooting.title">Lustre Troubleshooting</title>
- </info>
+ <title xml:id="lustretroubleshooting.title">Lustre Troubleshooting</title>
<para>This chapter provides information to troubleshoot Lustre, submit a Lustre bug, and Lustre performance tips. It includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438198_11171">
- <title>26.1 Lustre Error Messages</title>
+ <title><indexterm><primary>troubleshooting</primary></indexterm>
+ <indexterm><primary>lustre</primary><secondary>troubleshooting</secondary><see>troubleshooting</see></indexterm>
+ <indexterm><primary>lustre</primary><secondary>errors</secondary><see>troubleshooting</see></indexterm>
+ <indexterm><primary>errors</primary><see>troubleshooting</see></indexterm>
+ Lustre Error Messages</title>
<para>Several resources are available to help troubleshoot Lustre. This section describes error numbers, error messages and logs.</para>
<section remap="h3">
- <title>26.1.1 Error Numbers</title>
+ <title><indexterm><primary>troubleshooting</primary><secondary>error numbers</secondary></indexterm>Error Numbers</title>
<para>Error numbers for Lustre come from the Linux errno.h, and are located in <literal>/usr/include/asm/errno.h</literal>. Lustre does not use all of the available Linux error numbers. The exact meaning of an error number depends on where it is used. Here is a summary of the basic errors that Lustre users may encounter.</para>
<informaltable frame="all">
<tgroup cols="3">
<para> -1</para>
</entry>
<entry>
- <literal>
- -EPERM
- </literal>
+ <literal> -EPERM </literal>
</entry>
<entry>
<para> Permission is denied.</para>
</entry>
</row>
<row>
+ <entry> -2 </entry>
<entry>
- -2
- </entry>
- <entry>
- <literal>
- -ENOENT
- </literal>
+ <literal> -ENOENT </literal>
</entry>
<entry>
<para> The requested file or directory does not exist.</para>
<para> -4</para>
</entry>
<entry>
- <literal>
- -EINTR
- </literal>
+ <literal> -EINTR </literal>
</entry>
<entry>
<para> The operation was interrupted (usually CTRL-C or a killing process).</para>
<para> -5</para>
</entry>
<entry>
- <literal>
- -EIO
- </literal>
+ <literal> -EIO </literal>
</entry>
<entry>
<para> The operation failed with a read or write error.</para>
<para> -19</para>
</entry>
<entry>
- <literal>
- -ENODEV
- </literal>
+ <literal> -ENODEV </literal>
</entry>
<entry>
<para> No such device is available. The server stopped or failed over.</para>
<para> -22</para>
</entry>
<entry>
- <literal>
- -EINVAL
- </literal>
+ <literal> -EINVAL </literal>
</entry>
<entry>
<para> The parameter contains an invalid value.</para>
<para> -28</para>
</entry>
<entry>
- <literal>
- -ENOSPC
- </literal>
+ <literal> -ENOSPC </literal>
</entry>
<entry>
<para> The file system is out-of-space or out of inodes. Use <literal>lfs df</literal> (query the amount of file system space) or <literal>lfs df -i</literal> (query the number of inodes).</para>
<para> -30</para>
</entry>
<entry>
- <literal>
- -EROFS
- </literal>
+ <literal> -EROFS </literal>
</entry>
<entry>
<para> The file system is read-only, likely due to a detected error.</para>
<para> -43</para>
</entry>
<entry>
- <literal>
- -EIDRM
- </literal>
+ <literal> -EIDRM </literal>
</entry>
<entry>
<para> The UID/GID does not match any known UID/GID on the MDS. Update etc/hosts and etc/group on the MDS to add the missing user or group.</para>
<para> -107</para>
</entry>
<entry>
- <literal>
- -ENOTCONN
- </literal>
+ <literal> -ENOTCONN </literal>
</entry>
<entry>
<para> The client is not connected to this server.</para>
<para> -110</para>
</entry>
<entry>
- <literal>
- -ETIMEDOUT
- </literal>
+ <literal> -ETIMEDOUT </literal>
</entry>
<entry>
<para> The operation took too long and timed out.</para>
</informaltable>
</section>
<section xml:id="dbdoclet.50438198_40669">
- <title>26.1.2 Viewing Error Messages</title>
+ <title><indexterm><primary>troubleshooting</primary><secondary>error messages</secondary></indexterm>Viewing Error Messages</title>
<para>As Lustre code runs on the kernel, single-digit error codes display to the application; these error codes are an indication of the problem. Refer to the kernel console log (dmesg) for all recent kernel messages from that node. On the node, <literal>/var/log/messages</literal> holds a log of all messages for at least the past day.</para>
<para>The error message initiates with "LustreError" in the console log and provides a short description of:</para>
<itemizedlist>
<para>Lustre logs are dumped to <literal>/proc/sys/lnet/debug_pat</literal>h.</para>
<para>Collect the first group of messages related to a problem, and any messages that precede "LBUG" or "assertion failure" errors. Messages that mention server nodes (OST or MDS) are specific to that server; you must collect similar messages from the relevant server console logs.</para>
<para>Another Lustre debug log holds information for Lustre action for a short period of time which, in turn, depends on the processes on the node to use Lustre. Use the following command to extract debug logs on each of the nodes, run</para>
- <screen>$ lctl dk <filename>
-</screen>
+ <screen>$ lctl dk <filename></screen>
<note>
<para>LBUG freezes the thread to allow capture of the panic stack. A system reboot is needed to clear the thread.</para>
</note>
</section>
</section>
<section xml:id="dbdoclet.50438198_30989">
- <title>26.2 Reporting a Lustre Bug</title>
+ <title><indexterm><primary>troubleshooting</primary><secondary>reporting bugs</secondary></indexterm><indexterm><primary>reporting bugs</primary><see>troubleshooing</see></indexterm>
+ Reporting a Lustre Bug</title>
<para>If, after troubleshooting your Lustre system, you cannot resolve the problem, consider reporting a Lustre bug. The process for reporting a bug is described in the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</link>.</para>
<para>You can also post a question to the <link xl:href="http://wiki.lustre.org/index.php/Lustre_Mailing_Lists">lustre-discuss mailing list</link> or search the <link xl:href="http://groups.google.com/group/lustre-discuss-list">lustre-discuss Archives</link> for information about your issue.</para>
<para>A Lustre diagnostics tool is available for downloading at: <link xl:href="http://downloads.lustre.org/public/tools/lustre-diagnostics/">http://downloads.lustre.org/public/tools/lustre-diagnostics/</link></para>
<para>Output is sent directly to the terminal. Use normal file redirection to send the output to a file, and then manually attach the file to the bug you are submitting.</para>
</section>
<section xml:id="dbdoclet.50438198_93109">
- <title>26.3 Common Lustre Problems</title>
+ <title><indexterm><primary>troubleshooting</primary><secondary>common problems</secondary></indexterm>Common Lustre Problems</title>
<para>This section describes how to address common issues encountered with Lustre.</para>
<section remap="h3">
- <title>26.3.1 OST Object is Missing or Damaged</title>
+ <title>OST Object is Missing or Damaged</title>
<para>If the OSS fails to find an object or finds a damaged object, this message appears:</para>
<para><screen>OST object missing or damaged (OST "ost1", object 98148, error -2)</screen></para>
<para>If the reported error is -2 (<literal>-ENOENT</literal>, or "No such file or directory"), then the object is missing. This can occur either because the MDS and OST are out of sync, or because an OST object was corrupted and deleted.</para>
<para>If the reported error is anything else, then you should immediately inspect this server for storage problems.</para>
</section>
<section remap="h3">
- <title>26.3.2 OSTs Become Read-Only</title>
+ <title>OSTs Become Read-Only</title>
<para>If the SCSI devices are inaccessible to Lustre at the block device level, then <literal>ldiskfs</literal> remounts the device read-only to prevent file system corruption. This is a normal behavior. The status in <literal>/proc/fs/lustre/health_check</literal> also shows "not healthy" on the affected nodes.</para>
<para>To determine what caused the "not healthy" condition:</para>
<itemizedlist>
<para>To recover from this problem, you must restart Lustre services using these file systems. There is no other way to know that the I/O made it to disk, and the state of the cache may be inconsistent with what is on disk.</para>
</section>
<section remap="h3">
- <title>26.3.3 Identifying a Missing OST</title>
+ <title>Identifying a Missing OST</title>
<para>If an OST is missing for any reason, you may need to know what files are affected. Although an OST is missing, the files system should be operational. From any mounted client node, generate a list of files that reside on the affected OST. It is advisable to mark the missing OST as 'unavailable' so clients and the MDS do not time out trying to contact it.</para>
<orderedlist>
<listitem>
<screen>$ lctl --device <OST device name or number> deactivate</screen>
<para>The OST device number or device name is generated by the lctl dl command.</para>
<para>The <literal>deactivate</literal> command prevents clients from creating new objects on the specified OST, although you can still access the OST for reading.</para>
- <note>
- <para>If the OST later becomes available it needs to be reactivated, run:</para>
- <screen># lctl --device <OST device name or number> activate</screen>
- </note>
+ <note>
+
<para>If the OST later becomes available it needs to be reactivated, run:</para>
+ <screen># lctl --device <OST device name or number> activate</screen>
+ </note>
</listitem>
<listitem>
<para>Determine all files that are striped over the missing OST, run:</para>
<listitem>
<para>You can delete these files with the unlink or munlink command.</para>
<screen># unlink|munlink filename {filename ...} </screen>
- <note>
- <para>There is no functional difference between the <literal>unlink</literal> and <literal>munlink</literal> commands. The unlink command is for newer Linux distributions. You can run <literal>munlink</literal> if <literal>unlink</literal> is not available.</para>
- <para>When you run the <literal>unlink</literal> or <literal>munlink</literal> command, the file on the MDS is permanently removed.</para>
- </note>
+ <note>
+
<para>There is no functional difference between the <literal>unlink</literal> and <literal>munlink</literal> commands. The unlink command is for newer Linux distributions. You can run <literal>munlink</literal> if <literal>unlink</literal> is not available.</para>
+
<para>When you run the <literal>unlink</literal> or <literal>munlink</literal> command, the file on the MDS is permanently removed.</para>
+ </note>
</listitem>
<listitem>
<para>If you need to know, specifically, which parts of the file are missing data, then you first need to determine the file layout (striping pattern), which includes the index of the missing OST). Run:</para>
<para>If the file system cannot be mounted, currently there is no way that parses metadata directly from an MDS. If the bad OST does not start, options to mount the file system are to provide a loop device OST in its place or replace it with a newly-formatted OST. In that case, the missing objects are created and are read as zero-filled.</para>
</section>
<section xml:id="dbdoclet.50438198_69657">
- <title>26.3.4 Fixing a Bad LAST_ID on an OST</title>
+ <title>Fixing a Bad LAST_ID on an OST</title>
<para>Each OST contains a LAST_ID file, which holds the last object (pre-)created by the MDS <footnote>
<para>The contents of the LAST_ID file must be accurate regarding the actual objects that exist on the OST.</para>
</footnote>. The MDT contains a lov_objid file, with values that represent the last object the MDS has allocated to a file.</para>
</orderedlist>
</section>
<section remap="h3">
- <title>26.3.5 Handling/Debugging "<literal>Bind: Address already in use</literal>" Error</title>
+ <title><indexterm><primary>troubleshooting</primary><secondary>'Address already in use'</secondary></indexterm>Handling/Debugging "<literal>Bind: Address already in use</literal>" Error</title>
<para>During startup, Lustre may report a <literal>bind: Address already in use</literal> error and reject to start the operation. This is caused by a portmap service (often NFS locking) which starts before Lustre and binds to the default port 988. You must have port 988 open from firewall or IP tables for incoming connections on the client, OSS, and MDS nodes. LNET will create three outgoing connections on available, reserved ports to each client-server pair, starting with 1023, 1022 and 1021.</para>
<para>Unfortunately, you cannot set sunprc to avoid port 988. If you receive this error, do the following:</para>
<itemizedlist>
</note>
</section>
<section remap="h3">
- <title>26.3.6 Handling/Debugging Error "- 28"</title>
+ <title><indexterm><primary>troubleshooting</primary><secondary>'Error -28'</secondary></indexterm>Handling/Debugging Error "- 28"</title>
<para>A Linux error -28 (<literal>ENOSPC</literal>) that occurs during a write or sync operation indicates that an existing file residing on an OST could not be rewritten or updated because the OST was full, or nearly full. To verify if this is the case, on a client on which the OST is mounted, enter :</para>
<screen>lfs df -h</screen>
<para>To address this issue, you can do one of the following:</para>
</note>
</section>
<section remap="h3">
- <title>26.3.7 Triggering Watchdog for PID NNN</title>
+ <title>Triggering Watchdog for PID NNN</title>
<para>In some cases, a server node triggers a watchdog timer and this causes a process stack to be dumped to the console along with a Lustre kernel debug log being dumped into <literal>/tmp</literal> (by default). The presence of a watchdog timer does NOT mean that the thread OOPSed, but rather that it is taking longer time than expected to complete a given operation. In some cases, this situation is expected.</para>
<para>For example, if a RAID rebuild is really slowing down I/O on an OST, it might trigger watchdog timers to trip. But another message follows shortly thereafter, indicating that the thread in question has completed processing (after some number of seconds). Generally, this indicates a transient problem. In other cases, it may legitimately signal that a thread is stuck because of a software error (lock inversion, for example).</para>
<screen>Lustre: 0:0:(watchdog.c:122:lcw_cb()) </screen>
</screen>
</section>
<section remap="h3">
- <title>26.3.8 Handling Timeouts on Initial Lustre Setup</title>
+ <title><indexterm><primary>troubleshooting</primary><secondary>timeouts on setup</secondary></indexterm>Handling Timeouts on Initial Lustre Setup</title>
<para>If you come across timeouts or hangs on the initial setup of your Lustre system, verify that name resolution for servers and clients is working correctly. Some distributions configure <literal>/etc/hosts sts</literal> so the name of the local machine (as reported by the 'hostname' command) is mapped to local host (127.0.0.1) instead of a proper IP address.</para>
<para>This might produce this error:</para>
<screen>LustreError:(ldlm_handle_cancel()) received cancel for unknown lock cookie
</screen>
</section>
<section remap="h3">
- <title>26.3.9 Handling/Debugging "LustreError: xxx went back in time"</title>
+ <title>Handling/Debugging "LustreError: xxx went back in time"</title>
<para>Each time Lustre changes the state of the disk file system, it records a unique transaction number. Occasionally, when committing these transactions to the disk, the last committed transaction number displays to other nodes in the cluster to assist the recovery. Therefore, the promised transactions remain absolutely safe on the disappeared disk.</para>
<para>This situation arises when:</para>
<itemizedlist>
<para>If the error occurs during failover, examine your disk cache settings. If it occurs after a restart without failover, try to determine how the disk can report that a write succeeded, then lose the Data Device corruption or Disk Errors.</para>
</section>
<section remap="h3">
- <title>26.3.10 Lustre Error: "<literal>Slow Start_Page_Write</literal>"</title>
+ <title>Lustre Error: "<literal>Slow Start_Page_Write</literal>"</title>
<para>The slow <literal>start_page_write</literal> message appears when the operation takes an extremely long time to allocate a batch of memory pages. Use these pages to receive network traffic first, and then write to disk.</para>
</section>
<section remap="h3">
- <title>26.3.11 Drawbacks in Doing Multi-client O_APPEND Writes</title>
+ <title>Drawbacks in Doing Multi-client O_APPEND Writes</title>
<para>It is possible to do multi-client <literal>O_APPEND</literal> writes to a single file, but there are few drawbacks that may make this a sub-optimal solution. These drawbacks are:</para>
<itemizedlist>
<listitem>
</itemizedlist>
</section>
<section remap="h3">
- <title>26.3.12 Slowdown Occurs During Lustre Startup</title>
+ <title><indexterm><primary>troubleshooting</primary><secondary>slowdown during startup</secondary></indexterm>Slowdown Occurs During Lustre Startup</title>
<para>When Lustre starts, the Lustre file system needs to read in data from the disk. For the very first mdsrate run after the reboot, the MDS needs to wait on all the OSTs for object pre-creation. This causes a slowdown to occur when Lustre starts up.</para>
<para>After the file system has been running for some time, it contains more data in cache and hence, the variability caused by reading critical metadata from disk is mostly eliminated. The file system now reads data from the cache.</para>
</section>
<section remap="h3">
- <title>26.3.13 Log Message <literal>'Out of Memory</literal>' on OST</title>
+ <title><indexterm><primary>troubleshooting</primary><secondary>OST out of memory</secondary></indexterm>Log Message <literal>'Out of Memory</literal>' on OST</title>
<para>When planning the hardware for an OSS node, consider the memory usage of several components in the Lustre system. If insufficient memory is available, an 'out of memory' message can be logged.</para>
<para>During normal operation, several conditions indicate insufficient RAM on a server node:</para>
<itemizedlist>
<para>For information on determining the MDS memory and OSS memory requirements, see <xref linkend="dbdoclet.50438256_26456"/>.</para>
</section>
<section remap="h3">
- <title>26.3.14 Setting SCSI I/O Sizes</title>
+ <title>Setting SCSI I/O Sizes</title>
<para>Some SCSI drivers default to a maximum I/O size that is too small for good Lustre performance. we have fixed quite a few drivers, but you may still find that some drivers give unsatisfactory performance with Lustre. As the default value is hard-coded, you need to recompile the drivers to change their default. On the other hand, some drivers may have a wrong default set.</para>
<para>If you suspect bad I/O performance and an analysis of Lustre statistics indicates that I/O is not 1 MB, check <literal>/sys/block/<device>/queue/max_sectors_kb</literal>. If the <literal>max_sectors_kb</literal> value is less than 1024, set it to at least 1024 to improve performance. If changing <literal>max_sectors_kb</literal> does not change the I/O size as reported by Lustre, you may want to examine the SCSI driver code.</para>
</section>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretuning">
- <info>
- <title xml:id="lustretuning.title">Lustre Tuning</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretuning">
+ <title xml:id="lustretuning.title">Lustre Tuning</title>
<para>This chapter contains information about tuning Lustre for better performance and includes the following sections:</para>
<itemizedlist>
<listitem>
<para>Many options in Lustre are set by means of kernel module parameters. These parameters are contained in the <literal>modprobe.conf</literal> file.</para>
</note>
<section xml:id="dbdoclet.50438272_55226">
- <title>25.1 Optimizing the Number of Service Threads</title>
+ <title>
+ <indexterm><primary>tuning</primary></indexterm>
+<indexterm><primary>tuning</primary><secondary>service threads</secondary></indexterm>
+ Optimizing the Number of Service Threads</title>
<para>An OSS can have a minimum of 2 service threads and a maximum of 512 service threads. The number of service threads is a function of how much RAM and how many CPUs are on each OSS node (1 thread / 128MB * num_cpus). If the load on the OSS node is high, new service threads will be started in order to process more requests concurrently, up to 4x the initial number of threads (subject to the maximum of 512). For a 2GB 2-CPU system, the default thread count is 32 and the maximum thread count is 128.</para>
<para>Increasing the size of the thread pool may help when:</para>
<itemizedlist>
<para>If there are too many threads, the latency for individual I/O requests can become very high and should be avoided. Set the desired maximum thread count permanently using the method described above.</para>
</note>
<section remap="h3">
- <title>25.1.1 Specifying the OSS Service Thread Count</title>
+ <title><indexterm><primary>tuning</primary><secondary>OSS threads</secondary></indexterm>Specifying the OSS Service Thread Count</title>
<para>The <literal>oss_num_threads</literal> parameter enables the number of OST service threads to be specified at module load time on the OSS nodes:</para>
<screen>options ost oss_num_threads={N}</screen>
<para>After startup, the minimum and maximum number of OSS thread counts can be set via the <literal>{service}.thread_{min,max,started}</literal> tunable. To change the tunable at runtime, run:</para>
<para>For details, see <xref linkend="dbdoclet.50438271_87260"/>.</para>
</section>
<section remap="h3">
- <title>25.1.2 Specifying the MDS Service Thread Count</title>
+ <title><indexterm><primary>tuning</primary><secondary>MDS threads</secondary></indexterm>Specifying the MDS Service Thread Count</title>
<para>The <literal>mds_num_threads</literal> parameter enables the number of MDS service threads to be specified at module load time on the MDS node:</para>
<screen>options mds mds_num_threads={N}</screen>
<para>After startup, the minimum and maximum number of MDS thread counts can be set via the <literal>{service}.thread_{min,max,started}</literal> tunable. To change the tunable at runtime, run:</para>
</section>
</section>
<section xml:id="dbdoclet.50438272_73839">
- <title>25.2 Tuning LNET Parameters</title>
+ <title>
+ <indexterm><primary>LNET</primary><secondary>tuning</secondary>
+ </indexterm><indexterm><primary>tuning</primary><secondary>LNET</secondary></indexterm>
+ Tuning LNET Parameters</title>
<para>This section describes LNET tunables. that may be necessary on some systems to improve performance. To test the performance of your Lustre network, see <link xl:href="LNETSelfTest.html#50438223_71556">Chapter 23</link>: <link xl:href="LNETSelfTest.html#50438223_21832">Testing Lustre Network Performance (LNET Self-Test)</link>.</para>
<section remap="h3">
- <title>25.2.1 Transmit and Receive Buffer Size</title>
+ <title>Transmit and Receive Buffer Size</title>
<para>The kernel allocates buffers for sending and receiving messages on a network.</para>
<para><literal>ksocklnd</literal> has separate parameters for the transmit and receive buffers.</para>
<screen>options ksocklnd tx_buffer_size=0 rx_buffer_size=0
<para>If these parameters are left at the default value (0), the system automatically tunes the transmit and receive buffer size. In almost every case, this default produces the best performance. Do not attempt to tune these parameters unless you are a network expert.</para>
</section>
<section remap="h3">
- <title>25.2.2 Hardware Interrupts (<literal>enable_irq_affinity</literal>)</title>
+ <title>Hardware Interrupts (<literal>enable_irq_affinity</literal>)</title>
<para>The hardware interrupts that are generated by network adapters may be handled by any CPU in the system. In some cases, we would like network traffic to remain local to a single CPU to help keep the processor cache warm and minimize the impact of context switches. This is helpful when an SMP system has more than one network interface and ideal when the number of interfaces equals the number of CPUs. To enable the <literal>enable_irq_affinity</literal> parameter, enter:</para>
<screen>options ksocklnd enable_irq_affinity=1</screen>
<para>In other cases, if you have an SMP platform with a single fast interface such as 10Gb Ethernet and more than two CPUs, you may see performance improve by turning this parameter off.</para>
</section>
</section>
<section xml:id="dbdoclet.50438272_25884">
- <title>25.3 Lockless I/O Tunables</title>
+ <title><indexterm><primary>tuning</primary><secondary>lockless I/O</secondary></indexterm>Lockless I/O Tunables</title>
<para>The lockless I/O tunable feature allows servers to ask clients to do lockless I/O (liblustre-style where the server does the locking) on contended files.</para>
<para>The lockless I/O patch introduces these tunables:</para>
<itemizedlist>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438272_80545">
- <title>25.4 Improving Lustre Performance When Working with Small Files</title>
+ <title><indexterm><primary>tuning</primary><secondary>for small files</secondary></indexterm>Improving Lustre Performance When Working with Small Files</title>
<para>A Lustre environment where an application writes small file chunks from many clients to a single file will result in bad I/O performance. To improve Lustre'™s performance with small files:</para>
<itemizedlist>
<listitem>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438272_45406">
- <title>25.5 Understanding Why Write Performance Is Better Than Read Performance</title>
+ <title><indexterm><primary>tuning</primary><secondary>write performance</secondary></indexterm>Understanding Why Write Performance is Better Than Read Performance</title>
<para>Typically, the performance of write operations on a Lustre cluster is better than read operations. When doing writes, all clients are sending write RPCs asynchronously. The RPCs are allocated, and written to disk in the order they arrive. In many cases, this allows the back-end storage to aggregate writes efficiently.</para>
<para>In the case of read operations, the reads from clients may come in a different order and need a lot of seeking to get read from the disk. This noticeably hampers the read throughput.</para>
<para>Currently, there is no readahead on the OSTs themselves, though the clients do readahead. If there are lots of clients doing reads it would not be possible to do any readahead in any case because of memory consumption (consider that even a single RPC (1 MB) readahead for 1000 clients would consume 1 GB of RAM).</para>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingfailover">
- <info>
- <title xml:id="managingfailover.title">Managing Failover</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingfailover">
+ <title xml:id="managingfailover.title">Managing Failover</title>
<para>This chapter describes failover in a Lustre system and includes the following sections:</para>
<itemizedlist>
<listitem>
<para>For information about high availability(HA) management software, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Red_Hat_Cluster_Manager_with_Lustre">Using Red Hat Cluster Manager with Lustre</link> or the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre">LuUsing Pacemaker with stre</link>.</para>
</note>
<section xml:id="dbdoclet.50438213_13563">
- <title>20.1 Lustre Failover and <anchor xml:id="dbdoclet.50438213_marker-1301522" xreflabel=""/>Multiple-Mount Protection</title>
+ <title>
+ <indexterm><primary>multiple-mount protection</primary><see>failover</see></indexterm>
+ <indexterm><primary>failover</primary></indexterm>
+ Lustre Failover and Multiple-Mount Protection</title>
<para>The failover functionality in Lustre is implemented with the multiple-mount protection (MMP) feature, which protects the file system from being mounted simultaneously to more than one node. This feature is important in a shared storage environment (for example, when a failover pair of OSTs share a partition).</para>
<para>Lustre's backend file system, <literal>ldiskfs</literal>, supports the MMP mechanism. A block in the file system is updated by a <literal>kmmpd</literal> daemon at one second intervals, and a sequence number is written in this block. If the file system is cleanly unmounted, then a special "clean" sequence is written to this block. When mounting the file system, <literal>ldiskfs</literal> checks if the MMP block has a clean sequence or not.</para>
<para>Even if the MMP block has a clean sequence, <literal>ldiskfs</literal> waits for some interval to guard against the following situations:</para>
<para>The MMP feature is only supported on Linux kernel versions >= 2.6.9.</para>
</note>
<section remap="h3">
- <title>20.1.1 Working with Multiple-Mount Protection</title>
+ <title><indexterm><primary>failover</primary><secondary>multiple-mount protection</secondary></indexterm>Working with Multiple-Mount Protection</title>
<para>On a new Lustre file system, MMP is automatically enabled by mkfs.lustre at format time if failover is being used and the kernel and <literal>e2fsprogs</literal> version support it. On an existing file system, a Lustre administrator can manually enable MMP when the file system is unmounted.</para>
<para>Use the following commands to determine whether MMP is running in Lustre and to enable or disable the MMP feature.</para>
<para>To determine if MMP is enabled, run:</para>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingfilesystemio">
- <info>
- <title xml:id="managingfilesystemio.title">Managing the File System and I/O</title>
- </info>
+ <title xml:id="managingfilesystemio.title">Managing the File System and I/O</title>
<para>This chapter describes file striping and I/O options, and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438211_17536">
- <title>19.1 Handling Full OSTs</title>
+ <title><indexterm><primary>I/O</primary></indexterm>
+ <indexterm><primary>I/O</primary><secondary>full OSTs</secondary></indexterm>
+ Handling Full OSTs</title>
<para>Sometimes a Lustre file system becomes unbalanced, often due to incorrectly-specified stripe settings, or when very large files are created that are not striped over all of the OSTs. If an OST is full and an attempt is made to write more information to the file system, an error occurs. The procedures below describe how to handle a full OST.</para>
<para>The MDS will normally handle space balancing automatically at file creation time, and this procedure is normally not needed, but may be desirable in certain circumstances (e.g. when creating very large files that would consume more than the total free space of the full OSTs).</para>
<section remap="h3">
- <title>19.1.1 Checking OST Space Usage</title>
+ <title><indexterm><primary>I/O</primary><secondary>OST space usage</secondary></indexterm>Checking OST Space Usage</title>
<para>The example below shows an unbalanced file system:</para>
<screen>root@LustreClient01 ~]# lfs df -h
UUID bytes Used Available \
1017192448 bytes (1.0 GB) copied, 23.2411 seconds, 43.8 MB/s</screen>
</section>
<section remap="h3">
- <title>19.1.2 Taking a Full OST Offline</title>
+ <title><indexterm><primary>I/O</primary><secondary>taking OST offline</secondary></indexterm>Taking a Full OST Offline</title>
<para>To avoid running out of space in the file system, if the OST usage is imbalanced and one or more OSTs are close to being full while there are others that have a lot of space, the full OSTs may optionally be deactivated at the MDS to prevent the MDS from allocating new objects there.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Log into the MDS server:</emphasis></para>
+ <para>Log into the MDS server:</para>
<screen>[root@LustreClient01 ~]# ssh root@192.168.0.10
root@192.168.0.10's password:
Last login: Wed Nov 26 13:35:12 2008 from 192.168.0.6</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Use the <literal>lctl dl</literal> command to show the status of all file system components:</emphasis></para>
+ <para>Use the <literal>lctl dl</literal> command to show the status of all file system components:</para>
<screen>[root@mds ~]# lctl dl
0 UP mgs MGS MGS 9
1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Use <literal>lctl</literal> deactivate to take the full OST offline:</emphasis></para>
+ <para>Use <literal>lctl</literal> deactivate to take the full OST offline:</para>
<screen>[root@mds ~]# lctl --device 7 deactivate</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Display the status of the file system components:</emphasis></para>
+ <para>Display the status of the file system components:</para>
<screen>[root@mds ~]# lctl dl
0 UP mgs MGS MGS 9
1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
<para>The device list shows that OST0002 is now inactive. When new files are created in the file system, they will only use the remaining active OSTs. Either manual space rebalancing can be done by migrating data to other OSTs, as shown in the next section, or normal file deletion and creation can be allowed to passively rebalance the space usage.</para>
</section>
<section remap="h3">
- <title>19.1.3 Migrating Data within a File System</title>
+ <title>
+ <indexterm><primary>I/O</primary><secondary>migrating data</secondary></indexterm>
+ <indexterm><primary>maintance</primary><secondary>full OSTs</secondary></indexterm>
+ Migrating Data within a File System</title>
<para>As stripes cannot be moved within the file system, data must be migrated manually by copying and renaming the file, removing the original file, and renaming the new file with the original file name. The simplest way to do this is to use the <literal>lfs_migrate</literal> command (see <xref linkend="dbdoclet.50438206_42260"/>). However, the steps for migrating a file by hand are also shown here for reference.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Identify the file(s) to be moved.</emphasis></para>
+ <para>Identify the file(s) to be moved.</para>
<para>In the example below, output from the <literal>getstripe</literal> command indicates that the file <literal>test_2</literal> is located entirely on OST2:</para>
<screen>[client]# lfs getstripe /mnt/lustre/test_2
/mnt/lustre/test_2
2 8 0x8 0</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">To move single object(s), create a new copy and remove the original. Enter:</emphasis></para>
+ <para>To move single object(s), create a new copy and remove the original. Enter:</para>
<screen>[client]# cp -a /mnt/lustre/test_2 /mnt/lustre/test_2.tmp
[client]# mv /mnt/lustre/test_2.tmp /mnt/lustre/test_2</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">To migrate large files from one or more OSTs, enter:</emphasis></para>
+ <para>To migrate large files from one or more OSTs, enter:</para>
<screen>[client]# lfs find --ost {OST_UUID} -size +1G | lfs_migrate -y</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Check the file system balance.</emphasis></para>
+ <para>Check the file system balance.</para>
<para>The <literal>df</literal> output in the example below shows a more balanced system compared to the <literal>df</literal> output in the example in <xref linkend="dbdoclet.50438211_17536"/>.</para>
<screen>[client]# lfs df -h
UUID bytes Used Available Use% \
</orderedlist>
</section>
<section remap="h3">
- <title>19.1.4 Returning an Inactive OST Back Online</title>
+ <title>
+ <indexterm><primary>I/O</primary><secondary>bringing OST online</secondary></indexterm>
+ <indexterm><primary>maintance</primary><secondary>bringing OST online</secondary></indexterm>
+
+ Returning an Inactive OST Back Online</title>
<para>Once the deactivated OST(s) no longer are severely imbalanced, due to either active or passive data redistribution, they should be reactivated so they will again have new files allocated on them.</para>
<screen>[mds]# lctl --device 7 activate
[mds]# lctl dl
</section>
</section>
<section xml:id="dbdoclet.50438211_75549">
- <title>19.2 Creating and Managing OST Pools</title>
+ <title>
+ <indexterm><primary>I/O</primary><secondary>pools</secondary></indexterm>
+ <indexterm><primary>maintance</primary><secondary>pools</secondary></indexterm>
+ <indexterm><primary>pools</primary></indexterm>
+ Creating and Managing OST Pools</title>
<para>The OST pools feature enables users to group OSTs together to make object placement more flexible. A 'pool' is the name associated with an arbitrary subset of OSTs in a Lustre cluster.</para>
<para>OST pools follow these rules:</para>
<itemizedlist>
<listitem>
- <para> An OST can be a member of multiple pools.</para>
+ <para>An OST can be a member of multiple pools.</para>
</listitem>
<listitem>
- <para> No ordering of OSTs in a pool is defined or implied.</para>
+ <para>No ordering of OSTs in a pool is defined or implied.</para>
</listitem>
<listitem>
- <para> Stripe allocation within a pool follows the same rules as the normal stripe allocator.</para>
+ <para>Stripe allocation within a pool follows the same rules as the normal stripe allocator.</para>
</listitem>
<listitem>
- <para> OST membership in a pool is flexible, and can change over time.</para>
+ <para>OST membership in a pool is flexible, and can change over time.</para>
</listitem>
</itemizedlist>
<para>When an OST pool is defined, it can be used to allocate files. When file or directory striping is set to a pool, only OSTs in the pool are candidates for striping. If a stripe_index is specified which refers to an OST that is not a member of the pool, an error is returned.</para>
<para>If a directory has pool striping set and the pool is subsequently removed, the new files created in this directory have the (non-pool) default striping pattern for that directory applied and no error is returned.</para>
</note>
<section remap="h3">
- <title>19.2.1 Working with OST Pools</title>
+ <title>Working with OST Pools</title>
<para>OST pools are defined in the configuration log on the MGS. Use the lctl command to:</para>
<itemizedlist>
<listitem>
- <para> Create/destroy a pool</para>
+ <para>Create/destroy a pool</para>
</listitem>
<listitem>
- <para> Add/remove OSTs in a pool</para>
+ <para>Add/remove OSTs in a pool</para>
</listitem>
<listitem>
- <para> List pools and OSTs in a specific pool</para>
+ <para>List pools and OSTs in a specific pool</para>
</listitem>
</itemizedlist>
<para>The lctl command MUST be run on the MGS. Another requirement for managing OST pools is to either have the MDT and MGS on the same node or have a Lustre client mounted on the MGS node, if it is separate from the MDS. This is needed to validate the pool commands being run are correct.</para>
<para>Running the <literal>writeconf</literal> command on the MDS erases all pools information (as well as any other parameters set using <literal>lctl conf_param</literal>). We recommend that the pools definitions (and <literal>conf_param</literal> settings) be executed using a script, so they can be reproduced easily after a <literal>writeconf</literal> is performed.</para>
</caution>
<para>To create a new pool, run:</para>
- <screen>lctl pool_new <fsname>.<poolname>
-</screen>
+ <screen>lctl pool_new <fsname>.<poolname></screen>
<note>
<para>The pool name is an ASCII string up to 16 characters.</para>
</note>
<para>To add the named OST to a pool, run:</para>
- <screen>lctl pool_add <fsname>.<poolname> <ost_list>
-</screen>
+ <screen>lctl pool_add <fsname>.<poolname> <ost_list></screen>
<para>Where:</para>
<itemizedlist>
<listitem>
<para>To list OSTs in a named pool, run:</para>
<screen>lctl pool_list <fsname>.<poolname></screen>
<section remap="h4">
- <title>19.2.1.1 Using the lfs Command with OST Pools</title>
+ <title>Using the lfs Command with OST Pools</title>
<para>Several lfs commands can be run with OST pools. Use the <literal>lfs setstripe</literal> command to associate a directory with an OST pool. This causes all new regular files and directories in the directory to be created in the pool. The lfs command can be used to list pools in a file system and OSTs in a named pool.</para>
<para>To associate a directory with a pool, so all new files and directories will be created in the pool, run:</para>
<screen>lfs setstripe --pool|-p pool_name <filename|dirname> </screen>
</section>
</section>
<section remap="h3">
- <title>19.2.2 Tips for Using OST Pools</title>
+ <title><indexterm><primary>pools</primary><secondary>usage tips</secondary></indexterm>Tips for Using OST Pools</title>
<para>Here are several suggestions for using OST pools.</para>
<itemizedlist>
<listitem>
</section>
</section>
<section xml:id="dbdoclet.50438211_11204">
- <title>19.3 Adding an OST to a Lustre File System</title>
+ <title><indexterm><primary>I/O</primary><secondary>adding an OST</secondary></indexterm>Adding an OST to a Lustre File System</title>
<para>To add an OST to existing Lustre file system:</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Add a new OST by passing on the following commands, run:</emphasis></para>
+ <para>Add a new OST by passing on the following commands, run:</para>
<screen>$ mkfs.lustre --fsname=spfs --ost --mgsnode=mds16@tcp0 /dev/sda
$ mkdir -p /mnt/test/ost0
$ mount -t lustre /dev/sda /mnt/test/ost0</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Migrate the data (possibly).</emphasis></para>
+ <para>Migrate the data (possibly).</para>
<para>The file system is quite unbalanced when new empty OSTs are added. New file creations are automatically balanced. If this is a scratch file system or files are pruned at a regular interval, then no further work may be needed. Files existing prior to the expansion can be rebalanced with an in-place copy, which can be done with a simple script.</para>
<para>The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
<para>A very clever migration script would do the following:</para>
<para>If a Lustre administrator wants to explore this approach further, per-OST disk-usage statistics can be found under <literal>/proc/fs/lustre/osc/*/rpc_stats</literal></para>
</section>
<section xml:id="dbdoclet.50438211_80295">
- <title>19.4 Performing Direct I/O</title>
+ <title><indexterm><primary>I/O</primary><secondary>direct</secondary></indexterm>Performing Direct I/O</title>
<para>Lustre supports the <literal>O_DIRECT</literal> flag to open.</para>
<para>Applications using the <literal>read()</literal> and <literal>write()</literal> calls must supply buffers aligned on a page boundary (usually 4 K). If the alignment is not correct, the call returns <literal>-EINVAL</literal>. Direct I/O may help performance in cases where the client is doing a large amount of I/O and is CPU-bound (CPU utilization 100%).</para>
<section remap="h3">
- <title>19.4.1 Making File System Objects Immutable</title>
+ <title>Making File System Objects Immutable</title>
<para>An immutable file or directory is one that cannot be modified, renamed or removed. To do this:</para>
<screen>chattr +i <file></screen>
<para>To remove this flag, use <literal>chattr -i</literal></para>
</section>
</section>
<section xml:id="dbdoclet.50438211_61024">
- <title>19.5 Other I/O Options</title>
+ <title>Other I/O Options</title>
<para>This section describes other I/O options, including checksums.</para>
<section remap="h3">
- <title>19.5.1 Lustre Checksums</title>
+ <title>Lustre Checksums</title>
<para>To guard against network data corruption, a Lustre client can perform two types of data checksums: in-memory (for data in client memory) and wire (for data sent over the network). For each checksum type, a 32-bit checksum of the data read or written on both the client and server is computed, to ensure that the data has not been corrupted in transit over the network. The <literal>ldiskfs</literal> backing file system does NOT do any persistent checksumming, so it does not detect corruption of data in the OST file system.</para>
<para>The checksumming feature is enabled, by default, on individual client nodes. If the client or OST detects a checksum mismatch, then an error is logged in the syslog of the form:</para>
<screen>LustreError: BAD WRITE CHECKSUM: changed in transit before arrival at OST: \
<para>To check the status of a wire checksum, run:</para>
<screen>lctl get_param osc.*.checksums</screen>
<section remap="h4">
- <title>19.5.1.1 Changing Checksum Algorithms</title>
+ <title>Changing Checksum Algorithms</title>
<para>By default, Lustre uses the adler32 checksum algorithm, because it is robust and has a lower impact on performance than crc32. The Lustre administrator can change the checksum algorithm via /proc, depending on what is supported in the kernel.</para>
<para>To check which checksum algorithm is being used by Lustre, run:</para>
<screen>$ cat /proc/fs/lustre/osc/<fsname>-OST<index>-osc-*/checksum_type</screen>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managinglnet">
- <info>
- <title xml:id="managinglnet.title">Managing Lustre Networking (LNET)</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managinglnet">
+ <title xml:id="managinglnet.title">Managing Lustre Networking (LNET)</title>
<para>This chapter describes some tools for managing Lustre Networking (LNET) and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438203_51732">
- <title>15.1 Updating the Health Status of a Peer or Router</title>
+ <title><indexterm><primary>LNET</primary><secondary>management</secondary></indexterm>
+ Updating the Health Status of a Peer or Router</title>
<para>There are two mechanisms to update the health status of a peer or a router:</para>
<itemizedlist>
<listitem>
<para>The router pinger actively checks the router health by sending pings, but LNDs only notice a dead peer when there is network traffic going on.</para>
</listitem>
<listitem>
- <para> The router pinger can bring a router from alive to dead or vice versa, but LNDs can only bring a peer down.</para>
+ <para>The router pinger can bring a router from alive to dead or vice versa, but LNDs can only bring a peer down.</para>
</listitem>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438203_48703">
- <title>15.2 Starting and Stopping LNET</title>
+ <title><indexterm><primary>LNET</primary><secondary>starting/stopping</secondary></indexterm>Starting and Stopping LNET</title>
<para>Lustre automatically starts and stops LNET, but it can also be manually started in a standalone manner. This is particularly useful to verify that your networking setup is working correctly before you attempt to start Lustre.</para>
<section remap="h3">
- <title>15.2.1 Starting LNET</title>
+ <title>Starting LNET</title>
<para>To start LNET, run:</para>
<screen>$ modprobe lnet
$ lctl network up</screen>
<para>where <literal><NID list></literal> is the list of available NIDs.</para>
<para>This command takes the "best" NID from a list of the NIDs of a remote host. The "best" NID is the one that the local node uses when trying to communicate with the remote node.</para>
<section remap="h4">
- <title>15.2.1.1 Starting Clients</title>
+ <title>Starting Clients</title>
<para>To start a TCP client, run:</para>
<screen>mount -t lustre mdsnode:/mdsA/client /mnt/lustre/</screen>
<para>To start an Elan client, run:</para>
</section>
</section>
<section remap="h3">
- <title>15.2.2 Stopping LNET</title>
+ <title>Stopping LNET</title>
<para>Before the LNET modules can be removed, LNET references must be removed. In general, these references are removed automatically when Lustre is shut down, but for standalone routers, an explicit step is needed to stop LNET. Run:</para>
<screen>lctl network unconfigure</screen>
<note>
- <para>
-Attempting to remove Lustre modules prior to stopping the network may result in a crash or an LNET hang. if this occurs, the node must be rebooted (in most cases). Make sure that the Lustre network and Lustre are stopped prior to unloading the modules. Be extremely careful using rmmod -f.</para>
+ <para>Attempting to remove Lustre modules prior to stopping the network may result in a crash or an LNET hang. if this occurs, the node must be rebooted (in most cases). Make sure that the Lustre network and Lustre are stopped prior to unloading the modules. Be extremely careful using rmmod -f.</para>
</note>
<para>To unconfigure the LNET network, run:</para>
- <screen>modprobe -r <any lnd and the lnet modules>
-</screen>
+ <screen>modprobe -r <any lnd and the lnet modules></screen>
<note>
<para>
To remove all Lustre modules, run:</para>
</section>
</section>
<section xml:id="dbdoclet.50438203_82542">
- <title>15.3 Multi-Rail Configurations with LNET</title>
+ <title><indexterm><primary>LNET</primary><secondary>multi-rail configuration</secondary></indexterm>Multi-Rail Configurations with LNET</title>
<para>To aggregate bandwidth across both rails of a dual-rail IB cluster (o2iblnd) <footnote>
<para>Multi-rail configurations are only supported by o2iblnd; other IB LNDs do not support multiple interfaces.</para>
</footnote> using LNET, consider these points:</para>
<para>A Lustre node always uses the same local NID to communicate with a given peer NID. The criteria used to determine the local NID are:</para>
<itemizedlist>
<listitem>
- <para> Fewest hops (to minimize routing), and</para>
+ <para>Fewest hops (to minimize routing), and</para>
</listitem>
<listitem>
- <para> Appears first in the "<literal>networks</literal>" or "<literal>ip2nets</literal>" LNET configuration strings</para>
+ <para>Appears first in the "<literal>networks</literal>" or "<literal>ip2nets</literal>" LNET configuration strings</para>
</listitem>
</itemizedlist>
</listitem>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438203_78227">
- <title>15.4 Load Balancing with InfiniBand</title>
+ <title><indexterm><primary>LNET</primary><secondary>Infiniband load balancing</secondary></indexterm>Load Balancing with InfiniBand</title>
<para>A Lustre file system contains OSSs with two InfiniBand HCAs. Lustre clients have only one InfiniBand HCA using OFED Infiniband ''o2ib'' drivers. Load balancing between the HCAs on the OSS is accomplished through LNET.</para>
<section remap="h3">
- <title>15.4.1 Setting Up <literal>modprobe.conf</literal> for Load Balancing</title>
+ <title><indexterm><primary>LNET</primary><secondary>modprobe.conf</secondary></indexterm>Setting Up <literal>modprobe.conf</literal> for Load Balancing</title>
<para>To configure LNET for load balancing on clients and servers:</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Set the <literal>modprobe.conf</literal> options.</emphasis></para>
+ <para>Set the <literal>modprobe.conf</literal> options.</para>
<para>Depending on your configuration, set modprobe.conf options as follows:</para>
<itemizedlist>
<listitem>
- <para> Dual HCA OSS server</para>
+ <para>Dual HCA OSS server</para>
</listitem>
</itemizedlist>
<screen>options lnet networks="o2ib0(ib0),o2ib1(ib1) 192.168.10.1.[101-102] </screen>
<itemizedlist>
<listitem>
- <para> Client with the odd IP address</para>
+ <para>Client with the odd IP address</para>
</listitem>
</itemizedlist>
<screen>options lnet networks=o2ib0(ib0) 192.168.10.[103-253/2] </screen>
<itemizedlist>
<listitem>
- <para> Client with the even IP address</para>
+ <para>Client with the even IP address</para>
</listitem>
</itemizedlist>
<screen>options lnet networks=o2ib1(ib0) 192.168.10.[102-254/2]
</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Run the modprobe lnet command and create a combined MGS/MDT file system.</emphasis></para>
+ <para>Run the modprobe lnet command and create a combined MGS/MDT file system.</para>
<para>The following commands create the MGS/MDT file system and mount the servers (MGS/MDT and OSS).</para>
<screen>modprobe lnet
$ mkfs.lustre --fsname lustre --mgs --mdt <block device name>
$ mount -t lustre mgs@o2ib0:/lustre /mnt/ost</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Mount the clients.</emphasis></para>
+ <para>Mount the clients.</para>
<screen>mount -t lustre <MGS node>:/<fsname> <mount point></screen>
<para>This example shows an IB client being mounted.</para>
<screen>mount -t lustre
<para>This configuration gives every server two NIDs, one on each network, and statically load-balances clients between the rails.</para>
<itemizedlist>
<listitem>
- <para> A single client that must get two rails of bandwidth, and it does not matter if the maximum aggregate bandwidth is only (# servers) * (1 rail).</para>
+ <para>A single client that must get two rails of bandwidth, and it does not matter if the maximum aggregate bandwidth is only (# servers) * (1 rail).</para>
</listitem>
</itemizedlist>
<screen>ip2nets=" o2ib0(ib0) 192.168.[0-1].[0-252/2] \
<para>This configuration gives every server a single NID on one rail or the other. Clients have a NID on both rails.</para>
<itemizedlist>
<listitem>
- <para> All clients and all servers must get two rails of bandwidth.</para>
+ <para>All clients and all servers must get two rails of bandwidth.</para>
</listitem>
</itemizedlist>
<screen>ip2nets=†o2ib0(ib0),o2ib2(ib1) 192.168.[0-1].[0-252/2] \
o2ib0(ib0),o2ib3(ib1) 192.168.[2-253].[0-252/2) \
#even clients;\
o2ib1(ib0),o2ib2(ib1) 192.168.[2-253].[1-253/2) \
-#odd clients"
-</screen>
+#odd clients"</screen>
<para>This configuration includes two additional proxy o2ib networks to work around Lustre's simplistic NID selection algorithm. It connects "even" clients to "even" servers with <literal>o2ib0</literal> on <literal>rail0</literal>, and "odd" servers with <literal>o2ib3</literal> on <literal>rail1</literal>. Similarly, it connects "odd" clients to "odd" servers with <literal>o2ib1</literal> on <literal>rail0</literal>, and "even" servers with <literal>o2ib2</literal> on <literal>rail1</literal>.</para>
</section>
</section>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingsecurity">
- <info>
- <title xml:id="managingsecurity.title">Managing Lustre Security</title>
- </info>
+ <title xml:id="managingsecurity.title">Managing Lustre Security</title>
<para>This chapter describes Lustre security and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438221_16221">
- <title>22.1 Using ACLs</title>
+ <title><indexterm><primary>Access Control List (ACL)</primary></indexterm>Using ACLs</title>
<para>An access control list (ACL), is a set of data that informs an operating system about permissions or access rights that each user or group has to specific system objects, such as directories or files. Each object has a unique security attribute that identifies users who have access to it. The ACL lists each object and user access privileges such as read, write or execute.</para>
<section remap="h3">
- <title>22.1.1 How ACLs Work</title>
+ <title><indexterm><primary>Access Control List (ACL)</primary><secondary>how they work</secondary></indexterm>How ACLs Work</title>
<para>Implementing ACLs varies between operating systems. Systems that support the Portable Operating System Interface (POSIX) family of standards share a simple yet powerful file system permission model, which should be well-known to the Linux/Unix administrator. ACLs add finer-grained permissions to this model, allowing for more complicated permission schemes. For a detailed explanation of ACLs on Linux, refer to the SuSE Labs article, <emphasis>Posix Access Control Lists on Linux</emphasis>:</para>
<para><link xl:href="http://www.suse.de/~agruen/acl/linux-acls/online/">http://www.suse.de/~agruen/acl/linux-acls/online/</link></para>
<para>We have implemented ACLs according to this model. Lustre works with the standard Linux ACL tools, setfacl, getfacl, and the historical chacl, normally installed with the ACL package.</para>
</note>
</section>
<section remap="h3">
- <title>22.1.2 Using ACLs with Lustre</title>
+ <title><indexterm><primary>Access Control List (ACL)</primary><secondary>using</secondary></indexterm>Using ACLs with Lustre</title>
<para>POSIX Access Control Lists (ACLs) can be used with Lustre. An ACL consists of file entries representing permissions based on standard POSIX file system object permissions that define three classes of user (owner, group and other). Each class is associated with a set of permissions [read (r), write (w) and execute (x)].</para>
<itemizedlist>
<listitem>
<para>If ACLs are not enabled on the MDS, then any attempts to reference an ACL on a client return an Operation not supported error.</para>
</section>
<section remap="h3">
- <title>22.1.3 Examples</title>
+ <title><indexterm><primary>Access Control List (ACL)</primary><secondary>examples</secondary></indexterm>Examples</title>
<para>These examples are taken directly from the POSIX paper referenced above. ACLs on a Lustre file system work exactly like ACLs on any Linux file system. They are manipulated with the standard tools in the standard manner. Below, we create a directory and allow a specific user access.</para>
<screen>[root@client lustre]# umask 027
[root@client lustre]# mkdir rain
</section>
</section>
<section xml:id="dbdoclet.50438221_64726">
- <title>22.2 Using Root Squash</title>
+ <title><indexterm><primary>root squash</primary></indexterm>Using Root Squash</title>
<para>Lustre 1.6 introduced root squash functionality, a security feature which controls super user access rights to an Lustre file system. Before the root squash feature was added, Lustre users could run <literal>rm -rf *</literal> as root, and remove data which should not be deleted. Using the root squash feature prevents this outcome.</para>
<para>The root squash feature works by re-mapping the user ID (UID) and group ID (GID) of the root user to a UID and GID specified by the system administrator, via the Lustre configuration management server (MGS). The root squash feature also enables the Lustre administrator to specify a set of client for which UID/GID re-mapping does not apply.</para>
<section remap="h3">
- <title>22.2.1 Configuring Root Squash</title>
+ <title><indexterm><primary>root squash</primary><secondary>configuring</secondary></indexterm>Configuring Root Squash</title>
<para>Root squash functionality is managed by two configuration parameters, <literal>root_squash</literal> and <literal>nosquash_nids</literal>.</para>
<itemizedlist>
<listitem>
<para>In this example, root squash does not apply to TCP clients on subnet 172.16.245.0 that have an even number as the last component of their IP address.</para>
</section>
<section xml:id="dbdoclet.50438221_48757">
- <title >22.2.2 Enabling and Tuning Root Squash</title>
+ <title><indexterm><primary>root squash</primary><secondary>enabling</secondary></indexterm>Enabling and Tuning Root Squash</title>
<para>The default value for <literal>nosquash_nids</literal> is NULL, which means that root squashing applies to all clients. Setting the root squash UID and GID to 0 turns root squash off.</para>
<para>Root squash parameters can be set when the MDT is created (<literal>mkfs.lustre --mdt</literal>). For example:</para>
<screen>mkfs.lustre --reformat --fsname=Lustre --mdt --mgs \
</note>
</section>
<section remap="h3">
- <title>22.2.3 Tips on Using Root Squash</title>
+ <title><indexterm><primary>root squash</primary><secondary>tips</secondary></indexterm>Tips on Using Root Squash</title>
<para>Lustre configuration management limits root squash in several ways.</para>
<itemizedlist>
<listitem>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingstripingfreespace">
- <info>
- <title xml:id="managingstripingfreespace.title">Managing File Striping and Free Space</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingstripingfreespace">
+ <title xml:id="managingstripingfreespace.title">Managing File Striping and Free Space</title>
<para>This chapter describes file striping and I/O options, and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438209_79324">
- <title>18.1 How Lustre Striping Works</title>
+ <title>
+ <indexterm><primary>space</primary></indexterm>
+ <indexterm><primary>striping</primary><secondary>how it works</secondary></indexterm>
+ <indexterm><primary>striping</primary><see>space</see></indexterm>
+ <indexterm><primary>space</primary><secondary>striping</secondary></indexterm>
+ How Lustre Striping Works</title>
<para>Lustre uses a round-robin algorithm for selecting the next OST to which a stripe is to be written. Normally the usage of OSTs is well balanced. However, if users create a small number of exceptionally large files or incorrectly specify striping parameters, imbalanced OST usage may result.</para>
<para>The MDS allocates objects on seqential OSTs. Periodically, it will adjust the striping layout to eliminate some degenerated cases where applications that create very regular file layouts (striping patterns) would preferentially use a particular OST in the sequence.</para>
<para>Stripes are written to sequential OSTs until free space across the OSTs differs by more than 20%. The MDS will then use weighted random allocations with a preference for allocating objects on OSTs with more free space. This can reduce I/O performance until space usage is rebalanced to within 20% again.</para>
<para>For a more detailed description of stripe assignments, see <xref linkend="dbdoclet.50438209_10424"/>.</para>
</section>
<section xml:id="dbdoclet.50438209_48033">
- <title>18.2 Lustre File <anchor xml:id="dbdoclet.50438209_marker-1291832" xreflabel=""/>Striping Considerations</title>
+ <title><indexterm><primary>striping</primary><secondary>considerations</secondary></indexterm>
+ <indexterm><primary>space</primary><secondary>considerations</secondary></indexterm>
+ Lustre File Striping Considerations</title>
<para>Whether you should set up file striping and what parameter values you select depends on your need. A good rule of thumb is to stripe over as few objects as will meet those needs and no more.</para>
<para>Some reasons for using striping include:</para>
<itemizedlist>
</listitem>
</itemizedlist>
<section remap="h3">
- <title>18.2.1 Choosing a Stripe Size</title>
+ <title><indexterm><primary>striping</primary><secondary>size</secondary></indexterm>
+ Choosing a Stripe Size</title>
<para>Choosing a stripe size is a small balancing act, but there are reasonable defaults.</para>
<itemizedlist>
<listitem>
</section>
</section>
<section xml:id="dbdoclet.50438209_78664">
- <title>18.3 Setting the File Layout/Striping Configuration (<literal>lfs setstripe</literal>)</title>
+ <title><indexterm><primary>striping</primary><secondary>configuration</secondary></indexterm>
+ Setting the File Layout/Striping Configuration (<literal>lfs setstripe</literal>)</title>
<para>Use the <literal>lfs setstripe</literal> command to create new files with a specific file layout (stripe pattern) configuration.</para>
<screen>lfs setstripe [--size|-s stripe_size] [--count|-c stripe_count]
[--index|-i start_ost] [--pool|-p pool_name] <filename|dirname> </screen>
- <para>
- <emphasis role="bold"><literal>stripe_size</literal></emphasis>
+ <para><emphasis role="bold">
+ <literal>stripe_size</literal>
+ </emphasis>
</para>
<para>The <literal>stripe_size</literal> indicates how much data to write to one OST before moving to the next OST. The default <literal>stripe_size</literal> is 1 MB, and passing a stripe_size of 0 causes the default stripe size to be used. Otherwise, the <literal>stripe_size</literal> value must be a multiple of 64 KB.</para>
- <para>
- <emphasis role="bold"><literal>stripe_count</literal></emphasis>
+ <para><emphasis role="bold">
+ <literal>stripe_count</literal>
+ </emphasis>
</para>
<para>The <literal>stripe_count</literal> indicates how many OSTs to use. The default <literal>stripe_count</literal> value is 1. Setting <literal>stripe_count</literal> to 0 causes the default stripe count to be used. Setting <literal>stripe_count</literal> to -1 means stripe over all available OSTs (full OSTs are skipped).</para>
- <para>
- <emphasis role="bold"><literal>start_ost</literal></emphasis>
+ <para><emphasis role="bold">
+ <literal>start_ost</literal>
+ </emphasis>
</para>
<para>The start OST is the first OST to which files are written. The default value for <literal>start_ost</literal> is -1, which allows the MDS to choose the starting index. This setting is strongly recommended, as it allows space and load balancing to be done by the MDS as needed. Otherwise, the file starts on the specified OST index. The numbering of the OSTs starts at 0.</para>
<note>
<para>If you pass a <literal>start_ost</literal> value of 0 and a <literal>stripe_count</literal> value of <emphasis>1</emphasis>, all files are written to OST 0, until space is exhausted. This is probably not what you meant to do. If you only want to adjust the stripe count and keep the other parameters at their default settings, do not specify any of the other parameters:</para>
<para><screen>lfs setstripe -c <stripe_count> <file></screen></para>
</note>
- <para>
- <emphasis role="bold"><literal>pool_name</literal></emphasis>
+ <para><emphasis role="bold">
+ <literal>pool_name</literal>
+ </emphasis>
</para>
<para>Specify the OST pool on which the file will be written. This allows limiting the OSTs used to a subset of all OSTs in the file system. For more details about using OST pools, see <link xl:href="ManagingFileSystemIO.html#50438211_75549">Creating and Managing OST Pools</link>.</para>
<section remap="h3">
- <title>18.3.1 Using a Specific Striping Pattern/File Layout for a Single File</title>
+ <title>Using a Specific Striping Pattern/File Layout for a Single File</title>
<para>It is possible to specify the file layout when a new file is created using the command <literal>lfs setstripe</literal>. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an <literal>lfs setstripe</literal> command fails if the file already exists.</para>
<section xml:id="dbdoclet.50438209_60155">
- <title>18.3.1.1 Setting the Stripe Size</title>
+ <title>Setting the Stripe Size</title>
<para>The command to create a new file with a specified stripe size is similar to:</para>
<screen>[client]# lfs setstripe -s 4M /mnt/lustre/new_file</screen>
<para>This example command creates the new file <literal>/mnt/lustre/new_file</literal> with a stripe size of 4 MB.</para>
<para>As can be seen, the stripe size is 4 MB.</para>
</section>
<section remap="h4">
- <title>18.3.1.2 Setting the Stripe Count</title>
+ <title><indexterm><primary>striping</primary><secondary>count</secondary></indexterm>
+ Setting the Stripe Count</title>
<para>The command below creates a new file with a stripe count of -1 to specify striping over all available OSTs:</para>
<screen>[client]# lfs setstripe -c -1 /mnt/lustre/full_stripe</screen>
<para>The example below indicates that the file full_stripe is striped over all six active OSTs in the configuration:</para>
</section>
</section>
<section remap="h3">
- <title>18.3.2 Changing Striping for a Directory</title>
+ <title><indexterm><primary>striping</primary><secondary>per directory</secondary></indexterm>Changing Striping for a Directory</title>
<para>In a directory, the <literal>lfs setstripe</literal> command sets a default striping configuration for files created in the directory. The usage is the same as <literal>lfs setstripe</literal> for a regular file, except that the directory must exist prior to setting the default striping configuration. If a file is created in a directory with a default stripe configuration (without otherwise specifying striping), Lustre uses those striping parameters instead of the file system default for the new file.</para>
<para>To change the striping pattern (file layout) for a sub-directory, create a directory with desired file layout as described above. Sub-directories inherit the file layout of the root/parent directory.</para>
</section>
<section remap="h3">
- <title>18.3.3 Changing Striping for a File System</title>
+ <title><indexterm><primary>striping</primary><secondary>per file system</secondary></indexterm>Changing Striping for a File System</title>
<para>Change the striping on the file system root will change the striping for all newly created files that would otherwise have a striping parameter from the parent directory or explicitly on the command line.</para>
<note>
<para>Striping of new files and sub-directories is done per the striping parameter settings of the root directory. Once you set striping on the root directory, then, by default, it applies to any new child directories created in that root directory (unless they have their own striping settings).</para>
</note>
</section>
<section remap="h3">
- <title>18.3.4 Creating a File on a Specific OST</title>
+ <title><indexterm><primary>striping</primary><secondary>on specific OST</secondary></indexterm>Creating a File on a Specific OST</title>
<para>You can use <literal>lfs setstripe</literal> to create a file on a specific OST. In the following example, the file "<literal>bob</literal>" will be created on the first OST (id 0).</para>
<screen>$ lfs setstripe --count 1 --index 0 bob
$ dd if=/dev/zero of=bob count=1 bs=100M
</section>
</section>
<section xml:id="dbdoclet.50438209_44776">
- <title>18.4 Retrieving File Layout/Striping Information (<literal>getstripe</literal>)</title>
+ <title><indexterm><primary>striping</primary><secondary>getting information</secondary></indexterm>Retrieving File Layout/Striping Information (<literal>getstripe</literal>)</title>
<para>The <literal>lfs getstripe</literal> command is used to display information that shows over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along with the OST index and object ID for each stripe in the file. For directories, the default settings for files created in that directory are printed.</para>
<section remap="h3">
- <title>18.4.1 Displaying the Current Stripe Size</title>
+ <title>Displaying the Current Stripe Size</title>
<para>To see the current stripe size, use the <literal>lfs getstripe</literal> command on a Lustre file or directory. For example:</para>
<screen>[client]# lfs getstripe /mnt/lustre </screen>
<para>This command produces output similar to this:</para>
<para>In this example, the default stripe count is 1 (data blocks are striped over a single OSTs), the default stripe size is 1 MB, and objects are created over all available OSTs.</para>
</section>
<section remap="h3">
- <title>18.4.2 Inspecting the File Tree</title>
+ <title>Inspecting the File Tree</title>
<para>To inspect an entire tree of files, use the <literal>lfs</literal> find command:</para>
<screen>lfs find [--recursive | -r] <file or directory> ...</screen>
<para>You can also use <literal>ls -l /proc/<replaceable><pid></replaceable>/fd/</literal> to find open files using Lustre. For example:</para>
</section>
</section>
<section xml:id="dbdoclet.50438209_10424">
- <title>18.5 Managing Free Space</title>
+ <title><indexterm><primary>space</primary><secondary>free space</secondary></indexterm>Managing Free Space</title>
<para>The MDT assigns file stripes to OSTs based on location (which OSS) and size considerations (free space) to optimize file system performance. Emptier OSTs are preferentially selected for stripes, and stripes are preferentially spread out between OSSs to increase network bandwidth utilization. The weighting factor between these two optimizations can be adjusted by the user.</para>
- <section xml:id='dbdoclet.50438209_35838'>
- <title>18.5.1 Checking File System Free Space</title>
+ <section xml:id="dbdoclet.50438209_35838">
+ <title>Checking File System Free Space</title>
<para>Free space is an important consideration in assigning file stripes. The <literal>lfs df</literal> command shows available disk space on the mounted Lustre file system and space consumption per OST. If multiple Lustre file systems are mounted, a path may be specified, but is not required.</para>
<informaltable frame="all">
<tgroup cols="2">
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">
- <literal>-h</literal>
- </emphasis></para>
+ <para> <literal>-h</literal></para>
</entry>
<entry>
<para> Human-readable print sizes in human readable format (for example: 1K, 234M, 5G).</para>
</row>
<row>
<entry>
- <para> <emphasis>
- <literal role="bold">-i, --inodes</literal>
- </emphasis></para>
+ <para> <literal role="bold">-i, --inodes</literal></para>
</entry>
<entry>
<para> Lists inodes instead of block usage.</para>
</informaltable>
<note>
<para>The <literal>df -i</literal> and <literal>lfs df -i</literal> commands show the minimum number of inodes that can be created in the file system. Depending on the configuration, it may be possible to create more inodes than initially reported by <literal>df -i</literal>. Later, <literal>df -i</literal> operations will show the current, estimated free inode count.</para>
- <para> If the underlying file system has fewer free blocks than inodes, then the total inode count for the file system reports only as many inodes as there are free blocks. This is done because Lustre may need to store an external attribute for each new inode, and it is better to report a free inode count that is the guaranteed, minimum number of inodes that can be created.</para>
+ <para>If the underlying file system has fewer free blocks than inodes, then the total inode count for the file system reports only as many inodes as there are free blocks. This is done because Lustre may need to store an external attribute for each new inode, and it is better to report a free inode count that is the guaranteed, minimum number of inodes that can be created.</para>
</note>
<para><emphasis role="bold">Examples</emphasis></para>
<screen>[lin-cli1] $ lfs df
2169648 1% /mnt/lustre[OST:2]</screen>
</section>
<section remap="h3">
- <title>18.5.2 Using Stripe Allocations</title>
+ <title><indexterm><primary>striping</primary><secondary>allocations</secondary></indexterm>
+ Using Stripe Allocations</title>
<para>Two stripe allocation methods are provided: <emphasis>round-robin</emphasis> and <emphasis>weighted</emphasis>. By default, the allocation method is determined by the amount of free-space imbalance on the OSTs. The weighted allocator is used when any two OSTs are imbalanced by more than 20%. Otherwise, the faster round-robin allocator is used. (The round-robin order maximizes network balancing.)</para>
<itemizedlist>
<listitem>
</itemizedlist>
</section>
<section remap="h3">
- <title>18.5.3 Adjusting the Weighting Between Free Space and Location</title>
+ <title><indexterm><primary>space</primary><secondary>locationg weighting</secondary></indexterm>>Adjusting the Weighting Between Free Space and Location</title>
<para>The weighting priority can be adjusted in the <literal>/proc</literal> file <literal>/proc/fs/lustre/lov/lustre-mdtlov/qos_prio_free proc</literal>. The default value is 90%. Use this command on the MGS to permanently change this weighting:</para>
<screen>lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90</screen>
<para>Increasing this value puts more weighting on free space. When the free space priority is set to 100%, then location is no longer used in stripe-ordering calculations and weighting is based entirely on free space.</para>
<para>This operations manual provides detailed information and procedures to install, configure and tune the Lustre file system. The manual covers topics such as failover, quotas, striping and bonding. The Lustre manual also contains troubleshooting information and tips to improve Lustre operation and performance.</para>
<section remap="h2">
<title>About this Document</title>
- <para>This document is maintained by Whamcloud, Inc in Docbook format. The canonical version is available at <link xml:href="http://wiki.whamcloud.com">http://wiki.whamcloud.com/</link>. </para>
+ <para>This document is maintained by Whamcloud, Inc in Docbook format. The canonical version is available at <link xl:href="http://wiki.whamcloud.com">http://wiki.whamcloud.com/</link>. </para>
<section remap="h2">
<title>UNIX Commands</title>
<para>This document might not contain information about basic UNIX commands and procedures such as shutting down the system, booting the system, and configuring devices. Refer to the following for this information:</para>
<?xml version="1.0" encoding="UTF-8"?>
-<article version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink">
- <info>
- <title>Revision History</title>
- </info>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
-
-
- <tbody>
- <row>
- <entry align="left"><para>Lustre 2.0 Operations Manual</para></entry>
- <entry align="right" valign="top"><para><link xl:href="index.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/toc01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/toc01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="Preface.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/prev01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/prev01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="I_LustreIntro.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/next01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/next01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="ix.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/index01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/index01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link></para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><link xl:href=""/></para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
-
- <tbody>
- <row>
- <entry align="right"><para>Revision History</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para> </para>
- <informaltable frame="all">
- <tgroup cols="4">
- <colspec colname="c1" colwidth="25*"/>
- <colspec colname="c2" colwidth="25*"/>
- <colspec colname="c3" colwidth="25*"/>
- <colspec colname="c4" colwidth="25*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">BookTitle</emphasis></para></entry>
- <entry><para><emphasis role="bold">Part Number</emphasis></para></entry>
- <entry><para><emphasis role="bold">Date</emphasis></para></entry>
- <entry><para><emphasis role="bold">Comments</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> Lustre 2.0 Operations Manual</para></entry>
- <entry><para> 821-2076-10</para></entry>
- <entry><para> July 2010</para></entry>
- <entry><para> First release of Lustre 2.0 manual</para></entry>
- </row>
- <row>
- <entry><para> Lustre 2.0 Operations Manual</para></entry>
- <entry><para> 821-2076-10</para></entry>
- <entry><para> January 2011</para></entry>
- <entry><para> Second release of Lustre 2.0 manual</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <!--
-Begin SiteCatalyst code version: G.5.
--->
- <!--
-End SiteCatalyst code version: G.5.
--->
- <informaltable frame="none">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
-
-
-
- <tbody>
- <row>
- <entry align="left"><para>Lustre 2.0 Operations Manual</para></entry>
- <entry align="right"><para>821-2076-10</para></entry>
- <entry align="right" valign="top"><para><link xl:href="index.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/toc01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/toc01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="Preface.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/prev01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/prev01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="I_LustreIntro.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/next01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/next01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="ix.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/index01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/index01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link></para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><link xl:href=""/></para>
- <para><link xl:href="copyright.html">Copyright</link> © 2011, Oracle and/or its affiliates. All rights reserved.</para>
-</article>
+<revhistory>
+
+ <revision>
+ <revnumber>1.0</revnumber>
+ <date>June 2011</date>
+ <author>Whamcloud, Inc</author>
+ <revremark>First release of Lustre 2.x manual</revremark>
+ </revision>
+</revhistory>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settinglustreproperties">
- <info>
- <title xml:id="settinglustreproperties.title">Setting Lustre Properties in a C Program (<literal>llapi</literal>)</title>
- </info>
+ <title xml:id="settinglustreproperties.title">Setting Lustre Properties in a C Program (<literal>llapi</literal>)</title>
<para>This chapter describes the <literal>llapi</literal> library of commands used for setting Lustre file properties within a C program running in a cluster environment, such as a data processing or MPI application. The commands described in this chapter are:</para>
<itemizedlist>
<listitem>
<para>Lustre programming interface man pages are found in the <literal>lustre/doc</literal> folder.</para>
</note>
<section xml:id="dbdoclet.50438215_30970">
- <title>34.1 <literal>llapi_file_create</literal></title>
+ <title>
+ <literal>llapi_file_create</literal>
+ </title>
<para>Use <literal>llapi_file_create</literal> to set Lustre properties for a new file.</para>
<section remap="h5">
<title>Synopsis</title>
</section>
</section>
<section xml:id="dbdoclet.50438215_50149">
- <title>34.2 llapi_file_get_stripe</title>
+ <title>llapi_file_get_stripe</title>
<para>Use <literal>llapi_file_get_stripe</literal> to get striping information for a file or directory on a Lustre file system.</para>
<section remap="h5">
<title>Synopsis</title>
</section>
</section>
<section xml:id="dbdoclet.50438215_86607">
- <title>34.3 <literal>llapi_file_open</literal></title>
+ <title>
+ <literal>llapi_file_open</literal>
+ </title>
<para>The <literal>llapi_file_open</literal> command opens (or creates) a file or device on a Lustre filesystem.</para>
<section remap="h5">
<title>Synopsis</title>
</section>
</section>
<section xml:id="dbdoclet.50438215_12433">
- <title>34.4 <literal>llapi_quotactl</literal></title>
+ <title>
+ <literal>llapi_quotactl</literal>
+ </title>
<para>Use <literal>llapi_quotact</literal>l to manipulate disk quotas on a Lustre file system.</para>
<section remap="h5">
<title>Synopsis</title>
</tgroup>
</informaltable>
</section>
- <section remap="h5"><title>Return Values</title><para><literal>llapi_quotactl()</literal> returns:</para><para><literal>0</literal> On success</para><para><literal>
- -1
-</literal> On failure and sets error number (<literal>errno</literal>) to indicate the error</para></section>
+ <section remap="h5">
+ <title>Return Values</title>
+ <para><literal>llapi_quotactl()</literal> returns:</para>
+ <para><literal>0</literal> On success</para>
+ <para><literal> -1 </literal> On failure and sets error number (<literal>errno</literal>) to indicate the error</para>
+ </section>
<section remap="h5">
<title>Errors</title>
<para><literal>llapi_quotactl</literal> errors are described below.</para>
</section>
</section>
<section xml:id="dbdoclet.50438215_15718">
- <title>34.5 <literal>llapi_path2fid</literal></title>
+ <title>
+ <literal>llapi_path2fid</literal>
+ </title>
<para>Use <literal>llapi_path2fid</literal> to get the FID from the pathname.</para>
<section remap="h5">
<title>Synopsis</title>
</section>
</section>
<section xml:id="dbdoclet.50438215_marker-1297700">
- <title>34.6 Example Using the <literal>llapi</literal> Library</title>
+ <title>Example Using the <literal>llapi</literal> Library</title>
<para>Use <literal>llapi_file_create</literal> to set Lustre properties for a new file. For a synopsis and description of llapi_file_create and examples of how to use it, see <xref linkend="configurationfilesmoduleparameters"/>.</para>
<para>You can set striping from inside programs like <literal>ioctl</literal>. To compile the sample program, you need to download libtest.c and liblustreapi.c files from the Lustre source tree.</para>
<para><emphasis role="bold">A simple C program to demonstrate striping API - libtest.c</emphasis></para>
</screen>
<section remap="h5">
<title>See Also</title>
- <itemizedlist>
- <listitem>
- <para>
+ <itemizedlist>
+ <listitem>
<xref linkend="dbdoclet.50438215_30970"/>
</para>
- </listitem>
- <listitem>
- <para>
+ </listitem>
+ <listitem>
<xref linkend="dbdoclet.50438215_50149"/>
</para>
- </listitem>
- <listitem>
- <para>
+ </listitem>
+ <listitem>
<xref linkend="dbdoclet.50438215_86607"/>
</para>
- </listitem>
- <listitem>
- <para>
+ </listitem>
+ <listitem>
<xref linkend="dbdoclet.50438215_12433"/>
</para>
- </listitem>
- </itemizedlist>
+ </listitem>
+ </itemizedlist>
</section>
</section>
</chapter>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settingupbonding">
- <info>
- <title xml:id="settingupbonding.title">Setting Up Network Interface Bonding</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settingupbonding">
+ <title xml:id="settingupbonding.title">Setting Up Network Interface Bonding</title>
<para>This chapter describes how to use multiple network interfaces in parallel to increase bandwidth and/or redundancy. Topics include:</para>
<itemizedlist>
<listitem>
<para>Using network interface bonding is optional.</para>
</note>
<section xml:id="dbdoclet.50438258_95824">
- <title>7.1 Network Interface Bonding Overview</title>
+ <title>Network Interface Bonding Overview</title>
<para>Bonding, also known as link aggregation, trunking and port trunking, is a method of aggregating multiple physical network links into a single logical link for increased bandwidth.</para>
<para>Several different types of bonding are available in Linux. All these types are referred to as 'modes', and use the bonding kernel module.</para>
<para>Modes 0 to 3 allow load balancing and fault tolerance by using multiple interfaces. Mode 4 aggregates a group of interfaces into a single virtual interface where all members of the group share the same speed and duplex settings. This mode is described under IEEE spec 802.3ad, and it is referred to as either 'mode 4' or '802.3ad.'</para>
</section>
<section xml:id="dbdoclet.50438258_29107">
- <title>7.2 Requirements</title>
+ <title>Requirements</title>
<para>The most basic requirement for successful bonding is that both endpoints of the connection must be capable of bonding. In a normal case, the non-server endpoint is a switch. (Two systems connected via crossover cables can also use bonding.) Any switch used must explicitly handle 802.3ad Dynamic Link Aggregation.</para>
<para>The kernel must also be configured with bonding. All supported Lustre kernels have bonding functionality. The network driver for the interfaces to be bonded must have the ethtool functionality to determine slave speed and duplex settings. All recent network drivers implement it.</para>
<para>To verify that your interface works with ethtool, run:</para>
# ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
- Supported link modes: 10baseT/Half 10baseT/Full/
+ Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
</screen>
</section>
<section xml:id="dbdoclet.50438258_25353">
- <title>7.3 Bonding Module Parameters</title>
+ <title>Bonding Module Parameters</title>
<para>Bonding module parameters control various aspects of bonding.</para>
- <para>Outgoing traffic is mapped across the slave interfaces according to the transmit hash policy. For Lustre, we recommend that you set the xmit_hash_policy option to the layer3+4 option for bonding. This policy uses upper layer protocol information if available to generate the hash. This allows traffic to a particular network peer to span multiple slaves, although a single connection does not span multiple slaves.</para>
- <screen>$ xmit_hash_policy=layer3+4
-</screen>
+ <para>Outgoing traffic is mapped across the slave interfaces according to the transmit hash policy. For Lustre, we recommend that you set the <literal>xmit_hash_policy</literal> option to the layer3+4 option for bonding. This policy uses upper layer protocol information if available to generate the hash. This allows traffic to a particular network peer to span multiple slaves, although a single connection does not span multiple slaves.</para>
+ <screen>$ xmit_hash_policy=layer3+4</screen>
<para>The miimon option enables users to monitor the link status. (The parameter is a time interval in milliseconds.) It makes an interface failure transparent to avoid serious network degradation during link failures. A reasonable default setting is 100 milliseconds; run:</para>
- <screen>$ miimon=100
-</screen>
+ <screen>$ miimon=100</screen>
<para>For a busy network, increase the timeout.</para>
</section>
<section xml:id="dbdoclet.50438258_26899">
- <title>7.4 Setting Up Bonding</title>
+ <title>Setting Up Bonding</title>
<para>To set up bonding:</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Create a virtual 'bond' interface by creating a configuration file in:</emphasis></para>
+ <para>Create a virtual 'bond' interface by creating a configuration file in:</para>
<screen>/etc/sysconfig/network-scripts/ # vi /etc/sysconfig/ network-scripts/ifcfg-\
-bond0
-</screen>
+bond0</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Append the following lines to the file.</emphasis></para>
+ <para>Append the following lines to the file.</para>
<screen>DEVICE=bond0
IPADDR=192.168.10.79 # Use the free IP Address of your network
NETWORK=192.168.10.0
NETMASK=255.255.255.0
USERCTL=no
BOOTPROTO=none
-ONBOOT=yes
-</screen>
+ONBOOT=yes</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Attach one or more slave interfaces to the bond interface. Modify the eth0 and eth1 configuration files (using a VI text editor).</emphasis></para>
+ <para>Attach one or more slave interfaces to the bond interface. Modify the eth0 and eth1 configuration files (using a VI text editor).</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Use the VI text editor to open the eth0 configuration file.</emphasis></para>
- <screen># vi /etc/sysconfig/network-scripts/ifcfg-eth0
-</screen>
+ <para>Use the VI text editor to open the eth0 configuration file.</para>
+ <screen># vi /etc/sysconfig/network-scripts/ifcfg-eth0</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Modify/append the eth0 file as follows:</emphasis></para>
+ <para>Modify/append the eth0 file as follows:</para>
<screen>DEVICE=eth0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
-BOOTPROTO=none
-</screen>
+BOOTPROTO=none</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Use the VI text editor to open the eth1 configuration file.</emphasis></para>
- <screen># vi /etc/sysconfig/network-scripts/ifcfg-eth1
-</screen>
+ <para>Use the VI text editor to open the eth1 configuration file.</para>
+ <screen># vi /etc/sysconfig/network-scripts/ifcfg-eth1</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Modify/append the eth1 file as follows:</emphasis></para>
+ <para>Modify/append the eth1 file as follows:</para>
<screen>DEVICE=eth1
USERCTL=no
ONBOOT=yes
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Set up the bond interface and its options in /etc/modprobe.conf. Start the slave interfaces by your normal network method.</emphasis></para>
+ <para>Set up the bond interface and its options in /etc/modprobe.conf. Start the slave interfaces by your normal network method.</para>
<screen># vi /etc/modprobe.conf
</screen>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Append the following lines to the file.</emphasis></para>
+ <para>Append the following lines to the file.</para>
<screen>alias bond0 bonding
options bond0 mode=balance-alb miimon=100
</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Load the bonding module.</emphasis></para>
+ <para>Load the bonding module.</para>
<screen># modprobe bonding
# ifconfig bond0 up
# ifenslave bond0 eth0 eth1
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Start/restart the slave interfaces (using your normal network method).</emphasis></para>
+ <para>Start/restart the slave interfaces (using your normal network method).</para>
<note>
- <para>You must modprobe the bonding module for each bonded interface. If you wish to create bond0 and bond1, two entries in modprobe.conf are required.</para>
+ <para>You must <literal>modprobe</literal> the bonding module for each bonded interface. If you wish to create bond0 and bond1, two entries in <literal>modprobe.conf</literal> are required.</para>
</note>
<para>The examples below are from RedHat systems. For setup use: <literal>/etc/sysconfig/networking-scripts/ifcfg-*</literal> The website referenced below includes detailed instructions for other configuration methods, instructions to use DHCP with bonding, and other setup details. We strongly recommend you use this website.</para>
<para><link xl:href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding">http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding</link></para>
</listitem>
<listitem>
- <para><emphasis role="bold">Check /proc/net/bonding to determine status on bonding. There should be a file there for each bond interface.</emphasis></para>
+ <para>Check /proc/net/bonding to determine status on bonding. There should be a file there for each bond interface.</para>
<screen># cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.0.3 (March 23, 2006)
</screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Use ethtool or ifconfig to check the interface state. ifconfig lists the first bonded interface as 'bond0.'</emphasis></para>
+ <para>Use ethtool or ifconfig to check the interface state. ifconfig lists the first bonded interface as 'bond0.'</para>
<screen>ifconfig
bond0 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0
inet addr:192.168.10.79 Bcast:192.168.10.255 \ Mask:255.255.255.0
</listitem>
</orderedlist>
<section remap="h3">
- <title>7.4.1 Examples</title>
+ <title>Examples</title>
<para>This is an example showing <literal>modprobe.conf</literal> entries for bonding Ethernet interfaces <literal>eth1</literal> and <literal>eth2</literal> to <literal>bond0</literal>:</para>
<screen># cat /etc/modprobe.conf
alias eth0 8139too
SLAVE=yes
</screen>
<para>In the following example, the <literal>bond0</literal> interface is the master (MASTER) while <literal>eth0</literal> and <literal>eth1</literal> are slaves (SLAVE).</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry>
- <para><emphasis role="bold">Note -</emphasis>All slaves of <literal>bond0</literal> have the same MAC address (Hwaddr) - <literal>bond0</literal>. All modes, except TLB and ALB, have this MAC address. TLB and ALB require a unique MAC address for each slave.</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note>
+ <para>All slaves of <literal>bond0</literal> have the same MAC address (Hwaddr) - <literal>bond0</literal>. All modes, except TLB and ALB, have this MAC address. TLB and ALB require a unique MAC address for each slave.</para>
+ </note>
<screen>$ /sbin/ifconfig
bond0Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4
RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
-Interrupt:9 Base address:0x1400
-</screen>
+Interrupt:9 Base address:0x1400</screen>
</section>
</section>
<section xml:id="dbdoclet.50438258_54769">
- <title>7.5 Configuring Lustre with Bonding</title>
+ <title>Configuring Lustre with Bonding</title>
<para>Lustre uses the IP address of the bonded interfaces and requires no special configuration. It treats the bonded interface as a regular TCP/IP interface. If needed, specify bond0 using the Lustre networks parameter in /etc/modprobe.</para>
- <screen>options lnet networks=tcp(bond0)
-</screen>
+ <screen>options lnet networks=tcp(bond0)</screen>
</section>
<section xml:id="dbdoclet.50438258_92244">
- <title>7.6 Bonding References</title>
+ <title>Bonding References</title>
<para>We recommend the following bonding references:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<para>This is the most extensive reference and we highly recommend it. This website includes explanations of more complicated setups, including the use of DHCP with bonding.</para>
- <screen>
-</screen>
</section>
</chapter>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settinguplustresystem">
- <info>
- <title xml:id="settinguplustresystem.title">Setting Up a Lustre File System</title>
- </info>
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settinguplustresystem">
+ <title xml:id="settinguplustresystem.title">Setting Up a Lustre File System</title>
<para>This chapter describes hardware configuration requirements for a Lustre file system including:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438256_49017">
- <title>5.1 Hardware Considerations</title>
+ <title><indexterm><primary>setup</primary></indexterm>
+ <indexterm><primary>setup</primary><secondary>hardware</secondary></indexterm>
+ <indexterm><primary>design</primary><see>setup</see></indexterm>
+ Hardware Considerations</title>
<para>Lustre can work with any kind of block storage device such as single disks, software RAID, hardware RAID, or a logical volume manager. In contrast to some networked file systems, the block devices are only attached to the MDS and OSS nodes in Lustre and are not accessed by the clients directly.</para>
<para>Since the block devices are accessed by only one or two server nodes, a storage area network (SAN) that is accessible from all the servers is not required. Expensive switches are not needed because point-to-point connections between the servers and the storage arrays normally provide the simplest and best attachments. (If failover capability is desired, the storage must be attached to multiple servers.)</para>
<para>For a production environment, it is preferable that the MGS have separate storage to allow future expansion to multiple file systems. However, it is possible to run the MDS and MGS on the same machine and have them share the same storage device.</para>
<para>Performance and other issues can occur when an MDS or OSS and a client are running on the same machine:</para>
<itemizedlist>
<listitem>
- <para> Running the MDS and a client on the same machine can cause recovery and deadlock issues and impact the performance of other Lustre clients.</para>
+ <para>Running the MDS and a client on the same machine can cause recovery and deadlock issues and impact the performance of other Lustre clients.</para>
</listitem>
<listitem>
- <para> Running the OSS and a client on the same machine can cause issues with low memory and memory pressure. If the client consumes all the memory and then tries to write data to the file system, the OSS will need to allocate pages to receive data from the client but will not be able to perform this operation due to low memory. This can cause the client to hang.</para>
+ <para>Running the OSS and a client on the same machine can cause issues with low memory and memory pressure. If the client consumes all the memory and then tries to write data to the file system, the OSS will need to allocate pages to receive data from the client but will not be able to perform this operation due to low memory. This can cause the client to hang.</para>
</listitem>
</itemizedlist>
<para>Only servers running on 64-bit CPUs are tested and supported. 64-bit CPU clients are typically used for testing to match expected customer usage and avoid limitations due to the 4 GB limit for RAM size, 1 GB low-memory limitation, and 16 TB file size limit of 32-bit CPUs. Also, due to kernel API limitations, performing backups of Lustre 2.x. filesystems on 32-bit clients may cause backup tools to confuse files that have the same 32-bit inode number.</para>
<para>Lustre uses journaling file system technology on both the MDTs and OSTs. For a MDT, as much as a 20 percent performance gain can be obtained by placing the journal on a separate device.</para>
<para>The MDS can effectively utilize a lot of CPU cycles. A minimium of four processor cores are recommended. More are advisable for files systems with many clients.</para>
<note>
- <para>Lustre clients running on architectures with different endianness are supported. One limitation is that the PAGE_SIZE kernel macro on the client must be as large as the PAGE_SIZE of the server. In particular, ia64 or PPC clients with large pages (up to 64kB pages) can run with x86 servers (4kB pages). If you are running x86 clients with ia64 or PPC servers, you must compile the ia64 kernel with a 4kB PAGE_SIZE (so the server page size is not larger than the client page size). <anchor xml:id="dbdoclet.50438256_51943" xreflabel=""/></para>
+ <para>Lustre clients running on architectures with different endianness are supported. One limitation is that the PAGE_SIZE kernel macro on the client must be as large as the PAGE_SIZE of the server. In particular, ia64 or PPC clients with large pages (up to 64kB pages) can run with x86 servers (4kB pages). If you are running x86 clients with ia64 or PPC servers, you must compile the ia64 kernel with a 4kB PAGE_SIZE (so the server page size is not larger than the client page size). </para>
</note>
<section remap="h3">
- <title>5.1.1 MDT Storage Hardware Considerations</title>
+ <title><indexterm><primary>setup</primary><secondary>MDT</secondary></indexterm>
+ MDT Storage Hardware Considerations</title>
<para>The data access pattern for MDS storage is a database-like access pattern with many seeks and read-and-writes of small amounts of data. High throughput to MDS storage is not important. Storage types that provide much lower seek times, such as high-RPM SAS or SSD drives can be used for the MDT.</para>
<para>For maximum performance, the MDT should be configured as RAID1 with an internal journal and two disks from different controllers.</para>
<para>If you need a larger MDT, create multiple RAID1 devices from pairs of disks, and then make a RAID0 array of the RAID1 devices. This ensures maximum reliability because multiple disk failures only have a small chance of hitting both disks in the same RAID1 device.</para>
<para>Doing the opposite (RAID1 of a pair of RAID0 devices) has a 50% chance that even two disk failures can cause the loss of the whole MDT device. The first failure disables an entire half of the mirror and the second failure has a 50% chance of disabling the remaining mirror.</para>
</section>
<section remap="h3">
- <title>5.1.2 OST Storage Hardware Considerations</title>
+ <title><indexterm><primary>setup</primary><secondary>OST</secondary></indexterm>OST Storage Hardware Considerations</title>
<para>The data access pattern for the OSS storage is a streaming I/O pattern that is dependent on the access patterns of applications being used. Each OSS can manage multiple object storage targets (OSTs), one for each volume with I/O traffic load-balanced between servers and targets. An OSS should be configured to have a balance between the network bandwidth and the attached storage bandwidth to prevent bottlenecks in the I/O path. Depending on the server hardware, an OSS typically serves between 2 and 8 targets, with each target up to 16 terabytes (TBs) in size.</para>
<para>Lustre file system capacity is the sum of the capacities provided by the targets. For example, 64 OSSs, each with two 8 TB targets, provide a file system with a capacity of nearly 1 PB. If each OST uses ten 1 TB SATA disks (8 data disks plus 2 parity disks in a RAID 6 configuration), it may be possible to get 50 MB/sec from each drive, providing up to 400 MB/sec of disk bandwidth per OST. If this system is used as storage backend with a system network like InfiniBand that provides a similar bandwidth, then each OSS could provide 800 MB/sec of end-to-end I/O throughput. (Although the architectural constraints described here are simple, in practice it takes careful hardware selection, benchmarking and integration to obtain such results.)</para>
</section>
</section>
<section xml:id="dbdoclet.50438256_31079">
- <title>5.2 Determining Space Requirements</title>
+ <title><indexterm><primary>setup</primary><secondary>space</secondary></indexterm>
+ <indexterm><primary>space</primary><secondary>determining requirements</secondary></indexterm>
+ Determining Space Requirements</title>
<para>The desired performance characteristics of the backing file systems on the MDT and OSTs are independent of one another. The size of the MDT backing file system depends on the number of inodes needed in the total Lustre file system, while the aggregate OST space depends on the total amount of data stored on the file system.</para>
<para>Each time a file is created on a Lustre file system, it consumes one inode on the MDT and one inode for each OST object over which the file is striped. Normally, each file's stripe count is based on the system-wide default stripe count. However, this can be changed for individual files using the lfssetstripe option. For more details, see <xref linkend="managingstripingfreespace"/>.</para>
<para>In a Lustre ldiskfs file system, all the inodes are allocated on the MDT and OSTs when the file system is first formatted. The total number of inodes on a formatted MDT or OST cannot be easily changed, although it is possible to add OSTs with additional space and corresponding inodes. Thus, the number of inodes created at format time should be generous enough to anticipate future expansion.</para>
<para>When the file system is in use and a file is created, the metadata associated with that file is stored in one of the pre-allocated inodes and does not consume any of the free space used to store file data.</para>
<note>
- <para>By default, the ldiskfs file system used by Lustre servers to store user-data objects and system data reserves 5% of space that cannot be used by Lustre. Additionally, Lustre reserves up to 400 MB on each OST for journal use and a small amount of space outside the journal to store accounting data for Lustre. This reserved space is unusable for general storage. Thus, at least 400 MB of space is used on each OST before any file object data is saved.<anchor xml:id="dbdoclet.50438256_74070" xreflabel=""/></para>
+ <para>By default, the ldiskfs file system used by Lustre servers to store user-data objects and system data reserves 5% of space that cannot be used by Lustre. Additionally, Lustre reserves up to 400 MB on each OST for journal use and a small amount of space outside the journal to store accounting data for Lustre. This reserved space is unusable for general storage. Thus, at least 400 MB of space is used on each OST before any file object data is saved.</para>
</note>
<section xml:id="dbdoclet.50438256_87676">
- <title>5.2.1 Determining MDS/MDT Space Requirements</title>
+ <title><indexterm><primary>setup</primary><secondary>MDS/MDT</secondary></indexterm>
+ <indexterm><primary>space</primary><secondary>determining MDS/MDT requirements</secondary></indexterm>
+ Determining MDS/MDT Space Requirements</title>
<para>When calculating the MDT size, the important factor to consider is the number of files to be stored in the file system. This determines the number of inodes needed, which drives the MDT sizing. To be on the safe side, plan for 4 KB per inode on the MDT, which is the default value. Attached storage required for Lustre metadata is typically 1-2 percent of the file system capacity depending upon file size.</para>
<para>For example, if the average file size is 5 MB and you have 100 TB of usable OST space, then you can calculate the minimum number of inodes as follows:</para>
- <informalexample><para>(100 TB * 1024 GB/TB * 1024 MB/GB) / 5 MB/inode = 20 million inodes</para></informalexample>
+ <informalexample>
+ <para>(100 TB * 1024 GB/TB * 1024 MB/GB) / 5 MB/inode = 20 million inodes</para>
+ </informalexample>
<para>We recommend that you use at least twice the minimum number of inodes to allow for future expansion and allow for an average file size smaller than expected. Thus, the required space is:</para>
- <informalexample><para>4 KB/inode * 40 million inodes = 160 GB</para></informalexample>
+ <informalexample>
+ <para>4 KB/inode * 40 million inodes = 160 GB</para>
+ </informalexample>
<para>If the average file size is small, 4 KB for example, Lustre is not very efficient as the MDT uses as much space as the OSTs. However, this is not a common configuration for Lustre.</para>
<note>
<para>If the MDT is too small, this can cause all the space on the OSTs to be unusable. Be sure to determine the appropriate size of the MDT needed to support the file system before formatting the file system. It is difficult to increase the number of inodes after the file system is formatted.</para>
</note>
</section>
<section remap="h3">
- <title>5.2.2 Determining OSS/OST Space Requirements</title>
+ <title><indexterm><primary>setup</primary><secondary>OSS/OST</secondary></indexterm>
+ <indexterm><primary>space</primary><secondary>determining OSS/OST requirements</secondary></indexterm>
+ Determining OSS/OST Space Requirements</title>
<para>For the OST, the amount of space taken by each object depends on the usage pattern of the users/applications running on the system. Lustre defaults to a conservative estimate for the object size (16 KB per object). If you are confident that the average file size for your applications will be larger than this, you can specify a larger average file size (fewer total inodes) to reduce file system overhead and minimize file system check time. See <xref linkend="dbdoclet.50438256_53886"/> for more details.</para>
</section>
</section>
<section xml:id="dbdoclet.50438256_84701">
- <title>5.3 Setting File System Formatting Options</title>
+ <title>
+ <indexterm><primary>file system</primary><secondary>formatting options</secondary></indexterm>
+ <indexterm><primary>setup</primary><secondary>file system</secondary></indexterm>
+ Setting File System Formatting Options</title>
<para>To override the default formatting options for any of the Lustre backing file systems, use this argument to <literal>mkfs.lustre</literal> to pass formatting options to the backing <literal>mkfs</literal>:</para>
<screen>--mkfsoptions='backing fs options'</screen>
<para>For other options to format backing ldiskfs filesystems, see the Linux man page for <literal>mke2fs(8)</literal>.</para>
<section xml:id="dbdoclet.50438256_pgfId-1293228">
- <title>5.3.1 Setting the Number of Inodes for the MDS</title>
+ <title><indexterm><primary>inodes</primary><secondary>MDS</secondary></indexterm><indexterm><primary>setup</primary><secondary>inodes</secondary></indexterm>Setting the Number of Inodes for the MDS</title>
<para>The number of inodes on the MDT is determined at format time based on the total size of the file system to be created. The default MDT inode ratio is one inode for every 4096 bytes of file system space. To override the inode ratio, use the following option:</para>
<screen>-i <emphasis><bytes per inode></emphasis></screen>
<para>For example, use the following option to create one inode per 2048 bytes of file system space.</para>
- <screen>--mkfsoptions="-i 2048"
-</screen>
+ <screen>--mkfsoptions="-i 2048" </screen>
<para>To avoid mke2fs creating an unusable file system, do not specify the -i option with an inode ratio below one inode per 1024 bytes. Instead, specify an absolute number of inodes, using this option:</para>
<screen>-N<emphasis> <number of inodes></emphasis></screen>
<para>For example, by default, a 2 TB MDT will have 512M inodes. The largest currently-supported file system size is 16 TB, which would hold 4B inodes, the maximum possible number of inodes in a ldiskfs file system. With an MDS inode ratio of 1024 bytes per inode, a 2 TB MDT would hold 2B inodes, and a 4 TB MDT would hold 4B inodes.</para>
</section>
<section remap="h3">
- <title>5.3.2 <anchor xml:id="dbdoclet.50438256_34593" xreflabel=""/>Setting the Inode Size for the <anchor xml:id="dbdoclet.50438256_marker-1293231" xreflabel=""/>MDT</title>
+ <title><indexterm><primary>inodes</primary><secondary>MDT</secondary></indexterm>Setting the Inode Size for the MDT</title>
<para>Lustre uses "large" inodes on backing file systems to efficiently store Lustre metadata with each file. On the MDT, each inode is at least 512 bytes in size (by default), while on the OST each inode is 256 bytes in size.</para>
- <para> The backing ldiskfs file system also needs sufficient space for other metadata like the journal (up to 400 MB), bitmaps and directories and a few files that Lustre uses to maintain cluster consistency.</para>
+ <para>The backing ldiskfs file system also needs sufficient space for other metadata like the journal (up to 400 MB), bitmaps and directories and a few files that Lustre uses to maintain cluster consistency.</para>
<para>To specify a larger inode size, use the <literal>-I <inodesize></literal> option. We recommend you do NOT specify a smaller-than-default inode size, as this can lead to serious performance problems; and you cannot change this parameter after formatting the file system. The inode ratio must always be larger than the inode size.</para>
</section>
<section xml:id="dbdoclet.50438256_53886">
- <title>5.3.3 Setting the Number of Inodes for an <anchor xml:id="dbdoclet.50438256_marker-1293235" xreflabel=""/>OST</title>
+ <title><indexterm><primary>inodes</primary><secondary>OST</secondary></indexterm>Setting the Number of Inodes for an OST</title>
<para>When formatting OST file systems, it is normally advantageous to take local file system usage into account. Try to minimize the number of inodes on each OST, while keeping enough margin for potential variance in future usage. This helps reduce the format and file system check time, and makes more space available for data.</para>
- <para> The current default is to create one inode per 16 KB of space in the OST file system, but in many environments, this is far too many inodes for the average file size. As a good rule of thumb, the OSTs should have at least:</para>
+ <para>The current default is to create one inode per 16 KB of space in the OST file system, but in many environments, this is far too many inodes for the average file size. As a good rule of thumb, the OSTs should have at least:</para>
<screen>num_ost_inodes = 4 * <emphasis><num_mds_inodes></emphasis> * <emphasis><default_stripe_count></emphasis> / <emphasis><number_osts></emphasis></screen>
<para>You can specify the number of inodes on the OST file systems using the following option to the <literal>--mkfs</literal> option:</para>
<screen>-N <emphasis><num_inodes></emphasis></screen>
- <para></para>
<para> Alternately, if you know the average file size, then you can specify the OST inode count for the OST file systems using:</para>
<screen>-i <emphasis><average_file_size</emphasis> / (<emphasis>number_of_stripes</emphasis> * 4)<emphasis>></emphasis></screen>
<para>For example, if the average file size is 16 MB and there are, by default 4 stripes per file, then <literal>--mkfsoptions='-i 1048576'</literal> would be appropriate.</para>
<para>For more details on formatting MDT and OST file systems, see <xref linkend="dbdoclet.50438208_51921"/>.</para>
</section>
<section remap="h3">
- <title>5.3.4 File and File System Limits</title>
+ <title><indexterm><primary>setup</primary><secondary>limits</secondary></indexterm>File and File System Limits</title>
<para><xref linkend="settinguplustresystem.tab1"/> describes file and file system size limits. These limits are imposed by either the Lustre architecture or the Linux virtual file system (VFS) and virtual memory subsystems. In a few cases, a limit is defined within the code and can be changed by re-compiling Lustre (see <xref linkend="installinglustrefromsourcecode"/>). In these cases, the indicated limit was used for Lustre testing. </para>
<table frame="all">
<title xml:id="settinguplustresystem.tab1">File and file system limits</title>
<tbody>
<row>
<entry>
- <para> Maximum <anchor xml:id="dbdoclet.50438256_marker-1290714" xreflabel=""/>stripe count</para>
+ <para> Maximum stripe count</para>
</entry>
<entry>
<para> 160</para>
</entry>
<entry>
- <para> This limit is hard-coded, but is near the upper limit imposed by the underlying ldiskfs file system.</para>
+ <para>This limit is hard-coded, but is near the upper limit imposed by the underlying ldiskfs file system.</para>
</entry>
</row>
<row>
<entry>
- <para> Maximum <anchor xml:id="dbdoclet.50438256_marker-1290722" xreflabel=""/>stripe size</para>
+ <para> Maximum stripe size</para>
</entry>
<entry>
<para> < 4 GB</para>
</entry>
<entry>
- <para> The amount of data written to each object before moving on to next object.</para>
+ <para>The amount of data written to each object before moving on to next object.</para>
</entry>
</row>
<row>
<entry>
- <para> Minimum <anchor xml:id="dbdoclet.50438256_marker-1290730" xreflabel=""/>stripe size</para>
+ <para> Minimum stripe size</para>
</entry>
<entry>
<para> 64 KB</para>
</entry>
<entry>
- <para> Due to the 64 KB PAGE_SIZE on some 64-bit machines, the minimum stripe size is set to 64 KB.</para>
+ <para>Due to the 64 KB PAGE_SIZE on some 64-bit machines, the minimum stripe size is set to 64 KB.</para>
</entry>
</row>
<row>
<para> 2 TB</para>
</entry>
<entry>
- <para> The amount of data that can be stored in a single object. The ldiskfs limit of 2TB for a single file applies. Lustre allows 160 stripes of 2 TB each.</para>
+ <para>The amount of data that can be stored in a single object. The ldiskfs limit of 2TB for a single file applies. Lustre allows 160 stripes of 2 TB each.</para>
</entry>
</row>
<row>
<entry>
- <para> Maximum <anchor xml:id="dbdoclet.50438256_marker-1290738" xreflabel=""/>number of OSTs</para>
+ <para> Maximum number of OSTs</para>
</entry>
<entry>
<para> 8150</para>
</entry>
<entry>
- <para> The maximum number of OSTs is a constant that can be changed at compile time. Lustre has been tested with up to 4000 OSTs.</para>
+ <para>The maximum number of OSTs is a constant that can be changed at compile time. Lustre has been tested with up to 4000 OSTs.</para>
</entry>
</row>
<row>
<para> 1</para>
</entry>
<entry>
- <para> Maximum of 1 MDT per file system, but a single MDS can host multiple MDTs, each one for a separate file system.</para>
+ <para>Maximum of 1 MDT per file system, but a single MDS can host multiple MDTs, each one for a separate file system.</para>
</entry>
</row>
<row>
<entry>
- <para> Maximum <anchor xml:id="dbdoclet.50438256_marker-1290746" xreflabel=""/>number of clients</para>
+ <para> Maximum number of clients</para>
</entry>
<entry>
<para> 131072</para>
</entry>
<entry>
- <para> The number of clients is a constant that can be changed at compile time.</para>
+ <para>The number of clients is a constant that can be changed at compile time.</para>
</entry>
</row>
<row>
<entry>
- <para> Maximum size of a <anchor xml:id="dbdoclet.50438256_marker-1290753" xreflabel=""/>file system</para>
+ <para> Maximum size of a file system</para>
</entry>
<entry>
<para> 64 PB</para>
</entry>
<entry>
- <para> Each OST or MDT can have a file system up to 16 TB, regardless of whether 32-bit or 64-bit kernels are on the server.</para>
+ <para>Each OST or MDT can have a file system up to 16 TB, regardless of whether 32-bit or 64-bit kernels are on the server.</para>
<para>You can have multiple OST file systems on a single OSS node.</para>
</entry>
</row>
<para>320 TB on 64-bit systems</para>
</entry>
<entry>
- <para> Individual files have a hard limit of nearly 16 TB on 32-bit systems imposed by the kernel memory subsystem. On 64-bit systems this limit does not exist. Hence, files can be 64-bits in size. Lustre imposes an additional size limit of up to the number of stripes, where each stripe is 2 TB.</para>
+ <para>Individual files have a hard limit of nearly 16 TB on 32-bit systems imposed by the kernel memory subsystem. On 64-bit systems this limit does not exist. Hence, files can be 64-bits in size. Lustre imposes an additional size limit of up to the number of stripes, where each stripe is 2 TB.</para>
<para>A single file can have a maximum of 160 stripes, which gives an upper single file limit of 320 TB for 64-bit systems. The actual amount of data that can be stored in a file depends upon the amount of free space in each OST on which the file is striped.</para>
</entry>
</row>
<row>
<entry>
- <para> Maximum number of <anchor xml:id="dbdoclet.50438256_marker-1290770" xreflabel=""/>files or subdirectories in a single directory</para>
+ <para> Maximum number of files or subdirectories in a single directory</para>
</entry>
<entry>
<para> 10 million files</para>
</entry>
<entry>
- <para> Lustre uses the ldiskfs hashed directory code, which has a limit of about 10 million files depending on the length of the file name. The limit on subdirectories is the same as the limit on regular files.</para>
+ <para>Lustre uses the ldiskfs hashed directory code, which has a limit of about 10 million files depending on the length of the file name. The limit on subdirectories is the same as the limit on regular files.</para>
<para>Lustre is tested with ten million files in a single directory.</para>
</entry>
</row>
<para> 4 billion</para>
</entry>
<entry>
- <para> The ldiskfs file system imposes an upper limit of 4 billion inodes. By default, the MDS file system is formatted with 4 KB of space per inode, meaning 512 million inodes per file system of 2 TB.</para>
+ <para>The ldiskfs file system imposes an upper limit of 4 billion inodes. By default, the MDS file system is formatted with 4 KB of space per inode, meaning 512 million inodes per file system of 2 TB.</para>
<para>This can be increased initially, at the time of MDS file system creation. For more information, see <xref linkend="settinguplustresystem"/>.</para>
</entry>
</row>
<row>
<entry>
- <para> Maximum length of a <anchor xml:id="dbdoclet.50438256_marker-1290791" xreflabel=""/>filename</para>
+ <para> Maximum length of a filename</para>
</entry>
<entry>
<para> 255 bytes (filename)</para>
</entry>
<entry>
- <para> This limit is 255 bytes for a single filename, the same as in an ldiskfs file system.</para>
+ <para>This limit is 255 bytes for a single filename, the same as in an ldiskfs file system.</para>
</entry>
</row>
<row>
<entry>
- <para> Maximum length of a <anchor xml:id="dbdoclet.50438256_marker-1292019" xreflabel=""/>pathname</para>
+ <para> Maximum length of a pathname</para>
</entry>
<entry>
<para> 4096 bytes (pathname)</para>
</entry>
<entry>
- <para> The Linux VFS imposes a full pathname length of 4096 bytes.</para>
+ <para>The Linux VFS imposes a full pathname length of 4096 bytes.</para>
</entry>
</row>
<row>
<entry>
- <para> Maximum number of <anchor xml:id="dbdoclet.50438256_marker-1290800" xreflabel=""/>open files for Lustre file systems</para>
+ <para> Maximum number of open files for Lustre file systems</para>
</entry>
<entry>
<para> None</para>
</entry>
<entry>
- <para> Lustre does not impose a maximum for the number of open files, but the practical limit depends on the amount of RAM on the MDS. No "tables" for open files exist on the MDS, as they are only linked in a list to a given client's export. Each client process probably has a limit of several thousands of open files which depends on the ulimit.</para>
+ <para>Lustre does not impose a maximum for the number of open files, but the practical limit depends on the amount of RAM on the MDS. No "tables" for open files exist on the MDS, as they are only linked in a list to a given client's export. Each client process probably has a limit of several thousands of open files which depends on the ulimit.</para>
</entry>
</row>
</tbody>
</section>
</section>
<section xml:id="dbdoclet.50438256_26456">
- <title>5.4 Determining Memory<anchor xml:id="dbdoclet.50438256_marker-1292212" xreflabel=""/> Requirements</title>
+ <title><indexterm><primary>setup</primary><secondary>memory</secondary></indexterm>Determining Memory Requirements</title>
<para>This section describes the memory requirements for each Lustre file system component.</para>
<section remap="h3">
- <title>5.4.1 Client Memory Requirements</title>
+ <title>
+ <indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>client</tertiary></indexterm>
+ Client Memory Requirements</title>
<para>A minimum of 2 GB RAM is recommended for clients.</para>
</section>
<section remap="h3">
- <title>5.4.2 <anchor xml:id="dbdoclet.50438256_78447" xreflabel=""/>MDS Memory Requirements</title>
+ <title><indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>MDS</tertiary></indexterm>MDS Memory Requirements</title>
<para>MDS memory requirements are determined by the following factors:</para>
<itemizedlist>
<listitem>
- <para> Number of clients</para>
+ <para>Number of clients</para>
</listitem>
<listitem>
- <para> Size of the directories</para>
+ <para>Size of the directories</para>
</listitem>
<listitem>
- <para> Load placed on server</para>
+ <para>Load placed on server</para>
</listitem>
</itemizedlist>
<para>The amount of memory used by the MDS is a function of how many clients are on the system, and how many files they are using in their working set. This is driven, primarily, by the number of locks a client can hold at one time. The number of locks held by clients varies by load and memory availability on the server. Interactive clients can hold in excess of 10,000 locks at times. On the MDS, memory usage is approximately 2 KB per file, including the Lustre distributed lock manager (DLM) lock and kernel data structures for the files currently in use. Having file data in cache can improve metadata performance by a factor of 10x or more compared to reading it from disk.</para>
</listitem>
</itemizedlist>
<section remap="h4">
- <title>5.4.2.1 Calculating MDS Memory Requirements</title>
+ <title><indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>MDS</tertiary></indexterm>Calculating MDS Memory Requirements</title>
<para>By default, 400 MB are used for the file system journal. Additional RAM is used for caching file data for the larger working set, which is not actively in use by clients but should be kept "hot" for improved access times. Approximately 1.5 KB per file is needed to keep a file in cache without a lock.</para>
<para>For example, for a single MDT on an MDS with 1,000 clients, 16 interactive nodes, and a 2 million file working set (of which 400,000 files are cached on the clients):</para>
<informalexample>
</section>
</section>
<section remap="h3">
- <title>5.4.3 <anchor xml:id="dbdoclet.50438256_84553" xreflabel=""/>OSS Memory Requirements</title>
+ <title><indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>OSS</tertiary></indexterm>OSS Memory Requirements</title>
<para>When planning the hardware for an OSS node, consider the memory usage of several components in the Lustre system (i.e., journal, service threads, file system metadata, etc.). Also, consider the effect of the OSS read cache feature, which consumes memory as it caches data on the OSS node.</para>
<para>In addition to the MDS memory requirements mentioned in <xref linkend="dbdoclet.50438256_87676"/>, the OSS requirements include:</para>
<itemizedlist>
<para>The same calculation applies to files accessed from the OSS as for the MDS, but the load is distributed over many more OSSs nodes, so the amount of memory required for locks, inode cache, etc. listed under MDS is spread out over the OSS nodes.</para>
<para>Because of these memory requirements, the following calculations should be taken as determining the absolute minimum RAM required in an OSS node.</para>
<section remap="h4">
- <title>5.4.3.1 Calculating OSS Memory Requirements</title>
+ <title><indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>OSS</tertiary></indexterm>Calculating OSS Memory Requirements</title>
<para>The minimum recommended RAM size for an OSS with two OSTs is computed below:</para>
<informalexample>
<para>Ethernet/TCP send/receive buffers (1 MB * 512 threads) = 512 MB</para>
</section>
</section>
<section xml:id="dbdoclet.50438256_78272">
- <title>5.5 Implementing Networks To Be Used by Lustre</title>
+ <title><indexterm><primary>setup</primary><secondary>network</secondary></indexterm>Implementing Networks To Be Used by Lustre</title>
<para>As a high performance file system, Lustre places heavy loads on networks. Thus, a network interface in each Lustre server and client is commonly dedicated to Lustre traffic. This is often a dedicated TCP/IP subnet, although other network hardware can also be used.</para>
<para>A typical Lustre implementation may include the following:</para>
<itemizedlist>
<listitem>
- <para> A high-performance backend network for the Lustre servers, typically an InfiniBand (IB) network.</para>
+ <para>A high-performance backend network for the Lustre servers, typically an InfiniBand (IB) network.</para>
</listitem>
<listitem>
- <para> A larger client network.</para>
+ <para>A larger client network.</para>
</listitem>
<listitem>
- <para> Lustre routers to connect the two networks.</para>
+ <para>Lustre routers to connect the two networks.</para>
</listitem>
</itemizedlist>
<para>Lustre networks and routing are configured and managed by specifying parameters to the Lustre Networking (lnet) module in <literal>/etc/modprobe.conf</literal> or <literal>/etc/modprobe.conf.local</literal> (depending on your Linux distribution).</para>
<para>If you are using multiple network types, then you'll need a router. Any node with appropriate interfaces can route Lustre Networking (LNET) traffic between different network hardware types or topologies --the node may be a server, a client, or a standalone router. LNET can route messages between different network types (such as TCP-to-InfiniBand) or across different topologies (such as bridging two InfiniBand or TCP/IP networks). Routing will be configured in <xref linkend="configuringlnet"/>.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">
- Identify the network interfaces to include in or exclude from LNET.
- </emphasis>
+ <para><emphasis role="bold">Identify the network interfaces to include in or exclude from LNET. </emphasis>
</para>
<para>If not explicitly specified, LNET uses either the first available interface or a pre-defined default for a given network type. Interfaces that LNET should not use (such as an administrative network or IP-over-IB), can be excluded.</para>
<para>Network interfaces to be used or excluded will be specified using the lnet kernel module parameters networks and <literal>ip2netsas</literal> described in <xref linkend="configuringlnet"/>.</para>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='systemconfigurationutilities'>
- <info>
- <title xml:id='systemconfigurationutilities.title'>System Configuration Utilities</title>
- </info>
-
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. -->
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="systemconfigurationutilities">
+ <title xml:id="systemconfigurationutilities.title">System Configuration Utilities</title>
<para>This chapter includes system configuration utilities and includes the following sections:</para>
- <itemizedlist><listitem>
+ <itemizedlist>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_55923"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_76969"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_38274"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_58217"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_44971"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_84890"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_90386"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_23232"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_23648"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_64286"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_90218"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_54734"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_63667"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_75432"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_12635"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_82679"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_51496"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_39574"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438219_99928"/></para>
</listitem>
-
-</itemizedlist>
- <section xml:id="dbdoclet.50438219_55923">
- <title>36.1 e2<anchor xml:id="dbdoclet.50438219_marker-1317398" xreflabel=""/>scan</title>
- <para>The e2scan utility is an ext2 file system-modified inode scan program. The e2scan program uses libext2fs to find inodes with ctime or mtime newer than a given time and prints out their pathname. Use e2scan to efficiently generate lists of files that have been modified. The e2scan tool is included in the e2fsprogs package, located at:</para>
- <para><link xl:href="http://downloads.lustre.org/public/tools/e2fsprogs/">http://downloads.lustre.org/public/tools/e2fsprogs/</link></para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>e2scan [options] [-f file] block_device
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>When invoked, the e2scan utility iterates all inodes on the block device, finds modified inodes, and prints their inode numbers. A similar iterator, using libext2fs(5), builds a table (called parent database) which lists the parent node for each inode. With a lookup function, you can reconstruct modified pathnames from root.</para>
- </section>
- <section remap="h5">
- <title>Options</title>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">-b inode buffer blocks</emphasis></para></entry>
- <entry><para> Sets the readahead inode blocks to get excellent performance when scanning the block device.</para><para> </para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">-o output file</emphasis></para></entry>
- <entry><para> If an output file is specified, modified pathnames are written to this file. Otherwise, modified parameters are written to stdout.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">-t inode | pathname</emphasis></para></entry>
- <entry><para> Sets the e2scan type if type is inode. The e2scan utility prints modified inode numbers to stdout. By default, the type is set as pathname.</para><para>The e2scan utility lists modified pathnames based on modified inode numbers.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">-u</emphasis></para></entry>
- <entry><para> Rebuilds the parent database from scratch. Otherwise, the current parent database is used.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438219_55923">
+ <title><indexterm><primary>e2scan</primary></indexterm>
+ e2scan</title>
+ <para>The e2scan utility is an ext2 file system-modified inode scan program. The e2scan program uses libext2fs to find inodes with ctime or mtime newer than a given time and prints out their pathname. Use e2scan to efficiently generate lists of files that have been modified. The e2scan tool is included in the e2fsprogs package, located at:</para>
+ <para><link xl:href="http://downloads.lustre.org/public/tools/e2fsprogs/">http://downloads.lustre.org/public/tools/e2fsprogs/</link></para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>e2scan [options] [-f file] block_device</screen>
</section>
- <section xml:id="dbdoclet.50438219_76969">
- <title>36.2 l_<anchor xml:id="dbdoclet.50438219_marker-1318227" xreflabel=""/>getidentity</title>
- <para>The l_getidentity utility handles Lustre user / group cache upcall.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>l_getidentity {mdtname} {uid}
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The group upcall file contains the path to an executable file that, when properly installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the mds_grp_downcall_data structure and write it to the /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info pseudo-file.</para>
- <para>The l_getidentity utility is the reference implementation of the user or group cache upcall.</para>
- </section>
- <section remap="h5">
- <title>Options</title>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">mdtname</emphasis></para></entry>
- <entry><para> Metadata server target name</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">uid</emphasis></para></entry>
- <entry><para> User identifier</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Files</title>
- <para>The l_getidentity files are located at:</para>
- <screen>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall
-</screen>
- </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>When invoked, the e2scan utility iterates all inodes on the block device, finds modified inodes, and prints their inode numbers. A similar iterator, using libext2fs(5), builds a table (called parent database) which lists the parent node for each inode. With a lookup function, you can reconstruct modified pathnames from root.</para>
</section>
- <section xml:id="dbdoclet.50438219_38274">
- <title>36.3 lc<anchor xml:id="dbdoclet.50438219_marker-1302539" xreflabel=""/>tl</title>
- <para>The lctl utility is used for root control and configuration. With lctl you can directly control Lustre via an ioctl interface, allowing various configuration, maintenance and debugging features to be accessed.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>lctl
-lctl --device <devno><emphasis/><command [args]>
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The lctl utility can be invoked in interactive mode by issuing the lctl command. After that, commands are issued as shown below. The most common lctl commands are:</para>
- <screen>dl
+ <section remap="h5">
+ <title>Options</title>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-b inode buffer blocks</emphasis></para>
+ </entry>
+ <entry>
+ <para> Sets the readahead inode blocks to get excellent performance when scanning the block device.</para>
+ <para> </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-o output file</emphasis></para>
+ </entry>
+ <entry>
+ <para> If an output file is specified, modified pathnames are written to this file. Otherwise, modified parameters are written to stdout.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-t inode | pathname</emphasis></para>
+ </entry>
+ <entry>
+ <para> Sets the e2scan type if type is inode. The e2scan utility prints modified inode numbers to stdout. By default, the type is set as pathname.</para>
+ <para>The e2scan utility lists modified pathnames based on modified inode numbers.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-u</emphasis></para>
+ </entry>
+ <entry>
+ <para> Rebuilds the parent database from scratch. Otherwise, the current parent database is used.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438219_76969">
+ <title><indexterm><primary>l_getidentity</primary></indexterm>
+l_getidentity</title>
+ <para>The l_getidentity utility handles Lustre user / group cache upcall.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>l_getidentity {mdtname} {uid}</screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The group upcall file contains the path to an executable file that, when properly installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the mds_grp_downcall_data structure and write it to the /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info pseudo-file.</para>
+ <para>The l_getidentity utility is the reference implementation of the user or group cache upcall.</para>
+ </section>
+ <section remap="h5">
+ <title>Options</title>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">mdtname</emphasis></para>
+ </entry>
+ <entry>
+ <para> Metadata server target name</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">uid</emphasis></para>
+ </entry>
+ <entry>
+ <para> User identifier</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Files</title>
+ <para>The l_getidentity files are located at:</para>
+ <screen>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall</screen>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438219_38274">
+ <title><indexterm><primary>lctl</primary></indexterm>
+lctl</title>
+ <para>The lctl utility is used for root control and configuration. With lctl you can directly control Lustre via an ioctl interface, allowing various configuration, maintenance and debugging features to be accessed.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>lctl
+lctl --device <devno> <command [args]></screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The lctl utility can be invoked in interactive mode by issuing the lctl command. After that, commands are issued as shown below. The most common lctl commands are:</para>
+ <screen>dl
dk
device
network <<emphasis>up/down</emphasis>>
list_nids
ping <emphasis>nid</emphasis>help
-quit
-</screen>
- <para>For a complete list of available commands, type help at the lctl prompt. To get basic help on command meaning and syntax, type help<emphasis>command</emphasis>. Command completion is activated with the TAB key, and command history is available via the up- and down-arrow keys.</para>
- <para>For non-interactive use, use the second invocation, which runs the command after connecting to the device.</para>
- </section>
- <section remap="h5">
- <title>Setting Parameters with lctl</title>
- <para>Lustre parameters are not always accessible using the procfs interface, as it is platform-specific. As a solution, lctl {get,set}_param has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to /proc/{fs,sys}/{lustre,lnet}. For future portability, use lctl {get,set}_param .</para>
- <para>When the file system is running, use the lctl set_param command to set temporary parameters (mapping to items in /proc/{fs,sys}/{lnet,lustre}). The lctl set_param command uses this syntax:</para>
- <screen>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value>
-</screen>
- <para>For example:</para>
- <screen>$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
-</screen>
- <para>Many permanent parameters can be set with lctl conf_param. In general, lctl conf_param can be used to specify any parameter settable in a /proc/fs/lustre file, with its own OBD device. The lctl conf_param command uses this syntax:</para>
- <screen><obd|fsname>.<obdtype>.<proc_file_name>=<value>)
-</screen>
- <para>For example:</para>
- <screen>$ lctl conf_param testfs-MDT0000.mdt.group_upcall=NONE
-$ lctl conf_param testfs.llite.max_read_ahead_mb=16
-</screen>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="5*"/>
- <colspec colname="c2" colwidth="95*"/>
-
-
- <tbody>
- <row>
- <entry><para><inlinemediaobject><imageobject role="html">
- <imagedata fileref="./shared/caution.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis>The lctlconf_param command permanently sets parameters in the file system configuration.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para>To get current Lustre parameter settings, use the lctl get_param command with this syntax:</para>
- <screen>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name>
-</screen>
- <para>For example:</para>
- <screen>$ lctl get_param -n ost.*.ost_io.timeouts
-</screen>
- <para>To list Lustre parameters that are available to set, use the lctl list_param command, with this syntax:</para>
- <screen>lctl list_param [-n] <obdtype>.<obdname>
-</screen>
- <para>For example:</para>
- <screen>$ lctl list_param obdfilter.lustre-OST0000
-</screen>
-<para>For more information on using lctl to set temporary and permanent parameters, see <xref linkend="dbdoclet.50438194_51490"/> (Setting Parameters with lctl).</para>
- <para><emphasis role="bold">Network Configuration</emphasis></para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">network</emphasis> <<emphasis>up/down</emphasis>>|<<emphasis>tcp/elan/myrinet</emphasis>></para></entry>
- <entry><para> Starts or stops LNET, or selects a network type for other <emphasis role="bold">lctl</emphasis> LNET commands.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">list_nids</emphasis></para></entry>
- <entry><para> Prints all NIDs on the local node. LNET must be running.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">which_nid</emphasis> <<emphasis>nidlist</emphasis>></para></entry>
- <entry><para> From a list of NIDs for a remote node, identifies the NID on which interface communication will occur.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">ping</emphasis> <<emphasis>nid</emphasis>></para></entry>
- <entry><para> Checks LNET connectivity via an LNET ping. This uses the fabric appropriate to the specified NID.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">interface_list</emphasis></para></entry>
- <entry><para> Prints the network interface information for a given <emphasis role="bold">network</emphasis> type.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">peer_list</emphasis></para></entry>
- <entry><para> Prints the known peers for a given <emphasis role="bold">network</emphasis> type.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">conn_list</emphasis></para></entry>
- <entry><para> Prints all the connected remote NIDs for a given <emphasis role="bold">network</emphasis> type.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">active_tx</emphasis></para></entry>
- <entry><para> This command prints active transmits. It is only used for the Elan <emphasis role="bold">network</emphasis> type.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">route_list</emphasis></para></entry>
- <entry><para> Prints the complete routing table.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Device Selection</emphasis></para>
- <informaltable frame="all">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"> </emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">device</emphasis> <<emphasis>devname</emphasis>></para></entry>
- <entry><para> </para></entry>
- <entry><para> This selects the specified OBD device. All other commands depend on the device being set.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">device_list</emphasis></para></entry>
- <entry><para> </para></entry>
- <entry><para> Shows the local Lustre OBDs, a/k/a <emphasis role="bold">dl</emphasis>.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Device Operations</emphasis></para>
- <informaltable frame="all">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
- <thead>
- <row>
- <entry nameend="c2" namest="c1"><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">list_param</emphasis><emphasis>[-F|-R]</emphasis> <<emphasis>param_path ...</emphasis>></para></entry>
- <entry><para> Lists the Lustre or LNET parameter name.</para><para> </para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> -<emphasis role="bold">F</emphasis></para></entry>
- <entry><para> Adds '/', '@' or '=' for directories, symlinks and writeable files, respectively.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> -<emphasis role="bold">R</emphasis></para></entry>
- <entry><para> Recursively lists all parameters under the specified path. If <emphasis>param_path</emphasis> is unspecified, all parameters are shown.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">get_param</emphasis><emphasis>[-n|-N|-F]</emphasis> <<emphasis>param_path ...</emphasis>></para></entry>
- <entry><para> Gets the value of a Lustre or LNET parameter from the specified path.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">-n</emphasis></para></entry>
- <entry><para> Prints only the parameter value and not the parameter name.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">-N</emphasis></para></entry>
- <entry><para> Prints only matched parameter names and not the values; especially useful when using patterns.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">-F</emphasis></para></entry>
- <entry><para> When <emphasis role="bold">-N</emphasis> is specified, adds '/', '@' or '=' for directories, symlinks and writeable files, respectively.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">set_param</emphasis><emphasis>[-n]</emphasis><emphasis><param_path=value...></emphasis></para></entry>
- <entry><para> Sets the value of a Lustre or LNET parameter from the specified path.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">-n</emphasis></para></entry>
- <entry><para> Disables printing of the key name when printing values.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">conf_param</emphasis><emphasis>[-d] <device|fsname>.<parameter>=<value></emphasis></para></entry>
- <entry><para> Sets a permanent configuration parameter for any device via the MGS. This command must be run on the MGS node.</para><para>All writeable parameters under <emphasis role="bold">lctl list_param</emphasis> (e.g. <emphasis>lctl list_param -F osc.*.* | grep</emphasis> =) can be permanently set using <emphasis role="bold">lctl conf_param</emphasis>, but the format is slightly different. For <emphasis role="bold">conf_param</emphasis>, the device is specified first, then the obdtype. Wildcards are not supported. Additionally, failover nodes may be added (or removed), and some system-wide parameters may be set as well (sys.at_max, sys.at_min, sys.at_extra, sys.at_early_margin, sys.at_history, sys.timeout, sys.ldlm_timeout). For system-wide parameters, <device> is ignored.</para><para>For more information on setting permanent parameters and <emphasis role="bold">lctl conf_param</emphasis> command examples, see <xref linkend="dbdoclet.50438194_64195"/> (Setting Permanent Parameters).</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">-d <device|fsname>.<parameter></emphasis></para><para> </para></entry>
- <entry><para> Deletes a parameter setting (use the default value at the next restart). A null value for <emphasis role="bold"><value></emphasis> also deletes the parameter setting.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">activate</emphasis></para></entry>
- <entry><para> Re-activates an import after the deactivate operation. This setting is only effective until the next restart (see <emphasis role="bold">conf_param</emphasis>).</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">deactivate</emphasis></para></entry>
- <entry><para> Deactivates an import, in particular meaning do not assign new file stripes to an OSC. Running lctl deactivate on the MDS stops new objects from being allocated on the OST. Running lctl deactivate on Lustre clients causes them to return -EIO when accessing objects on the OST instead of waiting for recovery.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">abort_recovery</emphasis></para></entry>
- <entry><para> Aborts the recovery process on a re-starting MDT or OST.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <note><para>Lustre tunables are not always accessible using the procfs interface, as it is platform-specific. As a solution, lctl {get,set,list}_param has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to /proc/{fs,sys}/{lustre,lnet}. For future portability, use lctl {get,set,list}_param instead.</para></note>
- <para><emphasis role="bold">Virtual Block Device Operations</emphasis></para>
- <para>Lustre can emulate a virtual block device upon a regular file. This emulation is needed when you are trying to set up a swap space via the file.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">blockdev_attach</emphasis><emphasis><file name> <device node></emphasis></para></entry>
- <entry><para> Attaches a regular Lustre file to a block device. If the device node does not exist, <emphasis role="bold">lctl</emphasis> creates it. We recommend that you create the device node by <emphasis role="bold">lctl</emphasis> since the emulator uses a dynamical major number.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">blockdev_detach</emphasis><emphasis><device node></emphasis></para></entry>
- <entry><para> Detaches the virtual block device.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">blockdev_info</emphasis><emphasis><device node></emphasis></para></entry>
- <entry><para> Provides information about the Lustre file attached to the device node.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Changelogs</emphasis></para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">changelog_register</emphasis></para></entry>
- <entry><para> Registers a new changelog user for a particular device. Changelog entries are not purged beyond a registered user's set point (see <emphasis role="bold">lfs changelog_clear</emphasis>).</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">changelog_deregister</emphasis><emphasis><id></emphasis></para></entry>
- <entry><para> Unregisters an existing changelog user. If the user's "clear" record number is the minimum for the device, changelog records are purged until the next minimum.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Debug</emphasis></para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">debug_daemon</emphasis></para></entry>
- <entry><para> Starts and stops the debug daemon, and controls the output filename and size.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">debug_kernel</emphasis><emphasis>[file] [raw]</emphasis></para></entry>
- <entry><para> Dumps the kernel debug buffer to stdout or a file.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">debug_file</emphasis><emphasis><input> [output]</emphasis></para></entry>
- <entry><para> Converts the kernel-dumped debug log from binary to plain text format.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">clear</emphasis></para></entry>
- <entry><para> Clears the kernel debug buffer.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">mark</emphasis><emphasis><text></emphasis></para></entry>
- <entry><para> Inserts marker text in the kernel debug buffer.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">filter</emphasis><emphasis><subsystem id/debug mask></emphasis></para></entry>
- <entry><para> Filters kernel debug messages by subsystem or mask.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">show</emphasis><emphasis><subsystem id/debug mask></emphasis></para></entry>
- <entry><para> Shows specific types of messages.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">debug_list</emphasis><emphasis><subs/types></emphasis></para></entry>
- <entry><para> Lists all subsystem and debug types.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">modules</emphasis><emphasis><path></emphasis></para></entry>
- <entry><para> Provides GDB-friendly module information.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Options</title>
- <para>Use the following options to invoke lctl.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">--device</emphasis></para></entry>
- <entry><para> Device to be used for the operation (specified by name or number). See device_list.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--ignore_errors | ignore_errors</emphasis></para></entry>
- <entry><para> Ignores errors during script processing.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Examples</title>
- <para><emphasis role="bold">lctl</emphasis></para>
- <screen>$ lctl
+quit</screen>
+ <para>For a complete list of available commands, type help at the lctl prompt. To get basic help on command meaning and syntax, type help<emphasis>command</emphasis>. Command completion is activated with the TAB key, and command history is available via the up- and down-arrow keys.</para>
+ <para>For non-interactive use, use the second invocation, which runs the command after connecting to the device.</para>
+ </section>
+ <section remap="h5">
+ <title>Setting Parameters with lctl</title>
+ <para>Lustre parameters are not always accessible using the procfs interface, as it is platform-specific. As a solution, lctl {get,set}_param has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to /proc/{fs,sys}/{lustre,lnet}. For future portability, use lctl {get,set}_param .</para>
+ <para>When the file system is running, use the lctl set_param command to set temporary parameters (mapping to items in /proc/{fs,sys}/{lnet,lustre}). The lctl set_param command uses this syntax:</para>
+ <screen>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value></screen>
+ <para>For example:</para>
+ <screen>$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))</screen>
+ <para>Many permanent parameters can be set with lctl conf_param. In general, lctl conf_param can be used to specify any parameter settable in a /proc/fs/lustre file, with its own OBD device. The lctl conf_param command uses this syntax:</para>
+ <screen><obd|fsname>.<obdtype>.<proc_file_name>=<value>) </screen>
+ <para>For example:</para>
+ <screen>$ lctl conf_param testfs-MDT0000.mdt.group_upcall=NONE
+$ lctl conf_param testfs.llite.max_read_ahead_mb=16 </screen>
+ <caution>
+ <para>The lctlconf_param command permanently sets parameters in the file system configuration.</para>
+ </caution>
+ <para>To get current Lustre parameter settings, use the lctl get_param command with this syntax:</para>
+ <screen>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name></screen>
+ <para>For example:</para>
+ <screen>$ lctl get_param -n ost.*.ost_io.timeouts </screen>
+ <para>To list Lustre parameters that are available to set, use the lctl list_param command, with this syntax:</para>
+ <screen>lctl list_param [-n] <obdtype>.<obdname></screen>
+ <para>For example:</para>
+ <screen>$ lctl list_param obdfilter.lustre-OST0000</screen>
+ <para>For more information on using lctl to set temporary and permanent parameters, see <xref linkend="dbdoclet.50438194_51490"/> (Setting Parameters with lctl).</para>
+ <para><emphasis role="bold">Network Configuration</emphasis></para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">network</emphasis> <<emphasis>up/down</emphasis>>|<<emphasis>tcp/elan/myrinet</emphasis>></para>
+ </entry>
+ <entry>
+ <para> Starts or stops LNET, or selects a network type for other <emphasis role="bold">lctl</emphasis> LNET commands.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">list_nids</emphasis></para>
+ </entry>
+ <entry>
+ <para> Prints all NIDs on the local node. LNET must be running.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">which_nid</emphasis> <<emphasis>nidlist</emphasis>></para>
+ </entry>
+ <entry>
+ <para> From a list of NIDs for a remote node, identifies the NID on which interface communication will occur.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">ping</emphasis> <<emphasis>nid</emphasis>></para>
+ </entry>
+ <entry>
+ <para> Checks LNET connectivity via an LNET ping. This uses the fabric appropriate to the specified NID.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">interface_list</emphasis></para>
+ </entry>
+ <entry>
+ <para> Prints the network interface information for a given <emphasis role="bold">network</emphasis> type.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">peer_list</emphasis></para>
+ </entry>
+ <entry>
+ <para> Prints the known peers for a given <emphasis role="bold">network</emphasis> type.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">conn_list</emphasis></para>
+ </entry>
+ <entry>
+ <para> Prints all the connected remote NIDs for a given <emphasis role="bold">network</emphasis> type.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">active_tx</emphasis></para>
+ </entry>
+ <entry>
+ <para> This command prints active transmits. It is only used for the Elan <emphasis role="bold">network</emphasis> type.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">route_list</emphasis></para>
+ </entry>
+ <entry>
+ <para> Prints the complete routing table.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Device Selection</emphasis></para>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="33*"/>
+ <colspec colname="c2" colwidth="33*"/>
+ <colspec colname="c3" colwidth="33*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold"> </emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">device</emphasis> <<emphasis>devname</emphasis>></para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> This selects the specified OBD device. All other commands depend on the device being set.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">device_list</emphasis></para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> Shows the local Lustre OBDs, a/k/a <emphasis role="bold">dl</emphasis>.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Device Operations</emphasis></para>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="33*"/>
+ <colspec colname="c2" colwidth="33*"/>
+ <colspec colname="c3" colwidth="33*"/>
+ <thead>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">list_param</emphasis><emphasis>[-F|-R]</emphasis> <<emphasis>param_path ...</emphasis>></para>
+ </entry>
+ <entry>
+ <para> Lists the Lustre or LNET parameter name.</para>
+ <para> </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> -<emphasis role="bold">F</emphasis></para>
+ </entry>
+ <entry>
+ <para> Adds '/', '@' or '=' for directories, symlinks and writeable files, respectively.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> -<emphasis role="bold">R</emphasis></para>
+ </entry>
+ <entry>
+ <para> Recursively lists all parameters under the specified path. If <emphasis>param_path</emphasis> is unspecified, all parameters are shown.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">get_param</emphasis><emphasis>[-n|-N|-F]</emphasis> <<emphasis>param_path ...</emphasis>></para>
+ </entry>
+ <entry>
+ <para> Gets the value of a Lustre or LNET parameter from the specified path.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">-n</emphasis></para>
+ </entry>
+ <entry>
+ <para> Prints only the parameter value and not the parameter name.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">-N</emphasis></para>
+ </entry>
+ <entry>
+ <para> Prints only matched parameter names and not the values; especially useful when using patterns.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">-F</emphasis></para>
+ </entry>
+ <entry>
+ <para> When <emphasis role="bold">-N</emphasis> is specified, adds '/', '@' or '=' for directories, symlinks and writeable files, respectively.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">set_param</emphasis><emphasis>[-n]</emphasis><emphasis><param_path=value...></emphasis></para>
+ </entry>
+ <entry>
+ <para> Sets the value of a Lustre or LNET parameter from the specified path.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">-n</emphasis></para>
+ </entry>
+ <entry>
+ <para> Disables printing of the key name when printing values.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">conf_param</emphasis><emphasis>[-d] <device|fsname>.<parameter>=<value></emphasis></para>
+ </entry>
+ <entry>
+ <para> Sets a permanent configuration parameter for any device via the MGS. This command must be run on the MGS node.</para>
+ <para>All writeable parameters under <emphasis role="bold">lctl list_param</emphasis> (e.g. <emphasis>lctl list_param -F osc.*.* | grep</emphasis> =) can be permanently set using <emphasis role="bold">lctl conf_param</emphasis>, but the format is slightly different. For <emphasis role="bold">conf_param</emphasis>, the device is specified first, then the obdtype. Wildcards are not supported. Additionally, failover nodes may be added (or removed), and some system-wide parameters may be set as well (sys.at_max, sys.at_min, sys.at_extra, sys.at_early_margin, sys.at_history, sys.timeout, sys.ldlm_timeout). For system-wide parameters, <device> is ignored.</para>
+ <para>For more information on setting permanent parameters and <emphasis role="bold">lctl conf_param</emphasis> command examples, see <xref linkend="dbdoclet.50438194_64195"/> (Setting Permanent Parameters).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">-d <device|fsname>.<parameter></emphasis></para>
+ <para> </para>
+ </entry>
+ <entry>
+ <para> Deletes a parameter setting (use the default value at the next restart). A null value for <emphasis role="bold"><value></emphasis> also deletes the parameter setting.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">activate</emphasis></para>
+ </entry>
+ <entry>
+ <para> Re-activates an import after the deactivate operation. This setting is only effective until the next restart (see <emphasis role="bold">conf_param</emphasis>).</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">deactivate</emphasis></para>
+ </entry>
+ <entry>
+ <para> Deactivates an import, in particular meaning do not assign new file stripes to an OSC. Running lctl deactivate on the MDS stops new objects from being allocated on the OST. Running lctl deactivate on Lustre clients causes them to return -EIO when accessing objects on the OST instead of waiting for recovery.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">abort_recovery</emphasis></para>
+ </entry>
+ <entry>
+ <para> Aborts the recovery process on a re-starting MDT or OST.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <note>
+ <para>Lustre tunables are not always accessible using the procfs interface, as it is platform-specific. As a solution, lctl {get,set,list}_param has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to /proc/{fs,sys}/{lustre,lnet}. For future portability, use lctl {get,set,list}_param instead.</para>
+ </note>
+ <para><emphasis role="bold">Virtual Block Device Operations</emphasis></para>
+ <para>Lustre can emulate a virtual block device upon a regular file. This emulation is needed when you are trying to set up a swap space via the file.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">blockdev_attach</emphasis><emphasis><file name> <device node></emphasis></para>
+ </entry>
+ <entry>
+ <para> Attaches a regular Lustre file to a block device. If the device node does not exist, <emphasis role="bold">lctl</emphasis> creates it. We recommend that you create the device node by <emphasis role="bold">lctl</emphasis> since the emulator uses a dynamical major number.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">blockdev_detach</emphasis><emphasis><device node></emphasis></para>
+ </entry>
+ <entry>
+ <para> Detaches the virtual block device.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">blockdev_info</emphasis><emphasis><device node></emphasis></para>
+ </entry>
+ <entry>
+ <para> Provides information about the Lustre file attached to the device node.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Changelogs</emphasis></para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">changelog_register</emphasis></para>
+ </entry>
+ <entry>
+ <para> Registers a new changelog user for a particular device. Changelog entries are not purged beyond a registered user's set point (see <emphasis role="bold">lfs changelog_clear</emphasis>).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">changelog_deregister</emphasis><emphasis><id></emphasis></para>
+ </entry>
+ <entry>
+ <para> Unregisters an existing changelog user. If the user's "clear" record number is the minimum for the device, changelog records are purged until the next minimum.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Debug</emphasis></para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">debug_daemon</emphasis></para>
+ </entry>
+ <entry>
+ <para> Starts and stops the debug daemon, and controls the output filename and size.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">debug_kernel</emphasis><emphasis>[file] [raw]</emphasis></para>
+ </entry>
+ <entry>
+ <para> Dumps the kernel debug buffer to stdout or a file.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">debug_file</emphasis><emphasis><input> [output]</emphasis></para>
+ </entry>
+ <entry>
+ <para> Converts the kernel-dumped debug log from binary to plain text format.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">clear</emphasis></para>
+ </entry>
+ <entry>
+ <para> Clears the kernel debug buffer.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">mark</emphasis><emphasis><text></emphasis></para>
+ </entry>
+ <entry>
+ <para> Inserts marker text in the kernel debug buffer.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">filter</emphasis><emphasis><subsystem id/debug mask></emphasis></para>
+ </entry>
+ <entry>
+ <para> Filters kernel debug messages by subsystem or mask.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">show</emphasis><emphasis><subsystem id/debug mask></emphasis></para>
+ </entry>
+ <entry>
+ <para> Shows specific types of messages.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">debug_list</emphasis><emphasis><subs/types></emphasis></para>
+ </entry>
+ <entry>
+ <para> Lists all subsystem and debug types.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">modules</emphasis><emphasis><path></emphasis></para>
+ </entry>
+ <entry>
+ <para> Provides GDB-friendly module information.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Options</title>
+ <para>Use the following options to invoke lctl.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--device</emphasis></para>
+ </entry>
+ <entry>
+ <para> Device to be used for the operation (specified by name or number). See device_list.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--ignore_errors | ignore_errors</emphasis></para>
+ </entry>
+ <entry>
+ <para> Ignores errors during script processing.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Examples</title>
+ <para><emphasis role="bold">lctl</emphasis></para>
+ <screen>$ lctl
lctl > dl
0 UP mgc MGC192.168.0.20@tcp btbb24e3-7deb-2ffa-eab0-44dffe00f692 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter testfs-OST0000 testfs-OST0000_UUID 3
lctl > dk /tmp/log Debug log: 87 lines, 87 kept, 0 dropped.
-lctl > quit
-</screen>
- </section>
- <section remap="h5">
- <title>See Also</title>
- <itemizedlist>
- <listitem>
- <para> <xref linkend="dbdoclet.50438219_75432"/> </para>
- </listitem> <listitem>
- <para> <xref linkend="dbdoclet.50438219_12635"/> </para>
- </listitem> <listitem>
- <para> <xref linkend="dbdoclet.50438219_38274"/> </para>
- </listitem> <listitem>
- <para> <xref linkend="dbdoclet.50438206_94597"/> </para>
- </listitem>
- </itemizedlist>
- </section>
+lctl > quit</screen>
</section>
- <section xml:id="dbdoclet.50438219_58217">
- <title>36.4 ll_decode_filter_fid</title>
- <para>The ll_decode_filter_fid utility displays the Lustre object ID and MDT parent FID.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>ll_decode_filter_fid object_file [object_file ...]
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The ll_decode_filter_fid utility decodes and prints the Lustre OST object ID, MDT FID, stripe index for the specified OST object(s), which is stored in the "trusted.fid" attribute on each OST object. This is accessible to ll_decode_filter_fid when the OST filesystem is mounted locally as type ldiskfs for maintenance.</para>
- <para>The "trusted.fid" extended attribute is stored on each OST object when it is first modified (data written or attributes set), and is not accessed or modified by Lustre after that time.</para>
- <para>The OST object ID (objid) is useful in case of OST directory corruption, though normally the ll_recover_lost_found_objs(8) utility is able to reconstruct the entire OST object directory hierarchy. The MDS FID can be useful to determine which MDS inode an OST object is (or was) used by. The stripe index can be used in conjunction with other OST objects to reconstruct the layout of a file even if the MDT inode was lost.</para>
- </section>
- <section remap="h5">
- <title>Examples</title>
- <screen>root@oss1# cd /mnt/ost/lost+found
+ <section remap="h5">
+ <title>See Also</title>
+ <itemizedlist>
+ <listitem>
+ <para> <xref linkend="dbdoclet.50438219_75432"/> </para>
+ </listitem>
+ <listitem>
+ <para> <xref linkend="dbdoclet.50438219_12635"/> </para>
+ </listitem>
+ <listitem>
+ <para> <xref linkend="dbdoclet.50438219_38274"/> </para>
+ </listitem>
+ <listitem>
+ <para> <xref linkend="dbdoclet.50438206_94597"/> </para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438219_58217">
+ <title><indexterm><primary>ll_decode_filter_fid</primary></indexterm>
+ll_decode_filter_fid</title>
+ <para>The ll_decode_filter_fid utility displays the Lustre object ID and MDT parent FID.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>ll_decode_filter_fid object_file [object_file ...]</screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The ll_decode_filter_fid utility decodes and prints the Lustre OST object ID, MDT FID, stripe index for the specified OST object(s), which is stored in the "trusted.fid" attribute on each OST object. This is accessible to ll_decode_filter_fid when the OST filesystem is mounted locally as type ldiskfs for maintenance.</para>
+ <para>The "trusted.fid" extended attribute is stored on each OST object when it is first modified (data written or attributes set), and is not accessed or modified by Lustre after that time.</para>
+ <para>The OST object ID (objid) is useful in case of OST directory corruption, though normally the ll_recover_lost_found_objs(8) utility is able to reconstruct the entire OST object directory hierarchy. The MDS FID can be useful to determine which MDS inode an OST object is (or was) used by. The stripe index can be used in conjunction with other OST objects to reconstruct the layout of a file even if the MDT inode was lost.</para>
+ </section>
+ <section remap="h5">
+ <title>Examples</title>
+ <screen>root@oss1# cd /mnt/ost/lost+found
root@oss1# ll_decode_filter_fid #12345[4,5,8]
#123454: objid=690670 seq=0 parent=[0x751c5:0xfce6e605:0x0]
#123455: objid=614725 seq=0 parent=[0x18d11:0xebba84eb:0x1]
-#123458: objid=533088 seq=0 parent=[0x21417:0x19734d61:0x0]
-</screen>
- <para>This shows that the three files in lost+found have decimal object IDs - 690670, 614725, and 533088, respectively. The object sequence number (formerly object group) is 0 for all current OST objects.</para>
- <para>The MDT parent inode FIDs are hexdecimal numbers of the form sequence:oid:idx. Since the sequence number is below 0x100000000 in all these cases, the FIDs are in the legacy Inode and Generation In FID (IGIF) namespace and are mapped directly to the MDT inode = seq and generation = oid values; the MDT inodes are 0x751c5, 0x18d11, and 0x21417 respectively. For objects with MDT parent sequence numbers above 0x200000000, this indicates that the FID needs to be mapped via the MDT Object Index (OI) file on the MDT to determine the internal inode number.</para>
- <para>The idx field shows the stripe number of this OST object in the Lustre RAID-0 striped file.</para>
- </section>
- <section remap="h5">
- <title>See Also</title>
- <para><xref linkend="dbdoclet.50438219_44971"/></para>
- </section>
+#123458: objid=533088 seq=0 parent=[0x21417:0x19734d61:0x0]</screen>
+ <para>This shows that the three files in lost+found have decimal object IDs - 690670, 614725, and 533088, respectively. The object sequence number (formerly object group) is 0 for all current OST objects.</para>
+ <para>The MDT parent inode FIDs are hexdecimal numbers of the form sequence:oid:idx. Since the sequence number is below 0x100000000 in all these cases, the FIDs are in the legacy Inode and Generation In FID (IGIF) namespace and are mapped directly to the MDT inode = seq and generation = oid values; the MDT inodes are 0x751c5, 0x18d11, and 0x21417 respectively. For objects with MDT parent sequence numbers above 0x200000000, this indicates that the FID needs to be mapped via the MDT Object Index (OI) file on the MDT to determine the internal inode number.</para>
+ <para>The idx field shows the stripe number of this OST object in the Lustre RAID-0 striped file.</para>
</section>
- <section xml:id="dbdoclet.50438219_44971">
- <title>36.5 ll_recover_lost_<anchor xml:id="dbdoclet.50438219_marker-1317865" xreflabel=""/>found_objs</title>
- <para>The ll_recover_lost_found_objs utility helps recover Lustre OST objects (file data) from a lost and found directory and return them to their correct locations.</para>
- <note><para>Running the ll_recover_lost_found_objs tool is not strictly necessary to bring an OST back online, it just avoids losing access to objects that were moved to the lost and found directory due to directory corruption.</para></note>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>$ ll_recover_lost_found_objs [-hv] -d directory
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The first time Lustre writes to an object, it saves the MDS inode number and the objid as an extended attribute on the object, so in case of directory corruption of the OST, it is possible to recover the objects. Running e2fsck fixes the corrupted OST directory, but it puts all of the objects into a lost and found directory, where they are inaccessible to Lustre. Use the ll_recover_lost_found_objs utility to recover all (or at least most) objects from a lost and found directory and return them to the O/0/d* directories.</para>
- <para>To use ll_recover_lost_found_objs, mount the file system locally (using the -t ldiskfs command), run the utility and then unmount it again. The OST must not be mounted by Lustre when ll_recover_lost_found_objs is run.</para>
- </section>
- <section remap="h5">
- <title>Options</title>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"> Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">-h</emphasis></para></entry>
- <entry><para> Prints a help message</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">-v</emphasis></para></entry>
- <entry><para> Increases verbosity</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">-d</emphasis><emphasis>directory</emphasis></para></entry>
- <entry><para> Sets the lost and found directory path</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Example</title>
- <screen>ll_recover_lost_found_objs -d /mnt/ost/lost+found
-</screen>
- </section>
+ <section remap="h5">
+ <title>See Also</title>
+ <para><xref linkend="dbdoclet.50438219_44971"/></para>
</section>
- <section xml:id="dbdoclet.50438219_84890">
- <title>36.6 llobdstat</title>
- <para>The llobdstat utility displays OST statistics.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>llobdstat ost_name [interval]
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The llobdstat utility displays a line of OST statistics for the given ost_name every interval seconds. It should be run directly on an OSS node. Type <emphasis>CTRL-C</emphasis> to stop statistics printing.</para>
- </section>
- <section remap="h5">
- <title>Example</title>
- <screen># llobdstat liane-OST0002 1
+ </section>
+ <section xml:id="dbdoclet.50438219_44971">
+ <title><indexterm><primary>ll_recover_lost_found_objs</primary></indexterm>
+ll_recover_lost_found_objs</title>
+ <para>The ll_recover_lost_found_objs utility helps recover Lustre OST objects (file data) from a lost and found directory and return them to their correct locations.</para>
+ <note>
+ <para>Running the ll_recover_lost_found_objs tool is not strictly necessary to bring an OST back online, it just avoids losing access to objects that were moved to the lost and found directory due to directory corruption.</para>
+ </note>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>$ ll_recover_lost_found_objs [-hv] -d directory</screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The first time Lustre writes to an object, it saves the MDS inode number and the objid as an extended attribute on the object, so in case of directory corruption of the OST, it is possible to recover the objects. Running e2fsck fixes the corrupted OST directory, but it puts all of the objects into a lost and found directory, where they are inaccessible to Lustre. Use the ll_recover_lost_found_objs utility to recover all (or at least most) objects from a lost and found directory and return them to the O/0/d* directories.</para>
+ <para>To use ll_recover_lost_found_objs, mount the file system locally (using the -t ldiskfs command), run the utility and then unmount it again. The OST must not be mounted by Lustre when ll_recover_lost_found_objs is run.</para>
+ </section>
+ <section remap="h5">
+ <title>Options</title>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold"> Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-h</emphasis></para>
+ </entry>
+ <entry>
+ <para> Prints a help message</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-v</emphasis></para>
+ </entry>
+ <entry>
+ <para> Increases verbosity</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-d</emphasis><emphasis> directory</emphasis></para>
+ </entry>
+ <entry>
+ <para> Sets the lost and found directory path</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Example</title>
+ <screen>ll_recover_lost_found_objs -d /mnt/ost/lost+found </screen>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438219_84890">
+ <title><indexterm><primary>llodbstat</primary></indexterm>
+llobdstat</title>
+ <para>The llobdstat utility displays OST statistics.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>llobdstat ost_name [interval]</screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The llobdstat utility displays a line of OST statistics for the given ost_name every interval seconds. It should be run directly on an OSS node. Type <emphasis>CTRL-C</emphasis> to stop statistics printing.</para>
+ </section>
+ <section remap="h5">
+ <title>Example</title>
+ <screen># llobdstat liane-OST0002 1
/usr/bin/llobdstat on /proc/fs/lustre/obdfilter/liane-OST0002/stats
Processor counters run at 2800.189 MHz
Read: 1.21431e+07, Write: 9.93363e+08, create/destroy: 24/1499, stat: 34, p\
1217026056 0.00MB 0.00MB/s 0.00MB 0.00MB/s
1217026057 0.00MB 0.00MB/s 0.00MB 0.00MB/s
1217026058 0.00MB 0.00MB/s 0.00MB 0.00MB/s
-1217026059 0.00MB 0.00MB/s 0.00MB 0.00MB/s st:1
-</screen>
- </section>
- <section remap="h5">
- <title>Files</title>
- <screen>/proc/fs/lustre/obdfilter/<ostname>/stats
-</screen>
- </section>
+1217026059 0.00MB 0.00MB/s 0.00MB 0.00MB/s st:1</screen>
</section>
- <section xml:id="dbdoclet.50438219_90386">
- <title>36.7 llog_reader</title>
- <para>The llog_reader utility parses Lustre's on-disk configuration logs.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>llog_reader filename
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The llog_reader utility parses the binary format of Lustre's on-disk configuration logs. Llog_reader can only read logs; use tunefs.lustre to write to them.</para>
- <para>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs, then use llog_reader to dump the log file's contents, for example:</para>
- <screen>mount -t ldiskfs /dev/sda /mnt/mgs
-llog_reader /mnt/mgs/CONFIGS/tfs-client
-</screen>
- <para>To examine the same log file on a running Lustre server, use the ldiskfs-enabled debugfs utility (called debug.ldiskfs on some distributions) to extract the file, for example:</para>
- <screen>debugfs -c -R 'dump CONFIGS/tfs-client /tmp/tfs-client' /dev/sda
-llog_reader /tmp/tfs-client
-</screen>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="5*"/>
- <colspec colname="c2" colwidth="95*"/>
-
-
- <tbody>
- <row>
- <entry><para><inlinemediaobject><imageobject role="html">
- <imagedata fileref="./shared/caution.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis>Although they are stored in the CONFIGS directory, mountdata files do not use the configuration log format and will confuse the llog_reader utility.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>See Also</title>
- <para><xref linkend="dbdoclet.50438219_39574"/></para>
- </section>
+ <section remap="h5">
+ <title>Files</title>
+ <screen>/proc/fs/lustre/obdfilter/<ostname>/stats</screen>
</section>
- <section xml:id="dbdoclet.50438219_23232">
- <title>36.8 ll<anchor xml:id="dbdoclet.50438219_marker-1318052" xreflabel=""/>stat</title>
- <para>The llstat utility displays Lustre statistics.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>llstat [-c] [-g] [-i interval] stats_file
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The llstat utility displays statistics from any of the Lustre statistics files that share a common format and are updated at <emphasis>interval</emphasis> seconds. To stop statistics printing, use <emphasis>ctrl</emphasis>-<emphasis>c.</emphasis></para>
- </section>
- <section remap="h5">
- <title>Options</title>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"> Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">-c</emphasis></para></entry>
- <entry><para> Clears the statistics file.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">-i</emphasis></para></entry>
- <entry><para> Specifies the polling period (in seconds).</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">-g</emphasis></para></entry>
- <entry><para> Specifies graphable output format.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">-h</emphasis></para></entry>
- <entry><para> Displays help information.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">stats_file</emphasis></para></entry>
- <entry><para> Specifies either the full path to a statistics file or the shorthand reference, <emphasis role="bold">mds</emphasis> or <emphasis role="bold">ost</emphasis></para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Example</title>
- <para>To monitor /proc/fs/lustre/ost/OSS/ost/stats at 1 second intervals, run;</para>
- <screen>llstat -i 1 ost
+ </section>
+ <section xml:id="dbdoclet.50438219_90386">
+ <title><indexterm><primary>llog_reader</primary></indexterm>
+llog_reader</title>
+ <para>The llog_reader utility parses Lustre's on-disk configuration logs.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>llog_reader filename</screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The llog_reader utility parses the binary format of Lustre's on-disk configuration logs. Llog_reader can only read logs; use tunefs.lustre to write to them.</para>
+ <para>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs, then use llog_reader to dump the log file's contents, for example:</para>
+ <screen>mount -t ldiskfs /dev/sda /mnt/mgs
+llog_reader /mnt/mgs/CONFIGS/tfs-client</screen>
+ <para>To examine the same log file on a running Lustre server, use the ldiskfs-enabled debugfs utility (called debug.ldiskfs on some distributions) to extract the file, for example:</para>
+ <screen>debugfs -c -R 'dump CONFIGS/tfs-client /tmp/tfs-client' /dev/sda
+llog_reader /tmp/tfs-client</screen>
+ <caution>
+ <para>Although they are stored in the CONFIGS directory, mountdata files do not use the configuration log format and will confuse the llog_reader utility.</para>
+ </caution>
+ </section>
+ <section remap="h5">
+ <title>See Also</title>
+ <para><xref linkend="dbdoclet.50438219_39574"/></para>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438219_23232">
+ <title><indexterm><primary>llstat</primary></indexterm>
+llstat</title>
+ <para>The llstat utility displays Lustre statistics.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>llstat [-c] [-g] [-i interval] stats_file
</screen>
- </section>
- <section remap="h5">
- <title>Files</title>
- <para>The llstat files are located at:</para>
- <screen>/proc/fs/lustre/mdt/MDS/*/stats
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The llstat utility displays statistics from any of the Lustre statistics files that share a common format and are updated at <emphasis>interval</emphasis> seconds. To stop statistics printing, use <emphasis>ctrl</emphasis>-<emphasis>c.</emphasis></para>
+ </section>
+ <section remap="h5">
+ <title>Options</title>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold"> Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-c</emphasis></para>
+ </entry>
+ <entry>
+ <para> Clears the statistics file.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-i</emphasis></para>
+ </entry>
+ <entry>
+ <para> Specifies the polling period (in seconds).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-g</emphasis></para>
+ </entry>
+ <entry>
+ <para> Specifies graphable output format.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-h</emphasis></para>
+ </entry>
+ <entry>
+ <para> Displays help information.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">stats_file</emphasis></para>
+ </entry>
+ <entry>
+ <para> Specifies either the full path to a statistics file or the shorthand reference, <emphasis role="bold">mds</emphasis> or <emphasis role="bold">ost</emphasis></para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Example</title>
+ <para>To monitor /proc/fs/lustre/ost/OSS/ost/stats at 1 second intervals, run;</para>
+ <screen>llstat -i 1 ost</screen>
+ </section>
+ <section remap="h5">
+ <title>Files</title>
+ <para>The llstat files are located at:</para>
+ <screen>/proc/fs/lustre/mdt/MDS/*/stats
/proc/fs/lustre/mds/*/exports/*/stats
/proc/fs/lustre/mdc/*/stats
/proc/fs/lustre/ldlm/services/*/stats
/proc/fs/lustre/obdfilter/*/stats
/proc/fs/lustre/llite/*/stats
</screen>
- </section>
</section>
- <section xml:id="dbdoclet.50438219_23648">
- <title>36.9 llverdev<anchor xml:id="dbdoclet.50438219_marker-1317750" xreflabel=""/></title>
- <para>The llverdev verifies a block device is functioning properly over its full size.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>llverdev [-c <emphasis>chunksize</emphasis>] [-f] [-h] [-o <emphasis>offset</emphasis>] [-l] [-p] [-r] [-t <emphasis>timestamp</emphasis>] [-v] [-w] <emphasis>device</emphasis></screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>Sometimes kernel drivers or hardware devices have bugs that prevent them from accessing the full device size correctly, or possibly have bad sectors on disk or other problems which prevent proper data storage. There are often defects associated with major system boundaries such as 2^32 bytes, 2^31 sectors, 2^31 blocks, 2^32 blocks, etc.</para>
- <para>The llverdev utility writes and verifies a unique test pattern across the entire device to ensure that data is accessible after it was written, and that data written to one part of the disk is not overwriting data on another part of the disk.</para>
- <para>It is expected that llverdev will be run on large size devices (TB). It is always better to run llverdev in verbose mode, so that device testing can be easily restarted from the point where it was stopped.</para>
- <para>Running a full verification can be time-consuming for very large devices. We recommend starting with a partial verification to ensure that the device is minimally sane before investing in a full verification.</para>
- </section>
- <section remap="h5">
- <title>Options</title>
- <informaltable frame="all">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"> </emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-c|--chunksize</emphasis></para></entry>
- <entry><para> I/O chunk size in bytes (default value is 1048576).</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-f|--force</emphasis></para></entry>
- <entry><para> Forces the test to run without a confirmation that the device will be overwritten and all data will be permanently destroyed.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-h|--help</emphasis></para></entry>
- <entry><para> Displays a brief help message.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-o</emphasis><emphasis>offset</emphasis></para></entry>
- <entry><para> Offset (in kilobytes) of the start of the test (default value is 0).</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-l|--long</emphasis></para></entry>
- <entry><para> Runs a full check, writing and then reading and verifying every block on the disk.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-p|--partial</emphasis></para></entry>
- <entry><para> Runs a partial check, only doing periodic checks across the device (1 GB steps).</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-r|--read</emphasis></para></entry>
- <entry><para> Runs the test in read (verify) mode only, after having previously run the test in <emphasis role="bold">-w</emphasis> mode.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-t</emphasis><emphasis>timestamp</emphasis></para></entry>
- <entry><para> Sets the test start time as printed at the start of a previously-interrupted test to ensure that validation data is the same across the entire filesystem (default value is the current time()).</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-v|--verbose</emphasis></para></entry>
- <entry><para> Runs the test in verbose mode, listing each read and write operation.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-w|--write</emphasis></para></entry>
- <entry><para> Runs the test in write (test-pattern) mode (default runs both read and write).</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Examples</title>
- <para>Runs a partial device verification on /dev/sda:</para>
- <screen>llverdev -v -p /dev/sda
+ </section>
+ <section xml:id="dbdoclet.50438219_23648">
+ <title><indexterm><primary>llverdev</primary></indexterm>
+llverdev</title>
+ <para>The llverdev verifies a block device is functioning properly over its full size.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>llverdev [-c <emphasis>chunksize</emphasis>] [-f] [-h] [-o <emphasis>offset</emphasis>] [-l] [-p] [-r] [-t <emphasis>timestamp</emphasis>] [-v] [-w] <emphasis>device</emphasis></screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>Sometimes kernel drivers or hardware devices have bugs that prevent them from accessing the full device size correctly, or possibly have bad sectors on disk or other problems which prevent proper data storage. There are often defects associated with major system boundaries such as 2^32 bytes, 2^31 sectors, 2^31 blocks, 2^32 blocks, etc.</para>
+ <para>The llverdev utility writes and verifies a unique test pattern across the entire device to ensure that data is accessible after it was written, and that data written to one part of the disk is not overwriting data on another part of the disk.</para>
+ <para>It is expected that llverdev will be run on large size devices (TB). It is always better to run llverdev in verbose mode, so that device testing can be easily restarted from the point where it was stopped.</para>
+ <para>Running a full verification can be time-consuming for very large devices. We recommend starting with a partial verification to ensure that the device is minimally sane before investing in a full verification.</para>
+ </section>
+ <section remap="h5">
+ <title>Options</title>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="33*"/>
+ <colspec colname="c2" colwidth="33*"/>
+ <colspec colname="c3" colwidth="33*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold"> </emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">-c|--chunksize</emphasis></para>
+ </entry>
+ <entry>
+ <para> I/O chunk size in bytes (default value is 1048576).</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">-f|--force</emphasis></para>
+ </entry>
+ <entry>
+ <para> Forces the test to run without a confirmation that the device will be overwritten and all data will be permanently destroyed.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">-h|--help</emphasis></para>
+ </entry>
+ <entry>
+ <para> Displays a brief help message.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">-o</emphasis><emphasis>offset</emphasis></para>
+ </entry>
+ <entry>
+ <para> Offset (in kilobytes) of the start of the test (default value is 0).</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">-l|--long</emphasis></para>
+ </entry>
+ <entry>
+ <para> Runs a full check, writing and then reading and verifying every block on the disk.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">-p|--partial</emphasis></para>
+ </entry>
+ <entry>
+ <para> Runs a partial check, only doing periodic checks across the device (1 GB steps).</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">-r|--read</emphasis></para>
+ </entry>
+ <entry>
+ <para> Runs the test in read (verify) mode only, after having previously run the test in <emphasis role="bold">-w</emphasis> mode.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">-t</emphasis><emphasis>timestamp</emphasis></para>
+ </entry>
+ <entry>
+ <para> Sets the test start time as printed at the start of a previously-interrupted test to ensure that validation data is the same across the entire filesystem (default value is the current time()).</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">-v|--verbose</emphasis></para>
+ </entry>
+ <entry>
+ <para> Runs the test in verbose mode, listing each read and write operation.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">-w|--write</emphasis></para>
+ </entry>
+ <entry>
+ <para> Runs the test in write (test-pattern) mode (default runs both read and write).</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Examples</title>
+ <para>Runs a partial device verification on /dev/sda:</para>
+ <screen>llverdev -v -p /dev/sda
llverdev: permanently overwrite all data on /dev/sda (yes/no)? y
llverdev: /dev/sda is 4398046511104 bytes (4096.0 GB) in size
Timestamp: 1009839028
-Current write offset: 4096 kB
-</screen>
- <para>Continues an interrupted verification at offset 4096kB from the start of the device, using the same timestamp as the previous run:</para>
- <screen>llverdev -f -v -p --offset=4096 --timestamp=1009839028 /dev/sda
+Current write offset: 4096 kB</screen>
+ <para>Continues an interrupted verification at offset 4096kB from the start of the device, using the same timestamp as the previous run:</para>
+ <screen>llverdev -f -v -p --offset=4096 --timestamp=1009839028 /dev/sda
llverdev: /dev/sda is 4398046511104 bytes (4096.0 GB) in size
Timestamp: 1009839028
write complete
-read complete
-</screen>
- </section>
+read complete </screen>
</section>
- <section xml:id="dbdoclet.50438219_64286">
- <title>36.10 lshowmount<anchor xml:id="dbdoclet.50438219_marker-1317690" xreflabel=""/></title>
- <para>The lshowmount utility shows Lustre exports.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>lshowmount [-ehlv]
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The lshowmount utility shows the hosts that have Lustre mounted to a server. Ths utility looks for exports from the MGS, MDS, and obdfilter.</para>
- </section>
- <section remap="h5">
- <title>Options</title>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">-e|--enumerate</emphasis></para></entry>
- <entry><para> Causes lshowmount to list each client mounted on a separate line instead of trying to compress the list of clients into a hostrange string.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">-h|--help</emphasis></para></entry>
- <entry><para> Causes lshowmount to print out a usage message.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">-l|--lookup</emphasis></para></entry>
- <entry><para> Causes lshowmount to try to look up the hostname for NIDs that look like IP addresses.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">-v|--verbose</emphasis></para></entry>
- <entry><para> Causes lshowmount to output export information for each service instead of only displaying the aggregate information for all Lustre services on the server.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Files</title>
- <screen>/proc/fs/lustre/mgs/<server>/exports/<uuid>/nid /proc/fs/lustre/mds/<server>/expo\
-rts/<uuid>/nid /proc/fs/lustre/obdfilter/<server>/exports/<uuid>/nid
-</screen>
- </section>
+ </section>
+ <section xml:id="dbdoclet.50438219_64286">
+ <title><indexterm><primary>lshowmount</primary></indexterm>
+lshowmount</title>
+ <para>The lshowmount utility shows Lustre exports.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>lshowmount [-ehlv]</screen>
</section>
- <section xml:id="dbdoclet.50438219_90218">
- <title>36.11 l<anchor xml:id="dbdoclet.50438219_marker-1318121" xreflabel=""/><anchor xml:id="dbdoclet.50438219_marker-1318122" xreflabel=""/>st</title>
- <para>The lst utility starts LNET self-test.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>lst
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>LNET self-test helps site administrators confirm that Lustre Networking (LNET) has been properly installed and configured. The self-test also confirms that LNET and the network software and hardware underlying it are performing as expected.</para>
- <para>Each LNET self-test runs in the context of a session. A node can be associated with only one session at a time, to ensure that the session has exclusive use of the nodes on which it is running. A session is create, controlled and monitored from a single node; this is referred to as the self-test console.</para>
- <para>Any node may act as the self-test console. Nodes are named and allocated to a self-test session in groups. This allows all nodes in a group to be referenced by a single name.</para>
- <para>Test configurations are built by describing and running test batches. A test batch is a named collection of tests, with each test composed of a number of individual point-to-point tests running in parallel. These individual point-to-point tests are instantiated according to the test type, source group, target group and distribution specified when the test is added to the test batch.</para>
- </section>
- <section remap="h5">
- <title>Modules</title>
- <para>To run LNET self-test, load these modules: libcfs, lnet, lnet_selftest and any one of the klnds (ksocklnd, ko2iblnd...). To load all necessary modules, run modprobe lnet_selftest, which recursively loads the modules on which lnet_selftest depends.</para>
- <para>There are two types of nodes for LNET self-test: the console node and test nodes. Both node types require all previously-specified modules to be loaded. (The userspace test node does not require these modules).</para>
- <para>Test nodes can be in either kernel or in userspace. A console user can invite a kernel test node to join the test session by running lst add_group NID, but the user cannot actively add a userspace test node to the test session. However, the console user can passively accept a test node to the test session while the test node runs lst client to connect to the console.</para>
- </section>
- <section remap="h5">
- <title>Utilities</title>
- <para>LNET self-test includes two user utilities, lst and lstclient.</para>
- <para>lst is the user interface for the self-test console (run on the console node). It provides a list of commands to control the entire test system, such as create session, create test groups, etc.</para>
- <para>lstclient is the userspace self-test program which is linked with userspace LNDs and LNET. A user can invoke lstclient to join a self-test session:</para>
- <screen>lstclient -sesid CONSOLE_NID group NAME
-</screen>
- </section>
- <section remap="h5">
- <title>Example Script</title>
- <para>This is a sample LNET self-test script which simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an IB network (connected via LNET routers), with half the clients reading and half the clients writing.</para>
- <screen>#!/bin/bash
+ <section remap="h5">
+ <title>Description</title>
+ <para>The lshowmount utility shows the hosts that have Lustre mounted to a server. Ths utility looks for exports from the MGS, MDS, and obdfilter.</para>
+ </section>
+ <section remap="h5">
+ <title>Options</title>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-e|--enumerate</emphasis></para>
+ </entry>
+ <entry>
+ <para> Causes lshowmount to list each client mounted on a separate line instead of trying to compress the list of clients into a hostrange string.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-h|--help</emphasis></para>
+ </entry>
+ <entry>
+ <para> Causes lshowmount to print out a usage message.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-l|--lookup</emphasis></para>
+ </entry>
+ <entry>
+ <para> Causes lshowmount to try to look up the hostname for NIDs that look like IP addresses.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">-v|--verbose</emphasis></para>
+ </entry>
+ <entry>
+ <para> Causes lshowmount to output export information for each service instead of only displaying the aggregate information for all Lustre services on the server.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Files</title>
+ <screen>/proc/fs/lustre/mgs/<server>/exports/<uuid>/nid /proc/fs/lustre/mds/<server>/expo\
+rts/<uuid>/nid /proc/fs/lustre/obdfilter/<server>/exports/<uuid>/nid</screen>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438219_90218">
+ <title><indexterm><primary>lst</primary></indexterm>
+lst</title>
+ <para>The lst utility starts LNET self-test.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>lst</screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>LNET self-test helps site administrators confirm that Lustre Networking (LNET) has been properly installed and configured. The self-test also confirms that LNET and the network software and hardware underlying it are performing as expected.</para>
+ <para>Each LNET self-test runs in the context of a session. A node can be associated with only one session at a time, to ensure that the session has exclusive use of the nodes on which it is running. A session is create, controlled and monitored from a single node; this is referred to as the self-test console.</para>
+ <para>Any node may act as the self-test console. Nodes are named and allocated to a self-test session in groups. This allows all nodes in a group to be referenced by a single name.</para>
+ <para>Test configurations are built by describing and running test batches. A test batch is a named collection of tests, with each test composed of a number of individual point-to-point tests running in parallel. These individual point-to-point tests are instantiated according to the test type, source group, target group and distribution specified when the test is added to the test batch.</para>
+ </section>
+ <section remap="h5">
+ <title>Modules</title>
+ <para>To run LNET self-test, load these modules: libcfs, lnet, lnet_selftest and any one of the klnds (ksocklnd, ko2iblnd...). To load all necessary modules, run modprobe lnet_selftest, which recursively loads the modules on which lnet_selftest depends.</para>
+ <para>There are two types of nodes for LNET self-test: the console node and test nodes. Both node types require all previously-specified modules to be loaded. (The userspace test node does not require these modules).</para>
+ <para>Test nodes can be in either kernel or in userspace. A console user can invite a kernel test node to join the test session by running lst add_group NID, but the user cannot actively add a userspace test node to the test session. However, the console user can passively accept a test node to the test session while the test node runs lst client to connect to the console.</para>
+ </section>
+ <section remap="h5">
+ <title>Utilities</title>
+ <para>LNET self-test includes two user utilities, lst and lstclient.</para>
+ <para>lst is the user interface for the self-test console (run on the console node). It provides a list of commands to control the entire test system, such as create session, create test groups, etc.</para>
+ <para>lstclient is the userspace self-test program which is linked with userspace LNDs and LNET. A user can invoke lstclient to join a self-test session:</para>
+ <screen>lstclient -sesid CONSOLE_NID group NAME</screen>
+ </section>
+ <section remap="h5">
+ <title>Example Script</title>
+ <para>This is a sample LNET self-test script which simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an IB network (connected via LNET routers), with half the clients reading and half the clients writing.</para>
+ <screen>#!/bin/bash
export LST_SESSION=$$
lst new_session read/write
lst add_group servers 192.168.10.[8,10,12-16]@tcp
# display server stats for 30 seconds
lst stat servers & sleep 30; kill $!
# tear down
-lst end_session
-</screen>
- </section>
+lst end_session </screen>
</section>
- <section xml:id="dbdoclet.50438219_54734">
- <title>36.12 lustre_<anchor xml:id="dbdoclet.50438219_marker-1317379" xreflabel=""/>rmmod.sh</title>
- <para>The lustre_rmmod.sh utility removes all Lustre and LNET modules (assuming no Lustre services are running). It is located in /usr/bin.</para>
- <note><para>The lustre_rmmod.sh utility does not work if Lustre modules are being used or if you have manually run the lctl network up command.</para></note>
- </section>
- <section xml:id="dbdoclet.50438219_63667">
- <title>36.13 lustre_<anchor xml:id="dbdoclet.50438219_marker-1317224" xreflabel=""/>rsync</title>
- <para>The lustre_rsync utility synchronizes (replicates) a Lustre file system to a target file system.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>lustre_rsync --source|-s <src> --target|-t <tgt>
+ </section>
+ <section xml:id="dbdoclet.50438219_54734">
+ <title><indexterm><primary>lustre_rmmod.sh</primary></indexterm>
+lustre_rmmod.sh</title>
+ <para>The lustre_rmmod.sh utility removes all Lustre and LNET modules (assuming no Lustre services are running). It is located in /usr/bin.</para>
+ <note>
+ <para>The lustre_rmmod.sh utility does not work if Lustre modules are being used or if you have manually run the lctl network up command.</para>
+ </note>
+ </section>
+ <section xml:id="dbdoclet.50438219_63667">
+ <title><indexterm><primary>lustre_rsync</primary></indexterm>
+lustre_rsync</title>
+ <para>The lustre_rsync utility synchronizes (replicates) a Lustre file system to a target file system.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>lustre_rsync --source|-s <src> --target|-t <tgt>
--mdt|-m <mdt> [--user|-u <user id>]
[--xattr|-x <yes|no>] [--verbose|-v]
[--statuslog|-l <log>] [--dry-run] [--abort-on-err]
lustre_rsync --statuslog|-l <log>
lustre_rsync --statuslog|-l <log> --source|-s <source>
- --target|-t <tgt> --mdt|-m <mdt>
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The lustre_rsync utility is designed to synchronize (replicate) a Lustre file system (source) to another file system (target). The target can be a Lustre file system or any other type, and is a normal, usable file system. The synchronization operation is efficient and does not require directory walking, as lustre_rsync uses Lustre MDT changelogs to identify changes in the Lustre file system.</para>
- <para>Before using lustre_rsync:</para>
- <itemizedlist><listitem>
- <para> A changelog user must be registered (see lctl (8) changelog_register)</para>
- </listitem>
-
-</itemizedlist>
- <para>- AND -</para>
- <itemizedlist><listitem>
- <para> Verify that the Lustre file system (source) and the replica file system (target) are identical before the changelog user is registered. If the file systems are discrepant, use a utility, e.g. regular rsync (not lustre_rsync) to make them identical.</para>
- </listitem>
-
-</itemizedlist>
- </section>
- <section remap="h5">
- <title>Options</title>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">--source=</emphasis><emphasis><src></emphasis></para></entry>
- <entry><para> The path to the root of the Lustre file system (source) which will be synchronized. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--target=</emphasis><emphasis><tgt></emphasis></para></entry>
- <entry><para> The path to the root where the source file system will be synchronized (target). This is a mandatory option if the status log created during a previous synchronization operation (--statuslog) is not specified. This option can be repeated if multiple synchronization targets are desired.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--mdt=</emphasis><emphasis><mdt></emphasis></para></entry>
- <entry><para> The metadata device to be synchronized. A changelog user must be registered for this device. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--user=</emphasis><emphasis><user id></emphasis></para></entry>
- <entry><para> The changelog user ID for the specified MDT. To use lustre_rsync, the changelog user must be registered. For details, see the changelog_register parameter in the lctl man page. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--statuslog=</emphasis><emphasis><log></emphasis></para></entry>
- <entry><para> A log file to which synchronization status is saved. When lustre_rsync starts, the state of a previous replication is read from here. If the status log from a previous synchronization operation is specified, otherwise mandatory options like --source, --target and --mdt options may be skipped. By specifying options like --source, --target and/or --mdt in addition to the --statuslog option, parameters in the status log can be overriden. Command line options take precedence over options in the status log.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--xattr</emphasis><emphasis><yes|no></emphasis></para></entry>
- <entry><para> Specifies whether extended attributes (xattrs) are synchronized or not. The default is to synchronize extended attributes.</para><para>NOTE: Disabling xattrs causes Lustre striping information not to be synchronized.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--verbose</emphasis></para></entry>
- <entry><para> Produces a verbose output.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--dry-run</emphasis></para></entry>
- <entry><para> Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--abort-on-err</emphasis></para></entry>
- <entry><para> Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Examples</title>
- <para>Register a changelog user for an MDT (e.g., MDT lustre-MDT0000).</para>
- <screen>$ ssh
+ --target|-t <tgt> --mdt|-m <mdt></screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The lustre_rsync utility is designed to synchronize (replicate) a Lustre file system (source) to another file system (target). The target can be a Lustre file system or any other type, and is a normal, usable file system. The synchronization operation is efficient and does not require directory walking, as lustre_rsync uses Lustre MDT changelogs to identify changes in the Lustre file system.</para>
+ <para>Before using lustre_rsync:</para>
+ <itemizedlist>
+ <listitem>
+ <para>A changelog user must be registered (see lctl (8) changelog_register)</para>
+ </listitem>
+ </itemizedlist>
+ <para>- AND -</para>
+ <itemizedlist>
+ <listitem>
+ <para>Verify that the Lustre file system (source) and the replica file system (target) are identical before the changelog user is registered. If the file systems are discrepant, use a utility, e.g. regular rsync (not lustre_rsync) to make them identical.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section remap="h5">
+ <title>Options</title>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--source=</emphasis><emphasis><src></emphasis></para>
+ </entry>
+ <entry>
+ <para> The path to the root of the Lustre file system (source) which will be synchronized. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--target=</emphasis><emphasis><tgt></emphasis></para>
+ </entry>
+ <entry>
+ <para> The path to the root where the source file system will be synchronized (target). This is a mandatory option if the status log created during a previous synchronization operation (--statuslog) is not specified. This option can be repeated if multiple synchronization targets are desired.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--mdt=</emphasis><emphasis><mdt></emphasis></para>
+ </entry>
+ <entry>
+ <para> The metadata device to be synchronized. A changelog user must be registered for this device. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--user=</emphasis><emphasis><user id></emphasis></para>
+ </entry>
+ <entry>
+ <para> The changelog user ID for the specified MDT. To use lustre_rsync, the changelog user must be registered. For details, see the changelog_register parameter in the lctl man page. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--statuslog=</emphasis><emphasis><log></emphasis></para>
+ </entry>
+ <entry>
+ <para> A log file to which synchronization status is saved. When lustre_rsync starts, the state of a previous replication is read from here. If the status log from a previous synchronization operation is specified, otherwise mandatory options like --source, --target and --mdt options may be skipped. By specifying options like --source, --target and/or --mdt in addition to the --statuslog option, parameters in the status log can be overriden. Command line options take precedence over options in the status log.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--xattr</emphasis><emphasis><yes|no></emphasis></para>
+ </entry>
+ <entry>
+ <para> Specifies whether extended attributes (xattrs) are synchronized or not. The default is to synchronize extended attributes.</para>
+ <para>NOTE: Disabling xattrs causes Lustre striping information not to be synchronized.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--verbose</emphasis></para>
+ </entry>
+ <entry>
+ <para> Produces a verbose output.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--dry-run</emphasis></para>
+ </entry>
+ <entry>
+ <para> Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--abort-on-err</emphasis></para>
+ </entry>
+ <entry>
+ <para> Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Examples</title>
+ <para>Register a changelog user for an MDT (e.g., MDT lustre-MDT0000).</para>
+ <screen>$ ssh
$ MDS lctl changelog_register \
--device lustre-MDT0000 -n
-cl1
-</screen>
- <para> </para>
- <para>Synchronize/replicate a Lustre file system (/mnt/lustre) to a target file system (/mnt/target).</para>
- <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target \
+cl1</screen>
+ <para>Synchronize/replicate a Lustre file system (/mnt/lustre) to a target file system (/mnt/target).</para>
+ <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target \
--mdt=lustre-MDT0000 --user=cl1 \
--statuslog replicate.log --verbose
Lustre filesystem: lustre
lustre_rsync took 1 seconds
Changelog records consumed: 22
</screen>
- <para> </para>
- <para>After the file system undergoes changes, synchronize the changes with the target file system. Only the statuslog name needs to be specified, as it has all the parameters passed earlier.</para>
- <screen>$ lustre_rsync --statuslog replicate.log --verbose
+ <para>After the file system undergoes changes, synchronize the changes with the target file system. Only the statuslog name needs to be specified, as it has all the parameters passed earlier.</para>
+ <screen>$ lustre_rsync --statuslog replicate.log --verbose
Replicating Lustre filesystem: lustre
MDT device: lustre-MDT0000
Source: /mnt/lustre
Starting changelog record: 22
Errors: 0
lustre_rsync took 2 seconds
-Changelog records consumed: 42
-</screen>
- <para> </para>
- <para>Synchronize a Lustre file system (/mnt/lustre) to two target file systems (/mnt/target1 and /mnt/target2).</para>
- <screen>$ lustre_rsync --source=/mnt/lustre \
+Changelog records consumed: 42</screen>
+ <para>Synchronize a Lustre file system (/mnt/lustre) to two target file systems (/mnt/target1 and /mnt/target2).</para>
+ <screen>$ lustre_rsync --source=/mnt/lustre \
--target=/mnt/target1 --target=/mnt/target2 \
--mdt=lustre-MDT0000 --user=cl1
- --statuslog replicate.log
-</screen>
- </section>
- <section remap="h5">
- <title>See Also</title>
- <para><xref linkend="dbdoclet.50438206_94597"/></para>
- </section>
+ --statuslog replicate.log</screen>
</section>
- <section xml:id="dbdoclet.50438219_75432">
- <title>36.14 mkfs.<anchor xml:id="dbdoclet.50438219_marker-1316695" xreflabel=""/>lustre</title>
- <para>The mkfs.lustre utility formats a disk for a Lustre service.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>mkfs.lustre <target_type> [options] device
-</screen>
- <para>where <target_type> is one of the following:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">--ost</emphasis></para></entry>
- <entry><para> Object Storage Target (OST)</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--mdt</emphasis></para></entry>
- <entry><para> Metadata Storage Target (MDT)</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--network=</emphasis><emphasis>net,...</emphasis></para></entry>
- <entry><para> Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--mgs</emphasis></para></entry>
- <entry><para> Configuration Management Service (MGS), one per site. This service can be combined with one <emphasis role="bold">--mdt</emphasis> service by specifying both types.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>mkfs.lustre is used to format a disk device for use as part of a Lustre file system. After formatting, a disk can be mounted to start the Lustre service defined by this command.</para>
- <para>When the file system is created, parameters can simply be added as a --param option to the mkfs.lustre command. See <xref linkend="dbdoclet.50438194_17237"/>.</para>
- <informaltable frame="all">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
- <thead>
- <row>
- <entry nameend="c2" namest="c1"><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--backfstype</emphasis>=<emphasis>fstype</emphasis></para></entry>
- <entry><para> Forces a particular format for the backing file system (such as ext3, ldiskfs).</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--comment</emphasis>=<emphasis>comment</emphasis></para></entry>
- <entry><para> Sets a user comment about this disk, ignored by Lustre.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--device-size</emphasis>=<emphasis>KB</emphasis></para></entry>
- <entry><para> Sets the device size for loop devices.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--dryrun</emphasis></para></entry>
- <entry><para> Only prints what would be done; it does not affect the disk.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--failnode</emphasis>=<emphasis>nid</emphasis>,...</para></entry>
- <entry><para> Sets the NID(s) of a failover partner. This option can be repeated as needed.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--fsname</emphasis>=<emphasis>filesystem_name</emphasis></para></entry>
- <entry><para> The Lustre file system of which this service/node will be a part. The default file system name is 'lustreâ€.</para><para> </para><note><para>The file system name is limited to 8 characters.</para></note></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--index</emphasis>=<emphasis>index</emphasis></para></entry>
- <entry><para> Forces a particular OST or MDT index.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--mkfsoptions</emphasis>=<emphasis>opts</emphasis></para></entry>
- <entry><para> Formats options for the backing file system. For example, ext3 options could be set here.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--mountfsoptions</emphasis>=<emphasis>opts</emphasis></para></entry>
- <entry><para> Sets the mount options used when the backing file system is mounted.</para><para><emphasis role="bold">CAUTION</emphasis>: Unlike earlier versions of mkfs.lustre, this version completely replaces the default mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para><para>The defaults for ldiskfs are:</para><para>OST: <emphasis>errors=remount-ro,mballoc,extents</emphasis>;</para><para>MGS/MDT: <emphasis>errors=remount-ro,iopen_nopriv,user_xattr</emphasis></para><para>Do not alter the default mount options unless you know what you are doing.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--network</emphasis>=<emphasis>net</emphasis>,...</para><para> </para></entry>
- <entry><para> Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--mgsnode</emphasis>=<emphasis>nid</emphasis>,...</para></entry>
- <entry><para> Sets the NIDs of the MGS node, required for all targets other than the MGS.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--param</emphasis><emphasis>key</emphasis>=<emphasis>value</emphasis></para></entry>
- <entry><para> Sets the permanent parameter <emphasis>key</emphasis> to value <emphasis>value</emphasis>. This option can be repeated as necessary. Typical options might include:</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> --<emphasis>param sys.timeout=40</emphasis></para></entry>
- <entry><para> System obd timeout.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> --<emphasis>param lov.stripesize=2M</emphasis></para></entry>
- <entry><para> Default stripe size.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> --<emphasis>param lov.stripecount=2</emphasis></para></entry>
- <entry><para> Default stripe count.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> --<emphasis>param failover.mode=failout</emphasis></para></entry>
- <entry><para> Returns errors instead of waiting for recovery.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--quiet</emphasis></para></entry>
- <entry><para> Prints less information.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--reformat</emphasis></para></entry>
- <entry><para> Reformats an existing Lustre disk.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--stripe-count-hint=stripes</emphasis></para></entry>
- <entry><para> Used to optimize the MDT's inode size.</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--verbose</emphasis></para></entry>
- <entry><para> Prints more information.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Examples</title>
- <para>Creates a combined MGS and MDT for file system <emphasis role="bold">testfs</emphasis> on, e.g., node <emphasis role="bold">cfs21</emphasis>:</para>
- <screen>mkfs.lustre --fsname=testfs --mdt --mgs /dev/sda1
-</screen>
- <para>Creates an OST for file system <emphasis role="bold">testfs</emphasis> on any node (using the above MGS):</para>
- <screen>mkfs.lustre --fsname=testfs --ost --mgsnode=cfs21@tcp0 /dev/sdb
-</screen>
- <para>Creates a standalone MGS on, e.g., node <emphasis role="bold">cfs22</emphasis>:</para>
- <screen>mkfs.lustre --mgs /dev/sda1
-</screen>
- <para>Creates an MDT for file system <emphasis role="bold">myfs1</emphasis> on any node (using the above MGS):</para>
- <screen>mkfs.lustre --fsname=myfs1 --mdt --mgsnode=cfs22@tcp0 /dev/sda2
-</screen>
- </section>
- <section remap="h5">
- <title>See Also</title>
- <itemizedlist>
- <listitem>
- <para><xref linkend="dbdoclet.50438219_75432"/>mkfs.lustre, </para>
- </listitem> <listitem>
- <para><xref linkend="dbdoclet.50438219_12635"/>mount.lustre, </para>
- </listitem> <listitem>
- <para><xref linkend="dbdoclet.50438206_94597"/>lfs</para>
- </listitem>
- </itemizedlist>
- </section>
+ <section remap="h5">
+ <title>See Also</title>
+ <para><xref linkend="dbdoclet.50438206_94597"/></para>
</section>
- <section xml:id="dbdoclet.50438219_12635">
- <title>36.15 mount.<anchor xml:id="dbdoclet.50438219_marker-1310637" xreflabel=""/>lustre</title>
- <para>The mount.lustre utility starts a Lustre client or target service.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>mount -t lustre [-o options] directory
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The mount.lustre utility starts a Lustre client or target service. This program should not be called directly; rather, it is a helper program invoked through mount(8), as shown above. Use the umount command to stop Lustre clients and targets.</para>
- <para>There are two forms for the device option, depending on whether a client or a target service is started:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis><mgsspec>:/<fsname></emphasis></para><para> </para></entry>
- <entry><para> Mounts the Lustre file system named <emphasis>fsname</emphasis> on the client by contacting the Management Service at <emphasis>mgsspec</emphasis> on the pathname given by <emphasis>directory</emphasis>. The format for <emphasis>mgsspec</emphasis> is defined below. A mounted client file system appears in fstab(5) and is usable, like any local file system, and provides a full POSIX-compliant interface.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis><disk_device></emphasis></para></entry>
- <entry><para> Starts the target service defined by the mkfs.lustre command on the physical disk <emphasis>disk_device</emphasis>. A mounted target service file system is only useful for df(1) operations and appears in fstab(5) to show the device is in use.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Options</title>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold"><mgsspec>:</emphasis>=<emphasis><mgsnode>[:<mgsnode>]</emphasis></para><para> </para></entry>
- <entry><para> The MGS specification may be a colon-separated list of nodes.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold"><mgsnode>:</emphasis>=<emphasis><mgsnid>[,<mgsnid>]</emphasis></para></entry>
- <entry><para> Each node may be specified by a comma-separated list of NIDs.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para>In addition to the standard mount options, Lustre understands the following client-specific options:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">flock</emphasis></para></entry>
- <entry><para> Enables full flock support, coherent across all client nodes.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">localflock</emphasis></para></entry>
- <entry><para> Enables local flock support, using only client-local flock (faster, for applications that require flock, but do not run on multiple nodes).</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">noflock</emphasis></para></entry>
- <entry><para> Disables flock support entirely. Applications calling flock get an error. It is up to the administrator to choose either <emphasis role="bold">localflock</emphasis> (fastest, low impact, not coherent between nodes) or <emphasis role="bold">flock</emphasis> (slower, performance impact for use, coherent between nodes).</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">user_xattr</emphasis></para></entry>
- <entry><para> Enables get/set of extended attributes by regular users. See the attr(5) manual page.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">nouser_xattr</emphasis></para></entry>
- <entry><para> Disables use of extended attributes by regular users. Root and system processes can still use extended attributes.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">acl</emphasis></para></entry>
- <entry><para> Enables POSIX Access Control List support. See the acl(5) manual page.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">noacl</emphasis></para></entry>
- <entry><para> Disables Access Control List support.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para>In addition to the standard mount options and backing disk type (e.g. ext3) options, Lustre understands the following server-specific options:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">nosvc</emphasis></para></entry>
- <entry><para> Starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">nomsgs</emphasis></para></entry>
- <entry><para> Starts only the MDT (with a co-located MGS), without starting the MGS.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">exclude=<ostlist></emphasis></para></entry>
- <entry><para> Starts a client or MDT with a colon-separated list of known inactive OSTs.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">nosvc</emphasis></para></entry>
- <entry><para> Only starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">nomsgs</emphasis></para></entry>
- <entry><para> Starts a MDT with a co-located MGS, without starting the MGS.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">exclude</emphasis>=<emphasis>ostlist</emphasis></para></entry>
- <entry><para> Starts a client or MDT with a (colon-separated) list of known inactive OSTs.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">abort_recov</emphasis></para></entry>
- <entry><para> Aborts client recovery and starts the target service immediately.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">md_stripe_cache_size</emphasis></para></entry>
- <entry><para> Sets the stripe cache size for server-side disk with a striped RAID configuration.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">recovery_time_soft</emphasis>=<emphasis>timeout</emphasis></para></entry>
- <entry><para> Allows <emphasis>timeout</emphasis> seconds for clients to reconnect for recovery after a server crash. This timeout is incrementally extended if it is about to expire and the server is still handling new connections from recoverable clients. The default soft recovery timeout is 300 seconds (5 minutes).</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">recovery_time_hard</emphasis>=<emphasis>timeout</emphasis></para></entry>
- <entry><para> The server is allowed to incrementally extend its timeout up to a hard maximum of <emphasis>timeout</emphasis> seconds. The default hard recovery timeout is set to 900 seconds (15 minutes).</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Examples</title>
- <para>Starts a client for the Lustre file system testfs at mount point /mnt/myfilesystem. The Management Service is running on a node reachable from this client via the cfs21@tcp0 NID.</para>
- <screen>mount -t lustre cfs21@tcp0:/testfs /mnt/myfilesystem
-</screen>
- <para>Starts the Lustre metadata target service from /dev/sda1 on mount point /mnt/test/mdt.</para>
- <screen>mount -t lustre /dev/sda1 /mnt/test/mdt
-</screen>
- <para>Starts the testfs-MDT0000 service (using the disk label), but aborts the recovery process.</para>
- <screen>mount -t lustre -L testfs-MDT0000 -o abort_recov /mnt/test/mdt
-</screen>
- </section>
- <section remap="h5">
- <title>See Also</title>
-
- <itemizedlist><listitem>
- <para> <xref linkend="dbdoclet.50438219_75432"/></para>
- </listitem> <listitem>
- <para> <xref linkend="dbdoclet.50438219_39574"/></para>
- </listitem> <listitem>
- <para> <xref linkend="dbdoclet.50438219_38274"/></para>
- </listitem> <listitem>
- <para> <xref linkend="dbdoclet.50438206_94597"/></para>
- </listitem>
- </itemizedlist>
- </section>
+ </section>
+ <section xml:id="dbdoclet.50438219_75432">
+ <title><indexterm><primary>mkfs.lustre</primary></indexterm>
+mkfs.lustre</title>
+ <para>The mkfs.lustre utility formats a disk for a Lustre service.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>mkfs.lustre <target_type> [options] device</screen>
+ <para>where <target_type> is one of the following:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--ost</emphasis></para>
+ </entry>
+ <entry>
+ <para> Object Storage Target (OST)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--mdt</emphasis></para>
+ </entry>
+ <entry>
+ <para> Metadata Storage Target (MDT)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--network=</emphasis><emphasis>net,...</emphasis></para>
+ </entry>
+ <entry>
+ <para> Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--mgs</emphasis></para>
+ </entry>
+ <entry>
+ <para> Configuration Management Service (MGS), one per site. This service can be combined with one <emphasis role="bold">--mdt</emphasis> service by specifying both types.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
</section>
- <section xml:id="dbdoclet.50438219_82679">
- <title>36.16 plot-<anchor xml:id="dbdoclet.50438219_marker-1317989" xreflabel=""/>llstat</title>
- <para>The plot-llstat utility plots Lustre statistics.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>plot-llstat results_filename [parameter_index]
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The plot-llstat utility generates a CSV file and instruction files for gnuplot from the output of llstat. Since llstat is generic in nature, plot-llstat is also a generic script. The value of parameter_index can be 1 for count per interval, 2 for count per second (default setting) or 3 for total count.</para>
- <para>The plot-llstat utility creates a .dat (CSV) file using the number of operations specified by the user. The number of operations equals the number of columns in the CSV file. The values in those columns are equal to the corresponding value of parameter_index in the output file.</para>
- <para>The plot-llstat utility also creates a .scr file that contains instructions for gnuplot to plot the graph. After generating the .dat and .scr files, the plot-llstat tool invokes gnuplot to display the graph.</para>
- </section>
- <section remap="h5">
- <title>Options</title>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">results_filename</emphasis></para></entry>
- <entry><para> Output generated by plot-llstat</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">parameter_index</emphasis></para><para> </para></entry>
- <entry><para> Value of parameter_index can be:</para><para> 1 - count per interval</para><para> 2 - count per second (default setting)</para><para> 3 - total count</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Example</title>
- <screen>llstat -i2 -g -c lustre-OST0000 > log
-plot-llstat log 3
-</screen>
- </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>mkfs.lustre is used to format a disk device for use as part of a Lustre file system. After formatting, a disk can be mounted to start the Lustre service defined by this command.</para>
+ <para>When the file system is created, parameters can simply be added as a --param option to the mkfs.lustre command. See <xref linkend="dbdoclet.50438194_17237"/>.</para>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="33*"/>
+ <colspec colname="c2" colwidth="33*"/>
+ <colspec colname="c3" colwidth="33*"/>
+ <thead>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--backfstype</emphasis>=<emphasis>fstype</emphasis></para>
+ </entry>
+ <entry>
+ <para> Forces a particular format for the backing file system (such as ext3, ldiskfs).</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--comment</emphasis>=<emphasis>comment</emphasis></para>
+ </entry>
+ <entry>
+ <para> Sets a user comment about this disk, ignored by Lustre.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--device-size</emphasis>=<emphasis>KB</emphasis></para>
+ </entry>
+ <entry>
+ <para> Sets the device size for loop devices.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--dryrun</emphasis></para>
+ </entry>
+ <entry>
+ <para> Only prints what would be done; it does not affect the disk.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--failnode</emphasis>=<emphasis>nid</emphasis>,...</para>
+ </entry>
+ <entry>
+ <para> Sets the NID(s) of a failover partner. This option can be repeated as needed.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--fsname</emphasis>=<emphasis>filesystem_name</emphasis></para>
+ </entry>
+ <entry>
+ <para> The Lustre file system of which this service/node will be a part. The default file system name is 'lustreâ€.</para>
+ <para> </para>
+ <note>
+ <para>The file system name is limited to 8 characters.</para>
+ </note>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--index</emphasis>=<emphasis>index</emphasis></para>
+ </entry>
+ <entry>
+ <para> Forces a particular OST or MDT index.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--mkfsoptions</emphasis>=<emphasis>opts</emphasis></para>
+ </entry>
+ <entry>
+ <para> Formats options for the backing file system. For example, ext3 options could be set here.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--mountfsoptions</emphasis>=<emphasis>opts</emphasis></para>
+ </entry>
+ <entry>
+ <para> Sets the mount options used when the backing file system is mounted.</para>
+ <para><emphasis role="bold">CAUTION</emphasis>: Unlike earlier versions of mkfs.lustre, this version completely replaces the default mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para>
+ <para>The defaults for ldiskfs are:</para>
+ <para>OST: <emphasis>errors=remount-ro,mballoc,extents</emphasis>;</para>
+ <para>MGS/MDT: <emphasis>errors=remount-ro,iopen_nopriv,user_xattr</emphasis></para>
+ <para>Do not alter the default mount options unless you know what you are doing.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--network</emphasis>=<emphasis>net</emphasis>,...</para>
+ <para> </para>
+ </entry>
+ <entry>
+ <para> Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--mgsnode</emphasis>=<emphasis>nid</emphasis>,...</para>
+ </entry>
+ <entry>
+ <para> Sets the NIDs of the MGS node, required for all targets other than the MGS.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--param</emphasis><emphasis>key</emphasis>=<emphasis>value</emphasis></para>
+ </entry>
+ <entry>
+ <para> Sets the permanent parameter <emphasis>key</emphasis> to value <emphasis>value</emphasis>. This option can be repeated as necessary. Typical options might include:</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> --<emphasis>param sys.timeout=40</emphasis></para>
+ </entry>
+ <entry>
+ <para> System obd timeout.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> --<emphasis>param lov.stripesize=2M</emphasis></para>
+ </entry>
+ <entry>
+ <para> Default stripe size.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> --<emphasis>param lov.stripecount=2</emphasis></para>
+ </entry>
+ <entry>
+ <para> Default stripe count.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> --<emphasis>param failover.mode=failout</emphasis></para>
+ </entry>
+ <entry>
+ <para> Returns errors instead of waiting for recovery.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--quiet</emphasis></para>
+ </entry>
+ <entry>
+ <para> Prints less information.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--reformat</emphasis></para>
+ </entry>
+ <entry>
+ <para> Reformats an existing Lustre disk.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--stripe-count-hint=stripes</emphasis></para>
+ </entry>
+ <entry>
+ <para> Used to optimize the MDT's inode size.</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis role="bold">--verbose</emphasis></para>
+ </entry>
+ <entry>
+ <para> Prints more information.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
</section>
- <section xml:id="dbdoclet.50438219_51496">
- <title>36.17 router<anchor xml:id="dbdoclet.50438219_marker-1317921" xreflabel=""/>stat</title>
- <para>The routerstat utility prints Lustre router statistics.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>routerstat [interval]
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The routerstat utility watches LNET router statistics. If no <emphasis>interval</emphasis> is specified, then statistics are sampled and printed only one time. Otherwise, statistics are sampled and printed at the specified <emphasis>interval</emphasis> (in seconds).</para>
- </section>
- <section remap="h5">
- <title>Options</title>
- <para>The routerstat output includes the following fields:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">M</emphasis></para></entry>
- <entry><para> msgs_alloc(msgs_max)</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">E</emphasis></para></entry>
- <entry><para> errors</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">S</emphasis></para></entry>
- <entry><para> send_count/send_length</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">R</emphasis></para></entry>
- <entry><para> recv_count/recv_length</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">F</emphasis></para></entry>
- <entry><para> route_count/route_length</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">D</emphasis></para></entry>
- <entry><para> drop_count/drop_length</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Files</title>
- <para>The routerstat utility extracts statistics data from:</para>
- <screen>/proc/sys/lnet/stats
-</screen>
- </section>
+ <section remap="h5">
+ <title>Examples</title>
+ <para>Creates a combined MGS and MDT for file system <emphasis role="bold">testfs</emphasis> on, e.g., node <emphasis role="bold">cfs21</emphasis>:</para>
+ <screen>mkfs.lustre --fsname=testfs --mdt --mgs /dev/sda1</screen>
+ <para>Creates an OST for file system <emphasis role="bold">testfs</emphasis> on any node (using the above MGS):</para>
+ <screen>mkfs.lustre --fsname=testfs --ost --mgsnode=cfs21@tcp0 /dev/sdb</screen>
+ <para>Creates a standalone MGS on, e.g., node <emphasis role="bold">cfs22</emphasis>:</para>
+ <screen>mkfs.lustre --mgs /dev/sda1</screen>
+ <para>Creates an MDT for file system <emphasis role="bold">myfs1</emphasis> on any node (using the above MGS):</para>
+ <screen>mkfs.lustre --fsname=myfs1 --mdt --mgsnode=cfs22@tcp0 /dev/sda2</screen>
</section>
- <section xml:id="dbdoclet.50438219_39574">
- <title>36.18 tunefs.<anchor xml:id="dbdoclet.50438219_marker-1317076" xreflabel=""/>lustre</title>
- <para>The tunefs.lustre utility modifies configuration information on a Lustre target disk.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>tunefs.lustre [options] <device>
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>tunefs.lustre is used to modify configuration information on a Lustre target disk. This includes upgrading old (pre-Lustre 1.6) disks. This does not reformat the disk or erase the target information, but modifying the configuration information can result in an unusable file system.</para>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="5*"/>
- <colspec colname="c2" colwidth="95*"/>
-
-
- <tbody>
- <row>
- <entry><para><inlinemediaobject><imageobject role="html">
- <imagedata fileref="./shared/caution.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis>Changes made here affect a file system only when the target is mounted the next time.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para>With tunefs.lustre, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old tunefs.lustre parameters and just use newly-specified parameters, run:</para>
- <screen>$ tunefs.lustre --erase-params --param=<new parameters> </screen>
- <para>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <obd|fsname>.<obdtype>.<proc_file_name>=<value>. For example:</para>
- <screen>$ tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1
-</screen>
- </section>
- <section remap="h5">
- <title>Options</title>
- <para>The tunefs.lustre options are listed and explained below.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">--comment</emphasis>=<emphasis>comment</emphasis></para></entry>
- <entry><para> Sets a user comment about this disk, ignored by Lustre.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--dryrun</emphasis></para></entry>
- <entry><para> Only prints what would be done; does not affect the disk.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--erase-params</emphasis></para></entry>
- <entry><para> Removes all previous parameter information.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--failnode</emphasis>=<emphasis>nid</emphasis>,...</para></entry>
- <entry><para> Sets the NID(s) of a failover partner. This option can be repeated as needed.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--fsname</emphasis>=<emphasis>filesystem_name</emphasis></para></entry>
- <entry><para> The Lustre file system of which this service will be a part. The default file system name is 'lustreâ€.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--index</emphasis>=index</para></entry>
- <entry><para> Forces a particular OST or MDT index.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--mountfsoptions</emphasis>=opts</para></entry>
- <entry><para> Sets the mount options used when the backing file system is mounted.</para><para><emphasis role="bold">CAUTION</emphasis>: Unlike earlier versions of tunefs.lustre, this version completely replaces the existing mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para><para>The defaults for ldiskfs are:</para><para>OST: <emphasis>errors=remount-ro,mballoc,extents</emphasis>;</para><para>MGS/MDT: <emphasis>errors=remount-ro,iopen_nopriv,user_xattr</emphasis></para><para>Do not alter the default mount options unless you know what you are doing.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--network</emphasis>=net,...</para></entry>
- <entry><para> Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--mgs</emphasis></para></entry>
- <entry><para> Adds a configuration management service to this target.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--msgnode</emphasis>=<emphasis>nid</emphasis>,...</para></entry>
- <entry><para> Sets the NID(s) of the MGS node; required for all targets other than the MGS.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--nomgs</emphasis></para></entry>
- <entry><para> Removes a configuration management service to this target.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--quiet</emphasis></para></entry>
- <entry><para> Prints less information.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--verbose</emphasis></para></entry>
- <entry><para> Prints more information.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">--writeconf</emphasis></para></entry>
- <entry><para> Erases all configuration logs for the file system to which this MDT belongs, and regenerates them. This is dangerous operation. All clients must be unmounted and servers for this file system should be stopped. All targets (OSTs/MDTs) must then be restarted to regenerate the logs. No clients should be started until all targets have restarted.</para><para> </para><para>The correct order of operations is:</para><para>* Unmount all clients on the file system</para><para>* Unmount the MDT and all OSTs on the file system</para><para>* Run <emphasis role="bold">tunefs.lustre --writeconf <device></emphasis> on every server</para><para>* Mount the MDT and OSTs</para><para>* Mount the clients</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Examples</title>
- <para>Change the MGS's NID address. (This should be done on each target disk, since they should all contact the same MGS.)</para>
- <screen>tunefs.lustre --erase-param --mgsnode=<new_nid> --writeconf /dev/sda
+ <section remap="h5">
+ <title>See Also</title>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438219_75432"/>mkfs.lustre, </para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438219_12635"/>mount.lustre, </para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438206_94597"/>lfs</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438219_12635">
+ <title><indexterm><primary>mount.lustre</primary></indexterm>
+mount.lustre</title>
+ <para>The mount.lustre utility starts a Lustre client or target service.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>mount -t lustre [-o options] directory
</screen>
- <para>Add a failover NID location for this target.</para>
- <screen>tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The mount.lustre utility starts a Lustre client or target service. This program should not be called directly; rather, it is a helper program invoked through mount(8), as shown above. Use the umount command to stop Lustre clients and targets.</para>
+ <para>There are two forms for the device option, depending on whether a client or a target service is started:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis><mgsspec>:/<fsname></emphasis></para>
+ <para> </para>
+ </entry>
+ <entry>
+ <para> Mounts the Lustre file system named <emphasis>fsname</emphasis> on the client by contacting the Management Service at <emphasis>mgsspec</emphasis> on the pathname given by <emphasis>directory</emphasis>. The format for <emphasis>mgsspec</emphasis> is defined below. A mounted client file system appears in fstab(5) and is usable, like any local file system, and provides a full POSIX-compliant interface.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis><disk_device></emphasis></para>
+ </entry>
+ <entry>
+ <para> Starts the target service defined by the mkfs.lustre command on the physical disk <emphasis>disk_device</emphasis>. A mounted target service file system is only useful for df(1) operations and appears in fstab(5) to show the device is in use.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Options</title>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold"><mgsspec>:</emphasis>=<emphasis><mgsnode>[:<mgsnode>]</emphasis></para>
+ <para> </para>
+ </entry>
+ <entry>
+ <para> The MGS specification may be a colon-separated list of nodes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold"><mgsnode>:</emphasis>=<emphasis><mgsnid>[,<mgsnid>]</emphasis></para>
+ </entry>
+ <entry>
+ <para> Each node may be specified by a comma-separated list of NIDs.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para>In addition to the standard mount options, Lustre understands the following client-specific options:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">flock</emphasis></para>
+ </entry>
+ <entry>
+ <para> Enables full flock support, coherent across all client nodes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">localflock</emphasis></para>
+ </entry>
+ <entry>
+ <para> Enables local flock support, using only client-local flock (faster, for applications that require flock, but do not run on multiple nodes).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">noflock</emphasis></para>
+ </entry>
+ <entry>
+ <para> Disables flock support entirely. Applications calling flock get an error. It is up to the administrator to choose either <emphasis role="bold">localflock</emphasis> (fastest, low impact, not coherent between nodes) or <emphasis role="bold">flock</emphasis> (slower, performance impact for use, coherent between nodes).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">user_xattr</emphasis></para>
+ </entry>
+ <entry>
+ <para> Enables get/set of extended attributes by regular users. See the attr(5) manual page.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">nouser_xattr</emphasis></para>
+ </entry>
+ <entry>
+ <para> Disables use of extended attributes by regular users. Root and system processes can still use extended attributes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">acl</emphasis></para>
+ </entry>
+ <entry>
+ <para> Enables POSIX Access Control List support. See the acl(5) manual page.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">noacl</emphasis></para>
+ </entry>
+ <entry>
+ <para> Disables Access Control List support.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para>In addition to the standard mount options and backing disk type (e.g. ext3) options, Lustre understands the following server-specific options:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">nosvc</emphasis></para>
+ </entry>
+ <entry>
+ <para> Starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">nomsgs</emphasis></para>
+ </entry>
+ <entry>
+ <para> Starts only the MDT (with a co-located MGS), without starting the MGS.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">exclude=<ostlist></emphasis></para>
+ </entry>
+ <entry>
+ <para> Starts a client or MDT with a colon-separated list of known inactive OSTs.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">nosvc</emphasis></para>
+ </entry>
+ <entry>
+ <para> Only starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">nomsgs</emphasis></para>
+ </entry>
+ <entry>
+ <para> Starts a MDT with a co-located MGS, without starting the MGS.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">exclude</emphasis>=<emphasis>ostlist</emphasis></para>
+ </entry>
+ <entry>
+ <para> Starts a client or MDT with a (colon-separated) list of known inactive OSTs.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">abort_recov</emphasis></para>
+ </entry>
+ <entry>
+ <para> Aborts client recovery and starts the target service immediately.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">md_stripe_cache_size</emphasis></para>
+ </entry>
+ <entry>
+ <para> Sets the stripe cache size for server-side disk with a striped RAID configuration.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">recovery_time_soft</emphasis>=<emphasis>timeout</emphasis></para>
+ </entry>
+ <entry>
+ <para> Allows <emphasis>timeout</emphasis> seconds for clients to reconnect for recovery after a server crash. This timeout is incrementally extended if it is about to expire and the server is still handling new connections from recoverable clients. The default soft recovery timeout is 300 seconds (5 minutes).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">recovery_time_hard</emphasis>=<emphasis>timeout</emphasis></para>
+ </entry>
+ <entry>
+ <para> The server is allowed to incrementally extend its timeout up to a hard maximum of <emphasis>timeout</emphasis> seconds. The default hard recovery timeout is set to 900 seconds (15 minutes).</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Examples</title>
+ <para>Starts a client for the Lustre file system testfs at mount point /mnt/myfilesystem. The Management Service is running on a node reachable from this client via the cfs21@tcp0 NID.</para>
+ <screen>mount -t lustre cfs21@tcp0:/testfs /mnt/myfilesystem</screen>
+ <para>Starts the Lustre metadata target service from /dev/sda1 on mount point /mnt/test/mdt.</para>
+ <screen>mount -t lustre /dev/sda1 /mnt/test/mdt</screen>
+ <para>Starts the testfs-MDT0000 service (using the disk label), but aborts the recovery process.</para>
+ <screen>mount -t lustre -L testfs-MDT0000 -o abort_recov /mnt/test/mdt</screen>
+ </section>
+ <section remap="h5">
+ <title>See Also</title>
+ <itemizedlist>
+ <listitem>
+ <para> <xref linkend="dbdoclet.50438219_75432"/></para>
+ </listitem>
+ <listitem>
+ <para> <xref linkend="dbdoclet.50438219_39574"/></para>
+ </listitem>
+ <listitem>
+ <para> <xref linkend="dbdoclet.50438219_38274"/></para>
+ </listitem>
+ <listitem>
+ <para> <xref linkend="dbdoclet.50438206_94597"/></para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438219_82679">
+ <title><indexterm><primary>plot-llstat</primary></indexterm>
+plot-llstat</title>
+ <para>The plot-llstat utility plots Lustre statistics.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>plot-llstat results_filename [parameter_index]
</screen>
- </section>
- <section remap="h5">
- <title>See Also</title>
- <itemizedlist>
- <listitem>
- <para><xref linkend="dbdoclet.50438219_75432"/></para>
- </listitem> <listitem>
- <para><xref linkend="dbdoclet.50438219_12635"/></para>
- </listitem> <listitem>
- <para><xref linkend="dbdoclet.50438219_38274"/></para>
- </listitem> <listitem>
- <para><xref linkend="dbdoclet.50438206_94597"/></para>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The plot-llstat utility generates a CSV file and instruction files for gnuplot from the output of llstat. Since llstat is generic in nature, plot-llstat is also a generic script. The value of parameter_index can be 1 for count per interval, 2 for count per second (default setting) or 3 for total count.</para>
+ <para>The plot-llstat utility creates a .dat (CSV) file using the number of operations specified by the user. The number of operations equals the number of columns in the CSV file. The values in those columns are equal to the corresponding value of parameter_index in the output file.</para>
+ <para>The plot-llstat utility also creates a .scr file that contains instructions for gnuplot to plot the graph. After generating the .dat and .scr files, the plot-llstat tool invokes gnuplot to display the graph.</para>
+ </section>
+ <section remap="h5">
+ <title>Options</title>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">results_filename</emphasis></para>
+ </entry>
+ <entry>
+ <para> Output generated by plot-llstat</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">parameter_index</emphasis></para>
+ <para> </para>
+ </entry>
+ <entry>
+ <para> Value of parameter_index can be:</para>
+ <para> 1 - count per interval</para>
+ <para> 2 - count per second (default setting)</para>
+ <para> 3 - total count</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Example</title>
+ <screen>llstat -i2 -g -c lustre-OST0000 > log
+plot-llstat log 3</screen>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438219_51496">
+ <title><indexterm><primary>routerstat</primary></indexterm>
+routerstat</title>
+ <para>The routerstat utility prints Lustre router statistics.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>routerstat [interval]</screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The routerstat utility watches LNET router statistics. If no <emphasis>interval</emphasis> is specified, then statistics are sampled and printed only one time. Otherwise, statistics are sampled and printed at the specified <emphasis>interval</emphasis> (in seconds).</para>
+ </section>
+ <section remap="h5">
+ <title>Options</title>
+ <para>The routerstat output includes the following fields:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">M</emphasis></para>
+ </entry>
+ <entry>
+ <para> msgs_alloc(msgs_max)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">E</emphasis></para>
+ </entry>
+ <entry>
+ <para> errors</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">S</emphasis></para>
+ </entry>
+ <entry>
+ <para> send_count/send_length</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">R</emphasis></para>
+ </entry>
+ <entry>
+ <para> recv_count/recv_length</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">F</emphasis></para>
+ </entry>
+ <entry>
+ <para> route_count/route_length</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">D</emphasis></para>
+ </entry>
+ <entry>
+ <para> drop_count/drop_length</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Files</title>
+ <para>The routerstat utility extracts statistics data from:</para>
+ <screen>/proc/sys/lnet/stats</screen>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438219_39574">
+ <title><indexterm><primary>tunefs.lustre</primary></indexterm>
+tunefs.lustre</title>
+ <para>The tunefs.lustre utility modifies configuration information on a Lustre target disk.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>tunefs.lustre [options] <device></screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>tunefs.lustre is used to modify configuration information on a Lustre target disk. This includes upgrading old (pre-Lustre 1.6) disks. This does not reformat the disk or erase the target information, but modifying the configuration information can result in an unusable file system.</para>
+ <caution>
+ <para>Changes made here affect a file system only when the target is mounted the next time.</para>
+ </caution>
+ <para>With tunefs.lustre, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old tunefs.lustre parameters and just use newly-specified parameters, run:</para>
+ <screen>$ tunefs.lustre --erase-params --param=<new parameters> </screen>
+ <para>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <obd|fsname>.<obdtype>.<proc_file_name>=<value>. For example:</para>
+ <screen>$ tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1</screen>
+ </section>
+ <section remap="h5">
+ <title>Options</title>
+ <para>The tunefs.lustre options are listed and explained below.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--comment</emphasis>=<emphasis>comment</emphasis></para>
+ </entry>
+ <entry>
+ <para> Sets a user comment about this disk, ignored by Lustre.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--dryrun</emphasis></para>
+ </entry>
+ <entry>
+ <para> Only prints what would be done; does not affect the disk.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--erase-params</emphasis></para>
+ </entry>
+ <entry>
+ <para> Removes all previous parameter information.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--failnode</emphasis>=<emphasis>nid</emphasis>,...</para>
+ </entry>
+ <entry>
+ <para> Sets the NID(s) of a failover partner. This option can be repeated as needed.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--fsname</emphasis>=<emphasis>filesystem_name</emphasis></para>
+ </entry>
+ <entry>
+ <para> The Lustre file system of which this service will be a part. The default file system name is 'lustreâ€.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--index</emphasis>=index</para>
+ </entry>
+ <entry>
+ <para> Forces a particular OST or MDT index.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--mountfsoptions</emphasis>=opts</para>
+ </entry>
+ <entry>
+ <para> Sets the mount options used when the backing file system is mounted.</para>
+ <para><emphasis role="bold">CAUTION</emphasis>: Unlike earlier versions of tunefs.lustre, this version completely replaces the existing mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para>
+ <para>The defaults for ldiskfs are:</para>
+ <para>OST: <emphasis>errors=remount-ro,mballoc,extents</emphasis>;</para>
+ <para>MGS/MDT: <emphasis>errors=remount-ro,iopen_nopriv,user_xattr</emphasis></para>
+ <para>Do not alter the default mount options unless you know what you are doing.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--network</emphasis>=net,...</para>
+ </entry>
+ <entry>
+ <para> Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--mgs</emphasis></para>
+ </entry>
+ <entry>
+ <para> Adds a configuration management service to this target.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--msgnode</emphasis>=<emphasis>nid</emphasis>,...</para>
+ </entry>
+ <entry>
+ <para> Sets the NID(s) of the MGS node; required for all targets other than the MGS.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--nomgs</emphasis></para>
+ </entry>
+ <entry>
+ <para> Removes a configuration management service to this target.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--quiet</emphasis></para>
+ </entry>
+ <entry>
+ <para> Prints less information.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--verbose</emphasis></para>
+ </entry>
+ <entry>
+ <para> Prints more information.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">--writeconf</emphasis></para>
+ </entry>
+ <entry>
+ <para> Erases all configuration logs for the file system to which this MDT belongs, and regenerates them. This is dangerous operation. All clients must be unmounted and servers for this file system should be stopped. All targets (OSTs/MDTs) must then be restarted to regenerate the logs. No clients should be started until all targets have restarted.</para>
+ <para> </para>
+ <para>The correct order of operations is:</para>
+ <para>* Unmount all clients on the file system</para>
+ <para>* Unmount the MDT and all OSTs on the file system</para>
+ <para>* Run <emphasis role="bold">tunefs.lustre --writeconf <device></emphasis> on every server</para>
+ <para>* Mount the MDT and OSTs</para>
+ <para>* Mount the clients</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Examples</title>
+ <para>Change the MGS's NID address. (This should be done on each target disk, since they should all contact the same MGS.)</para>
+ <screen>tunefs.lustre --erase-param --mgsnode=<new_nid> --writeconf /dev/sda</screen>
+ <para>Add a failover NID location for this target.</para>
+ <screen>tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda </screen>
+ </section>
+ <section remap="h5">
+ <title>See Also</title>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438219_75432"/></para>
</listitem>
- </itemizedlist>
- </section>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438219_12635"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438219_38274"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438206_94597"/></para>
+ </listitem>
+ </itemizedlist>
</section>
- <section xml:id="dbdoclet.50438219_99928">
- <title>36.19 Additional System <anchor xml:id="dbdoclet.50438219_marker-1313048" xreflabel=""/>Configuration Utilities</title>
- <para>This section describes additional system configuration utilities for Lustre.</para>
- <section remap="h3">
- <title>36.19.1 Application <anchor xml:id="dbdoclet.50438219_marker-1313051" xreflabel=""/>Profiling Utilities</title>
- <para>The following utilities are located in /usr/bin.</para>
- <para><emphasis role="bold">lustre_req_history.sh</emphasis></para>
- <para>The lustre_req_history.sh utility (run from a client), assembles as much Lustre RPC request history as possible from the local node and from the servers that were contacted, providing a better picture of the coordinated network activity.</para>
- <para><emphasis role="bold">llstat.sh</emphasis></para>
- <para>The llstat.sh utility (improved in Lustre 1.6), handles a wider range of /proc files, and has command line switches to produce more graphable output.</para>
- <para><emphasis role="bold">plot-llstat.sh</emphasis></para>
- <para>The plot-llstat.sh utility plots the output from llstat.sh using gnuplot.</para>
- </section>
- <section remap="h3">
- <title>36.19.2 More /proc Statistics for <anchor xml:id="dbdoclet.50438219_marker-1313052" xreflabel=""/>Application Profiling</title>
- <para>The following utilities provide additional statistics.</para>
- <para><emphasis role="bold">vfs_ops_stats</emphasis></para>
- <para>The client vfs_ops_stats utility tracks Linux VFS operation calls into Lustre for a single PID, PPID, GID or everything.</para>
- <screen>/proc/fs/lustre/llite/*/vfs_ops_stats
+ </section>
+ <section xml:id="dbdoclet.50438219_99928">
+ <title><indexterm><primary>utilities</primary><secondary>system config</secondary></indexterm>
+Additional System Configuration Utilities</title>
+ <para>This section describes additional system configuration utilities for Lustre.</para>
+ <section remap="h3">
+ <title><indexterm><primary>utilities</primary><secondary>application profiling</secondary></indexterm>
+Application Profiling Utilities</title>
+ <para>The following utilities are located in /usr/bin.</para>
+ <para><emphasis role="bold">lustre_req_history.sh</emphasis></para>
+ <para>The lustre_req_history.sh utility (run from a client), assembles as much Lustre RPC request history as possible from the local node and from the servers that were contacted, providing a better picture of the coordinated network activity.</para>
+ <para><emphasis role="bold">llstat.sh</emphasis></para>
+ <para>The llstat.sh utility (improved in Lustre 1.6), handles a wider range of /proc files, and has command line switches to produce more graphable output.</para>
+ <para><emphasis role="bold">plot-llstat.sh</emphasis></para>
+ <para>The plot-llstat.sh utility plots the output from llstat.sh using gnuplot.</para>
+ </section>
+ <section remap="h3">
+ <title>More /proc Statistics for Application Profiling</title>
+ <para>The following utilities provide additional statistics.</para>
+ <para><emphasis role="bold">vfs_ops_stats</emphasis></para>
+ <para>The client vfs_ops_stats utility tracks Linux VFS operation calls into Lustre for a single PID, PPID, GID or everything.</para>
+ <screen>/proc/fs/lustre/llite/*/vfs_ops_stats
/proc/fs/lustre/llite/*/vfs_track_[pid|ppid|gid]
</screen>
-
<para><emphasis role="bold">extents_stats</emphasis></para>
-
<para>The client extents_stats utility shows the size distribution of I/O calls from the client (cumulative and by process).</para>
- <screen>/proc/fs/lustre/llite/*/extents_stats, extents_stats_per_process
+ <para><emphasis role="bold">extents_stats</emphasis></para>
+ <para>The client extents_stats utility shows the size distribution of I/O calls from the client (cumulative and by process).</para>
+ <screen>/proc/fs/lustre/llite/*/extents_stats, extents_stats_per_process
</screen>
-
<para><emphasis role="bold">offset_stats</emphasis></para>
-
<para>The client offset_stats utility shows the read/write seek activity of a client by offsets and ranges.</para>
- <screen>/proc/fs/lustre/llite/*/offset_stats
+ <para><emphasis role="bold">offset_stats</emphasis></para>
+ <para>The client offset_stats utility shows the read/write seek activity of a client by offsets and ranges.</para>
+ <screen>/proc/fs/lustre/llite/*/offset_stats
</screen>
-
<para>Lustre 1.6 included per-client and improved MDT statistics:</para>
- <itemizedlist><listitem>
- <para> Per-client statistics tracked on the servers</para>
- </listitem>
-
-</itemizedlist>
-
<para>Each MDT and OST now tracks LDLM and operations statistics for every connected client, for comparisons and simpler collection of distributed job statistics.</para>
- <screen>/proc/fs/lustre/mds|obdfilter/*/exports/
+ <para>Lustre 1.6 included per-client and improved MDT statistics:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Per-client statistics tracked on the servers</para>
+ </listitem>
+ </itemizedlist>
+ <para>Each MDT and OST now tracks LDLM and operations statistics for every connected client, for comparisons and simpler collection of distributed job statistics.</para>
+ <screen>/proc/fs/lustre/mds|obdfilter/*/exports/
</screen>
- <itemizedlist><listitem>
- <para> Improved MDT statistics</para>
- </listitem>
-
-</itemizedlist>
-
<para>More detailed MDT operations statistics are collected for better profiling.</para>
- <screen>/proc/fs/lustre/mds/*/stats
+ <itemizedlist>
+ <listitem>
+ <para> Improved MDT statistics</para>
+ </listitem>
+ </itemizedlist>
+ <para>More detailed MDT operations statistics are collected for better profiling.</para>
+ <screen>/proc/fs/lustre/mds/*/stats
</screen>
+ </section>
+ <section remap="h3">
+ <title><indexterm><primary>utilities</primary><secondary>debugging</secondary></indexterm><indexterm><primary>debug</primary><secondary>utilities</secondary></indexterm>
+
+Testing / Debugging Utilities</title>
+ <para>Lustre offers the following test and debugging utilities.</para>
+ <section remap="h5">
+ <title><indexterm><primary>loadgen</primary></indexterm>
+loadgen</title>
+ <para>The Load Generator (loadgen) is a test program designed to simulate large numbers of Lustre clients connecting and writing to an OST. The loadgen utility is located at lustre/utils/loadgen (in a build directory) or at /usr/sbin/loadgen (from an RPM).</para>
+ <para>Loadgen offers the ability to run this test:</para>
+ <orderedlist>
+ <listitem>
+ <para>Start an arbitrary number of (echo) clients.</para>
+ </listitem>
+ <listitem>
+ <para>Start and connect to an echo server, instead of a real OST.</para>
+ </listitem>
+ <listitem>
+ <para>Create/bulk_write/delete objects on any number of echo clients simultaneously.</para>
+ </listitem>
+ </orderedlist>
+ <para>Currently, the maximum number of clients is limited by MAX_OBD_DEVICES and the amount of memory available.</para>
</section>
- <section remap="h3">
- <title>36.19.3 Testing / Debugging <anchor xml:id="dbdoclet.50438219_marker-1313053" xreflabel=""/>Utilities</title>
- <para>Lustre offers the following test and debugging utilities.</para>
- <section remap="h5">
- <title>loadgen</title>
- <para>The Load Generator (loadgen) is a test program designed to simulate large numbers of Lustre clients connecting and writing to an OST. The loadgen utility is located at lustre/utils/loadgen (in a build directory) or at /usr/sbin/loadgen (from an RPM).</para>
- <para>Loadgen offers the ability to run this test:</para>
- <para> 1. Start an arbitrary number of (echo) clients.</para>
- <para> 2. Start and connect to an echo server, instead of a real OST.</para>
- <para> 3. Create/bulk_write/delete objects on any number of echo clients simultaneously.</para>
- <para>Currently, the maximum number of clients is limited by MAX_OBD_DEVICES and the amount of memory available.</para>
- </section>
- <section remap="h5">
- <title>Usage</title>
- <para>The loadgen utility can be run locally on the OST server machine or remotely from any LNET host. The device command can take an optional NID as a parameter; if unspecified, the first local NID found is used.</para>
- <para>The obdecho module must be loaded by hand before running loadgen.</para>
- <screen># cd lustre/utils/
+ <section remap="h5">
+ <title>Usage</title>
+ <para>The loadgen utility can be run locally on the OST server machine or remotely from any LNET host. The device command can take an optional NID as a parameter; if unspecified, the first local NID found is used.</para>
+ <para>The obdecho module must be loaded by hand before running loadgen.</para>
+ <screen># cd lustre/utils/
# insmod ../obdecho/obdecho.ko
# ./loadgen
loadgen> h
wrote 25MB in 1.419s (17.623 MB/s)
loadgen>
</screen>
-
<para>The loadgen utility prints periodic status messages; message output can be controlled with the verbose command.</para>
-
<para>To insure that a file can be written to (a requirement of write cache), OSTs reserve ("grants"), chunks of space for each newly-created file. A grant may cause an OST to report that it is out of space, even though there is plenty of space on the disk, because the space is "reserved" by other files. The loadgen utility estimates the number of simultaneous open files as the disk size divided by the grant size and reports that number when the write tests are first started.</para>
-
<para><emphasis role="bold">Echo Server</emphasis></para>
-
<para>The loadgen utility can start an echo server. On another node, loadgen can specify the echo server as the device, thus creating a network-only test environment.</para>
- <screen>loadgen> echosrv
+ <para>The loadgen utility prints periodic status messages; message output can be controlled with the verbose command.</para>
+ <para>To insure that a file can be written to (a requirement of write cache), OSTs reserve ("grants"), chunks of space for each newly-created file. A grant may cause an OST to report that it is out of space, even though there is plenty of space on the disk, because the space is "reserved" by other files. The loadgen utility estimates the number of simultaneous open files as the disk size divided by the grant size and reports that number when the write tests are first started.</para>
+ <para><emphasis role="bold">Echo Server</emphasis></para>
+ <para>The loadgen utility can start an echo server. On another node, loadgen can specify the echo server as the device, thus creating a network-only test environment.</para>
+ <screen>loadgen> echosrv
loadgen> dl
0 UP obdecho echosrv echosrv 3
1 UP ost OSS OSS 3
</screen>
-
<para>On another node:</para>
- <screen>loadgen> device echosrv cfs21@tcp
+ <para>On another node:</para>
+ <screen>loadgen> device echosrv cfs21@tcp
Added uuid OSS_UUID: 192.168.0.21@tcp
Target OST name is 'echosrv'
loadgen> st 1
1: done (0)
wrote 1MB in 0.029s (34.023 MB/s)
</screen>
-
<para>The threads all perform their actions in non-blocking mode; use the wait command to block for the idle state. For example:</para>
- <screen>#!/bin/bash
+ <para>Scripting</para>
+ <para>The threads all perform their actions in non-blocking mode; use the wait command to block for the idle state. For example:</para>
+ <screen>#!/bin/bash
./loadgen << EOF
device lustre-OST0000
st 1
quit
EOF
</screen>
- <para><emphasis role="bold">Feature Requests</emphasis></para>
- <para>The loadgen utility is intended to grow into a more comprehensive test tool; feature requests are encouraged. The current feature requests include:</para>
- <itemizedlist><listitem>
- <para> Locking simulation</para>
- </listitem>
-<listitem>
- <itemizedlist><listitem>
- <para> Many (echo) clients cache locks for the specified resource at the same time.</para>
- </listitem>
-
-<listitem>
- <para> Many (echo) clients enqueue locks for the specified resource simultaneously.</para>
- </listitem>
-
-</itemizedlist>
- </listitem>
-<listitem>
- <para> obdsurvey functionality</para>
- </listitem>
-<listitem>
- <itemizedlist><listitem>
- <para> Fold the Lustre I/O kit's obdsurvey script functionality into loadgen</para>
- </listitem>
-
-</itemizedlist>
- </listitem>
-</itemizedlist>
- </section>
- <section remap="h5">
- <title>llog_reader</title>
- <para>The llog_reader utility translates a Lustre configuration log into human-readable form.</para>
- </section>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>llog_reader filename
+ <para><emphasis role="bold">Feature Requests</emphasis></para>
+ <para>The loadgen utility is intended to grow into a more comprehensive test tool; feature requests are encouraged. The current feature requests include:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Locking simulation</para>
+ </listitem>
+ <listitem>
+ <itemizedlist>
+ <listitem>
+ <para> Many (echo) clients cache locks for the specified resource at the same time.</para>
+ </listitem>
+ <listitem>
+ <para> Many (echo) clients enqueue locks for the specified resource simultaneously.</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ <listitem>
+ <para> obdsurvey functionality</para>
+ </listitem>
+ <listitem>
+ <itemizedlist>
+ <listitem>
+ <para> Fold the Lustre I/O kit's obdsurvey script functionality into loadgen</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section remap="h5">
+ <title><indexterm><primary>llog_reader</primary></indexterm>
+llog_reader</title>
+ <para>The llog_reader utility translates a Lustre configuration log into human-readable form.</para>
+ </section>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>llog_reader filename
</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>llog_reader parses the binary format of Lustre's on-disk configuration logs. It can only read the logs. Use tunefs.lustre to write to them.</para>
- <para>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs, then use llog_reader to dump the log file's contents. For example:</para>
- <screen>mount -t ldiskfs /dev/sda /mnt/mgs
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>llog_reader parses the binary format of Lustre's on-disk configuration logs. It can only read the logs. Use tunefs.lustre to write to them.</para>
+ <para>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs, then use llog_reader to dump the log file's contents. For example:</para>
+ <screen>mount -t ldiskfs /dev/sda /mnt/mgs
llog_reader /mnt/mgs/CONFIGS/tfs-client
</screen>
-
<para>To examine the same log file on a running Lustre server, use the ldiskfs-enabled debugfs utility (called debug.ldiskfs on some distributions) to extract the file. For example:</para>
- <screen>debugfs -c -R 'dump CONFIGS/tfs-client /tmp/tfs-client' /dev/sda
+ <para>To examine the same log file on a running Lustre server, use the ldiskfs-enabled debugfs utility (called debug.ldiskfs on some distributions) to extract the file. For example:</para>
+ <screen>debugfs -c -R 'dump CONFIGS/tfs-client /tmp/tfs-client' /dev/sda
llog_reader /tmp/tfs-client
</screen>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="5*"/>
- <colspec colname="c2" colwidth="95*"/>
-
-
- <tbody>
- <row>
- <entry><para><inlinemediaobject><imageobject role="html">
- <imagedata fileref="./shared/caution.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis>Although they are stored in the CONFIGS directory, mountdata files do not use the config log format and will confuse llog_reader.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">See Also</emphasis></para>
- <para><xref linkend="dbdoclet.50438219_39574"/></para>
- </section>
- <section remap="h5">
- <title>lr_reader</title>
- <para>The lr_reader utility translates a last received (last_rcvd) file into human-readable form.</para>
- <para>The following utilites are part of the Lustre I/O kit. For more information, see <xref linkend='benchmarkingtests'/>.</para>
- </section>
- <section remap="h5">
- <title>sgpdd_survey</title>
- <para>The sgpdd_survey utility tests 'bare metal' performance, bypassing as much of the kernel as possible. The sgpdd_survey tool does not require Lustre, but it does require the sgp_dd package.</para>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="5*"/>
- <colspec colname="c2" colwidth="95*"/>
-
-
- <tbody>
- <row>
- <entry><para><inlinemediaobject><imageobject role="html">
- <imagedata fileref="./shared/caution.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis>The sgpdd_survey utility erases all data on the device.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>obdfilter_survey</title>
- <para>The obdfilter_survey utility is a shell script that tests performance of isolated OSTS, the network via echo clients, and an end-to-end test.</para>
- </section>
- <section remap="h5">
- <title>ior-survey</title>
- <para>The ior-survey utility is a script used to run the IOR benchmark. Lustre includes IOR version 2.8.6.</para>
- </section>
- <section remap="h5">
- <title>ost_survey</title>
- <para>The ost_survey utility is an OST performance survey that tests client-to-disk performance of the individual OSTs in a Lustre file system.</para>
- </section>
- <section remap="h5">
- <title>stats-collect</title>
- <para>The stats-collect utility contains scripts used to collect application profiling information from Lustre clients and servers.</para>
- </section>
+ <caution>
+ <para>Although they are stored in the CONFIGS directory, mountdata files do not use the config log format and will confuse llog_reader.</para>
+ </caution>
+ <para><emphasis role="bold">See Also</emphasis></para>
+ <para><xref linkend="dbdoclet.50438219_39574"/></para>
</section>
- <section remap="h3">
- <title>36.19.4 Flock <anchor xml:id="dbdoclet.50438219_marker-1295910" xreflabel=""/>Feature</title>
- <para>Lustre now includes the flock feature, which provides file locking support. Flock describes classes of file locks known as 'flocks'. Flock can apply or remove a lock on an open file as specified by the user. However, a single file may not, simultaneously, have both shared and exclusive locks.</para>
- <para>By default, the flock utility is disabled on Lustre. Two modes are available.</para>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">local mode</emphasis></para></entry>
- <entry><para> In this mode, locks are coherent on one node (a single-node flock), but not across all clients. To enable it, use -o localflock. This is a client-mount option.</para><note><para>This mode does not impact performance and is appropriate for single-node databases.</para></note></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">consistent mode</emphasis></para></entry>
- <entry><para> In this mode, locks are coherent across all clients.</para><para> To enable it, use the -o flock. This is a client-mount option.</para><para><emphasis role="bold">CAUTION</emphasis>: This mode affects the performance of the file being flocked and may affect stability, depending on the Lustre version used. Consider using a newer Lustre version which is more stable. If the consistent mode is enabled and no applications are using flock, then it has no effect.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para>A call to use flock may be blocked if another process is holding an incompatible lock. Locks created using flock are applicable for an open file table entry. Therefore, a single process may hold only one type of lock (shared or exclusive) on a single file. Subsequent flock calls on a file that is already locked converts the existing lock to the new lock mode.</para>
- <section remap="h4">
- <title>36.19.4.1 Example</title>
- <screen>$ mount -t lustre -o flock mds@tcp0:/lustre /mnt/client
-</screen>
- <para>You can check it in /etc/mtab. It should look like,</para>
- <screen>mds@tcp0:/lustre /mnt/client lustre rw,flock 0 0
-</screen>
- </section>
+ <section remap="h5">
+ <title><indexterm><primary>lr_reader</primary></indexterm>
+lr_reader</title>
+ <para>The lr_reader utility translates a last received (last_rcvd) file into human-readable form.</para>
+ <para>The following utilites are part of the Lustre I/O kit. For more information, see <xref linkend="benchmarkingtests"/>.</para>
+ </section>
+ <section remap="h5">
+ <title><indexterm><primary>sgpdd_survey</primary></indexterm>
+sgpdd_survey</title>
+ <para>The sgpdd_survey utility tests 'bare metal' performance, bypassing as much of the kernel as possible. The sgpdd_survey tool does not require Lustre, but it does require the sgp_dd package.</para>
+ <caution>
+ <para>The sgpdd_survey utility erases all data on the device.</para>
+ </caution>
+ </section>
+ <section remap="h5">
+ <title><indexterm><primary>obdfilter_survey</primary></indexterm>obdfilter_survey</title>
+ <para>The obdfilter_survey utility is a shell script that tests performance of isolated OSTS, the network via echo clients, and an end-to-end test.</para>
+ </section>
+ <section remap="h5">
+ <title><indexterm><primary>ior-survey</primary></indexterm>ior-survey</title>
+ <para>The ior-survey utility is a script used to run the IOR benchmark. Lustre includes IOR version 2.8.6.</para>
+ </section>
+ <section remap="h5">
+ <title><indexterm><primary>ost_survey</primary></indexterm>ost_survey</title>
+ <para>The ost_survey utility is an OST performance survey that tests client-to-disk performance of the individual OSTs in a Lustre file system.</para>
+ </section>
+ <section remap="h5">
+ <title><indexterm><primary>stats-collect</primary></indexterm>stats-collect</title>
+ <para>The stats-collect utility contains scripts used to collect application profiling information from Lustre clients and servers.</para>
+ </section>
+ </section>
+ <section remap="h3">
+ <title><indexterm><primary>flock</primary></indexterm>Flock Feature</title>
+ <para>Lustre now includes the flock feature, which provides file locking support. Flock describes classes of file locks known as 'flocks'. Flock can apply or remove a lock on an open file as specified by the user. However, a single file may not, simultaneously, have both shared and exclusive locks.</para>
+ <para>By default, the flock utility is disabled on Lustre. Two modes are available.</para>
+ <informaltable frame="none">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">local mode</emphasis></para>
+ </entry>
+ <entry>
+ <para> In this mode, locks are coherent on one node (a single-node flock), but not across all clients. To enable it, use -o localflock. This is a client-mount option.</para>
+ <note>
+ <para>This mode does not impact performance and is appropriate for single-node databases.</para>
+ </note>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">consistent mode</emphasis></para>
+ </entry>
+ <entry>
+ <para> In this mode, locks are coherent across all clients.</para>
+ <para> To enable it, use the -o flock. This is a client-mount option.</para>
+ <para><emphasis role="bold">CAUTION</emphasis>: This mode affects the performance of the file being flocked and may affect stability, depending on the Lustre version used. Consider using a newer Lustre version which is more stable. If the consistent mode is enabled and no applications are using flock, then it has no effect.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para>A call to use flock may be blocked if another process is holding an incompatible lock. Locks created using flock are applicable for an open file table entry. Therefore, a single process may hold only one type of lock (shared or exclusive) on a single file. Subsequent flock calls on a file that is already locked converts the existing lock to the new lock mode.</para>
+ <section remap="h4">
+ <title>Example</title>
+ <screen>$ mount -t lustre -o flock mds@tcp0:/lustre /mnt/client</screen>
+ <para>You can check it in /etc/mtab. It should look like,</para>
+ <screen>mds@tcp0:/lustre /mnt/client lustre rw,flock 0 0</screen>
</section>
</section>
+ </section>
</chapter>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="troubleshootingrecovery">
- <info>
- <title xml:id="troubleshootingrecovery.title">Troubleshooting Recovery</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="troubleshootingrecovery">
+ <title xml:id="troubleshootingrecovery.title">Troubleshooting Recovery</title>
<para>This chapter describes what to do if something goes wrong during recovery. It describes:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438225_12316"/></para>
</listitem>
</itemizedlist>
-
<section xml:id="dbdoclet.50438225_71141">
- <title>27.1 Recovering from Errors or Corruption on a Backing File System</title>
+ <title><indexterm><primary>recovery</primary><secondary>corruption of backing file system</secondary></indexterm>Recovering from Errors or Corruption on a Backing File System</title>
<para>When an OSS, MDS, or MGS server crash occurs, it is not necessary to run e2fsck on the file system. <literal>ldiskfs</literal> journaling ensures that the file system remains coherent. The backing file systems are never accessed directly from the client, so client crashes are not relevant.</para>
<para>The only time it is REQUIRED that <literal>e2fsck</literal> be run on a device is when an event causes problems that ldiskfs journaling is unable to handle, such as a hardware device failure or I/O error. If the ldiskfs kernel code detects corruption on the disk, it mounts the file system as read-only to prevent further corruption, but still allows read access to the device. This appears as error "-30" (<literal>EROFS</literal>) in the syslogs on the server, e.g.:</para>
<screen>Dec 29 14:11:32 mookie kernel: LDISKFS-fs error (device sdz):
<para>In addition, the <literal>e2fsprogs</literal> package contains the <literal>lfsck</literal> tool, which does distributed coherency checking for the Lustre file system after <literal>e2fsck</literal> has been run. Running <literal>lfsck</literal> is NOT required in a large majority of cases, at a small risk of having some leaked space in the file system. To avoid a lengthy downtime, it can be run (with care) after Lustre is started.</para>
</section>
<section xml:id="dbdoclet.50438225_37365">
- <title>27.2 Recovering from Corruption in the Lustre File System</title>
+ <title><indexterm><primary>recovery</primary><secondary>corruption of Lustre file system</secondary></indexterm>Recovering from Corruption in the Lustre File System</title>
<para>In cases where the MDS or an OST becomes corrupt, you can run a distributed check on the file system to determine what sort of problems exist. Use <literal>lfsck</literal> to correct any defects found.</para>
<orderedlist>
<listitem>
<para>If the file system is in use and being modified while the <literal>--mdsdb</literal> and <literal>--ostdb</literal> steps are running, <literal>lfsck</literal> may report inconsistencies where none exist due to files and objects being created/removed after the database files were collected. Examine the <literal>lfsck</literal> results closely. You may want to re-run the test.</para>
</listitem>
</orderedlist>
- <section xml:id='dbdoclet.50438225_13916'>
- <title>27.2.1 Working with Orphaned Objects</title>
+ <section xml:id="dbdoclet.50438225_13916">
+ <title><indexterm><primary>recovery</primary><secondary>orphaned objects</secondary></indexterm>Working with Orphaned Objects</title>
<para>The easiest problem to resolve is that of orphaned objects. When the <literal>-l</literal> option for <literal>lfsck</literal> is used, these objects are linked to new files and put into <literal>lost+found</literal> in the Lustre file system, where they can be examined and saved or deleted as necessary. If you are certain the objects are not useful, run <literal>lfsck</literal> with the <literal>-d</literal> option to delete orphaned objects and free up any space they are using.</para>
<para>To fix dangling inodes, use <literal>lfsck</literal> with the <literal>-c</literal> option to create new, zero-length objects on the OSTs. These files read back with binary zeros for stripes that had objects re-created. Even without <literal>lfsck</literal> repair, these files can be read by entering:</para>
<screen>dd if=/lustre/bad/file of=/new/file bs=4k conv=sync,noerror</screen>
</section>
</section>
<section xml:id="dbdoclet.50438225_12316">
- <title>27.3 Recovering from an Unavailable OST</title>
+ <title><indexterm><primary>recovery</primary><secondary>unavailable OST</secondary></indexterm>Recovering from an Unavailable OST</title>
<para>One of the most common problems encountered in a Lustre environment is when an OST becomes unavailable, because of a network partition, OSS node crash, etc. When this happens, the OST's clients pause and wait for the OST to become available again, either on the primary OSS or a failover OSS. When the OST comes back online, Lustre starts a recovery process to enable clients to reconnect to the OST. Lustre servers put a limit on the time they will wait in recovery for clients to reconnect. The timeout length is determined by the <literal>obd_timeout</literal> parameter.</para>
<para>During recovery, clients reconnect and replay their requests serially, in the same order they were done originally. Until a client receives a confirmation that a given transaction has been written to stable storage, the client holds on to the transaction, in case it needs to be replayed. Periodically, a progress message prints to the log, stating how_many/expected clients have reconnected. If the recovery is aborted, this log shows how many clients managed to reconnect. When all clients have completed recovery, or if the recovery timeout is reached, the recovery period ends and the OST resumes normal request processing.</para>
<para>If some clients fail to replay their requests during the recovery period, this will not stop the recovery from completing. You may have a situation where the OST recovers, but some clients are not able to participate in recovery (e.g. network problems or client failure), so they are evicted and their requests are not replayed. This would result in any operations on the evicted clients failing, including in-progress writes, which would cause cached writes to be lost. This is a normal outcome; the recovery cannot wait indefinitely, or the file system would be hung any time a client failed. The lost transactions are an unfortunate result of the recovery process.</para>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="understandingfailover">
- <info>
- <title xml:id="understandingfailover.title">Understanding Failover in Lustre</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="understandingfailover">
+ <title xml:id="understandingfailover.title">Understanding Failover in Lustre</title>
<para>This chapter describes failover in a Lustre system. It includes:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50540653_59957">
- <title>3.1 What is Failover?</title>
+ <title><indexterm><primary>failover</primary></indexterm>
+ What is Failover?</title>
<para>A computer system is ''highly available'' when the services it provides are available with minimal downtime. In a highly-available system, if a failure condition occurs, such as the loss of a server or a network or software fault, the system's services continue without interruption. Generally, we measure availability by the percentage of time the system is required to be available.</para>
<para>Availability is accomplished by replicating hardware and/or software so that when a primary server fails or is unavailable, a standby server can be switched into its place to run applications and associated resources. This process, called <emphasis role="italic">failover</emphasis>, should be automatic and, in most cases, completely application-transparent.</para>
<para>A failover hardware setup requires a pair of servers with a shared resource (typically a physical storage device, which may be based on SAN, NAS, hardware RAID, SCSI or FC technology). The method of sharing storage should be essentially transparent at the device level; the same physical logical unit number (LUN) should be visible from both servers. To ensure high availability at the physical storage level, we encourage the use of RAID arrays to protect against drive-level failures.</para>
<para>Lustre does not provide redundancy for data; it depends exclusively on redundancy of backing storage devices. The backing OST storage should be RAID 5 or, preferably, RAID 6 storage. MDT storage should be RAID 1 or RAID 0+1.</para>
</note>
<section remap="h3">
- <title>3.1.1 Failover <anchor xml:id="dbdoclet.50540653_marker-1304336" xreflabel=""/>Capabilities</title>
+ <title><indexterm><primary>failover</primary><secondary>capabilities</secondary></indexterm>Failover Capabilities</title>
<para>To establish a highly-available Lustre file system, power management software or hardware and high availability (HA) software are used to provide the following failover capabilities:</para>
<itemizedlist>
<listitem>
<para>HA software is responsible for detecting failure of the primary Lustre server node and controlling the failover. Lustre works with any HA software that includes resource (I/O) fencing. For proper resource fencing, the HA software must be able to completely power off the failed server or disconnect it from the shared storage device. If two active nodes have access to the same storage device, data may be severely corrupted.</para>
</section>
<section remap="h3">
- <title>3.1.2 Types of Failover<anchor xml:id="dbdoclet.50540653_marker-1304346" xreflabel=""/> Configurations</title>
+ <title><indexterm><primary>failover</primary><secondary>configuration</secondary></indexterm>Types of Failover Configurations</title>
<para>Nodes in a cluster can be configured for failover in several ways. They are often configured in pairs (for example, two OSTs attached to a shared storage device), but other failover configurations are also possible. Failover configurations include:</para>
<itemizedlist>
<listitem>
</section>
</section>
<section xml:id="dbdoclet.50540653_97944">
- <title>3.2 Failover Functionality in Lustre</title>
+ <title><indexterm><primary>failover</primary><secondary>and Lustre</secondary></indexterm>Failover Functionality in Lustre</title>
<para>The failover functionality provided in Lustre can be used for the following failover scenario. When a client attempts to do I/O to a failed Lustre target, it continues to try until it receives an answer from any of the configured failover nodes for the Lustre target. A user-space application does not detect anything unusual, except that the I/O may take longer to complete.</para>
<para>Lustre failover requires two nodes configured as a failover pair, which must share one or more storage devices. Lustre can be configured to provide MDT or OST failover.</para>
<itemizedlist>
<para>For OST failover, multiple OSS nodes are configured to be able to serve the same OST. However, only one OSS node can serve the OST at a time. An OST can be moved between OSS nodes that have access to the same storage device using <literal>umount/mount</literal> commands.</para>
</listitem>
</itemizedlist>
- <para>To add a failover partner to a Lustre configuration, the <literal>--failnode</literal> option is used. This can be done at creation time (using <literal>mkfs.lustre</literal>) or later when the Lustre system is active (using <literal>tunefs.lustre</literal>). For explanations of these utilities, see <xref linkend='dbdoclet.50438219_75432'/> and <xref linkend='dbdoclet.50438219_39574'/>.</para>
+ <para>To add a failover partner to a Lustre configuration, the <literal>--failnode</literal> option is used. This can be done at creation time (using <literal>mkfs.lustre</literal>) or later when the Lustre system is active (using <literal>tunefs.lustre</literal>). For explanations of these utilities, see <xref linkend="dbdoclet.50438219_75432"/> and <xref linkend="dbdoclet.50438219_39574"/>.</para>
<para>Lustre failover capability can be used to upgrade the Lustre software between successive minor versions without cluster downtime. For more information, see <xref linkend="upgradinglustre"/>.</para>
- <para>For information about configuring failover, see <xref linkend='configuringfailover'/>.</para>
+ <para>For information about configuring failover, see <xref linkend="configuringfailover"/>.</para>
<note>
<para>Failover functionality in Lustre is provided only at the file system level. In a complete failover solution, failover functionality for system-level components, such as node failure detection or power control, must be provided by a third-party tool.</para>
</note>
<para>OST failover functionality does not protect against corruption caused by a disk failure. If the storage media (i.e., physical disk) used for an OST fails, Lustre cannot recover it. We strongly recommend that some form of RAID be used for OSTs. Lustre functionality assumes that the storage is reliable, so it adds no extra reliability features.</para>
</caution>
<section remap="h3">
- <title>3.2.1 MDT Failover Configuration <anchor xml:id="dbdoclet.50540653_marker-1304373" xreflabel=""/>(Active/Passive)</title>
+ <title><indexterm><primary>failover</primary><secondary>MDT</secondary></indexterm>MDT Failover Configuration (Active/Passive)</title>
<para>Two MDSs are typically configured as an active/passive failover pair as shown in <xref linkend="understandingfailover.fig.configmdt"/>. Note that both nodes must have access to shared storage for the MDT(s) and the MGS. The primary (active) MDS manages the Lustre system metadata resources. If the primary MDS fails, the secondary (passive) MDS takes over these resources and serves the MDTs and the MGS.</para>
<note>
<para>In an environment with multiple file systems, the MDSs can be configured in a quasi active/active configuration, with each MDS managing metadata for a subset of the Lustre file system.</para>
</figure>
</section>
<section remap="h3">
- <title>3.2.2 OST Failover Configuration <anchor xml:id="dbdoclet.50540653_marker-1304382" xreflabel=""/>(Active/Active)</title>
+ <title><indexterm><primary>failover</primary><secondary>OST</secondary></indexterm>OST Failover Configuration (Active/Active)</title>
<para>OSTs are usually configured in a load-balanced, active/active failover configuration. A failover cluster is built from two OSSs as shown in <xref linkend="understandingfailover.fig.configost"/>.</para>
<note>
<para>OSSs configured as a failover pair must have shared disks/RAID.</para>
</listitem>
</itemizedlist>
<section xml:id="understandinglustre.whatislustre">
- <title>What Lustre Is (and What It Isn't)</title>
+ <title><indexterm><primary>Lustre</primary></indexterm>What Lustre Is (and What It Isn't)</title>
<para>Lustre is a storage architecture for clusters. The central component of the Lustre architecture is the Lustre file system, which is supported on the Linux operating system and provides a POSIX-compliant UNIX file system interface.</para>
<para>The Lustre storage architecture is used for many different kinds of clusters. It is best known for powering seven of the ten largest high-performance computing (HPC) clusters worldwide, with tens of thousands of client systems, petabytes (PB) of storage and hundreds of gigabytes per second (GB/sec) of I/O throughput. Many HPC sites use Lustre as a site-wide global file system, serving dozens of clusters.</para>
<para>The ability of a Lustre file system to scale capacity and performance for any need reduces the need to deploy many separate file systems, such as one for each compute cluster. Storage management is simplified by avoiding the need to copy data between compute clusters. In addition to aggregating storage capacity of many servers, the I/O throughput is also aggregated and scales with additional servers. Moreover, throughput and/or capacity can be easily increased by adding servers dynamically.</para>
<para>While Lustre can function in many work environments, it is not necessarily the best choice for all applications. It is best suited for uses that exceed the capacity that a single server can provide, though in some use cases Lustre can perform better with a single server than other filesystems due to its strong locking and data coherency.</para>
<para>Lustre is currently not particularly well suited for "peer-to-peer" usage models where there are clients and servers running on the same node, each sharing a small amount of storage, due to the lack of Lustre-level data replication. In such uses, if one client/server fails, then the data stored on that node will not be accessible until the node is restarted.</para>
<section remap="h3">
- <title>Lustre Features</title>
+ <title><indexterm><primary>Lustre</primary><secondary>features</secondary></indexterm>Lustre Features</title>
<para>Lustre runs on a variety of vendor's kernels. For more details, see <link xl:href="http://wiki.whamcloud.com/">Lustre Release Information</link> on the Whamcloud wiki.</para>
<para>A Lustre installation can be scaled up or down with respect to the number of client nodes, disk storage and bandwidth. Scalability and performance are dependent on available disk and network bandwith and the processing power of the servers in the system. Lustre can be deployed in a wide variety of configurations that can be scaled well beyond the size and performance observed in production systems to date.</para>
<para><xref linkend="understandinglustre.tab1"/> shows the practical range of scalability and performance characteristics of the Lustre file system and some test results in production systems.</para>
</section>
</section>
<section xml:id="understandinglustre.components">
- <title>Lustre Components</title>
+ <title><indexterm><primary>Lustre</primary><secondary>components</secondary></indexterm>Lustre Components</title>
<para>An installation of the Lustre software includes a management server (MGS) and one or more Lustre file systems interconnected with Lustre networking (LNET).</para>
<para>A basic configuration of Lustre components is shown in <xref linkend="understandinglustre.fig.cluster"/>.</para>
<figure>
</mediaobject>
</figure>
<section remap="h3">
- <title>Management Server (MGS)</title>
+ <title><indexterm><primary>Lustre</primary><secondary>MGS</secondary></indexterm>Management Server (MGS)</title>
<para>The MGS stores configuration information for all the Lustre file systems in a cluster and provides this information to other Lustre components. Each Lustre target contacts the MGS to provide information, and Lustre clients contact the MGS to retrieve information.</para>
<para>It is preferable that the MGS have its own storage space so that it can be managed independently. However, the MGS can be co-located and share storage space with an MDS as shown in <xref linkend="understandinglustre.fig.cluster"/>.</para>
</section>
<para>A logical object volume (LOV) aggregates the OSCs to provide transparent access across all the OSTs. Thus, a client with the Lustre file system mounted sees a single, coherent, synchronized namespace. Several clients can write to different parts of the same file simultaneously, while, at the same time, other clients can read from the file.</para>
<para><xref linkend="understandinglustre.tab.storagerequire"/> provides the requirements for attached storage for each Lustre file system component and describes desirable characterics of the hardware used.</para>
<table frame="all">
- <title xml:id="understandinglustre.tab.storagerequire">Storage and hardware requirements for Lustre components</title>
+ <title xml:id="understandinglustre.tab.storagerequire"><indexterm><primary>Lustre</primary><secondary>requirements</secondary></indexterm>Storage and hardware requirements for Lustre components</title>
<tgroup cols="3">
<colspec colname="c1" colwidth="1*"/>
<colspec colname="c2" colwidth="3*"/>
<para>For additional hardware requirements and considerations, see <xref linkend="settinguplustresystem"/>.</para>
</section>
<section remap="h3">
- <title>Lustre Networking (LNET)</title>
+ <title><indexterm><primary>Lustre</primary><secondary>LNET</secondary></indexterm>Lustre Networking (LNET)</title>
<para>Lustre Networking (LNET) is a custom networking API that provides the communication infrastructure that handles metadata and file I/O data for the Lustre file system servers and clients. For more information about LNET, see something <xref linkend="understandinglustrenetworking"/>.</para>
</section>
<section remap="h3">
- <title>Lustre Cluster</title>
+ <title><indexterm><primary>Lustre</primary><secondary>cluster</secondary></indexterm>Lustre Cluster</title>
<para>At scale, the Lustre cluster can include hundreds of OSSs and thousands of clients (see <xref linkend="understandinglustre.fig.lustrescale"/>). More than one type of network can be used in a Lustre cluster. Shared storage between OSSs enables failover capability. For more details about OSS failover, see <xref linkend="understandingfailover"/>.</para>
<figure>
- <title xml:id="understandinglustre.fig.lustrescale">Lustre cluster at scale</title>
+ <title xml:id="understandinglustre.fig.lustrescale"><indexterm><primary>Lustre</primary><secondary>at scale</secondary></indexterm>Lustre cluster at scale</title>
<mediaobject>
<imageobject>
<imagedata fileref="./figures/Scaled_Cluster.png"/>
</section>
</section>
<section xml:id="understandinglustre.storageio">
- <title>Lustre Storage and I/O</title>
+ <title><indexterm><primary>Lustre</primary><secondary>storage</secondary></indexterm>
+ <indexterm><primary>Lustre</primary><secondary>I/O</secondary></indexterm>
+ Lustre Storage and I/O</title>
<para>In a Lustre file system, a file stored on the MDT points to one or more objects associated with a data file, as shown in <xref linkend="understandinglustre.fig.mdtost"/>. Each object contains data and is stored on an OST. If the MDT file points to one object, all the file data is stored in that object. If the file points to more than one object, the file data is 'striped' across the objects (using RAID 0) and each object is stored on a different OST. (For more information about how striping is implemented in Lustre, see <xref linkend="dbdoclet.50438250_89922"/>)</para>
<para>In <xref linkend="understandinglustre.fig.mdtost"/>, each filename points to an inode. The inode contains all of the file attributes, such as owner, access permissions, Lustre striping layout, access time, and access control. Multiple filenames may point to the same inode.</para>
<figure>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438250_89922">
- <title>1.3.1 Lustre File System and Striping</title>
+ <title>
+ <indexterm><primary>Lustre</primary><secondary>striping</secondary></indexterm>
+ <indexterm><primary>striping</primary><secondary>overview</secondary></indexterm>
+ Lustre File System and Striping</title>
<para>One of the main factors leading to the high performance of Lustre file systems is the ability to stripe data across multiple OSTs in a round-robin fashion. Users can optionally configure for each file the number of stripes, stripe size, and OSTs that are used.</para>
<para>Striping can be used to improve performance when the aggregate bandwidth to a single file exeeds the bandwidth of a single OST. The ability to stripe is also useful when a single OST does not have anough free space to hold an entire file. For more information about benefits and drawbacks of file striping, see <xref linkend="dbdoclet.50438209_48033"/>.</para>
<para>Striping allows segments or 'chunks' of data in a file to be stored on different OSTs, as shown in <xref linkend="understandinglustre.fig.filestripe"/>. In the Lustre file system, a RAID 0 pattern is used in which data is "striped" across a certain number of objects. The number of objects in a single file is called the <literal>stripe_count</literal>.</para>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438191_22878">
- <title>Introducing LNET</title>
+ <title>
+ <indexterm><primary>LNET</primary></indexterm>
+ <indexterm><primary>LNET</primary><secondary>understanding</secondary></indexterm>
+ Introducing LNET</title>
<para>In a cluster with a Lustre file system, the system network connecting the servers and the clients is implemented using Lustre Networking (LNET), which provides the communication infrastructure required by the Lustre file system.</para>
<para>LNET supports many commonly-used network types, such as InfiniBand and IP networks, and allows simultaneous availability across multiple network types with routing between them. Remote Direct Memory Access (RDMA) is permitted when supported by underlying networks using the appropriate Lustre network driver (LND). High availability and recovery features enable transparent recovery in conjunction with failover servers.</para>
<para>An LND is a pluggable driver that provides support for a particular network type. LNDs are loaded into the driver stack, with one LND for each network type in use.</para>
<para>For information about administering LNET, see <xref linkend="adminlustrepart3"/>.</para>
</section>
<section xml:id="dbdoclet.50438191_19625">
- <title>Key Features of LNET</title>
+ <title><indexterm><primary>LNET</primary><secondary>features</secondary></indexterm>Key Features of LNET</title>
<para>Key features of LNET include:</para>
<itemizedlist>
<listitem>
<para>Lustre can use bonded networks, such as bonded Ethernet networks, when the underlying network technology supports bonding. For more information, see <xref linkend="settingupbonding"/>.</para>
</section>
<section xml:id="dbdoclet.50438191_20721">
- <title>Supported Network Types</title>
+ <title><indexterm><primary>LNET</primary><secondary>supported networks</secondary></indexterm>Supported Network Types</title>
<para>LNET includes LNDs to support many network types including:</para>
<itemizedlist>
<listitem>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="upgradinglustre">
- <info>
- <title xml:id="upgradinglustre.title">Upgrading Lustre</title>
- </info>
+ <title xml:id="upgradinglustre.title">Upgrading Lustre</title>
<para>This chapter describes Lustre interoperability and how to upgrade to Lustre 2.0, and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438205_82079">
- <title>16.1 Lustre Interoperability</title>
+ <title><indexterm><primary>Lustre</primary><secondary>upgrading</secondary><see>upgrading</see></indexterm>
+ <indexterm><primary>upgrading</primary></indexterm>
+
+ Lustre Interoperability</title>
<para>Lustre 2.0 is built on a new architectural code base, which is different than the one used with Lustre 1.8. These architectural changes require existing Lustre 1.8.x users to follow a slightly different procedure to upgrade to Lustre 2.0 - requiring clients to be unmounted and the file system be shut down. Once the servers are upgraded and restarted, then the clients can be remounted. After the upgrade, Lustre 2.0 servers can interoperate with compatible 1.8 clients and servers. Lustre 2.0 does <emphasis>not</emphasis> support 2.0 clients interoperating with 1.8 servers.</para>
<note>
<para>Lustre 1.8 clients support a mix of 1.8 and 2.0 OSTs, not all OSSs need to be upgraded at the same time.</para>
</note>
</section>
<section xml:id="dbdoclet.50438205_51369">
- <title>16.2 Upgrading Lustre 1.8.x to 2.0</title>
+ <title><indexterm><primary>upgrading</primary><secondary>1.8 to 2.x</secondary></indexterm>Upgrading Lustre 1.8.x to 2.0</title>
<para>Upgrading to Lustre 2.0 involves shutting down the file system and upgrading servers, and optionally clients, all at the same time. Lustre 2.0 does not support a rolling upgrade in which the file system operates continuously while individual servers (or their failover partners) and clients are upgraded one at a time.</para>
<note>
<para>Although the Lustre 1.8 to 2.0 upgrade path has been tested, for best results we recommend performing a fresh Lustre 2.0 installation, rather than upgrading from 1.8 to 2.0.</para>
</note>
<section remap="h3">
- <title>16.2.1 Performing a File System Upgrade</title>
+ <title><indexterm><primary>upgrading</primary><secondary>file system</secondary></indexterm>Performing a File System Upgrade</title>
<para>This procedure describes a file system upgrade in which Lustre 2.0 packages are installed on multiple 1.8.x servers and, optionally, clients, requiring a file system shut down. You can choose to upgrade the entire Lustre file system to 2.0 or just upgrade the Lustre servers to 2.0 and leave the clients at 1.8.x. Lustre 2.0 servers can interoperate with compatible 1.8 clients and servers.</para>
<tip>
<para>In a Lustre upgrade, the package install and file system unmount steps are reversible; you can do either step first. To minimize downtime, this procedure first performs the 2.0 package installation, and then unmounts the file system.</para>
</tip>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Make a complete, restorable file system backup before upgrading Lustre.</emphasis></para>
+ <para>Make a complete, restorable file system backup before upgrading Lustre.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">If any Lustre nodes will not be upgraded to 2.0, make sure that these client and server nodes are at version 1.8.4.</emphasis></para>
+ <para>If any Lustre nodes will not be upgraded to 2.0, make sure that these client and server nodes are at version 1.8.4.</para>
<para>Lustre 2.0 is compatible with version 1.8.4 and above. If you are planning a heterogeneous environment (mixed 1.8 and 2.0 clients and servers), make sure that version 1.8.4 is installed on nodes that are not upgraded to 2.0.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Install the 2.0 packages on the Lustre servers and, optionally, the clients.</emphasis></para>
+ <para>Install the 2.0 packages on the Lustre servers and, optionally, the clients.</para>
<para>Some or all servers can be upgraded. Some or all clients can be upgraded.</para>
<para>For help determining where to install a specific package, see <xref linkend="installinglustre.tab.req"/>.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Install the kernel, modules and ldiskfs packages. For example:</emphasis></para>
+ <para>Install the kernel, modules and ldiskfs packages. For example:</para>
<screen>$ rpm -ivh
kernel-lustre-smp-<ver> \
kernel-ib-<ver> \
lustre-ldiskfs-<ver></screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Upgrade the utilities/userspace packages. For example:</emphasis></para>
+ <para>Upgrade the utilities/userspace packages. For example:</para>
<screen>$ rpm -Uvh lustre-<ver></screen>
</listitem>
<listitem>
- <para><emphasis role="bold">If a new <literal>e2fsprogs</literal> package is available, upgrade it. For example:</emphasis></para>
+ <para>If a new <literal>e2fsprogs</literal> package is available, upgrade it. For example:</para>
<screen>$ rpm -Uvh e2fsprogs-<ver>
</screen>
<para>Use e2fsprogs-1.41-10 or later, available at:</para>
<para><link xl:href="http://downloads.lustre.org/public/tools/e2fsprogs/">http://downloads.lustre.org/public/tools/e2fsprogs/</link></para>
</listitem>
<listitem>
- <para><emphasis role="bold">(Optional) If you want to add optional packages to your Lustre system, install them now.</emphasis></para>
+ <para>(Optional) If you want to add optional packages to your Lustre system, install them now.</para>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Shut down the file system.</emphasis></para>
+ <para>Shut down the file system.</para>
<para>Shut down the components in this order: clients, then the MDT, then OSTs. Unmounting a block device causes Lustre to be shut down on that node.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Unmount the clients. On each client node, run:</emphasis></para>
+ <para>Unmount the clients. On each client node, run:</para>
<screen>umount <mount point></screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Unmount the MDT. On the MDS node, run:</emphasis></para>
+ <para>Unmount the MDT. On the MDS node, run:</para>
<screen>umount <mount point></screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Unmount the OSTs (be sure to unmount all OSTs). On each OSS node, run:</emphasis></para>
+ <para>Unmount the OSTs (be sure to unmount all OSTs). On each OSS node, run:</para>
<screen>umount <mount point></screen>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><emphasis role="bold">Unload the old Lustre modules by rebooting the node or manually removing the Lustre modules.</emphasis></para>
+ <para>Unload the old Lustre modules by rebooting the node or manually removing the Lustre modules.</para>
<para>Run <literal>lustre_rmmod</literal> several times and use <literal>lsmod</literal> to check the currently loaded modules.</para>
</listitem>
<listitem>
- <para><emphasis role="bold"><emphasis role="bold">Start the upgraded file system</emphasis>.</emphasis></para>
+ <para>Start the upgraded file system.</para>
<para>Start the components in this order: OSTs, then the MDT, then clients.</para>
<orderedlist>
<listitem>
- <para><emphasis role="bold">Mount the OSTs (be sure to mount all OSTs). On each OSS node, run:</emphasis></para>
+ <para>Mount the OSTs (be sure to mount all OSTs). On each OSS node, run:</para>
<screen>mount -t lustre <block device name> <mount point></screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Mount the MDT. On the MDS node, run:</emphasis></para>
+ <para>Mount the MDT. On the MDS node, run:</para>
<screen>mount -t lustre <block device name> <mount point> </screen>
</listitem>
<listitem>
- <para><emphasis role="bold">Mount the file system on the clients. On each client node, run:</emphasis></para>
+ <para>Mount the file system on the clients. On each client node, run:</para>
<screen>mount -t lustre <MGS node>:/<fsname> <mount point> </screen>
</listitem>
</orderedlist>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="userutilities">
- <info>
- <title xml:id="userutilities.title">User Utilities</title>
- </info>
+ <title xml:id="userutilities.title">User Utilities</title>
<para>This chapter describes user utilities and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438206_94597">
- <title>32.1 <literal>lfs</literal></title>
+ <title><indexterm><primary>lfs</primary></indexterm>
+ <literal>lfs</literal>
+ </title>
<para>The <literal>lfs</literal> utility can be used for user configuration routines and monitoring.</para>
<section remap="h5">
<title>Synopsis</title>
<para> <literal>--pool <pool></literal></para>
</entry>
<entry>
-
<para>Name of the pre-defined pool of OSTs (see <xref linkend="dbdoclet.50438219_38274"/>) that will be used for striping. The <literal>stripe_cnt</literal>, <literal>stripe_size</literal> and <literal>start_ost</literal> values are used as well. The start-ost value must be part of the pool or an error is returned.</para>
+ <para>Name of the pre-defined pool of OSTs (see <xref linkend="dbdoclet.50438219_38274"/>) that will be used for striping. The <literal>stripe_cnt</literal>, <literal>stripe_size</literal> and <literal>start_ost</literal> values are used as well. The start-ost value must be part of the pool or an error is returned.</para>
</entry>
</row>
<row>
</row>
<row>
<entry nameend="c2" namest="c1">
-
<para><literal> setquota -t <-u|-g> [--block-grace <block-grace>] [--inode-grace <inode-grace>] <filesystem></literal></para>
+ <para><literal> setquota -t <-u|-g> [--block-grace <block-grace>] [--inode-grace <inode-grace>] <filesystem></literal></para>
</entry>
<entry>
<para>Sets the file system quota grace times for users or groups. Grace time is specified in '<literal>XXwXXdXXhXXmXXs</literal>' format or as an integer seconds value. See <xref linkend="dbdoclet.50438206_11903"/>.</para>
</section>
</section>
<section xml:id="dbdoclet.50438206_42260">
- <title>32.2 <literal>lfs_migrate</literal></title>
+ <title><indexterm><primary>lfs_migrate</primary></indexterm>
+ <literal>lfs_migrate</literal>
+ </title>
<para>The <literal>lfs_migrate</literal> utility is a simple tool to migrate files between Lustre OSTs.</para>
<section remap="h5">
<title>Synopsis</title>
</row>
<row>
<entry>
- <literal>-l</literal>
+ <literal>-l</literal>
</entry>
<entry>
<para>Migrates files with hard links (skips, by default). Files with multiple hard links are split into multiple separate files by <literal>lfs_migrate</literal>, so they are skipped, by default, to avoid breaking the hard links.</para>
</row>
<row>
<entry>
- <literal>-n</literal>
+ <literal>-n</literal>
</entry>
<entry>
<para>Only prints the names of files to be migrated.</para>
</section>
</section>
<section xml:id="dbdoclet.50438206_91700">
- <title>32.3 <literal>lfsck</literal></title>
+ <title><indexterm><primary>lfsck</primary></indexterm>
+ <literal>lfsck</literal>
+ </title>
<para><literal>lfsck</literal> ensures that objects are not referenced by multiple MDS files, that there are no orphan objects on the OSTs (objects that do not have any file on the MDS which references them), and that all of the objects referenced by the MDS exist. Under normal circumstances, Lustre maintains such coherency by distributed logging mechanisms, but under exceptional circumstances that may fail (e.g. disk failure, file system corruption leading to e2fsck repair). To avoid lengthy downtime, you can also run <literal>lfsck</literal> once Lustre is already started.</para>
<para>The <literal>e2fsck</literal> utility is run on each of the local MDS and OST device file systems and verifies that the underlying <literal>ldiskfs</literal> is consistent. After <literal>e2fsck</literal> is run, <literal>lfsck</literal> does distributed coherency checking for the Lustre file system. In most cases, <literal>e2fsck</literal> is sufficient to repair any file system issues and <literal>lfsck</literal> is not required.</para>
<section remap="h5">
</section>
</section>
<section xml:id="dbdoclet.50438206_75125">
- <title>32.4 <literal>filefrag</literal></title>
+ <title><indexterm><primary>filefrag</primary></indexterm>
+ <literal>filefrag</literal>
+ </title>
<para>The <literal>e2fsprogs</literal> package contains the <literal>filefrag</literal> tool which reports the extent of file fragmentation.</para>
<section remap="h5">
<title>Synopsis</title>
</section>
</section>
<section xml:id="dbdoclet.50438206_86244">
- <title>32.5 <literal>mount</literal></title>
+ <title><indexterm><primary>mount</primary></indexterm>
+ <literal>mount</literal>
+ </title>
<para>Lustre uses the standard <literal>mount(8)</literal> Linux command. When mounting a Lustre file system, mount(8) executes the <literal>/sbin/mount.lustre</literal> command to complete the mount. The mount command supports these Lustre-specific options:</para>
<informaltable frame="all">
<tgroup cols="2">
</informaltable>
</section>
<section xml:id="dbdoclet.50438206_56217">
- <title>32.6 Handling Timeouts</title>
+ <title>Handling Timeouts</title>
<para>Timeouts are the most common cause of hung applications. After a timeout involving an MDS or failover OST, applications attempting to access the disconnected resource wait until the connection gets established.</para>
<para>When a client performs any remote operation, it gives the server a reasonable amount of time to respond. If a server does not reply either due to a down network, hung server, or any other reason, a timeout occurs which requires a recovery.</para>
<para>If a timeout occurs, a message (similar to this one), appears on the console of the client, and in <literal>/var/log/messages</literal>:</para>
req@a2d45200 x5886/t0 o38->mds_svc_UUID@NID_mds_UUID:12 lens 168/64 ref 1 fl
-RPC:/0/0 rc 0
-</screen>
+RPC:/0/0 rc 0</screen>
</section>
</chapter>
<subtitle>Operations Manual</subtitle>
<xi:include href="copyright.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
+ <xi:include href="Revision.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
</info>
+
<xi:include href="Preface.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
<xi:include href="I_LustreIntro.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
<xi:include href="Glossary.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
+
+ <xi:include href="ix.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
</book>
<?xml version="1.0" encoding="UTF-8"?>
-<article version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink">
- <info>
- <title>Index</title>
- </info>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
-
-
- <tbody>
- <row>
- <entry align="left"><para>Lustre 2.0 Operations Manual</para></entry>
- <entry align="right" valign="top"><para><link xl:href="index.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/toc01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/toc01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="Glossary.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/prev01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/prev01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/next02.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/next02.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject><link xl:href="ix.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/index01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/index01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link></para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><link xl:href=""/></para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
-
- <tbody>
- <row>
- <entry align="right"><para>Index</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <section remap="h3">
- <title>A<anchor xml:id="dbdoclet.50438146_Symbols" xreflabel=""/><anchor xml:id="dbdoclet.50438146_Numerics" xreflabel=""/><anchor xml:id="dbdoclet.50438146_A" xreflabel=""/></title>
- <para></para>
- <para>access control list (ACL), <link xl:href="ManagingSecurity.html#50438221_marker-1292306">1</link></para>
- <para></para>
- <para>ACL, using, <link xl:href="ManagingSecurity.html#50438221_marker-1292306">1</link></para>
- <para>ACLs</para>
- <para></para>
- <para>examples, <link xl:href="ManagingSecurity.html#50438221_marker-1292325">1</link></para>
- <para></para>
- <para>Lustre support, <link xl:href="ManagingSecurity.html#50438221_marker-1292314">1</link></para>
- <para>adaptive timeouts</para>
- <para></para>
- <para>configuring, <link xl:href="LustreProc.html#50438271_marker-1293381">1</link></para>
- <para></para>
- <para>interpreting, <link xl:href="LustreProc.html#50438271_marker-1293383">1</link></para>
- <para></para>
- <para>introduction, <link xl:href="LustreProc.html#50438271_marker-1293380">1</link></para>
- <para>adding</para>
- <para></para>
- <para>clients, <link xl:href="ConfiguringLustre.html#50438267_marker-1292440">1</link></para>
- <para></para>
- <para>OSTs, <link xl:href="ConfiguringLustre.html#50438267_marker-1292440">1</link></para>
- <para></para>
- <para>allocating quotas, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290226">1</link></para>
- <section remap="h3">
- <title>B<anchor xml:id="dbdoclet.50438146_B" xreflabel=""/></title>
- <para></para>
- <para>bonding, <link xl:href="SettingUpBonding.html#50438258_marker-1301678">1</link></para>
- <para></para>
- <para>configuring Lustre, <link xl:href="SettingUpBonding.html#50438258_marker-1301911">1</link></para>
- <para></para>
- <para>module parameters, <link xl:href="SettingUpBonding.html#50438258_marker-1301747">1</link></para>
- <para></para>
- <para>references, <link xl:href="SettingUpBonding.html#50438258_marker-1301915">1</link></para>
- <para></para>
- <para>requirements, <link xl:href="SettingUpBonding.html#50438258_marker-1301684">1</link></para>
- <para></para>
- <para>setting up, <link xl:href="SettingUpBonding.html#50438258_marker-1301757">1</link></para>
- </section>
- <section remap="h3">
- <title>C<anchor xml:id="dbdoclet.50438146_C" xreflabel=""/></title>
- <para>calculating</para>
- <para></para>
- <para>OSS memory requirements, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1292245">1</link></para>
- <para></para>
- <para>changelogs, <link xl:href="LustreMonitoring.html#50438273_marker-1297383">1</link></para>
- <para></para>
- <para>checksums, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1291974">1</link></para>
- <para>client</para>
- <para></para>
- <para>eviction, <link xl:href="LustreRecovery.html#50438268_marker-1292164">1</link></para>
- <para>client read/write</para>
- <para></para>
- <para>extents survey, <link xl:href="LustreProc.html#50438271_marker-1290612">1</link></para>
- <para></para>
- <para>offset survey, <link xl:href="LustreProc.html#50438271_marker-1290564">1</link></para>
- <para>clients</para>
- <para></para>
- <para>adding, <link xl:href="ConfiguringLustre.html#50438267_marker-1292440">1</link></para>
- <para></para>
- <para>CollectL, <link xl:href="LustreMonitoring.html#50438273_marker-1297391">1</link></para>
- <para>command</para>
- <para></para>
- <para>filefrag, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305920">1</link>, <link xl:href="LustreProgrammingInterfaces.html#50438291_marker-1294565">2</link></para>
- <para></para>
- <para>lfs, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305209">1</link></para>
- <para></para>
- <para>lfsck, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305843">1</link></para>
- <para></para>
- <para>mount, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305992">1</link></para>
- <para></para>
- <para>Commit on Share (COS), <link xl:href="LustreRecovery.html#50438268_marker-1292182">1</link></para>
- <para></para>
- <para>components, Lustre, <link xl:href="UnderstandingLustre.html#50438250_marker-1293887">1</link></para>
- <para></para>
- <para>configuration example, Lustre, <link xl:href="ConfiguringLustre.html#50438267_marker-1290955">1</link></para>
- <para></para>
- <para>configuration, logs, regenerating, <link xl:href="LustreMaintenance.html#50438199_marker-1305736">1</link></para>
- <para>configuration, more complex</para>
- <para></para>
- <para>failover, <link xl:href="LustreOperations.html#50438194_marker-1306313">1</link></para>
- <para>configuring</para>
- <para></para>
- <para>adaptive timeouts, <link xl:href="LustreProc.html#50438271_marker-1293381">1</link></para>
- <para></para>
- <para>root squash, <link xl:href="ManagingSecurity.html#50438221_marker-1294610">1</link></para>
- </section>
- <section remap="h3">
- <title>D<anchor xml:id="dbdoclet.50438146_D" xreflabel=""/></title>
- <para></para>
- <para>debug_mb, <link xl:href="LustreDebugging.html#50438274_marker-1295885">1</link></para>
- <para>debugging</para>
- <para></para>
- <para>adding debugging to source code, <link xl:href="LustreDebugging.html#50438274_marker-1296026">1</link></para>
- <para></para>
- <para>buffer, <link xl:href="LustreDebugging.html#50438274_marker-1295885">1</link></para>
- <para></para>
- <para>controlling the kernel debug log, <link xl:href="LustreDebugging.html#50438274_marker-1295955">1</link></para>
- <para></para>
- <para>finding Lustre UUID of an OST, <link xl:href="LustreDebugging.html#50438274_marker-1296007">1</link></para>
- <para></para>
- <para>finding memory leaks, <link xl:href="LustreDebugging.html#50438274_marker-1296153">1</link></para>
- <para></para>
- <para>looking at disk content, <link xl:href="LustreDebugging.html#50438274_marker-1295982">1</link></para>
- <para></para>
- <para>messages, <link xl:href="LustreDebugging.html#50438274_marker-1295746">1</link></para>
- <para></para>
- <para>Ptlrpc request history, <link xl:href="LustreDebugging.html#50438274_marker-1296099">1</link></para>
- <para></para>
- <para>sample lctl run, <link xl:href="LustreDebugging.html#50438274_marker-1295914">1</link></para>
- <para></para>
- <para>tracing lock traffic, <link xl:href="LustreDebugging.html#50438274_marker-1296017">1</link></para>
- <para></para>
- <para>troubleshooting with strace, <link xl:href="LustreDebugging.html#50438274_marker-1295969">1</link></para>
- <para></para>
- <para>directory statahead, using, <link xl:href="LustreProc.html#50438271_marker-1294292">1</link></para>
- <para></para>
- <para>downed routers, <link xl:href="ManagingLNET.html#50438203_marker-1288828">1</link></para>
- </section>
- <section remap="h3">
- <title>E<anchor xml:id="dbdoclet.50438146_E" xreflabel=""/></title>
- <para></para>
- <para>e2fsprogs, <link xl:href="InstallingLustre.html#50438261_marker-1292893">1</link></para>
- <para></para>
- <para>e2scan, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317398">1</link></para>
- <para></para>
- <para>Elan (Quadrics Elan), <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289889">1</link></para>
- <para></para>
- <para>environmental requirements, <link xl:href="InstallingLustre.html#50438261_marker-1292914">1</link></para>
- <para></para>
- <para>error messages, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291323">1</link></para>
- <para></para>
- <para>error numbers, <link xl:href="LustreTroubleshooting.html#50438198_marker-1296744">1</link></para>
- <para></para>
- <para>external journal, creating, <link xl:href="ConfiguringStorage.html#50438208_marker-1289927">1</link></para>
- </section>
- <section remap="h3">
- <title>F<anchor xml:id="dbdoclet.50438146_F" xreflabel=""/></title>
- <para></para>
- <para>failover, <link xl:href="UnderstandingFailover.html#50438253_marker-1304330">1</link></para>
- <para></para>
- <para>capabilities, <link xl:href="UnderstandingFailover.html#50438253_marker-1304336">1</link></para>
- <para></para>
- <para>configuration types, <link xl:href="UnderstandingFailover.html#50438253_marker-1304346">1</link></para>
- <para></para>
- <para>configuring, <link xl:href="LustreOperations.html#50438194_marker-1306313">1</link></para>
- <para></para>
- <para>failover and MMP, <link xl:href="ManagingFailover.html#50438213_marker-1301522">1</link></para>
- <para></para>
- <para>MDT (active/passive), <link xl:href="UnderstandingFailover.html#50438253_marker-1304373">1</link></para>
- <para></para>
- <para>OST (active/active), <link xl:href="UnderstandingFailover.html#50438253_marker-1304382">1</link></para>
- <para></para>
- <para>file formats, quotas, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290326">1</link></para>
- <para></para>
- <para>file readahead, using, <link xl:href="LustreProc.html#50438271_marker-1294292">1</link></para>
- <para></para>
- <para>file size, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290761">1</link></para>
- <para></para>
- <para>file striping, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1291832">1</link></para>
- <para></para>
- <para>file system size, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290753">1</link></para>
- <para></para>
- <para>filefrag command, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305920">1</link>, <link xl:href="LustreProgrammingInterfaces.html#50438291_marker-1294565">2</link></para>
- <para></para>
- <para>filename length, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290791">1</link></para>
- <para></para>
- <para>flock utility, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1295910">1</link></para>
- <para>free space management</para>
- <para></para>
- <para>adjusting weighting between free space and location, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1294033">1</link></para>
- <para></para>
- <para>round-robin allocator, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1293988">1</link></para>
- <para></para>
- <para>weighted allocator, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1294020">1</link></para>
- <para></para>
- <para>free space, managing, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1295520">1</link></para>
- </section>
- <section remap="h3">
- <title>H<anchor xml:id="dbdoclet.50438146_G" xreflabel=""/><anchor xml:id="dbdoclet.50438146_H" xreflabel=""/></title>
- <para></para>
- <para>HA software, <link xl:href="InstallingLustre.html#50438261_marker-1292902">1</link></para>
- <para></para>
- <para>handling full OSTs, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1295529">1</link></para>
- <para></para>
- <para>handling timeouts, <link xl:href="UserUtilities_HTML.html#50438206_marker-1306030">1</link></para>
- </section>
- <section remap="h3">
- <title>I<anchor xml:id="dbdoclet.50438146_I" xreflabel=""/></title>
- <para>I/O options</para>
- <para></para>
- <para>checksums, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1291974">1</link></para>
- <para></para>
- <para>I/O tunables, <link xl:href="LustreProc.html#50438271_marker-1290508">1</link></para>
- <para></para>
- <para>I/O, direct, performing, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1291962">1</link></para>
- <para></para>
- <para>inode number, OST, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1293235">1</link></para>
- <para></para>
- <para>inode size, MDT, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1293231">1</link></para>
- <para></para>
- <para>installing Lustre, environmental requirements, <link xl:href="InstallingLustre.html#50438261_marker-1292914">1</link></para>
- <para></para>
- <para>installing Lustre, HA software, <link xl:href="InstallingLustre.html#50438261_marker-1292902">1</link></para>
- <para></para>
- <para>installing Lustre, memory requirements, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1292212">1</link></para>
- <para></para>
- <para>installing Lustre, required tools / utilities, <link xl:href="InstallingLustre.html#50438261_marker-1292890">1</link></para>
- <para></para>
- <para>interoperability, <link xl:href="UpgradingLustre.html#50438205_marker-1293321">1</link></para>
- <para>interpreting</para>
- <para></para>
- <para>adaptive timeouts, <link xl:href="LustreProc.html#50438271_marker-1293383">1</link></para>
- </section>
- <section remap="h3">
- <title>K<anchor xml:id="dbdoclet.50438146_J" xreflabel=""/><anchor xml:id="dbdoclet.50438146_K" xreflabel=""/></title>
- <para></para>
- <para>key features, <link xl:href="UnderstandingLustre.html#50438250_marker-1293792">1</link></para>
- </section>
- <section remap="h3">
- <title>L<anchor xml:id="dbdoclet.50438146_L" xreflabel=""/></title>
- <para></para>
- <para>l_getidentity, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1318227">1</link>, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317750">2</link>, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317690">3</link></para>
- <para></para>
- <para>lctl, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1302539">1</link></para>
- <para></para>
- <para>setting parameters, <link xl:href="LustreOperations.html#50438194_marker-1305718">1</link></para>
- <para></para>
- <para>lfs command, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305209">1</link></para>
- <para></para>
- <para>lfsck command, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305843">1</link></para>
- <para></para>
- <para>ll_recover_lost_found_objs, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317865">1</link></para>
- <para></para>
- <para>llapi, <link xl:href="SettingLustreProperties_HTML.html#50438215_marker-1297700">1</link></para>
- <para></para>
- <para>llstat, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1318052">1</link></para>
- <para>LNET</para>
- <para></para>
- <para>starting, <link xl:href="ManagingLNET.html#50438203_marker-1287400">1</link></para>
- <para></para>
- <para>stopping, <link xl:href="ManagingLNET.html#50438203_marker-1288543">1</link></para>
- <para>LNET self-test</para>
- <para></para>
- <para>commands, <link xl:href="LNETSelfTest.html#50438223_marker-1298562">1</link></para>
- <para>Load balancing with InfiniBand</para>
- <para></para>
- <para>modprobe.conf, <link xl:href="ManagingLNET.html#50438203_marker-1290316">1</link></para>
- <para></para>
- <para>locking proc entries, <link xl:href="LustreProc.html#50438271_marker-1290874">1</link></para>
- <para></para>
- <para>lst, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1318121">1</link></para>
- <para>Lustre</para>
- <para></para>
- <para>administration, aborting recovery, <link xl:href="LustreMaintenance.html#50438199_marker-1299023">1</link></para>
- <para></para>
- <para>administration, changing a server NID, <link xl:href="LustreMaintenance.html#50438199_marker-1305737">1</link></para>
- <para></para>
- <para>administration, determining which machine is serving an OST, <link xl:href="LustreMaintenance.html#50438199_marker-1305741">1</link></para>
- <para></para>
- <para>administration, failout / failover mode for OSTs, <link xl:href="LustreOperations.html#50438194_marker-1298926">1</link></para>
- <para></para>
- <para>administration, finding nodes in the file system, <link xl:href="LustreMaintenance.html#50438199_marker-1298897">1</link></para>
- <para></para>
- <para>administration, mounting a server, <link xl:href="LustreOperations.html#50438194_marker-1298863">1</link></para>
- <para></para>
- <para>administration, mounting a server without Lustre service, <link xl:href="LustreMaintenance.html#50438199_marker-1298918">1</link></para>
- <para></para>
- <para>administration, regenerating Lustre configuration logs, <link xl:href="LustreMaintenance.html#50438199_marker-1305736">1</link></para>
- <para></para>
- <para>administration, removing and restoring OSTs, <link xl:href="LustreMaintenance.html#50438199_marker-1298976">1</link></para>
- <para></para>
- <para>administration, running multiple Lustre file systems, <link xl:href="LustreOperations.html#50438194_marker-1298939">1</link></para>
- <para></para>
- <para>administration, setting and retrieving Lustre parameters, <link xl:href="LustreOperations.html#50438194_marker-1302467">1</link></para>
- <para></para>
- <para>administration, starting Lustre, <link xl:href="LustreOperations.html#50438194_marker-1305696">1</link></para>
- <para></para>
- <para>administration, working with inactive OSTs, <link xl:href="LustreMaintenance.html#50438199_marker-1298888">1</link></para>
- <para></para>
- <para>adminstration, unmounting a server, <link xl:href="LustreOperations.html#50438194_marker-1298879">1</link></para>
- <para></para>
- <para>components, <link xl:href="UnderstandingLustre.html#50438250_marker-1293887">1</link></para>
- <para></para>
- <para>configuration example, <link xl:href="ConfiguringLustre.html#50438267_marker-1290955">1</link></para>
- <para></para>
- <para>COS, <link xl:href="LustreRecovery.html#50438268_marker-1292182">1</link></para>
- <para></para>
- <para>failover, <link xl:href="UnderstandingFailover.html#50438253_marker-1304330">1</link></para>
- <para></para>
- <para>installing, environmental requirements, <link xl:href="InstallingLustre.html#50438261_marker-1292914">1</link></para>
- <para></para>
- <para>installing, HA software, <link xl:href="InstallingLustre.html#50438261_marker-1292902">1</link></para>
- <para></para>
- <para>installing, memory requirements, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1292212">1</link></para>
- <para></para>
- <para>installing, required tools / utilities, <link xl:href="InstallingLustre.html#50438261_marker-1292890">1</link></para>
- <para></para>
- <para>interoperability, <link xl:href="UpgradingLustre.html#50438205_marker-1293321">1</link></para>
- <para></para>
- <para>key features, <link xl:href="UnderstandingLustre.html#50438250_marker-1293792">1</link></para>
- <para></para>
- <para>metadata replay, <link xl:href="LustreRecovery.html#50438268_marker-1292175">1</link></para>
- <para></para>
- <para>orphaned objects, <link xl:href="TroubleShootingRecovery.html#50438225_marker-1292187">1</link></para>
- <para></para>
- <para>parameters, reporting current, <link xl:href="LustreOperations.html#50438194_marker-1302474">1</link></para>
- <para></para>
- <para>parameters, setting and retrieving, <link xl:href="LustreOperations.html#50438194_marker-1302467">1</link></para>
- <para></para>
- <para>parameters, setting with lctl, <link xl:href="LustreOperations.html#50438194_marker-1305718">1</link></para>
- <para></para>
- <para>parameters, setting with mkfs.lustre, <link xl:href="LustreOperations.html#50438194_marker-1305722">1</link></para>
- <para></para>
- <para>parameters, setting with tunefs.lustre, <link xl:href="LustreOperations.html#50438194_marker-1305720">1</link></para>
- <para></para>
- <para>recovering from corruption in Lustre file system, <link xl:href="TroubleShootingRecovery.html#50438225_marker-1292186">1</link></para>
- <para></para>
- <para>recovering from errors on backing file system, <link xl:href="TroubleShootingRecovery.html#50438225_marker-1292184">1</link></para>
- <para></para>
- <para>recovery, client eviction, <link xl:href="LustreRecovery.html#50438268_marker-1292164">1</link></para>
- <para></para>
- <para>reply reconstruction, <link xl:href="LustreRecovery.html#50438268_marker-1292176">1</link></para>
- <para></para>
- <para>scaling, <link xl:href="ConfiguringLustre.html#50438267_marker-1292440">1</link></para>
- <para></para>
- <para>starting, <link xl:href="LustreOperations.html#50438194_marker-1305696">1</link></para>
- <para></para>
- <para>upgrading, 1.6.x to 1.8.x, <link xl:href="UpgradingLustre.html#50438205_marker-1294652">1</link></para>
- <para></para>
- <para>VBR, introduction, <link xl:href="LustreRecovery.html#50438268_marker-1288580">1</link></para>
- <para></para>
- <para>VBR, tips, <link xl:href="LustreRecovery.html#50438268_marker-1288584">1</link></para>
- <para></para>
- <para>VBR, working with, <link xl:href="LustreRecovery.html#50438268_marker-1288583">1</link></para>
- <para>Lustre I/O kit</para>
- <para></para>
- <para>downloading, <link xl:href="BenchmarkingTests.html#50438212_marker-1289913">1</link></para>
- <para></para>
- <para>obdfilter_survey tool, <link xl:href="BenchmarkingTests.html#50438212_marker-1289957">1</link></para>
- <para></para>
- <para>ost_survey tool, <link xl:href="BenchmarkingTests.html#50438212_marker-1290067">1</link></para>
- <para></para>
- <para>sgpdd_survey tool, <link xl:href="BenchmarkingTests.html#50438212_marker-1302844">1</link></para>
- <para></para>
- <para>Lustre Monitoring Tool (LMT), <link xl:href="LustreMonitoring.html#50438273_marker-1297386">1</link></para>
- <para></para>
- <para>lustre_rmmod.sh, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317379">1</link></para>
- <para></para>
- <para>lustre_rsync, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317224">1</link></para>
- </section>
- <section remap="h3">
- <title>M<anchor xml:id="dbdoclet.50438146_M" xreflabel=""/></title>
- <para>man1</para>
- <para></para>
- <para>filefrag, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305920">1</link>, <link xl:href="LustreProgrammingInterfaces.html#50438291_marker-1294565">2</link></para>
- <para></para>
- <para>lfs, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305209">1</link></para>
- <para></para>
- <para>lfsck, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305843">1</link></para>
- <para></para>
- <para>mount, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305992">1</link></para>
- <para>man2</para>
- <para></para>
- <para>user/group cache upcall, <link xl:href="LustreProgrammingInterfaces.html#50438291_marker-1293215">1</link></para>
- <para>man5</para>
- <para></para>
- <para>LNET options, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1293320">1</link></para>
- <para></para>
- <para>module options, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1293311">1</link></para>
- <para></para>
- <para>MX LND, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1295997">1</link></para>
- <para></para>
- <para>Portals LND (Linux), <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1293719">1</link></para>
- <para></para>
- <para>SOCKLND kernel TCP/IP LND, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1293448">1</link></para>
- <para>man8</para>
- <para></para>
- <para>application profiling utilties, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1313051">1</link></para>
- <para></para>
- <para>l_getidentity, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1318227">1</link>, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317750">2</link>, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317690">3</link></para>
- <para></para>
- <para>lctl, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1302539">1</link></para>
- <para></para>
- <para>ll_recover_lost_found_objs, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317865">1</link></para>
- <para></para>
- <para>llstat, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1318052">1</link></para>
- <para></para>
- <para>lst, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1318121">1</link></para>
- <para></para>
- <para>lustre_rsync, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317224">1</link></para>
- <para></para>
- <para>mkfs.lustre, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1316695">1</link></para>
- <para></para>
- <para>mount.lustre, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1310637">1</link></para>
- <para></para>
- <para>plot-llstat, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317989">1</link></para>
- <para></para>
- <para>proc statistics, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1313052">1</link></para>
- <para></para>
- <para>routerstat, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317921">1</link></para>
- <para></para>
- <para>system configuration utilities, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1313048">1</link></para>
- <para></para>
- <para>test/debug utilities, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1313053">1</link></para>
- <para></para>
- <para>tunefs.lustre, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317076">1</link></para>
- <para></para>
- <para>managing free space, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1295520">1</link></para>
- <para>maximum</para>
- <para></para>
- <para>file size, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290761">1</link></para>
- <para></para>
- <para>file system size, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290753">1</link></para>
- <para></para>
- <para>filename/pathname length, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290791">1</link></para>
- <para></para>
- <para>number of clients, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290746">1</link></para>
- <para></para>
- <para>number of files in a directory, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290770">1</link></para>
- <para></para>
- <para>number of open files, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290800">1</link></para>
- <para></para>
- <para>number of OSTs and MDTs, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290738">1</link></para>
- <para></para>
- <para>stripe count, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290714">1</link></para>
- <para></para>
- <para>stripe size, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290722">1</link></para>
- <para>mballoc</para>
- <para></para>
- <para>history, <link xl:href="LustreProc.html#50438271_marker-1297045">1</link></para>
- <para>mballoc3</para>
- <para></para>
- <para>tunables, <link xl:href="LustreProc.html#50438271_marker-1290795">1</link></para>
- <para>MDS</para>
- <para></para>
- <para>service thread count, <link xl:href="LustreTuning.html#50438272_marker-1293755">1</link></para>
- <para>MDT</para>
- <para></para>
- <para>failover, <link xl:href="UnderstandingFailover.html#50438253_marker-1304373">1</link></para>
- <para></para>
- <para>inode size, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1293231">1</link></para>
- <para></para>
- <para>memory requirements, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1292212">1</link></para>
- <para></para>
- <para>metadata replay, <link xl:href="LustreRecovery.html#50438268_marker-1292175">1</link></para>
- <para>minimum</para>
- <para></para>
- <para>stripe size, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290730">1</link></para>
- <para></para>
- <para>mkfs.lustre, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1316695">1</link></para>
- <para></para>
- <para>setting parameters, <link xl:href="LustreOperations.html#50438194_marker-1305722">1</link></para>
- <para>MMP</para>
- <para></para>
- <para>MMP and failover, <link xl:href="ManagingFailover.html#50438213_marker-1301522">1</link></para>
- <para></para>
- <para>modprobe.conf, <link xl:href="ManagingLNET.html#50438203_marker-1290316">1</link></para>
- <para>monitoring</para>
- <para></para>
- <para>changelogs, <link xl:href="LustreMonitoring.html#50438273_marker-1297383">1</link></para>
- <para></para>
- <para>CollectL, <link xl:href="LustreMonitoring.html#50438273_marker-1297391">1</link></para>
- <para></para>
- <para>Lustre Monitoring Tool, <link xl:href="LustreMonitoring.html#50438273_marker-1297386">1</link></para>
- <para></para>
- <para>mount command, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305992">1</link></para>
- <para></para>
- <para>mount.lustre, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1310637">1</link></para>
- <para></para>
- <para>MX LND, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1295997">1</link></para>
- </section>
- <section remap="h3">
- <title>N<anchor xml:id="dbdoclet.50438146_N" xreflabel=""/></title>
- <para>network</para>
- <para></para>
- <para>bonding, <link xl:href="SettingUpBonding.html#50438258_marker-1301678">1</link></para>
- <para>networks, supported</para>
- <para></para>
- <para>Elan (Quadrics Elan), <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289889">1</link></para>
- <para></para>
- <para>ra (RapidArray), <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289887">1</link></para>
- <para></para>
- <para>NID, server, changing, <link xl:href="LustreMaintenance.html#50438199_marker-1305737">1</link></para>
- <para></para>
- <para>number of clients, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290746">1</link></para>
- <para></para>
- <para>number of files in a directory, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290770">1</link></para>
- <para></para>
- <para>number of open files, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290800">1</link></para>
- </section>
- <section remap="h3">
- <title>O<anchor xml:id="dbdoclet.50438146_O" xreflabel=""/></title>
- <para></para>
- <para>obdfilter_survey tool, <link xl:href="BenchmarkingTests.html#50438212_marker-1289957">1</link></para>
- <para></para>
- <para>orphaned objects, working with, <link xl:href="TroubleShootingRecovery.html#50438225_marker-1292187">1</link></para>
- <para>OSS</para>
- <para></para>
- <para>memory, determining, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1292245">1</link></para>
- <para></para>
- <para>service thread count, <link xl:href="LustreTuning.html#50438272_marker-1294858">1</link></para>
- <para></para>
- <para>OSS read cache, <link xl:href="LustreProc.html#50438271_marker-1296183">1</link></para>
- <para>OST</para>
- <para></para>
- <para>failover, <link xl:href="UnderstandingFailover.html#50438253_marker-1304382">1</link></para>
- <para></para>
- <para>number of inodes, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1293235">1</link></para>
- <para></para>
- <para>removing and restoring, <link xl:href="LustreMaintenance.html#50438199_marker-1298976">1</link></para>
- <para></para>
- <para>OST block I/O stream, watching, <link xl:href="LustreProc.html#50438271_marker-1290631">1</link></para>
- <para></para>
- <para>OST pools, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1295531">1</link></para>
- <para></para>
- <para>OST, adding, <link xl:href="LustreMaintenance.html#50438199_marker-1306353">1</link></para>
- <para></para>
- <para>OST, determining which machine is serving, <link xl:href="LustreMaintenance.html#50438199_marker-1305741">1</link></para>
- <para></para>
- <para>ost_survey tool, <link xl:href="BenchmarkingTests.html#50438212_marker-1290067">1</link></para>
- <para>OSTs</para>
- <para></para>
- <para>adding, <link xl:href="ConfiguringLustre.html#50438267_marker-1292440">1</link></para>
- <para></para>
- <para>OSTs and MDTs, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290738">1</link></para>
- <para></para>
- <para>OSTs, full, handling, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1295529">1</link></para>
- </section>
- <section remap="h3">
- <title>P<anchor xml:id="dbdoclet.50438146_P" xreflabel=""/></title>
- <para></para>
- <para>parameters, setting with lctl, <link xl:href="LustreOperations.html#50438194_marker-1305718">1</link></para>
- <para></para>
- <para>parameters, setting with mkfs.lustre, <link xl:href="LustreOperations.html#50438194_marker-1305722">1</link></para>
- <para></para>
- <para>parameters, setting with tunefs.lustre, <link xl:href="LustreOperations.html#50438194_marker-1305720">1</link></para>
- <para></para>
- <para>pathname length, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290791">1</link></para>
- <para></para>
- <para>performing direct I/O, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1291962">1</link></para>
- <para></para>
- <para>Perl, <link xl:href="InstallingLustre.html#50438261_marker-1292899">1</link></para>
- <para></para>
- <para>plot-llstat, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317989">1</link></para>
- <para></para>
- <para>pools, OST, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1295531">1</link></para>
- <para>Portals LND</para>
- <para></para>
- <para>Linux, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1293719">1</link></para>
- <para>proc entries</para>
- <para></para>
- <para>free space distribution, <link xl:href="LustreProc.html#50438271_marker-1296165">1</link></para>
- <para></para>
- <para>LNET information, <link xl:href="LustreProc.html#50438271_marker-1296164">1</link></para>
- <para></para>
- <para>locating file systems and servers, <link xl:href="LustreProc.html#50438271_marker-1296151">1</link></para>
- <para></para>
- <para>locking, <link xl:href="LustreProc.html#50438271_marker-1290874">1</link></para>
- <para></para>
- <para>timeouts, <link xl:href="LustreProc.html#50438271_marker-1296153">1</link></para>
- </section>
- <section remap="h3">
- <title>Q<anchor xml:id="dbdoclet.50438146_Q" xreflabel=""/></title>
- <para></para>
- <para>Quadrics Elan, <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289889">1</link></para>
- <para></para>
- <para>quota limits, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290282">1</link></para>
- <para></para>
- <para>quota statistics, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290343">1</link></para>
- <para>quotas</para>
- <para></para>
- <para>administering, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290170">1</link></para>
- <para></para>
- <para>allocating, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290226">1</link></para>
- <para></para>
- <para>creating files, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290170">1</link></para>
- <para></para>
- <para>enabling, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290128">1</link></para>
- <para></para>
- <para>file formats, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290326">1</link></para>
- <para></para>
- <para>granted cache, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290272">1</link></para>
- <para></para>
- <para>known issues, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290269">1</link></para>
- <para></para>
- <para>limits, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290282">1</link></para>
- <para></para>
- <para>statistics, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290343">1</link></para>
- <para></para>
- <para>working with, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290118">1</link></para>
- </section>
- <section remap="h3">
- <title>R<anchor xml:id="dbdoclet.50438146_R" xreflabel=""/></title>
- <para></para>
- <para>ra (RapidArray), <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289887">1</link></para>
- <para>RAID</para>
- <para></para>
- <para>creating an external journal, <link xl:href="ConfiguringStorage.html#50438208_marker-1289927">1</link></para>
- <para></para>
- <para>formatting options, <link xl:href="ConfiguringStorage.html#50438208_marker-1291599">1</link></para>
- <para></para>
- <para>handling degraded arrays, <link xl:href="LustreOperations.html#50438194_marker-1307136">1</link></para>
- <para></para>
- <para>performance tradeoffs, <link xl:href="ConfiguringStorage.html#50438208_marker-1291593">1</link></para>
- <para></para>
- <para>reliability best practices, <link xl:href="ConfiguringStorage.html#50438208_marker-1291581">1</link></para>
- <para></para>
- <para>selecting storage for MDS or OSTs, <link xl:href="ConfiguringStorage.html#50438208_marker-1291567">1</link></para>
- <para></para>
- <para>RapidArray, <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289887">1</link></para>
- <para></para>
- <para>readahead, tuning, <link xl:href="LustreProc.html#50438271_marker-1295183">1</link></para>
- <para>recovery mode, failure types</para>
- <para></para>
- <para>client failure, <link xl:href="LustreRecovery.html#50438268_marker-1287394">1</link></para>
- <para></para>
- <para>MDS failure/failover, <link xl:href="LustreRecovery.html#50438268_marker-1287397">1</link></para>
- <para></para>
- <para>network partition, <link xl:href="LustreRecovery.html#50438268_marker-1289388">1</link></para>
- <para></para>
- <para>OST failure, <link xl:href="LustreRecovery.html#50438268_marker-1289240">1</link></para>
- <para></para>
- <para>recovery, aborting, <link xl:href="LustreMaintenance.html#50438199_marker-1299023">1</link></para>
- <para></para>
- <para>regenerating configuration logs, <link xl:href="LustreMaintenance.html#50438199_marker-1305736">1</link></para>
- <para></para>
- <para>reply reconstruction, <link xl:href="LustreRecovery.html#50438268_marker-1292176">1</link></para>
- <para></para>
- <para>reporting current Lustre parameters, <link xl:href="LustreOperations.html#50438194_marker-1302474">1</link></para>
- <para></para>
- <para>required tools / utilities, <link xl:href="InstallingLustre.html#50438261_marker-1292890">1</link></para>
- <para>root squash</para>
- <para></para>
- <para>configuring, <link xl:href="ManagingSecurity.html#50438221_marker-1294610">1</link></para>
- <para></para>
- <para>tips, <link xl:href="ManagingSecurity.html#50438221_marker-1296366">1</link></para>
- <para></para>
- <para>tuning, <link xl:href="ManagingSecurity.html#50438221_marker-1294611">1</link></para>
- <para></para>
- <para>root squash, using, <link xl:href="ManagingSecurity.html#50438221_marker-1294644">1</link></para>
- <para></para>
- <para>round-robin allocator, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1293988">1</link></para>
- <para></para>
- <para>routers, downed, <link xl:href="ManagingLNET.html#50438203_marker-1288828">1</link></para>
- <para></para>
- <para>routerstat, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317921">1</link></para>
- <para></para>
- <para>RPC stream tunables, <link xl:href="LustreProc.html#50438271_marker-1290514">1</link></para>
- <para></para>
- <para>RPC stream, watching, <link xl:href="LustreProc.html#50438271_marker-1290535">1</link></para>
- </section>
- <section remap="h3">
- <title>S<anchor xml:id="dbdoclet.50438146_S" xreflabel=""/></title>
- <para></para>
- <para>scaling Lustre, <link xl:href="ConfiguringLustre.html#50438267_marker-1292440">1</link></para>
- <para>server</para>
- <para></para>
- <para>mounting, <link xl:href="LustreOperations.html#50438194_marker-1298863">1</link></para>
- <para></para>
- <para>unmounting, <link xl:href="LustreOperations.html#50438194_marker-1298879">1</link></para>
- <para></para>
- <para>server NID, changing, <link xl:href="LustreMaintenance.html#50438199_marker-1305737">1</link></para>
- <para>service threads</para>
- <para></para>
- <para>MDS, <link xl:href="LustreTuning.html#50438272_marker-1293755">1</link></para>
- <para></para>
- <para>OSS, <link xl:href="LustreTuning.html#50438272_marker-1294858">1</link></para>
- <para>setting</para>
- <para></para>
- <para>SCSI I/O sizes, <link xl:href="LustreTroubleshooting.html#50438198_marker-1294800">1</link></para>
- <para></para>
- <para>setting and retrieving Lustre parameters, <link xl:href="LustreOperations.html#50438194_marker-1302467">1</link></para>
- <para></para>
- <para>sgpdd_survey tool, <link xl:href="BenchmarkingTests.html#50438212_marker-1302844">1</link></para>
- <para></para>
- <para>SOCKLND kernel TCP/IP LND, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1293448">1</link></para>
- <para>starting</para>
- <para></para>
- <para>LNET, <link xl:href="ManagingLNET.html#50438203_marker-1287400">1</link></para>
- <para></para>
- <para>statahead, tuning, <link xl:href="LustreProc.html#50438271_marker-1295184">1</link></para>
- <para>stopping</para>
- <para></para>
- <para>LNET, <link xl:href="ManagingLNET.html#50438203_marker-1288543">1</link></para>
- <para></para>
- <para>strace, <link xl:href="LustreDebugging.html#50438274_marker-1295969">1</link></para>
- <para></para>
- <para>stripe count, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290714">1</link></para>
- <para></para>
- <para>stripe size, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290722">1</link></para>
- <para></para>
- <para>stripe size, minimum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290730">1</link></para>
- <para>striping</para>
- <para></para>
- <para>managing free space, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1295520">1</link></para>
- <para></para>
- <para>size, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1291859">1</link></para>
- <para></para>
- <para>striping using llapi, <link xl:href="SettingLustreProperties_HTML.html#50438215_marker-1297700">1</link></para>
- <para>supported networks</para>
- <para></para>
- <para>Elan (Quadrics Elan), <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289889">1</link></para>
- <para></para>
- <para>ra (RapidArray), <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289887">1</link></para>
- </section>
- <section remap="h3">
- <title>T<anchor xml:id="dbdoclet.50438146_T" xreflabel=""/></title>
- <para></para>
- <para>timeouts, handling, <link xl:href="UserUtilities_HTML.html#50438206_marker-1306030">1</link></para>
- <para>troubleshooting</para>
- <para></para>
- <para>drawbacks in doing multi-client O_APPEND writes, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291520">1</link></para>
- <para></para>
- <para>erasing a file system, <link xl:href="LustreOperations.html#50438194_marker-1307237">1</link></para>
- <para></para>
- <para>error messages, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291323">1</link></para>
- <para></para>
- <para>error numbers, <link xl:href="LustreTroubleshooting.html#50438198_marker-1296744">1</link></para>
- <para></para>
- <para>handling timeouts on initial Lustre setup, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291503">1</link></para>
- <para>handling/debugging "bind</para>
- <para></para>
- <para>address already in use" error, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291446">1</link></para>
- <para>handling/debugging "Lustre Error</para>
- <para></para>
- <para>xxx went back in time", <link xl:href="LustreTroubleshooting.html#50438198_marker-1291509">1</link></para>
- <para></para>
- <para>handling/debugging error "28", <link xl:href="LustreTroubleshooting.html#50438198_marker-1291470">1</link></para>
- <para></para>
- <para>identifying a missing OST, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291366">1</link></para>
- <para></para>
- <para>improving Lustre performance when working with small files, <link xl:href="LustreTuning.html#50438272_marker-1295851">1</link></para>
- <para></para>
- <para>log message 'out of memory' on OST, <link xl:href="LustreTroubleshooting.html#50438198_marker-1292113">1</link></para>
- <para>Lustre Error</para>
- <para></para>
- <para>"slow start_page_write", <link xl:href="LustreTroubleshooting.html#50438198_marker-1291517">1</link></para>
- <para></para>
- <para>OST object missing or damaged, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291349">1</link></para>
- <para></para>
- <para>OSTs become read-only, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291361">1</link></para>
- <para></para>
- <para>reclaiming reserved disk space, <link xl:href="LustreOperations.html#50438194_marker-1307251">1</link></para>
- <para></para>
- <para>recovering from an unavailable OST, <link xl:href="TroubleShootingRecovery.html#50438225_marker-1292768">1</link></para>
- <para></para>
- <para>replacing an existing OST or MDS, <link xl:href="LustreOperations.html#50438194_marker-1307278">1</link></para>
- <para></para>
- <para>reporting a Lustre bug, <link xl:href="LustreTroubleshooting.html#50438198_marker-1296753">1</link></para>
- <para></para>
- <para>setting SCSI I/O sizes, <link xl:href="LustreTroubleshooting.html#50438198_marker-1294800">1</link></para>
- <para></para>
- <para>slowdown occurs during Lustre startup, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291921">1</link></para>
- <para></para>
- <para>triggering watchdog for PID NNN, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291480">1</link></para>
- <para>tunables</para>
- <para></para>
- <para>RPC stream, <link xl:href="LustreProc.html#50438271_marker-1290514">1</link></para>
- <para></para>
- <para>tunables, lockless, <link xl:href="LustreTuning.html#50438272_marker-1291703">1</link></para>
- <para></para>
- <para>tunefs.lustre, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317076">1</link></para>
- <para></para>
- <para>setting parameters, <link xl:href="LustreOperations.html#50438194_marker-1305720">1</link></para>
- <para>tuning</para>
- <para></para>
- <para>directory statahead, <link xl:href="LustreProc.html#50438271_marker-1295184">1</link></para>
- <para></para>
- <para>file readahead, <link xl:href="LustreProc.html#50438271_marker-1295183">1</link></para>
- <para></para>
- <para>lockless tunables, <link xl:href="LustreTuning.html#50438272_marker-1291703">1</link></para>
- <para></para>
- <para>MDS threads, <link xl:href="LustreTuning.html#50438272_marker-1293755">1</link></para>
- <para></para>
- <para>OSS threads, <link xl:href="LustreTuning.html#50438272_marker-1294858">1</link></para>
- <para></para>
- <para>root squash, <link xl:href="ManagingSecurity.html#50438221_marker-1294611">1</link></para>
- </section>
- <section remap="h3">
- <title>U<anchor xml:id="dbdoclet.50438146_U" xreflabel=""/></title>
- <para>upgrade</para>
- <para></para>
- <para>1.6.x to 1.8.x, <link xl:href="UpgradingLustre.html#50438205_marker-1294652">1</link></para>
- <para></para>
- <para>complete file system, <link xl:href="UpgradingLustre.html#50438205_marker-1293327">1</link></para>
- <para>utilities, third-party</para>
- <para></para>
- <para>e2fsprogs, <link xl:href="InstallingLustre.html#50438261_marker-1292893">1</link></para>
- <para></para>
- <para>Perl, <link xl:href="InstallingLustre.html#50438261_marker-1292899">1</link></para>
- </section>
- <section remap="h3">
- <title>V<anchor xml:id="dbdoclet.50438146_V" xreflabel=""/></title>
- <para></para>
- <para>VBR, introduction, <link xl:href="LustreRecovery.html#50438268_marker-1288580">1</link></para>
- <para></para>
- <para>VBR, tips, <link xl:href="LustreRecovery.html#50438268_marker-1288584">1</link></para>
- <para></para>
- <para>VBR, working with, <link xl:href="LustreRecovery.html#50438268_marker-1288583">1</link></para>
- <para></para>
- <para>Version-based recovery (VBR), <link xl:href="LustreRecovery.html#50438268_marker-1288580">1</link></para>
- </section>
- <section remap="h3">
- <title>W<anchor xml:id="dbdoclet.50438146_W" xreflabel=""/></title>
- <para></para>
- <para>weighted allocator, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1294020">1</link></para>
- <para></para>
- <para>weighting, adjusting between free space and location, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1294033">1</link></para>
- <!--
-Begin SiteCatalyst code version: G.5.
--->
- <!--
-End SiteCatalyst code version: G.5.
--->
- <informaltable frame="none">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
-
-
-
- <tbody>
- <row>
- <entry align="left"><para>Lustre 2.0 Operations Manual</para></entry>
- <entry align="right"><para>821-2076-10</para></entry>
- <entry align="right" valign="top"><para><link xl:href="index.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/toc01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/toc01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="Glossary.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/prev01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/prev01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/next02.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/next02.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject><link xl:href="ix.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/index01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/index01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link></para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><link xl:href=""/></para>
- <para><link xl:href="copyright.html">Copyright</link> © 2011, Oracle and/or its affiliates. All rights reserved.</para>
- </section>
- </section>
-</article>
+<index version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink">
+</index>
git://git.whamcloud.com / doc / manual.git / commitdiff