<info>
<title xml:id='backupandrestore.title'>Backing Up and Restoring a File System</title>
</info>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292628" xreflabel=""/>Lustre provides backups at the file system-level, device-level and file-level. This chapter describes how to backup and restore on Lustre, and includes the following sections:</para>
+ <para>Lustre provides backups at the file system-level, device-level and file-level. This chapter describes how to backup and restore on Lustre, and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438207_56395"/></para>
</listitem>
<section xml:id="dbdoclet.50438207_56395">
<title>17.1 Backing up a File System</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292651" xreflabel=""/>Backing up a complete file system gives you full control over the files to back up, and allows restoration of individual files as needed. File system-level backups are also the easiest to integrate into existing backup solutions.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292652" xreflabel=""/>File system backups are performed from a Lustre client (or many clients working parallel in different directories) rather than on individual server nodes; this is no different than backing up any other file system.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292653" xreflabel=""/>However, due to the large size of most Lustre file systems, it is not always possible to get a complete backup. We recommend that you back up subsets of a file system. This includes subdirectories of the entire file system, filesets for a single user, files incremented by date, and so on.</para>
+ <para>Backing up a complete file system gives you full control over the files to back up, and allows restoration of individual files as needed. File system-level backups are also the easiest to integrate into existing backup solutions.</para>
+ <para>File system backups are performed from a Lustre client (or many clients working parallel in different directories) rather than on individual server nodes; this is no different than backing up any other file system.</para>
+ <para>However, due to the large size of most Lustre file systems, it is not always possible to get a complete backup. We recommend that you back up subsets of a file system. This includes subdirectories of the entire file system, filesets for a single user, files incremented by date, and so on.</para>
<note><para>In order to allow Lustre to scale the filesystem namespace for future applications, Lustre 2.x internally uses a 128-bit file identifier for all files. To interface with user applications, Lustre presents 64-bit inode numbers for the stat(), fstat(), and readdir() system calls on 64-bit applications, and 32-bit inode numbers to 32-bit applications.</para><para> Some 32-bit applications accessing Lustre filesystems (on both 32-bit and 64-bit CPUs) may experience problems with the stat(), fstat() or readdir() system calls under certain circumstances, though the Lustre client should return 32-bit inode numbers to these applications.</para><para> In particular, if the Lustre filesystem is exported from a 64-bit client via NFS to a 32-bit client, the Linux NFS server will export 64-bit inode numbers to applications running on the NFS client. If the 32-bit applications are not compiled with Large File Support (LFS), then they return EOVERFLOW errors when accessing the Lustre files. To avoid this problem, Linux NFS clients can use the kernel command-line option "nfs.enable_ino64=0" in order to force the NFS client to export 32-bit inode numbers to the client.</para><para><emphasis role="bold">Workaround</emphasis>: We very strongly recommend that backups using tar(1) and other utilities that depend on the inode number to uniquely identify an inode to be run on 64-bit clients. The 128-bit Lustre file identifiers cannot be uniquely mapped to a 32-bit inode number, and as a result these utilities may operate incorrectly on 32-bit clients.</para></note>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438207_pgfId-1293842" xreflabel=""/>17.1.1 Lustre_rsync</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293843" xreflabel=""/>The lustre_rsync feature keeps the entire file system in sync on a backup by replicating the file system's changes to a second file system (the second file system need not be a Lustre file system, but it must be sufficiently large). Lustre_rsync uses Lustre changelogs to efficiently synchronize the file systems without having to scan (directory walk) the Lustre file system. This efficiency is critically important for large file systems, and distinguishes the Lustre lustre_rsync feature from other replication/backup solutions.</para>
+ <title>17.1.1 Lustre_rsync</title>
+ <para>The lustre_rsync feature keeps the entire file system in sync on a backup by replicating the file system's changes to a second file system (the second file system need not be a Lustre file system, but it must be sufficiently large). Lustre_rsync uses Lustre changelogs to efficiently synchronize the file systems without having to scan (directory walk) the Lustre file system. This efficiency is critically important for large file systems, and distinguishes the Lustre lustre_rsync feature from other replication/backup solutions.</para>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438207_pgfId-1293847" xreflabel=""/>17.1.1.1 Using Lustre_rsync</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293848" xreflabel=""/>The lustre_rsync feature works by periodically running lustre_rsync, a userspace program used to synchronize changes in the Lustre file system onto the target file system. The lustre_rsync utility keeps a status file, which enables it to be safely interrupted and restarted without losing synchronization between the file systems.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293849" xreflabel=""/>The first time that lustre_rsync is run, the user must specify a set of parameters for the program to use. These parameters are described in the following table and in <link xl:href="SystemConfigurationUtilities.html#50438219_63667">lustre_rsync</link>. On subsequent runs, these parameters are stored in the the status file, and only the name of the status file needs to be passed to lustre_rsync.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293853" xreflabel=""/>Before using lustre_rsync:</para>
+ <title>17.1.1.1 Using Lustre_rsync</title>
+ <para>The lustre_rsync feature works by periodically running lustre_rsync, a userspace program used to synchronize changes in the Lustre file system onto the target file system. The lustre_rsync utility keeps a status file, which enables it to be safely interrupted and restarted without losing synchronization between the file systems.</para>
+ <para>The first time that lustre_rsync is run, the user must specify a set of parameters for the program to use. These parameters are described in the following table and in <link xl:href="SystemConfigurationUtilities.html#50438219_63667">lustre_rsync</link>. On subsequent runs, these parameters are stored in the the status file, and only the name of the status file needs to be passed to lustre_rsync.</para>
+ <para>Before using lustre_rsync:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293857" xreflabel=""/> Register the changelog user. For details, see the <xref linkend='systemconfigurationutilities'/> (changelog_register) parameter in the <xref linkend='systemconfigurationutilities'/> (lctl).</para>
+ <para> Register the changelog user. For details, see the <xref linkend='systemconfigurationutilities'/> (changelog_register) parameter in the <xref linkend='systemconfigurationutilities'/> (lctl).</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293858" xreflabel=""/>- AND -</para>
+ <para>- AND -</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293859" xreflabel=""/> Verify that the Lustre file system (source) and the replica file system (target) are identical <emphasis>before</emphasis> registering the changelog user. If the file systems are discrepant, use a utility, e.g. regular rsync (not lustre_rsync), to make them identical.</para>
+ <para> Verify that the Lustre file system (source) and the replica file system (target) are identical <emphasis>before</emphasis> registering the changelog user. If the file systems are discrepant, use a utility, e.g. regular rsync (not lustre_rsync), to make them identical.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293930" xreflabel=""/>The lustre_rsync utility uses the following parameters:</para>
+ <para>The lustre_rsync utility uses the following parameters:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="3*"/>
<colspec colname="c2" colwidth="10*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438207_pgfId-1294813" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438207_pgfId-1294815" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294817" xreflabel=""/>--source=<src></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294819" xreflabel=""/>The path to the root of the Lustre file system (source) which will be synchronized. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
+ <entry><para> --source=<src></para></entry>
+ <entry><para> The path to the root of the Lustre file system (source) which will be synchronized. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294821" xreflabel=""/>--target=<tgt></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294823" xreflabel=""/>The path to the root where the source file system will be synchronized (target). This is a mandatory option if the status log created during a previous synchronization operation (--statuslog) is not specified. This option can be repeated if multiple synchronization targets are desired.</para></entry>
+ <entry><para> --target=<tgt></para></entry>
+ <entry><para> The path to the root where the source file system will be synchronized (target). This is a mandatory option if the status log created during a previous synchronization operation (--statuslog) is not specified. This option can be repeated if multiple synchronization targets are desired.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294825" xreflabel=""/>--mdt=<mdt></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294827" xreflabel=""/>The metadata device to be synchronized. A changelog user must be registered for this device. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
+ <entry><para> --mdt=<mdt></para></entry>
+ <entry><para> The metadata device to be synchronized. A changelog user must be registered for this device. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294829" xreflabel=""/>--user=<user id></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294831" xreflabel=""/>The changelog user ID for the specified MDT. To use lustre_rsync, the changelog user must be registered. For details, see the changelog_register parameter in <xref linkend='systemconfigurationutilities'/> (lctl). This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
+ <entry><para> --user=<user id></para></entry>
+ <entry><para> The changelog user ID for the specified MDT. To use lustre_rsync, the changelog user must be registered. For details, see the changelog_register parameter in <xref linkend='systemconfigurationutilities'/> (lctl). This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294833" xreflabel=""/>--statuslog=<log></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294835" xreflabel=""/>A log file to which synchronization status is saved. When the lustre_rsync utility starts, if the status log from a previous synchronization operation is specified, then the state is read from the log and otherwise mandatory --source, --target and --mdt options can be skipped. Specifying the --source, --target and/or --mdt options, in addition to the --statuslog option, causes the specified parameters in the status log to be overriden. Command line options take precedence over options in the status log.</para></entry>
+ <entry><para> --statuslog=<log></para></entry>
+ <entry><para> A log file to which synchronization status is saved. When the lustre_rsync utility starts, if the status log from a previous synchronization operation is specified, then the state is read from the log and otherwise mandatory --source, --target and --mdt options can be skipped. Specifying the --source, --target and/or --mdt options, in addition to the --statuslog option, causes the specified parameters in the status log to be overriden. Command line options take precedence over options in the status log.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294837" xreflabel=""/>--xattr <yes|no></para></entry>
- <entry><para>
<anchor xml:id="dbdoclet.50438207_pgfId-1294839" xreflabel=""/>Specifies whether extended attributes (xattrs) are synchronized or not. The default is to synchronize extended attributes.</para><para><anchor xml:id="dbdoclet.50438207_pgfId-1294840" xreflabel=""/><note><para>Disabling xattrs causes Lustre striping information not to be synchronized.</para></note></para></entry>
+ <entry><para> --xattr <yes|no></para></entry>
+ <entry><para>
Specifies whether extended attributes (xattrs) are synchronized or not. The default is to synchronize extended attributes.</para><para><note><para>Disabling xattrs causes Lustre striping information not to be synchronized.</para></note></para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294842" xreflabel=""/>--verbose</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294844" xreflabel=""/>Produces verbose output.</para></entry>
+ <entry><para> --verbose</para></entry>
+ <entry><para> Produces verbose output.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294846" xreflabel=""/>--dry-run</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294848" xreflabel=""/>Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para></entry>
+ <entry><para> --dry-run</para></entry>
+ <entry><para> Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294850" xreflabel=""/>--abort-on-err</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438207_pgfId-1294852" xreflabel=""/>Stops processing the lustre_rsync operation if an error occurs. The default is to continue the operation.</para></entry>
+ <entry><para> --abort-on-err</para></entry>
+ <entry><para> Stops processing the lustre_rsync operation if an error occurs. The default is to continue the operation.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438207_pgfId-1293938" xreflabel=""/>17.1.1.2 lustre_rsync Examples</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293939" xreflabel=""/>Sample lustre_rsync commands are listed below.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293940" xreflabel=""/>Register a changelog user for an MDT (e.g. lustre-MDT0000).</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1293941" xreflabel=""/># lctl --device lustre-MDT0000 changelog_register lustre-MDT0000 Registered\
+ <title>17.1.1.2 lustre_rsync Examples</title>
+ <para>Sample lustre_rsync commands are listed below.</para>
+ <para>Register a changelog user for an MDT (e.g. lustre-MDT0000).</para>
+ <screen># lctl --device lustre-MDT0000 changelog_register lustre-MDT0000 Registered\
changelog userid 'cl1'
</screen>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293942" xreflabel=""/>Synchronize a Lustre file system (/mnt/lustre) to a target file system (/mnt/target).</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1293943" xreflabel=""/>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target --mdt=lustre-MDT00\
+ <para>Synchronize a Lustre file system (/mnt/lustre) to a target file system (/mnt/target).</para>
+ <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target --mdt=lustre-MDT00\
00 --user=cl1 --statuslog sync.log --verbose
-<anchor xml:id="dbdoclet.50438207_pgfId-1293946" xreflabel=""/>Lustre filesystem: lustre
-<anchor xml:id="dbdoclet.50438207_pgfId-1293947" xreflabel=""/>MDT device: lustre-MDT0000
-<anchor xml:id="dbdoclet.50438207_pgfId-1293948" xreflabel=""/>Source: /mnt/lustre
-<anchor xml:id="dbdoclet.50438207_pgfId-1293949" xreflabel=""/>Target: /mnt/target
-<anchor xml:id="dbdoclet.50438207_pgfId-1293950" xreflabel=""/>Statuslog: sync.log
-<anchor xml:id="dbdoclet.50438207_pgfId-1293951" xreflabel=""/>Changelog registration: cl1
-<anchor xml:id="dbdoclet.50438207_pgfId-1293952" xreflabel=""/>Starting changelog record: 0
-<anchor xml:id="dbdoclet.50438207_pgfId-1293953" xreflabel=""/>Errors: 0
-<anchor xml:id="dbdoclet.50438207_pgfId-1293954" xreflabel=""/>lustre_rsync took 1 seconds
-<anchor xml:id="dbdoclet.50438207_pgfId-1293955" xreflabel=""/>Changelog records consumed: 22
-</screen>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293956" xreflabel=""/>After the file system undergoes changes, synchronize the changes onto the target file system. Only the statuslog name needs to be specified, as it has all the parameters passed earlier.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1293957" xreflabel=""/>$ lustre_rsync --statuslog sync.log --verbose
-<anchor xml:id="dbdoclet.50438207_pgfId-1293958" xreflabel=""/>Replicating Lustre filesystem: lustre
-<anchor xml:id="dbdoclet.50438207_pgfId-1293959" xreflabel=""/>MDT device: lustre-MDT0000
-<anchor xml:id="dbdoclet.50438207_pgfId-1293960" xreflabel=""/>Source: /mnt/lustre
-<anchor xml:id="dbdoclet.50438207_pgfId-1293961" xreflabel=""/>Target: /mnt/target
-<anchor xml:id="dbdoclet.50438207_pgfId-1293962" xreflabel=""/>Statuslog: sync.log
-<anchor xml:id="dbdoclet.50438207_pgfId-1293963" xreflabel=""/>Changelog registration: cl1
-<anchor xml:id="dbdoclet.50438207_pgfId-1293964" xreflabel=""/>Starting changelog record: 22
-<anchor xml:id="dbdoclet.50438207_pgfId-1293965" xreflabel=""/>Errors: 0
-<anchor xml:id="dbdoclet.50438207_pgfId-1293966" xreflabel=""/>lustre_rsync took 2 seconds
-<anchor xml:id="dbdoclet.50438207_pgfId-1293967" xreflabel=""/>Changelog records consumed: 42
-</screen>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293968" xreflabel=""/>To synchronize a Lustre file system (/mnt/lustre) to two target file systems (/mnt/target1 and /mnt/target2).</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1293969" xreflabel=""/>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target1 --target=/mnt/tar\
+Lustre filesystem: lustre
+MDT device: lustre-MDT0000
+Source: /mnt/lustre
+Target: /mnt/target
+Statuslog: sync.log
+Changelog registration: cl1
+Starting changelog record: 0
+Errors: 0
+lustre_rsync took 1 seconds
+Changelog records consumed: 22
+</screen>
+ <para>After the file system undergoes changes, synchronize the changes onto the target file system. Only the statuslog name needs to be specified, as it has all the parameters passed earlier.</para>
+ <screen>$ lustre_rsync --statuslog sync.log --verbose
+Replicating Lustre filesystem: lustre
+MDT device: lustre-MDT0000
+Source: /mnt/lustre
+Target: /mnt/target
+Statuslog: sync.log
+Changelog registration: cl1
+Starting changelog record: 22
+Errors: 0
+lustre_rsync took 2 seconds
+Changelog records consumed: 42
+</screen>
+ <para>To synchronize a Lustre file system (/mnt/lustre) to two target file systems (/mnt/target1 and /mnt/target2).</para>
+ <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target1 --target=/mnt/tar\
get2 \
-<anchor xml:id="dbdoclet.50438207_pgfId-1293971" xreflabel=""/> --mdt=lustre-MDT0000 --user=cl1
-<anchor xml:id="dbdoclet.50438207_pgfId-1293972" xreflabel=""/> --statuslog sync.log
+ --mdt=lustre-MDT0000 --user=cl1
+ --statuslog sync.log
</screen>
</section>
</section>
</section>
<section xml:id="dbdoclet.50438207_71633">
<title>17.2 Backing Up and Restoring an MDS or OST (Device Level)</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1294928" xreflabel=""/>In some cases, it is useful to do a full device-level backup of an individual device (MDT or OST), before replacing hardware, performing maintenance, etc. Doing full device-level backups ensures that all of the data and configuration files is preserved in the original state and is the easiest method of doing a backup. For the MDT file system, it may also be the fastest way to perform the backup and restore, since it can do large streaming read and write operations at the maximum bandwidth of the underlying devices.</para>
+ <para>In some cases, it is useful to do a full device-level backup of an individual device (MDT or OST), before replacing hardware, performing maintenance, etc. Doing full device-level backups ensures that all of the data and configuration files is preserved in the original state and is the easiest method of doing a backup. For the MDT file system, it may also be the fastest way to perform the backup and restore, since it can do large streaming read and write operations at the maximum bandwidth of the underlying devices.</para>
<note><para>Keeping an updated full backup of the MDT is especially important because a permanent failure of the MDT file system renders the much larger amount of data in all the OSTs largely inaccessible and unusable.</para></note>
<note><para>In Lustre 2.0 and 2.1 the only correct way to perform an MDT backup and restore is to do a device-level backup as is described in this section. The ability to do MDT file-level backups is not functional in these releases because of the inability to restore the Object Index (OI) file correctly (see bug 22741 for details).</para></note>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292658" xreflabel=""/>If hardware replacement is the reason for the backup or if a spare storage device is available, it is possible to do a raw copy of the MDT or OST from one block device to the other, as long as the new device is at least as large as the original device. To do this, run:</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292659" xreflabel=""/>dd if=/dev/{original} of=/dev/{new} bs=1M
+ <para>If hardware replacement is the reason for the backup or if a spare storage device is available, it is possible to do a raw copy of the MDT or OST from one block device to the other, as long as the new device is at least as large as the original device. To do this, run:</para>
+ <screen>dd if=/dev/{original} of=/dev/{new} bs=1M
</screen>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292660" xreflabel=""/>If hardware errors cause read problems on the original device, use the command below to allow as much data as possible to be read from the original device while skipping sections of the disk with errors:</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292661" xreflabel=""/>dd if=/dev/{original} of=/dev/{new} bs=4k conv=sync,noerror count={original\
+ <para>If hardware errors cause read problems on the original device, use the command below to allow as much data as possible to be read from the original device while skipping sections of the disk with errors:</para>
+ <screen>dd if=/dev/{original} of=/dev/{new} bs=4k conv=sync,noerror count={original\
size in 4kB blocks}
</screen>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292662" xreflabel=""/>Even in the face of hardware errors, the ldiskfs file system is very robust and it may be possible to recover the file system data after running e2fsck -f on the new device.</para>
+ <para>Even in the face of hardware errors, the ldiskfs file system is very robust and it may be possible to recover the file system data after running e2fsck -f on the new device.</para>
</section>
<section xml:id="dbdoclet.50438207_21638">
<title>17.3 Making a File-Level Backup of an OST File System</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1294992" xreflabel=""/>This procedure provides another way to backup or migrate the data of an OST at the file level, so that the unused space of the OST does not need to be backed up. Backing up a single OST device is not necessarily the best way to perform backups of the Lustre file system, since the files stored in the backup are not usable without metadata stored on the MDT. However, it is the preferred method for migration of OST devices, especially when it is desirable to reformat the underlying file system with different configuration options or to reduce fragmentation.</para>
+ <para>This procedure provides another way to backup or migrate the data of an OST at the file level, so that the unused space of the OST does not need to be backed up. Backing up a single OST device is not necessarily the best way to perform backups of the Lustre file system, since the files stored in the backup are not usable without metadata stored on the MDT. However, it is the preferred method for migration of OST devices, especially when it is desirable to reformat the underlying file system with different configuration options or to reduce fragmentation.</para>
<note><para>In Lustre 2.0 and 2.1 the only correct way to perform an MDT backup and restore is to do a device-level backup as is described in this section. The ability to do MDT file-level backups is not functional in these releases because of the inability to restore the Object Index (OI) file correctly (see bug 22741 for details).</para></note>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292698" xreflabel=""/>Make a mountpoint for the file system.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292699" xreflabel=""/>[oss]# mkdir -p /mnt/ost
+ <para>Make a mountpoint for the file system.</para>
+ <screen>[oss]# mkdir -p /mnt/ost
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292700" xreflabel=""/>Mount the file system.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292701" xreflabel=""/>[oss]# mount -t ldiskfs /<emphasis>dev</emphasis>/{ostdev} /mnt/ost
+ <para>Mount the file system.</para>
+ <screen>[oss]# mount -t ldiskfs /<emphasis>dev</emphasis>/{ostdev} /mnt/ost
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292702" xreflabel=""/>Change to the mountpoint being backed up.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292703" xreflabel=""/>[oss]# cd /mnt/ost
+ <para>Change to the mountpoint being backed up.</para>
+ <screen>[oss]# cd /mnt/ost
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292704" xreflabel=""/>Back up the extended attributes.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1295034" xreflabel=""/>[oss]# getfattr -R -d -m '.*' -e hex -P . > ea-$(date +%Y%m%d).bak
+ <para>Back up the extended attributes.</para>
+ <screen>[oss]# getfattr -R -d -m '.*' -e hex -P . > ea-$(date +%Y%m%d).bak
</screen>
<note><para>If the tar(1) command supports the --xattr option, the getfattr step may be unnecessary as long as it does a backup of the "trusted" attributes. However, completing this step is not harmful and can serve as an added safety measure.</para></note>
<note><para>In most distributions, the getfattr command is part of the "attr" package. If the getfattr command returns errors like Operation not supported, then the kernel does not correctly support EAs. Stop and use a different backup method.</para></note>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292707" xreflabel=""/>Verify that the ea-$date.bak file has properly backed up the EA data on the OST.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1295104" xreflabel=""/>Without this attribute data, the restore process may be missing extra data that can be very useful in case of later file system corruption. Look at this file with more or a text editor. Each object file should hae a corresponding item similar to this:</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1295089" xreflabel=""/>[oss]# file: O/0/d0/100992
-<anchor xml:id="dbdoclet.50438207_pgfId-1295090" xreflabel=""/>trusted.fid= \
-<anchor xml:id="dbdoclet.50438207_pgfId-1295210" xreflabel=""/>0x0d822200000000004a8a73e500000000808a0100000000000000000000000000
+ <para>Verify that the ea-$date.bak file has properly backed up the EA data on the OST.</para>
+ <para>Without this attribute data, the restore process may be missing extra data that can be very useful in case of later file system corruption. Look at this file with more or a text editor. Each object file should hae a corresponding item similar to this:</para>
+ <screen>[oss]# file: O/0/d0/100992
+trusted.fid= \
+0x0d822200000000004a8a73e500000000808a0100000000000000000000000000
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292710" xreflabel=""/>Back up all file system data.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292711" xreflabel=""/>[oss]# tar czvf {backup file}.tgz --sparse .
+ <para>Back up all file system data.</para>
+ <screen>[oss]# tar czvf {backup file}.tgz --sparse .
</screen>
<note><para>In Lustre 1.6.7 and later, the --sparse option reduces the size of the backup file. Be sure to use it so the tar command does not mistakenly create an archive full of zeros.</para></note>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292712" xreflabel=""/>Change directory out of the file system.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292713" xreflabel=""/>[oss]# cd -
+ <para>Change directory out of the file system.</para>
+ <screen>[oss]# cd -
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292714" xreflabel=""/>Unmount the file system.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292715" xreflabel=""/>[oss]# umount /mnt/ost
+ <para>Unmount the file system.</para>
+ <screen>[oss]# umount /mnt/ost
</screen>
<note><para>When restoring an OST backup on a different node as part of an OST migration, you also have to change server NIDs and use the --writeconf command to re-generate the configuration logs. See <xref linkend='lustremaintenance'/> (Changing a Server NID).</para></note>
</section>
<section xml:id="dbdoclet.50438207_22325">
<title>17.4 Restoring a File-Level Backup</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292721" xreflabel=""/>To restore data from a file-level backup, you need to format the device, restore the file data and then restore the EA data.</para>
+ <para>To restore data from a file-level backup, you need to format the device, restore the file data and then restore the EA data.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292722" xreflabel=""/>Format the new device.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1295250" xreflabel=""/>[oss]# mkfs.lustre --ost --index {<emphasis>OST index</emphasis>} {<emphasis>other options</emphasis>} newdev}
+ <para>Format the new device.</para>
+ <screen>[oss]# mkfs.lustre --ost --index {<emphasis>OST index</emphasis>} {<emphasis>other options</emphasis>} newdev}
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293049" xreflabel=""/>Mount the file system.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1293051" xreflabel=""/>[oss]# mount -t ldiskfs {<emphasis>newdev</emphasis>} /mnt/ost
+ <para>Mount the file system.</para>
+ <screen>[oss]# mount -t ldiskfs {<emphasis>newdev</emphasis>} /mnt/ost
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293053" xreflabel=""/>Change to the new file system mount point.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1293055" xreflabel=""/>[oss]# cd /mnt/ost
+ <para>Change to the new file system mount point.</para>
+ <screen>[oss]# cd /mnt/ost
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293057" xreflabel=""/>Restore the file system backup.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1293059" xreflabel=""/>[oss]# tar xzvpf {<emphasis>backup file</emphasis>} --sparse
+ <para>Restore the file system backup.</para>
+ <screen>[oss]# tar xzvpf {<emphasis>backup file</emphasis>} --sparse
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293061" xreflabel=""/>Restore the file system extended attributes.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1293063" xreflabel=""/>[oss]# setfattr --restore=ea-${date}.bak
+ <para>Restore the file system extended attributes.</para>
+ <screen>[oss]# setfattr --restore=ea-${date}.bak
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293041" xreflabel=""/>Verify that the extended attributes were restored.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1295296" xreflabel=""/>[oss]# getfattr -d -m ".*" -e hex O/0/d0/100992 trusted.fid= \
-<anchor xml:id="dbdoclet.50438207_pgfId-1295303" xreflabel=""/>0x0d822200000000004a8a73e500000000808a0100000000000000000000000000
+ <para>Verify that the extended attributes were restored.</para>
+ <screen>[oss]# getfattr -d -m ".*" -e hex O/0/d0/100992 trusted.fid= \
+0x0d822200000000004a8a73e500000000808a0100000000000000000000000000
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293161" xreflabel=""/>Change directory out of the file system.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1293163" xreflabel=""/>[oss]# cd -
+ <para>Change directory out of the file system.</para>
+ <screen>[oss]# cd -
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1293165" xreflabel=""/>Unmount the new file system.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1293167" xreflabel=""/>[oss]# umount /mnt/ost
+ <para>Unmount the new file system.</para>
+ <screen>[oss]# umount /mnt/ost
</screen>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1295335" xreflabel=""/>If the file system was used between the time the backup was made and when it was restored, then the lfsck tool (part of Lustre e2fsprogs) can optionally be run to ensure the file system is coherent. If all of the device file systems were backed up at the same time after the entire Lustre file system was stopped, this is not necessary. In either case, the file system should be immediately usable even if lfsck is not run, though there may be I/O errors reading from files that are present on the MDT but not the OSTs, and files that were created after the MDT backup will not be accessible/visible.</para>
+ <para>If the file system was used between the time the backup was made and when it was restored, then the lfsck tool (part of Lustre e2fsprogs) can optionally be run to ensure the file system is coherent. If all of the device file systems were backed up at the same time after the entire Lustre file system was stopped, this is not necessary. In either case, the file system should be immediately usable even if lfsck is not run, though there may be I/O errors reading from files that are present on the MDT but not the OSTs, and files that were created after the MDT backup will not be accessible/visible.</para>
</section>
<section xml:id="dbdoclet.50438207_31553">
<title>17.5 Using LVM Snapshots with Lustre</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292749" xreflabel=""/>If you want to perform disk-based backups (because, for example, access to the backup system needs to be as fast as to the primary Lustre file system), you can use the Linux LVM snapshot tool to maintain multiple, incremental file system backups.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292750" xreflabel=""/>Because LVM snapshots cost CPU cycles as new files are written, taking snapshots of the main Lustre file system will probably result in unacceptable performance losses. You should create a new, backup Lustre file system and periodically (e.g., nightly) back up new/changed files to it. Periodic snapshots can be taken of this backup file system to create a series of "full" backups.</para>
+ <para>If you want to perform disk-based backups (because, for example, access to the backup system needs to be as fast as to the primary Lustre file system), you can use the Linux LVM snapshot tool to maintain multiple, incremental file system backups.</para>
+ <para>Because LVM snapshots cost CPU cycles as new files are written, taking snapshots of the main Lustre file system will probably result in unacceptable performance losses. You should create a new, backup Lustre file system and periodically (e.g., nightly) back up new/changed files to it. Periodic snapshots can be taken of this backup file system to create a series of "full" backups.</para>
<note><para>Creating an LVM snapshot is not as reliable as making a separate backup, because the LVM snapshot shares the same disks as the primary MDT device, and depends on the primary MDT device for much of its data. If the primary MDT device becomes corrupted, this may result in the snapshot being corrupted.</para></note>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438207_pgfId-1292752" xreflabel=""/>17.5.1 Creating an LVM-based Backup File System</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292753" xreflabel=""/>Use this procedure to create a backup Lustre file system for use with the LVM snapshot mechanism.</para>
+ <title>17.5.1 Creating an LVM-based Backup File System</title>
+ <para>Use this procedure to create a backup Lustre file system for use with the LVM snapshot mechanism.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292754" xreflabel=""/>Create LVM volumes for the MDT and OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292755" xreflabel=""/>Create LVM devices for your MDT and OST targets. Make sure not to use the entire disk for the targets; save some room for the snapshots. The snapshots start out as 0 size, but grow as you make changes to the current file system. If you expect to change 20% of the file system between backups, the most recent snapshot will be 20% of the target size, the next older one will be 40%, etc. Here is an example:</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292756" xreflabel=""/>cfs21:~# pvcreate /dev/sda1
-<anchor xml:id="dbdoclet.50438207_pgfId-1292757" xreflabel=""/> Physical volume "/dev/sda1" successfully created
-<anchor xml:id="dbdoclet.50438207_pgfId-1292758" xreflabel=""/>cfs21:~# vgcreate volgroup /dev/sda1
-<anchor xml:id="dbdoclet.50438207_pgfId-1292759" xreflabel=""/> Volume group "volgroup" successfully created
-<anchor xml:id="dbdoclet.50438207_pgfId-1292760" xreflabel=""/>cfs21:~# lvcreate -L200M -nMDT volgroup
-<anchor xml:id="dbdoclet.50438207_pgfId-1292761" xreflabel=""/> Logical volume "MDT" created
-<anchor xml:id="dbdoclet.50438207_pgfId-1292762" xreflabel=""/>cfs21:~# lvcreate -L200M -nOST0 volgroup
-<anchor xml:id="dbdoclet.50438207_pgfId-1292763" xreflabel=""/> Logical volume "OST0" created
-<anchor xml:id="dbdoclet.50438207_pgfId-1292764" xreflabel=""/>cfs21:~# lvscan
-<anchor xml:id="dbdoclet.50438207_pgfId-1292765" xreflabel=""/> ACTIVE '/dev/volgroup/MDT' [200.00 MB] inherit
-<anchor xml:id="dbdoclet.50438207_pgfId-1292766" xreflabel=""/> ACTIVE '/dev/volgroup/OST0' [200.00 MB] inherit
+ <para>Create LVM volumes for the MDT and OSTs.</para>
+ <para>Create LVM devices for your MDT and OST targets. Make sure not to use the entire disk for the targets; save some room for the snapshots. The snapshots start out as 0 size, but grow as you make changes to the current file system. If you expect to change 20% of the file system between backups, the most recent snapshot will be 20% of the target size, the next older one will be 40%, etc. Here is an example:</para>
+ <screen>cfs21:~# pvcreate /dev/sda1
+ Physical volume "/dev/sda1" successfully created
+cfs21:~# vgcreate volgroup /dev/sda1
+ Volume group "volgroup" successfully created
+cfs21:~# lvcreate -L200M -nMDT volgroup
+ Logical volume "MDT" created
+cfs21:~# lvcreate -L200M -nOST0 volgroup
+ Logical volume "OST0" created
+cfs21:~# lvscan
+ ACTIVE '/dev/volgroup/MDT' [200.00 MB] inherit
+ ACTIVE '/dev/volgroup/OST0' [200.00 MB] inherit
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292767" xreflabel=""/>Format the LVM volumes as Lustre targets.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292768" xreflabel=""/>In this example, the backup file system is called 'main' and designates the current, most up-to-date backup.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292769" xreflabel=""/>cfs21:~# mkfs.lustre --mdt --fsname=main /dev/volgroup/MDT
-<anchor xml:id="dbdoclet.50438207_pgfId-1292770" xreflabel=""/> No management node specified, adding MGS to this MDT.
-<anchor xml:id="dbdoclet.50438207_pgfId-1292771" xreflabel=""/> Permanent disk data:
-<anchor xml:id="dbdoclet.50438207_pgfId-1292772" xreflabel=""/> Target: main-MDTffff
-<anchor xml:id="dbdoclet.50438207_pgfId-1292773" xreflabel=""/> Index: unassigned
-<anchor xml:id="dbdoclet.50438207_pgfId-1292774" xreflabel=""/> Lustre FS: main
-<anchor xml:id="dbdoclet.50438207_pgfId-1292775" xreflabel=""/> Mount type: ldiskfs
-<anchor xml:id="dbdoclet.50438207_pgfId-1292776" xreflabel=""/> Flags: 0x75
-<anchor xml:id="dbdoclet.50438207_pgfId-1292777" xreflabel=""/> (MDT MGS needs_index first_time update )
-<anchor xml:id="dbdoclet.50438207_pgfId-1292778" xreflabel=""/> Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
-<anchor xml:id="dbdoclet.50438207_pgfId-1292779" xreflabel=""/> Parameters:
-<anchor xml:id="dbdoclet.50438207_pgfId-1292780" xreflabel=""/>checking for existing Lustre data
-<anchor xml:id="dbdoclet.50438207_pgfId-1292781" xreflabel=""/> device size = 200MB
-<anchor xml:id="dbdoclet.50438207_pgfId-1292782" xreflabel=""/> formatting backing filesystem ldiskfs on /dev/volgroup/MDT
-<anchor xml:id="dbdoclet.50438207_pgfId-1292783" xreflabel=""/> target name main-MDTffff
-<anchor xml:id="dbdoclet.50438207_pgfId-1292784" xreflabel=""/> 4k blocks 0
-<anchor xml:id="dbdoclet.50438207_pgfId-1292785" xreflabel=""/> options -i 4096 -I 512 -q -O dir_index -F
-<anchor xml:id="dbdoclet.50438207_pgfId-1292786" xreflabel=""/> mkfs_cmd = mkfs.ext2 -j -b 4096 -L main-MDTffff -i 4096 -I 512 -q -O dir_\
+ <para>Format the LVM volumes as Lustre targets.</para>
+ <para>In this example, the backup file system is called 'main' and designates the current, most up-to-date backup.</para>
+ <screen>cfs21:~# mkfs.lustre --mdt --fsname=main /dev/volgroup/MDT
+ No management node specified, adding MGS to this MDT.
+ Permanent disk data:
+ Target: main-MDTffff
+ Index: unassigned
+ Lustre FS: main
+ Mount type: ldiskfs
+ Flags: 0x75
+ (MDT MGS needs_index first_time update )
+ Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
+ Parameters:
+checking for existing Lustre data
+ device size = 200MB
+ formatting backing filesystem ldiskfs on /dev/volgroup/MDT
+ target name main-MDTffff
+ 4k blocks 0
+ options -i 4096 -I 512 -q -O dir_index -F
+ mkfs_cmd = mkfs.ext2 -j -b 4096 -L main-MDTffff -i 4096 -I 512 -q -O dir_\
index -F /dev/volgroup/MDT
-<anchor xml:id="dbdoclet.50438207_pgfId-1292787" xreflabel=""/> Writing CONFIGS/mountdata
-<anchor xml:id="dbdoclet.50438207_pgfId-1292788" xreflabel=""/>cfs21:~# mkfs.lustre --ost --mgsnode=cfs21 --fsname=main /dev/volgroup/OST0
-<anchor xml:id="dbdoclet.50438207_pgfId-1292789" xreflabel=""/> Permanent disk data:
-<anchor xml:id="dbdoclet.50438207_pgfId-1292790" xreflabel=""/> Target: main-OSTffff
-<anchor xml:id="dbdoclet.50438207_pgfId-1292791" xreflabel=""/>Index: unassigned
-<anchor xml:id="dbdoclet.50438207_pgfId-1292792" xreflabel=""/> Lustre FS: main
-<anchor xml:id="dbdoclet.50438207_pgfId-1292793" xreflabel=""/> Mount type: ldiskfs
-<anchor xml:id="dbdoclet.50438207_pgfId-1292794" xreflabel=""/> Flags: 0x72
-<anchor xml:id="dbdoclet.50438207_pgfId-1292795" xreflabel=""/> (OST needs_index first_time update )
-<anchor xml:id="dbdoclet.50438207_pgfId-1292796" xreflabel=""/> Persistent mount opts: errors=remount-ro,extents,mballoc
-<anchor xml:id="dbdoclet.50438207_pgfId-1292797" xreflabel=""/> Parameters: mgsnode=192.168.0.21@tcp
-<anchor xml:id="dbdoclet.50438207_pgfId-1292798" xreflabel=""/>checking for existing Lustre data
-<anchor xml:id="dbdoclet.50438207_pgfId-1292799" xreflabel=""/> device size = 200MB
-<anchor xml:id="dbdoclet.50438207_pgfId-1292800" xreflabel=""/> formatting backing filesystem ldiskfs on /dev/volgroup/OST0
-<anchor xml:id="dbdoclet.50438207_pgfId-1292801" xreflabel=""/> target name main-OSTffff
-<anchor xml:id="dbdoclet.50438207_pgfId-1292802" xreflabel=""/> 4k blocks 0
-<anchor xml:id="dbdoclet.50438207_pgfId-1292803" xreflabel=""/> options -I 256 -q -O dir_index -F
-<anchor xml:id="dbdoclet.50438207_pgfId-1292804" xreflabel=""/> mkfs_cmd = mkfs.ext2 -j -b 4096 -L main-OSTffff -I 256 -q -O dir_index -F\
+ Writing CONFIGS/mountdata
+cfs21:~# mkfs.lustre --ost --mgsnode=cfs21 --fsname=main /dev/volgroup/OST0
+ Permanent disk data:
+ Target: main-OSTffff
+Index: unassigned
+ Lustre FS: main
+ Mount type: ldiskfs
+ Flags: 0x72
+ (OST needs_index first_time update )
+ Persistent mount opts: errors=remount-ro,extents,mballoc
+ Parameters: mgsnode=192.168.0.21@tcp
+checking for existing Lustre data
+ device size = 200MB
+ formatting backing filesystem ldiskfs on /dev/volgroup/OST0
+ target name main-OSTffff
+ 4k blocks 0
+ options -I 256 -q -O dir_index -F
+ mkfs_cmd = mkfs.ext2 -j -b 4096 -L main-OSTffff -I 256 -q -O dir_index -F\
/dev/ volgroup/OST0
-<anchor xml:id="dbdoclet.50438207_pgfId-1292805" xreflabel=""/> Writing CONFIGS/mountdata
-<anchor xml:id="dbdoclet.50438207_pgfId-1292806" xreflabel=""/>cfs21:~# mount -t lustre /dev/volgroup/MDT /mnt/mdt
-<anchor xml:id="dbdoclet.50438207_pgfId-1292807" xreflabel=""/>cfs21:~# mount -t lustre /dev/volgroup/OST0 /mnt/ost
-<anchor xml:id="dbdoclet.50438207_pgfId-1292808" xreflabel=""/>cfs21:~# mount -t lustre cfs21:/main /mnt/main
+ Writing CONFIGS/mountdata
+cfs21:~# mount -t lustre /dev/volgroup/MDT /mnt/mdt
+cfs21:~# mount -t lustre /dev/volgroup/OST0 /mnt/ost
+cfs21:~# mount -t lustre cfs21:/main /mnt/main
</screen>
</listitem></orderedlist>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438207_pgfId-1292809" xreflabel=""/>17.5.2 Backing up New/Changed Files to the Backup File System</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292810" xreflabel=""/>At periodic intervals e.g., nightly, back up new and changed files to the LVM-based backup file system.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292811" xreflabel=""/>cfs21:~# cp /etc/passwd /mnt/main
-<anchor xml:id="dbdoclet.50438207_pgfId-1292812" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438207_pgfId-1292813" xreflabel=""/>cfs21:~# cp /etc/fstab /mnt/main
-<anchor xml:id="dbdoclet.50438207_pgfId-1292814" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438207_pgfId-1292815" xreflabel=""/>cfs21:~# ls /mnt/main
-<anchor xml:id="dbdoclet.50438207_pgfId-1292816" xreflabel=""/>fstab passwd
+ <title>17.5.2 Backing up New/Changed Files to the Backup File System</title>
+ <para>At periodic intervals e.g., nightly, back up new and changed files to the LVM-based backup file system.</para>
+ <screen>cfs21:~# cp /etc/passwd /mnt/main
+
+cfs21:~# cp /etc/fstab /mnt/main
+
+cfs21:~# ls /mnt/main
+fstab passwd
</screen>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438207_pgfId-1292817" xreflabel=""/>17.5.3 Creating Snapshot Volumes</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292818" xreflabel=""/>Whenever you want to make a "checkpoint" of the main Lustre file system, create LVM snapshots of all target MDT and OSTs in the LVM-based backup file system. You must decide the maximum size of a snapshot ahead of time, although you can dynamically change this later. The size of a daily snapshot is dependent on the amount of data changed daily in the main Lustre file system. It is likely that a two-day old snapshot will be twice as big as a one-day old snapshot.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292819" xreflabel=""/>You can create as many snapshots as you have room for in the volume group. If necessary, you can dynamically add disks to the volume group.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292820" xreflabel=""/>The snapshots of the target MDT and OSTs should be taken at the same point in time. Make sure that the cronjob updating the backup file system is not running, since that is the only thing writing to the disks. Here is an example:</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292821" xreflabel=""/>cfs21:~# modprobe dm-snapshot
-<anchor xml:id="dbdoclet.50438207_pgfId-1292822" xreflabel=""/>cfs21:~# lvcreate -L50M -s -n MDTb1 /dev/volgroup/MDT
-<anchor xml:id="dbdoclet.50438207_pgfId-1292823" xreflabel=""/> Rounding up size to full physical extent 52.00 MB
-<anchor xml:id="dbdoclet.50438207_pgfId-1292824" xreflabel=""/> Logical volume "MDTb1" created
-<anchor xml:id="dbdoclet.50438207_pgfId-1292825" xreflabel=""/>cfs21:~# lvcreate -L50M -s -n OSTb1 /dev/volgroup/OST0
-<anchor xml:id="dbdoclet.50438207_pgfId-1292826" xreflabel=""/> Rounding up size to full physical extent 52.00 MB
-<anchor xml:id="dbdoclet.50438207_pgfId-1292827" xreflabel=""/> Logical volume "OSTb1" created
-</screen>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292828" xreflabel=""/>After the snapshots are taken, you can continue to back up new/changed files to "main". The snapshots will not contain the new files.</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292829" xreflabel=""/>cfs21:~# cp /etc/termcap /mnt/main
-<anchor xml:id="dbdoclet.50438207_pgfId-1292830" xreflabel=""/>cfs21:~# ls /mnt/main
-<anchor xml:id="dbdoclet.50438207_pgfId-1292831" xreflabel=""/>fstab passwd termcap
+ <title>17.5.3 Creating Snapshot Volumes</title>
+ <para>Whenever you want to make a "checkpoint" of the main Lustre file system, create LVM snapshots of all target MDT and OSTs in the LVM-based backup file system. You must decide the maximum size of a snapshot ahead of time, although you can dynamically change this later. The size of a daily snapshot is dependent on the amount of data changed daily in the main Lustre file system. It is likely that a two-day old snapshot will be twice as big as a one-day old snapshot.</para>
+ <para>You can create as many snapshots as you have room for in the volume group. If necessary, you can dynamically add disks to the volume group.</para>
+ <para>The snapshots of the target MDT and OSTs should be taken at the same point in time. Make sure that the cronjob updating the backup file system is not running, since that is the only thing writing to the disks. Here is an example:</para>
+ <screen>cfs21:~# modprobe dm-snapshot
+cfs21:~# lvcreate -L50M -s -n MDTb1 /dev/volgroup/MDT
+ Rounding up size to full physical extent 52.00 MB
+ Logical volume "MDTb1" created
+cfs21:~# lvcreate -L50M -s -n OSTb1 /dev/volgroup/OST0
+ Rounding up size to full physical extent 52.00 MB
+ Logical volume "OSTb1" created
+</screen>
+ <para>After the snapshots are taken, you can continue to back up new/changed files to "main". The snapshots will not contain the new files.</para>
+ <screen>cfs21:~# cp /etc/termcap /mnt/main
+cfs21:~# ls /mnt/main
+fstab passwd termcap
</screen>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438207_pgfId-1292832" xreflabel=""/>17.5.4 Restoring the File System From a Snapshot</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292833" xreflabel=""/>Use this procedure to restore the file system from an LVM snapshot.</para>
+ <title>17.5.4 Restoring the File System From a Snapshot</title>
+ <para>Use this procedure to restore the file system from an LVM snapshot.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292834" xreflabel=""/>Rename the LVM snapshot.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292835" xreflabel=""/>Rename the file system snapshot from "main" to "back" so you can mount it without unmounting "main". This is recommended, but not required. Use the --reformat flag to tunefs.lustre to force the name change. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292836" xreflabel=""/>cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/volgroup/M\
+ <para>Rename the LVM snapshot.</para>
+ <para>Rename the file system snapshot from "main" to "back" so you can mount it without unmounting "main". This is recommended, but not required. Use the --reformat flag to tunefs.lustre to force the name change. For example:</para>
+ <screen>cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/volgroup/M\
DTb1
-<anchor xml:id="dbdoclet.50438207_pgfId-1292837" xreflabel=""/> checking for existing Lustre data
-<anchor xml:id="dbdoclet.50438207_pgfId-1292838" xreflabel=""/> found Lustre data
-<anchor xml:id="dbdoclet.50438207_pgfId-1292839" xreflabel=""/> Reading CONFIGS/mountdata
-<anchor xml:id="dbdoclet.50438207_pgfId-1292840" xreflabel=""/>Read previous values:
-<anchor xml:id="dbdoclet.50438207_pgfId-1292841" xreflabel=""/> Target: main-MDT0000
-<anchor xml:id="dbdoclet.50438207_pgfId-1292842" xreflabel=""/> Index: 0
-<anchor xml:id="dbdoclet.50438207_pgfId-1292843" xreflabel=""/> Lustre FS: main
-<anchor xml:id="dbdoclet.50438207_pgfId-1292844" xreflabel=""/> Mount type: ldiskfs
-<anchor xml:id="dbdoclet.50438207_pgfId-1292845" xreflabel=""/> Flags: 0x5
-<anchor xml:id="dbdoclet.50438207_pgfId-1292846" xreflabel=""/> (MDT MGS )
-<anchor xml:id="dbdoclet.50438207_pgfId-1292847" xreflabel=""/> Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
-<anchor xml:id="dbdoclet.50438207_pgfId-1292848" xreflabel=""/> Parameters:
-<anchor xml:id="dbdoclet.50438207_pgfId-1292849" xreflabel=""/>Permanent disk data:
-<anchor xml:id="dbdoclet.50438207_pgfId-1292850" xreflabel=""/> Target: back-MDT0000
-<anchor xml:id="dbdoclet.50438207_pgfId-1292851" xreflabel=""/> Index: 0
-<anchor xml:id="dbdoclet.50438207_pgfId-1292852" xreflabel=""/> Lustre FS: back
-<anchor xml:id="dbdoclet.50438207_pgfId-1292853" xreflabel=""/> Mount type: ldiskfs
-<anchor xml:id="dbdoclet.50438207_pgfId-1292854" xreflabel=""/> Flags: 0x105
-<anchor xml:id="dbdoclet.50438207_pgfId-1292855" xreflabel=""/> (MDT MGS writeconf )
-<anchor xml:id="dbdoclet.50438207_pgfId-1292856" xreflabel=""/> Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
-<anchor xml:id="dbdoclet.50438207_pgfId-1292857" xreflabel=""/> Parameters:
-<anchor xml:id="dbdoclet.50438207_pgfId-1292858" xreflabel=""/>Writing CONFIGS/mountdata
-<anchor xml:id="dbdoclet.50438207_pgfId-1292859" xreflabel=""/>cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/volgroup/O\
+ checking for existing Lustre data
+ found Lustre data
+ Reading CONFIGS/mountdata
+Read previous values:
+ Target: main-MDT0000
+ Index: 0
+ Lustre FS: main
+ Mount type: ldiskfs
+ Flags: 0x5
+ (MDT MGS )
+ Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
+ Parameters:
+Permanent disk data:
+ Target: back-MDT0000
+ Index: 0
+ Lustre FS: back
+ Mount type: ldiskfs
+ Flags: 0x105
+ (MDT MGS writeconf )
+ Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
+ Parameters:
+Writing CONFIGS/mountdata
+cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/volgroup/O\
STb1
-<anchor xml:id="dbdoclet.50438207_pgfId-1292860" xreflabel=""/> checking for existing Lustre data
-<anchor xml:id="dbdoclet.50438207_pgfId-1292861" xreflabel=""/> found Lustre data
-<anchor xml:id="dbdoclet.50438207_pgfId-1292862" xreflabel=""/> Reading CONFIGS/mountdata
-<anchor xml:id="dbdoclet.50438207_pgfId-1292863" xreflabel=""/>Read previous values:
-<anchor xml:id="dbdoclet.50438207_pgfId-1292864" xreflabel=""/> Target: main-OST0000
-<anchor xml:id="dbdoclet.50438207_pgfId-1292865" xreflabel=""/> Index: 0
-<anchor xml:id="dbdoclet.50438207_pgfId-1292866" xreflabel=""/> Lustre FS: main
-<anchor xml:id="dbdoclet.50438207_pgfId-1292867" xreflabel=""/> Mount type: ldiskfs
-<anchor xml:id="dbdoclet.50438207_pgfId-1292868" xreflabel=""/> Flags: 0x2
-<anchor xml:id="dbdoclet.50438207_pgfId-1292869" xreflabel=""/> (OST )
-<anchor xml:id="dbdoclet.50438207_pgfId-1292870" xreflabel=""/> Persistent mount opts: errors=remount-ro,extents,mballoc
-<anchor xml:id="dbdoclet.50438207_pgfId-1292871" xreflabel=""/> Parameters: mgsnode=192.168.0.21@tcp
-<anchor xml:id="dbdoclet.50438207_pgfId-1292872" xreflabel=""/>Permanent disk data:
-<anchor xml:id="dbdoclet.50438207_pgfId-1292873" xreflabel=""/> Target: back-OST0000
-<anchor xml:id="dbdoclet.50438207_pgfId-1292874" xreflabel=""/> Index: 0
-<anchor xml:id="dbdoclet.50438207_pgfId-1292875" xreflabel=""/> Lustre FS: back
-<anchor xml:id="dbdoclet.50438207_pgfId-1292876" xreflabel=""/> Mount type: ldiskfs
-<anchor xml:id="dbdoclet.50438207_pgfId-1292877" xreflabel=""/> Flags: 0x102
-<anchor xml:id="dbdoclet.50438207_pgfId-1292878" xreflabel=""/> (OST writeconf )
-<anchor xml:id="dbdoclet.50438207_pgfId-1292879" xreflabel=""/> Persistent mount opts: errors=remount-ro,extents,mballoc
-<anchor xml:id="dbdoclet.50438207_pgfId-1292880" xreflabel=""/> Parameters: mgsnode=192.168.0.21@tcp
-<anchor xml:id="dbdoclet.50438207_pgfId-1292881" xreflabel=""/>Writing CONFIGS/mountdata
-<anchor xml:id="dbdoclet.50438207_pgfId-1292882" xreflabel=""/>When renaming an FS, we must also erase the last_rcvd file from the snapsho\
+ checking for existing Lustre data
+ found Lustre data
+ Reading CONFIGS/mountdata
+Read previous values:
+ Target: main-OST0000
+ Index: 0
+ Lustre FS: main
+ Mount type: ldiskfs
+ Flags: 0x2
+ (OST )
+ Persistent mount opts: errors=remount-ro,extents,mballoc
+ Parameters: mgsnode=192.168.0.21@tcp
+Permanent disk data:
+ Target: back-OST0000
+ Index: 0
+ Lustre FS: back
+ Mount type: ldiskfs
+ Flags: 0x102
+ (OST writeconf )
+ Persistent mount opts: errors=remount-ro,extents,mballoc
+ Parameters: mgsnode=192.168.0.21@tcp
+Writing CONFIGS/mountdata
+When renaming an FS, we must also erase the last_rcvd file from the snapsho\
ts
-<anchor xml:id="dbdoclet.50438207_pgfId-1292883" xreflabel=""/>cfs21:~# mount -t ldiskfs /dev/volgroup/MDTb1 /mnt/mdtback
-<anchor xml:id="dbdoclet.50438207_pgfId-1292884" xreflabel=""/>cfs21:~# rm /mnt/mdtback/last_rcvd
-<anchor xml:id="dbdoclet.50438207_pgfId-1292885" xreflabel=""/>cfs21:~# umount /mnt/mdtback
-<anchor xml:id="dbdoclet.50438207_pgfId-1292886" xreflabel=""/>cfs21:~# mount -t ldiskfs /dev/volgroup/OSTb1 /mnt/ostback
-<anchor xml:id="dbdoclet.50438207_pgfId-1292887" xreflabel=""/>cfs21:~# rm /mnt/ostback/last_rcvd
-<anchor xml:id="dbdoclet.50438207_pgfId-1292888" xreflabel=""/>cfs21:~# umount /mnt/ostback
+cfs21:~# mount -t ldiskfs /dev/volgroup/MDTb1 /mnt/mdtback
+cfs21:~# rm /mnt/mdtback/last_rcvd
+cfs21:~# umount /mnt/mdtback
+cfs21:~# mount -t ldiskfs /dev/volgroup/OSTb1 /mnt/ostback
+cfs21:~# rm /mnt/ostback/last_rcvd
+cfs21:~# umount /mnt/ostback
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292889" xreflabel=""/>Mount the file system from the LVM snapshot.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292890" xreflabel=""/>For example:</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292891" xreflabel=""/>cfs21:~# mount -t lustre /dev/volgroup/MDTb1 /mnt/mdtback \
+ <para>Mount the file system from the LVM snapshot.</para>
+ <para>For example:</para>
+ <screen>cfs21:~# mount -t lustre /dev/volgroup/MDTb1 /mnt/mdtback \
-<anchor xml:id="dbdoclet.50438207_pgfId-1292892" xreflabel=""/>cfs21:~# mount -t lustre /dev/volgroup/OSTb1 /mnt/ostback
-<anchor xml:id="dbdoclet.50438207_pgfId-1292893" xreflabel=""/>cfs21:~# mount -t lustre cfs21:/back /mnt/back
+cfs21:~# mount -t lustre /dev/volgroup/OSTb1 /mnt/ostback
+cfs21:~# mount -t lustre cfs21:/back /mnt/back
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292894" xreflabel=""/> 3. Note the old directory contents, as of the snapshot time.</para>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292895" xreflabel=""/>For example:</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292896" xreflabel=""/>cfs21:~/cfs/b1_5/lustre/utils# ls /mnt/back
-<anchor xml:id="dbdoclet.50438207_pgfId-1292897" xreflabel=""/>fstab passwds
+ <para> 3. Note the old directory contents, as of the snapshot time.</para>
+ <para>For example:</para>
+ <screen>cfs21:~/cfs/b1_5/lustre/utils# ls /mnt/back
+fstab passwds
</screen>
</listitem></orderedlist>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438207_pgfId-1292898" xreflabel=""/>17.5.5 Deleting Old Snapshots</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292899" xreflabel=""/>To reclaim disk space, you can erase old snapshots as your backup policy dictates. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292900" xreflabel=""/>lvremove /dev/volgroup/MDTb1
+ <title>17.5.5 Deleting Old Snapshots</title>
+ <para>To reclaim disk space, you can erase old snapshots as your backup policy dictates. Run:</para>
+ <screen>lvremove /dev/volgroup/MDTb1
</screen>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438207_pgfId-1292901" xreflabel=""/>17.5.6 Changing Snapshot Volume Size</title>
- <para><anchor xml:id="dbdoclet.50438207_pgfId-1292902" xreflabel=""/>You can also extend or shrink snapshot volumes if you find your daily deltas are smaller or larger than expected. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438207_pgfId-1292903" xreflabel=""/>lvextend -L10G /dev/volgroup/MDTb1
+ <title>17.5.6 Changing Snapshot Volume Size</title>
+ <para>You can also extend or shrink snapshot volumes if you find your daily deltas are smaller or larger than expected. Run:</para>
+ <screen>lvextend -L10G /dev/volgroup/MDTb1
</screen>
<note><para>Extending snapshots seems to be broken in older LVM. It is working in LVM v2.02.01.</para></note>
</section>
<title xml:id='benchmarkingtests.title'>Benchmarking Lustre Performance (Lustre I/O Kit)</title>
</info>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1289895" xreflabel=""/>This chapter describes the Lustre I/O kit, a collection of I/O benchmarking tools for a Lustre cluster, and PIOS, a parallel I/O simulator for Linux and Solaris. It includes:</para>
+ <para>This chapter describes the Lustre I/O kit, a collection of I/O benchmarking tools for a Lustre cluster, and PIOS, a parallel I/O simulator for Linux and Solaris. It includes:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438212_44437"/></para>
</listitem>
<section xml:id="dbdoclet.50438212_44437">
<title>24.1 Using Lustre I/O Kit Tools</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1302857" xreflabel=""/>The tools in the Lustre I/O Kit are used to benchmark Lustre hardware and validate that it is working as expected before you install the Lustre software. It can also be used to to validate the performance of the various hardware and software layers in the cluster and also to find and troubleshoot I/O issues.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300612" xreflabel=""/>Typically, performance is measured starting with single raw devices and then proceeding to groups of devices. Once raw performance has been established, other software layers are then added incrementally and tested.</para>
+ <para>The tools in the Lustre I/O Kit are used to benchmark Lustre hardware and validate that it is working as expected before you install the Lustre software. It can also be used to to validate the performance of the various hardware and software layers in the cluster and also to find and troubleshoot I/O issues.</para>
+ <para>Typically, performance is measured starting with single raw devices and then proceeding to groups of devices. Once raw performance has been established, other software layers are then added incrementally and tested.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438212_pgfId-1289911" xreflabel=""/>24.1.1 Contents of the Lustre I/O Kit</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303002" xreflabel=""/>The I/O kit contains three tests, each of which tests a progressively higher layer in the Lustre stack:</para>
+ <title>24.1.1 Contents of the Lustre I/O Kit</title>
+ <para>The I/O kit contains three tests, each of which tests a progressively higher layer in the Lustre stack:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300548" xreflabel=""/>sgpdd_survey - Measure basic 'bare metal' performance of devices while bypassing the kernel block device layers, buffer cache, and file system.</para>
+ <para>sgpdd_survey - Measure basic 'bare metal' performance of devices while bypassing the kernel block device layers, buffer cache, and file system.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300572" xreflabel=""/>obdfilter_survey - Measure the performance of one or more OSTs directly on the OSS node or alternately over the network from a Lustre client.</para>
+ <para>obdfilter_survey - Measure the performance of one or more OSTs directly on the OSS node or alternately over the network from a Lustre client.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300557" xreflabel=""/>ost_survey - Performs I/O against OSTs individually to allow performance comparisons to detect if an OST is performing suboptimally due to hardware issues.</para>
+ <para>ost_survey - Performs I/O against OSTs individually to allow performance comparisons to detect if an OST is performing suboptimally due to hardware issues.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300595" xreflabel=""/>Typically with these tests, Lustre should deliver 85-90% of the raw device performance.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300756" xreflabel=""/>A utility stats-collect is also provided to collect application profiling information from Lustre clients and servers. See <link xl:href="BenchmarkingTests.html#50438212_58201">Collecting Application Profiling Information (stats-collect)</link> for more information.</para>
+ <para>Typically with these tests, Lustre should deliver 85-90% of the raw device performance.</para>
+ <para>A utility stats-collect is also provided to collect application profiling information from Lustre clients and servers. See <link xl:href="BenchmarkingTests.html#50438212_58201">Collecting Application Profiling Information (stats-collect)</link> for more information.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438212_pgfId-1289914" xreflabel=""/>24.1.2 Preparing to Use the Lustre <anchor xml:id="dbdoclet.50438212_marker-1289913" xreflabel=""/>I/O Kit</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300699" xreflabel=""/>The following prerequisites must be met to use the tests in the Lustre I/O kit:</para>
+ <title>24.1.2 Preparing to Use the Lustre <anchor xml:id="dbdoclet.50438212_marker-1289913" xreflabel=""/>I/O Kit</title>
+ <para>The following prerequisites must be met to use the tests in the Lustre I/O kit:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300700" xreflabel=""/> Password-free remote access to nodes in the system (provided by ssh or rsh).</para>
+ <para> Password-free remote access to nodes in the system (provided by ssh or rsh).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303852" xreflabel=""/> LNET self-test completed to test that Lustre Networking has been properly installed and configured. See <xref linkend='lnetselftest'/>.</para>
+ <para> LNET self-test completed to test that Lustre Networking has been properly installed and configured. See <xref linkend='lnetselftest'/>.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300701" xreflabel=""/> Lustre file system software installed.</para>
+ <para> Lustre file system software installed.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300702" xreflabel=""/>sg3_utils package providing the sgp_dd tool (sg3_utils is a separate RPM package available online using YUM).</para>
+ <para>sg3_utils package providing the sgp_dd tool (sg3_utils is a separate RPM package available online using YUM).</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1289915" xreflabel=""/>Download the Lustre I/O kit (lustre-iokit)from:</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300015" xreflabel=""/><link xl:href="http://downloads.lustre.org/public/tools/lustre-iokit/">http://downloads.lustre.org/public/tools/lustre-iokit/</link></para>
+ <para>Download the Lustre I/O kit (lustre-iokit)from:</para>
+ <para><link xl:href="http://downloads.lustre.org/public/tools/lustre-iokit/">http://downloads.lustre.org/public/tools/lustre-iokit/</link></para>
</section>
</section>
<section xml:id="dbdoclet.50438212_51053">
<title>24.2 Testing I/O Performance of Raw Hardware (sgpdd_survey<anchor xml:id="dbdoclet.50438212_marker-1302844" xreflabel=""/>)</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1289936" xreflabel=""/>The sgpdd_survey tool is used to test bare metal I/O performance of the raw hardware, while bypassing as much of the kernel as possible. This survey may be used to characterize the performance of a SCSI device by simulating an OST serving multiple stripe files. The data gathered by this survey can help set expectations for the performance of a Lustre OST using this device.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300866" xreflabel=""/>The script uses sgp_dd to carry out raw sequential disk I/O. It runs with variable numbers of sgp_dd threads to show how performance varies with different request queue depths.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300867" xreflabel=""/>The script spawns variable numbers of sgp_dd instances, each reading or writing a separate area of the disk to demonstrate performance variance within a number of concurrent stripe files.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305925" xreflabel=""/>Several tips and insights for disk performance measurement are described below. Some of this information is specific to RAID arrays and/or the Linux RAID implementation.</para>
+ <para>The sgpdd_survey tool is used to test bare metal I/O performance of the raw hardware, while bypassing as much of the kernel as possible. This survey may be used to characterize the performance of a SCSI device by simulating an OST serving multiple stripe files. The data gathered by this survey can help set expectations for the performance of a Lustre OST using this device.</para>
+ <para>The script uses sgp_dd to carry out raw sequential disk I/O. It runs with variable numbers of sgp_dd threads to show how performance varies with different request queue depths.</para>
+ <para>The script spawns variable numbers of sgp_dd instances, each reading or writing a separate area of the disk to demonstrate performance variance within a number of concurrent stripe files.</para>
+ <para>Several tips and insights for disk performance measurement are described below. Some of this information is specific to RAID arrays and/or the Linux RAID implementation.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305926" xreflabel=""/><emphasis>Performance is limited by the slowest disk.</emphasis></para>
+ <para><emphasis>Performance is limited by the slowest disk.</emphasis></para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305927" xreflabel=""/>Before creating a RAID array, benchmark all disks individually. We have frequently encountered situations where drive performance was not consistent for all devices in the array. Replace any disks that are significantly slower than the rest.</para>
+ <para>Before creating a RAID array, benchmark all disks individually. We have frequently encountered situations where drive performance was not consistent for all devices in the array. Replace any disks that are significantly slower than the rest.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305928" xreflabel=""/><emphasis>Disks and arrays are very sensitive to request size.</emphasis></para>
+ <para><emphasis>Disks and arrays are very sensitive to request size.</emphasis></para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305929" xreflabel=""/>To identify the optimal request size for a given disk, benchmark the disk with different record sizes ranging from 4 KB to 1 to 2 MB.</para>
+ <para>To identify the optimal request size for a given disk, benchmark the disk with different record sizes ranging from 4 KB to 1 to 2 MB.</para>
<caution><para>The sgpdd_survey script overwrites the device being tested, which results in the <emphasis>LOSS OF ALL DATA</emphasis> on that device. Exercise caution when selecting the device to be tested.</para></caution>
<note><para>Array performance with all LUNs loaded does not always match the performance of a single LUN when tested in isolation.</para></note>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303815" xreflabel=""/></para>
+ <para></para>
<para><emphasis role="bold">Prequisites:</emphasis></para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300842" xreflabel=""/>sgp_dd tool in the sg3_utils package</para>
+ <para>sgp_dd tool in the sg3_utils package</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300845" xreflabel=""/> Lustre software is <emphasis>NOT</emphasis> required</para>
+ <para> Lustre software is <emphasis>NOT</emphasis> required</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1289939" xreflabel=""/>The device(s) being tested must meet one of these two requirements:</para>
+ <para>The device(s) being tested must meet one of these two requirements:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300909" xreflabel=""/> If the device is a SCSI device, it must appear in the output of sg_map (make sure the kernel module sg is loaded).</para>
+ <para> If the device is a SCSI device, it must appear in the output of sg_map (make sure the kernel module sg is loaded).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300920" xreflabel=""/> If the device is a raw device, it must appear in the output of raw -qa.</para>
+ <para> If the device is a raw device, it must appear in the output of raw -qa.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1300930" xreflabel=""/>Raw and SCSI devices cannot be mixed in the test specification.</para>
+ <para>Raw and SCSI devices cannot be mixed in the test specification.</para>
<note><para>If you need to create raw devices to use the sgpdd_survey tool, note that raw device 0 cannot be used due to a bug in certain versions of the "raw" utility (including that shipped with RHEL4U4.)</para></note>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438212_pgfId-1289945" xreflabel=""/>24.2.1 Tuning Linux Storage Devices</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303368" xreflabel=""/>To get large I/O transfers (1 MB) to disk, it may be necessary to tune several kernel parameters as specified:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1303369" xreflabel=""/>/sys/block/sdN/queue/max_sectors_kb = 4096
-<anchor xml:id="dbdoclet.50438212_pgfId-1303370" xreflabel=""/>/sys/block/sdN/queue/max_phys_segments = 256
-<anchor xml:id="dbdoclet.50438212_pgfId-1303371" xreflabel=""/>/proc/scsi/sg/allow_dio = 1
-<anchor xml:id="dbdoclet.50438212_pgfId-1303372" xreflabel=""/>/sys/module/ib_srp/parameters/srp_sg_tablesize = 255
-<anchor xml:id="dbdoclet.50438212_pgfId-1304119" xreflabel=""/>/sys/block/sdN/queue/scheduler</screen>
+ <title>24.2.1 Tuning Linux Storage Devices</title>
+ <para>To get large I/O transfers (1 MB) to disk, it may be necessary to tune several kernel parameters as specified:</para>
+ <screen>/sys/block/sdN/queue/max_sectors_kb = 4096
+/sys/block/sdN/queue/max_phys_segments = 256
+/proc/scsi/sg/allow_dio = 1
+/sys/module/ib_srp/parameters/srp_sg_tablesize = 255
+/sys/block/sdN/queue/scheduler</screen>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438212_pgfId-1303366" xreflabel=""/>24.2.2 Running sgpdd_survey</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304151" xreflabel=""/>The sgpdd_survey script must be customized for the particular device being tested and for the location where the script saves its working and result files (by specifying the ${rslt} variable). Customization variables are described at the beginning of the script.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301030" xreflabel=""/>When the sgpdd_survey script runs, it creates a number of working files and a pair of result files. The names of all the files created start with the prefixdefined in the variable ${rslt}. (The default value is /tmp.) The files include:</para>
+ <title>24.2.2 Running sgpdd_survey</title>
+ <para>The sgpdd_survey script must be customized for the particular device being tested and for the location where the script saves its working and result files (by specifying the ${rslt} variable). Customization variables are described at the beginning of the script.</para>
+ <para>When the sgpdd_survey script runs, it creates a number of working files and a pair of result files. The names of all the files created start with the prefixdefined in the variable ${rslt}. (The default value is /tmp.) The files include:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301035" xreflabel=""/> File containing standard output data (same as stdout)</para>
+ <para> File containing standard output data (same as stdout)</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1301031" xreflabel=""/>${rslt}_<emphasis><date/time></emphasis>.summary<emphasis/></screen>
+ <screen>${rslt}_<emphasis><date/time></emphasis>.summary<emphasis/></screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301053" xreflabel=""/> Temporary (tmp) files</para>
+ <para> Temporary (tmp) files</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1301032" xreflabel=""/>${rslt}_<emphasis><date/time></emphasis>_*
+ <screen>${rslt}_<emphasis><date/time></emphasis>_*
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301063" xreflabel=""/> Collected tmp files for post-mortem</para>
+ <para> Collected tmp files for post-mortem</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1289951" xreflabel=""/>${rslt}_<emphasis><date/time></emphasis>.detail
+ <screen>${rslt}_<emphasis><date/time></emphasis>.detail
</screen>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1289952" xreflabel=""/>The stdout and the .summary file will contain lines like this:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1289953" xreflabel=""/>total_size 8388608K rsz 1024 thr 1 crg 1 180.45 MB/s 1 x 180.50 \=/ 180.50 \
+ <para>The stdout and the .summary file will contain lines like this:</para>
+ <screen>total_size 8388608K rsz 1024 thr 1 crg 1 180.45 MB/s 1 x 180.50 \=/ 180.50 \
MB/s
</screen>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1289954" xreflabel=""/>Each line corresponds to a run of the test. Each test run will have a different number of threads, record size, or number of regions.</para>
+ <para>Each line corresponds to a run of the test. Each test run will have a different number of threads, record size, or number of regions.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303307" xreflabel=""/>total_size - Size of file being tested in KBs (8 GB in above example).</para>
+ <para>total_size - Size of file being tested in KBs (8 GB in above example).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303308" xreflabel=""/>rsz - Record size in KBs (1 MB in above example).</para>
+ <para>rsz - Record size in KBs (1 MB in above example).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303311" xreflabel=""/>thr - Number of threads generating I/O (1 thread in above example).</para>
+ <para>thr - Number of threads generating I/O (1 thread in above example).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303312" xreflabel=""/> crg - Current regions, the number of disjount areas on the disk to which I/O is being sent (1 region in above example, indicating that no seeking is done).</para>
+ <para> crg - Current regions, the number of disjount areas on the disk to which I/O is being sent (1 region in above example, indicating that no seeking is done).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303313" xreflabel=""/>MB/s - Aggregate bandwidth measured by dividing the total amount of data by the elapsed time (180.45 MB/s in the above example).</para>
+ <para>MB/s - Aggregate bandwidth measured by dividing the total amount of data by the elapsed time (180.45 MB/s in the above example).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303314" xreflabel=""/>MB/s - The remaining numbers show the number of regions X performance of the slowest disk as a sanity check on the aggregate bandwidth.</para>
+ <para>MB/s - The remaining numbers show the number of regions X performance of the slowest disk as a sanity check on the aggregate bandwidth.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1289955" xreflabel=""/>If there are so many threads that the sgp_dd script is unlikely to be able to allocate I/O buffers, then ENOMEM is printed in place of the aggregate bandwidth result.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1289956" xreflabel=""/>If one or more sgp_dd instances do not successfully report a bandwidth number, then FAILED is printed in place of the aggregate bandwidth result.</para>
+ <para>If there are so many threads that the sgp_dd script is unlikely to be able to allocate I/O buffers, then ENOMEM is printed in place of the aggregate bandwidth result.</para>
+ <para>If one or more sgp_dd instances do not successfully report a bandwidth number, then FAILED is printed in place of the aggregate bandwidth result.</para>
</section>
</section>
<section xml:id="dbdoclet.50438212_26516">
<title>24.3 <anchor xml:id="dbdoclet.50438212_40624" xreflabel=""/>Testing OST Performance (obdfilter_survey<anchor xml:id="dbdoclet.50438212_marker-1289957" xreflabel=""/>)</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301303" xreflabel=""/>The obdfilter_survey script generates sequential I/O from varying numbers of threads and objects (files) to simulate the I/O patterns of a Lustre client.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301200" xreflabel=""/>The obdfilter_survey script can be run directly on the OSS node to measure the OST storage performance without any intervening network, or it can be run remotely on a Lustre client to measure the OST performance including network overhead.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303417" xreflabel=""/>The obdfilter_survey is used to characterize the performance of the following:</para>
+ <para>The obdfilter_survey script generates sequential I/O from varying numbers of threads and objects (files) to simulate the I/O patterns of a Lustre client.</para>
+ <para>The obdfilter_survey script can be run directly on the OSS node to measure the OST storage performance without any intervening network, or it can be run remotely on a Lustre client to measure the OST performance including network overhead.</para>
+ <para>The obdfilter_survey is used to characterize the performance of the following:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1289960" xreflabel=""/><emphasis role="bold">Local file system</emphasis> - In this mode, the obdfilter_survey script exercises one or more instances of the obdfilter directly. The script may run on one or more OSS nodes, for example, when the OSSs are all attached to the same multi-ported disk subsystem.</para>
+ <para><emphasis role="bold">Local file system</emphasis> - In this mode, the obdfilter_survey script exercises one or more instances of the obdfilter directly. The script may run on one or more OSS nodes, for example, when the OSSs are all attached to the same multi-ported disk subsystem.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1289962" xreflabel=""/>Run the script using the case=disk parameter to run the test against all the local OSTs. The script automatically detects all local OSTs and includes them in the survey.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303453" xreflabel=""/>To run the test against only specific OSTs, run the script using the target= parameter to list the OSTs to be tested explicitly. If some OSTs are on remote nodes, specify their hostnames in addition to the OST name (for example, oss2:lustre-OST0004).</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303461" xreflabel=""/>All obdfilter instances are driven directly. The script automatically loads the obdecho module (if required) and creates one instance of echo_client for each obdfilter instance in order to generate I/O requests directly to the OST.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304398" xreflabel=""/>For more details, see <link xl:href="BenchmarkingTests.html#50438212_59319">Testing Local Disk Performance</link>.</para>
+ <para>Run the script using the case=disk parameter to run the test against all the local OSTs. The script automatically detects all local OSTs and includes them in the survey.</para>
+ <para>To run the test against only specific OSTs, run the script using the target= parameter to list the OSTs to be tested explicitly. If some OSTs are on remote nodes, specify their hostnames in addition to the OST name (for example, oss2:lustre-OST0004).</para>
+ <para>All obdfilter instances are driven directly. The script automatically loads the obdecho module (if required) and creates one instance of echo_client for each obdfilter instance in order to generate I/O requests directly to the OST.</para>
+ <para>For more details, see <link xl:href="BenchmarkingTests.html#50438212_59319">Testing Local Disk Performance</link>.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1289963" xreflabel=""/><emphasis role="bold">Network</emphasis> - In this mode, the Lustre client generates I/O requests over the network but these requests are not sent to the OST file system. The OSS node runs the obdecho server to receive the requests but discards them before they are sent to the disk.</para>
+ <para><emphasis role="bold">Network</emphasis> - In this mode, the Lustre client generates I/O requests over the network but these requests are not sent to the OST file system. The OSS node runs the obdecho server to receive the requests but discards them before they are sent to the disk.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303500" xreflabel=""/>Pass the parameters case=network and target=<emphasis><hostname</emphasis>|<emphasis>IP_of_server></emphasis> to the script. For each network case, the script does the required setup.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304403" xreflabel=""/>For more details, see <link xl:href="BenchmarkingTests.html#50438212_36037">Testing Network Performance</link></para>
+ <para>Pass the parameters case=network and target=<emphasis><hostname</emphasis>|<emphasis>IP_of_server></emphasis> to the script. For each network case, the script does the required setup.</para>
+ <para>For more details, see <link xl:href="BenchmarkingTests.html#50438212_36037">Testing Network Performance</link></para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304334" xreflabel=""/><emphasis role="bold">Remote file system over the network</emphasis> - In this mode the obdfilter_survey script generates I/O from a Lustre client to a remote OSS to write the data to the file system.</para>
+ <para><emphasis role="bold">Remote file system over the network</emphasis> - In this mode the obdfilter_survey script generates I/O from a Lustre client to a remote OSS to write the data to the file system.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304336" xreflabel=""/>To run the test against all the local OSCs, pass the parameter case=netdisk to the script. Alternately you can pass the target= parameter with one or more OSC devices (e.g., lustre-OST0000-osc-ffff88007754bc00) against which the tests are to be run.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304408" xreflabel=""/>For more details, see <link xl:href="BenchmarkingTests.html#50438212_62662">Testing Remote Disk Performance</link>.</para>
+ <para>To run the test against all the local OSCs, pass the parameter case=netdisk to the script. Alternately you can pass the target= parameter with one or more OSC devices (e.g., lustre-OST0000-osc-ffff88007754bc00) against which the tests are to be run.</para>
+ <para>For more details, see <link xl:href="BenchmarkingTests.html#50438212_62662">Testing Remote Disk Performance</link>.</para>
<caution><para>The obdfilter_survey script is destructive and should not be run on devices that containing existing data that needs to be preserved. Thus, tests using obdfilter_survey should be run before the Lustre file system is placed in production.</para></caution>
<note><para>If the obdfilter_survey test is terminated before it completes, some small amount of space is leaked. you can either ignore it or reformat the file system.</para></note>
<note><para>The obdfilter_survey script must be customized, depending on the components under test and where the script's working files should be kept. Customization variables are described at the beginning of the obdfilter_survey script. In particular, pay attention to the listed maximum values listed for each parameter in the script.</para></note>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438212_pgfId-1289969" xreflabel=""/>24.3.1 <anchor xml:id="dbdoclet.50438212_59319" xreflabel=""/>Testing Local Disk Performance</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1298366" xreflabel=""/>The obdfilter_survey script can be run automatically or manually against a local disk. This script profiles the overall throughput of storage hardware, including the file system and RAID layers managing the storage, by sending workloads to the OSTs that vary in thread count, object count, and I/O size.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1298465" xreflabel=""/>When the obdfilter_survey script is run, it provides information about the performance abilities of the storage hardware and shows the saturation points.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303569" xreflabel=""/>The plot-obdfilter script generates from the output of the obdfilter_survey a CSV file and parameters for importing into a spreadsheet or gnuplot to visualize the data.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1297883" xreflabel=""/>To run the obdfilter_survey script, create a standard Lustre configuration; no special setup is needed.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1297701" xreflabel=""/><emphasis role="bold">To perform an automatic run:</emphasis></para>
+ <title>24.3.1 <anchor xml:id="dbdoclet.50438212_59319" xreflabel=""/>Testing Local Disk Performance</title>
+ <para>The obdfilter_survey script can be run automatically or manually against a local disk. This script profiles the overall throughput of storage hardware, including the file system and RAID layers managing the storage, by sending workloads to the OSTs that vary in thread count, object count, and I/O size.</para>
+ <para>When the obdfilter_survey script is run, it provides information about the performance abilities of the storage hardware and shows the saturation points.</para>
+ <para>The plot-obdfilter script generates from the output of the obdfilter_survey a CSV file and parameters for importing into a spreadsheet or gnuplot to visualize the data.</para>
+ <para>To run the obdfilter_survey script, create a standard Lustre configuration; no special setup is needed.</para>
+ <para><emphasis role="bold">To perform an automatic run:</emphasis></para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1297574" xreflabel=""/> 1. Start the Lustre OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304489" xreflabel=""/>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
+ <para> 1. Start the Lustre OSTs.</para>
+ <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1297587" xreflabel=""/> 2. Verify that the obdecho module is loaded. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1304502" xreflabel=""/>modprobe obdecho</screen>
+ <para> 2. Verify that the obdecho module is loaded. Run:</para>
+ <screen>modprobe obdecho</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1297591" xreflabel=""/> 3. Run the obdfilter_survey script with the parameter case=disk.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304517" xreflabel=""/>For example, to run a local test with up to two objects (nobjhi), up to two threads (thrhi), and 1024 MB transfer size (size):</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1297595" xreflabel=""/>$ nobjhi=2 thrhi=2 size=1024 case=disk sh obdfilter-survey
-<anchor xml:id="dbdoclet.50438212_pgfId-1304711" xreflabel=""/>
+ <para> 3. Run the obdfilter_survey script with the parameter case=disk.</para>
+ <para>For example, to run a local test with up to two objects (nobjhi), up to two threads (thrhi), and 1024 MB transfer size (size):</para>
+ <screen>$ nobjhi=2 thrhi=2 size=1024 case=disk sh obdfilter-survey
+
</screen>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1299344" xreflabel=""/><emphasis role="bold">To perform a manual run:</emphasis></para>
+ <para><emphasis role="bold">To perform a manual run:</emphasis></para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303657" xreflabel=""/> 1. Start the Lustre OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304535" xreflabel=""/>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
+ <para> 1. Start the Lustre OSTs.</para>
+ <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305056" xreflabel=""/> 2. Verify that the obdecho module is loaded. Run:</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305057" xreflabel=""/>modprobe obdecho</para>
+ <para> 2. Verify that the obdecho module is loaded. Run:</para>
+ <para>modprobe obdecho</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303655" xreflabel=""/> 3. Determine the OST names.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304551" xreflabel=""/>On the OSS nodes to be tested, run the lctldl command. The OST device names are listed in the fourth column of the output. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1303717" xreflabel=""/>$ lctl dl |grep obdfilter
-<anchor xml:id="dbdoclet.50438212_pgfId-1303733" xreflabel=""/>0 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 1159
-<anchor xml:id="dbdoclet.50438212_pgfId-1303751" xreflabel=""/>2 UP obdfilter lustre-OST0002 lustre-OST0002_UUID 1159
-<anchor xml:id="dbdoclet.50438212_pgfId-1303742" xreflabel=""/>...
+ <para> 3. Determine the OST names.</para>
+ <para>On the OSS nodes to be tested, run the lctldl command. The OST device names are listed in the fourth column of the output. For example:</para>
+ <screen>$ lctl dl |grep obdfilter
+0 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 1159
+2 UP obdfilter lustre-OST0002 lustre-OST0002_UUID 1159
+...
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303687" xreflabel=""/> 4. List all OSTs you want to test.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304568" xreflabel=""/>Use the target= parameter to list the OSTs separated by spaces. List the individual OSTs by name using the format <emphasis><fsname>-<OSTnumber></emphasis> (for example, lustre-OST0001). You do not have to specify an MDS or LOV.</para>
+ <para> 4. List all OSTs you want to test.</para>
+ <para>Use the target= parameter to list the OSTs separated by spaces. List the individual OSTs by name using the format <emphasis><fsname>-<OSTnumber></emphasis> (for example, lustre-OST0001). You do not have to specify an MDS or LOV.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1303805" xreflabel=""/> 5. Run the obdfilter_survey script with the target= parameter.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304650" xreflabel=""/>For example, to run a local test with up to two objects (nobjhi), up to two threads (thrhi), and 1024 Mb (size) transfer size:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1303806" xreflabel=""/>$ nobjhi=2 thrhi=2 size=1024 targets='lustre-OST0001 \
-<anchor xml:id="dbdoclet.50438212_pgfId-1304757" xreflabel=""/>lustre-OST0002' sh obdfilter-survey
+ <para> 5. Run the obdfilter_survey script with the target= parameter.</para>
+ <para>For example, to run a local test with up to two objects (nobjhi), up to two threads (thrhi), and 1024 Mb (size) transfer size:</para>
+ <screen>$ nobjhi=2 thrhi=2 size=1024 targets='lustre-OST0001 \
+lustre-OST0002' sh obdfilter-survey
</screen>
</listitem></orderedlist>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438212_pgfId-1289982" xreflabel=""/>24.3.2 <anchor xml:id="dbdoclet.50438212_36037" xreflabel=""/>Testing Network Performance</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1289983" xreflabel=""/>The obdfilter_survey script can only be run automatically against a network; no manual test is provided.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1297847" xreflabel=""/>To run the network test, a specific Lustre setup is needed. Make sure that these configuration requirements have been met.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305077" xreflabel=""/><emphasis role="bold">To perform an automatic run:</emphasis></para>
+ <title>24.3.2 <anchor xml:id="dbdoclet.50438212_36037" xreflabel=""/>Testing Network Performance</title>
+ <para>The obdfilter_survey script can only be run automatically against a network; no manual test is provided.</para>
+ <para>To run the network test, a specific Lustre setup is needed. Make sure that these configuration requirements have been met.</para>
+ <para><emphasis role="bold">To perform an automatic run:</emphasis></para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305363" xreflabel=""/> 1. Start the Lustre OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305364" xreflabel=""/>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
+ <para> 1. Start the Lustre OSTs.</para>
+ <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305365" xreflabel=""/> 2. Verify that the obdecho module is loaded. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1305366" xreflabel=""/>modprobe obdecho</screen>
+ <para> 2. Verify that the obdecho module is loaded. Run:</para>
+ <screen>modprobe obdecho</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305389" xreflabel=""/> 3. Start lctl and check the device list, which must be empty. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1305940" xreflabel=""/>lctl dl
+ <para> 3. Start lctl and check the device list, which must be empty. Run:</para>
+ <screen>lctl dl
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305639" xreflabel=""/> 4. Run the obdfilter_survey script with the parameters case=network and <emphasis role="bold">targets=</emphasis><emphasis><hostname</emphasis>|<emphasis>ip_of_server></emphasis>. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1304777" xreflabel=""/>$ nobjhi=2 thrhi=2 size=1024 targets='oss1 oss2' case=network sh obdfilte\
+ <para> 4. Run the obdfilter_survey script with the parameters case=network and <emphasis role="bold">targets=</emphasis><emphasis><hostname</emphasis>|<emphasis>ip_of_server></emphasis>. For example:</para>
+ <screen>$ nobjhi=2 thrhi=2 size=1024 targets='oss1 oss2' case=network sh obdfilte\
r-survey
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1297769" xreflabel=""/> 5. On the server side, view the statistics at:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1297820" xreflabel=""/>/proc/fs/lustre/obdecho/<emphasis><echo_srv></emphasis>/stats
+ <para> 5. On the server side, view the statistics at:</para>
+ <screen>/proc/fs/lustre/obdecho/<emphasis><echo_srv></emphasis>/stats
</screen>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1297824" xreflabel=""/>where <emphasis><echo_srv></emphasis> is the obdecho server created by the script.</para>
+ <para>where <emphasis><echo_srv></emphasis> is the obdecho server created by the script.</para>
</listitem></orderedlist>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438212_pgfId-1297766" xreflabel=""/>24.3.3 <anchor xml:id="dbdoclet.50438212_62662" xreflabel=""/>Testing Remote Disk Performance</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1297832" xreflabel=""/>The obdfilter_survey script can be run automatically or manually against a network disk. To run the network disk test, start with a standard Lustre configuration. No special setup is needed.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1297904" xreflabel=""/><emphasis role="bold">To perform an automatic run:</emphasis></para>
+ <title>24.3.3 <anchor xml:id="dbdoclet.50438212_62662" xreflabel=""/>Testing Remote Disk Performance</title>
+ <para>The obdfilter_survey script can be run automatically or manually against a network disk. To run the network disk test, start with a standard Lustre configuration. No special setup is needed.</para>
+ <para><emphasis role="bold">To perform an automatic run:</emphasis></para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305421" xreflabel=""/> 1. Start the Lustre OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305422" xreflabel=""/>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
+ <para> 1. Start the Lustre OSTs.</para>
+ <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305423" xreflabel=""/> 2. Verify that the obdecho module is loaded. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1305424" xreflabel=""/>modprobe obdecho</screen>
+ <para> 2. Verify that the obdecho module is loaded. Run:</para>
+ <screen>modprobe obdecho</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1297925" xreflabel=""/> 3. Run the obdfilter_survey script with the parameter case=netdisk. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1297926" xreflabel=""/>$ nobjhi=2 thrhi=2 size=1024 case=netdisk sh obdfilter-survey
+ <para> 3. Run the obdfilter_survey script with the parameter case=netdisk. For example:</para>
+ <screen>$ nobjhi=2 thrhi=2 size=1024 case=netdisk sh obdfilter-survey
</screen>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1297909" xreflabel=""/><emphasis role="bold">To perform a manual run:</emphasis></para>
+ <para><emphasis role="bold">To perform a manual run:</emphasis></para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305433" xreflabel=""/> 1. Start the Lustre OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305434" xreflabel=""/>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
+ <para> 1. Start the Lustre OSTs.</para>
+ <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305435" xreflabel=""/> 2. Verify that the obdecho module is loaded. Run:</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305436" xreflabel=""/>modprobe obdecho</para>
+ <para> 2. Verify that the obdecho module is loaded. Run:</para>
+ <para>modprobe obdecho</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305437" xreflabel=""/> 3. Determine the OSC names.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305438" xreflabel=""/>On the OSS nodes to be tested, run the lctldl command. The OSC device names are listed in the fourth column of the output. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1305483" xreflabel=""/>$ lctl dl |grep obdfilter
-<anchor xml:id="dbdoclet.50438212_pgfId-1305461" xreflabel=""/>3 UP osc lustre-OST0000-osc-ffff88007754bc00 54b91eab-0ea9-1516-b571-5e6df3\
+ <para> 3. Determine the OSC names.</para>
+ <para>On the OSS nodes to be tested, run the lctldl command. The OSC device names are listed in the fourth column of the output. For example:</para>
+ <screen>$ lctl dl |grep obdfilter
+3 UP osc lustre-OST0000-osc-ffff88007754bc00 54b91eab-0ea9-1516-b571-5e6df3\
49592e 5
-<anchor xml:id="dbdoclet.50438212_pgfId-1305462" xreflabel=""/>4 UP osc lustre-OST0001-osc-ffff88007754bc00 54b91eab-0ea9-1516-b571-5e6df3\
+4 UP osc lustre-OST0001-osc-ffff88007754bc00 54b91eab-0ea9-1516-b571-5e6df3\
49592e 5
-<anchor xml:id="dbdoclet.50438212_pgfId-1305515" xreflabel=""/>...
+...
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305443" xreflabel=""/> 4. List all OSCs you want to test.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305444" xreflabel=""/>Use the target= parameter to list the OSCs separated by spaces. List the individual OSCs by name seperated by spaces using the format <emphasis><fsname>-<OST_name></emphasis>-osc-<emphasis><OSC_number></emphasis> (for example, lustre-OST0000-osc-ffff88007754bc00). You <emphasis role="bold">do not have to specify an MDS or LOV.</emphasis></para>
+ <para> 4. List all OSCs you want to test.</para>
+ <para>Use the target= parameter to list the OSCs separated by spaces. List the individual OSCs by name seperated by spaces using the format <emphasis><fsname>-<OST_name></emphasis>-osc-<emphasis><OSC_number></emphasis> (for example, lustre-OST0000-osc-ffff88007754bc00). You <emphasis role="bold">do not have to specify an MDS or LOV.</emphasis></para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305445" xreflabel=""/> 5. Run the <emphasis role="bold">o</emphasis>bdfilter_survey script with the target= parameter and case=netdisk.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305446" xreflabel=""/>An example of a local test run with up to two objects (nobjhi), up to two threads (thrhi), and 1024 Mb (size) transfer size is shown below:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1305576" xreflabel=""/>$ nobjhi=2 thrhi=2 size=1024 \
-<anchor xml:id="dbdoclet.50438212_pgfId-1305577" xreflabel=""/>targets="lustre-OST0000-osc-ffff88007754bc00 \
-<anchor xml:id="dbdoclet.50438212_pgfId-1305578" xreflabel=""/>lustre-OST0001-osc-ffff88007754bc00" \
-<anchor xml:id="dbdoclet.50438212_pgfId-1305579" xreflabel=""/>sh obdfilter-survey
+ <para> 5. Run the <emphasis role="bold">o</emphasis>bdfilter_survey script with the target= parameter and case=netdisk.</para>
+ <para>An example of a local test run with up to two objects (nobjhi), up to two threads (thrhi), and 1024 Mb (size) transfer size is shown below:</para>
+ <screen>$ nobjhi=2 thrhi=2 size=1024 \
+targets="lustre-OST0000-osc-ffff88007754bc00 \
+lustre-OST0001-osc-ffff88007754bc00" \
+sh obdfilter-survey
</screen>
</listitem></orderedlist>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438212_pgfId-1290021" xreflabel=""/>24.3.4 Output Files</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1298013" xreflabel=""/>When the obdfilter_survey script runs, it creates a number of working files and a pair of result files. All files start with the prefix defined in the variable ${rslt}.</para>
+ <title>24.3.4 Output Files</title>
+ <para>When the obdfilter_survey script runs, it creates a number of working files and a pair of result files. All files start with the prefix defined in the variable ${rslt}.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438212_pgfId-1303053" xreflabel=""/>File</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438212_pgfId-1303055" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">File</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1303057" xreflabel=""/>${rslt}.summary</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1303059" xreflabel=""/>Same as stdout</para></entry>
+ <entry><para> ${rslt}.summary</para></entry>
+ <entry><para> Same as stdout</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1303061" xreflabel=""/>${rslt}.script_*</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1303063" xreflabel=""/>Per-host test script files</para></entry>
+ <entry><para> ${rslt}.script_*</para></entry>
+ <entry><para> Per-host test script files</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1303065" xreflabel=""/>${rslt}.detail_tmp*</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1303067" xreflabel=""/>Per-OST result files</para></entry>
+ <entry><para> ${rslt}.detail_tmp*</para></entry>
+ <entry><para> Per-OST result files</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1303069" xreflabel=""/>${rslt}.detail</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1303071" xreflabel=""/>Collected result files for post-mortem</para></entry>
+ <entry><para> ${rslt}.detail</para></entry>
+ <entry><para> Collected result files for post-mortem</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1290022" xreflabel=""/>The obdfilter_survey script iterates over the given number of threads and objects performing the specified tests and checks that all test processes have completed successfully.</para>
+ <para>The obdfilter_survey script iterates over the given number of threads and objects performing the specified tests and checks that all test processes have completed successfully.</para>
<note><para>The obdfilter_survey script may not clean up properly if it is aborted or if it encounters an unrecoverable error. In this case, a manual cleanup may be required, possibly including killing any running instances of lctl (local or remote), removing echo_client instances created by the script and unloading obdecho.</para></note>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438212_pgfId-1298104" xreflabel=""/>24.3.4.1 Script Output</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1298134" xreflabel=""/>The .summary file and stdout of the obdfilter_survey script contain lines like:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1294098" xreflabel=""/>ost 8 sz 67108864K rsz 1024 obj 8 thr 8 write 613.54 [ 64.00, 82.00]
+ <title>24.3.4.1 Script Output</title>
+ <para>The .summary file and stdout of the obdfilter_survey script contain lines like:</para>
+ <screen>ost 8 sz 67108864K rsz 1024 obj 8 thr 8 write 613.54 [ 64.00, 82.00]
</screen>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1294150" xreflabel=""/>Where:</para>
+ <para>Where:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438212_pgfId-1294115" xreflabel=""/>Parameter and value</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438212_pgfId-1294117" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter and value</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294119" xreflabel=""/>ost 8</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294121" xreflabel=""/>Total number of OSTs being tested.</para></entry>
+ <entry><para> ost 8</para></entry>
+ <entry><para> Total number of OSTs being tested.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294123" xreflabel=""/>sz 67108864K</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294125" xreflabel=""/>Total amount of data read or written (in KB).</para></entry>
+ <entry><para> sz 67108864K</para></entry>
+ <entry><para> Total amount of data read or written (in KB).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294127" xreflabel=""/>rsz 1024</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294129" xreflabel=""/>Record size (size of each echo_client I/O, in KB).</para></entry>
+ <entry><para> rsz 1024</para></entry>
+ <entry><para> Record size (size of each echo_client I/O, in KB).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294131" xreflabel=""/>obj 8</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294133" xreflabel=""/>Total number of objects over all OSTs.</para></entry>
+ <entry><para> obj 8</para></entry>
+ <entry><para> Total number of objects over all OSTs.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294135" xreflabel=""/>thr 8</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294137" xreflabel=""/>Total number of threads over all OSTs and objects.</para></entry>
+ <entry><para> thr 8</para></entry>
+ <entry><para> Total number of threads over all OSTs and objects.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294139" xreflabel=""/>write</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294141" xreflabel=""/>Test name. If more tests have been specified, they all appear on the same line.</para></entry>
+ <entry><para> write</para></entry>
+ <entry><para> Test name. If more tests have been specified, they all appear on the same line.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294143" xreflabel=""/>613.54</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294145" xreflabel=""/>Aggregate bandwidth over all OSTs (measured by dividing the total number of MB by the elapsed time).</para></entry>
+ <entry><para> 613.54</para></entry>
+ <entry><para> Aggregate bandwidth over all OSTs (measured by dividing the total number of MB by the elapsed time).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294147" xreflabel=""/>[64, 82.00]</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438212_pgfId-1294149" xreflabel=""/>Minimum and maximum instantaneous bandwidths on an individual OST.</para></entry>
+ <entry><para> [64, 82.00]</para></entry>
+ <entry><para> Minimum and maximum instantaneous bandwidths on an individual OST.</para></entry>
</row>
</tbody>
</tgroup>
<note><para>Although the numbers of threads and objects are specified per-OST in the customization section of the script, the reported results are aggregated over all OSTs.</para></note>
</section>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438212_pgfId-1290063" xreflabel=""/>24.3.4.2 Visualizing Results</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1290064" xreflabel=""/>It is useful to import the obdfilter_survey script summary data (it is fixed width) into Excel (or any graphing package) and graph the bandwidth versus the number of threads for varying numbers of concurrent regions. This shows how the OSS performs for a given number of concurrently-accessed objects (files) with varying numbers of I/Os in flight.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305127" xreflabel=""/>It is also useful to monitor and record average disk I/O sizes during each test using the 'disk io size' histogram in the file /proc/fs/lustre/obdfilter/ (see <link xl:href="LustreProc.html#50438271_55057">Watching the OST Block I/O Stream</link> for details). These numbers help identify problems in the system when full-sized I/Os are not submitted to the underlying disk. This may be caused by problems in the device driver or Linux block layer.</para>
+ <title>24.3.4.2 Visualizing Results</title>
+ <para>It is useful to import the obdfilter_survey script summary data (it is fixed width) into Excel (or any graphing package) and graph the bandwidth versus the number of threads for varying numbers of concurrent regions. This shows how the OSS performs for a given number of concurrently-accessed objects (files) with varying numbers of I/Os in flight.</para>
+ <para>It is also useful to monitor and record average disk I/O sizes during each test using the 'disk io size' histogram in the file /proc/fs/lustre/obdfilter/ (see <link xl:href="LustreProc.html#50438271_55057">Watching the OST Block I/O Stream</link> for details). These numbers help identify problems in the system when full-sized I/Os are not submitted to the underlying disk. This may be caused by problems in the device driver or Linux block layer.</para>
<para> */brw_stats</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1290066" xreflabel=""/>The plot-obdfilter script included in the I/O toolkit is an example of processing output files to a .csv format and plotting a graph using gnuplot.</para>
+ <para>The plot-obdfilter script included in the I/O toolkit is an example of processing output files to a .csv format and plotting a graph using gnuplot.</para>
</section>
</section>
</section>
<section xml:id="dbdoclet.50438212_85136">
<title>24.4 Testing OST I/O Performance (ost_<anchor xml:id="dbdoclet.50438212_marker-1290067" xreflabel=""/>survey)</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1290069" xreflabel=""/>The ost_survey tool is a shell script that uses lfs setstripe to perform I/O against a single OST. The script writes a file (currently using dd) to each OST in the Lustre file system, and compares read and write speeds. The ost_survey tool is used to detect anomalies between otherwise identical disk subsystems.</para>
+ <para>The ost_survey tool is a shell script that uses lfs setstripe to perform I/O against a single OST. The script writes a file (currently using dd) to each OST in the Lustre file system, and compares read and write speeds. The ost_survey tool is used to detect anomalies between otherwise identical disk subsystems.</para>
<note><para>We have frequently discovered wide performance variations across all LUNs in a cluster. This may be caused by faulty disks, RAID parity reconstruction during the test, or faulty network hardware.</para></note>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1290071" xreflabel=""/>To run the ost_survey script, supply a file size (in KB) and the Lustre mount point. For example, run:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1290072" xreflabel=""/>$ ./ost-survey.sh 10 /mnt/lustre
+ <para>To run the ost_survey script, supply a file size (in KB) and the Lustre mount point. For example, run:</para>
+ <screen>$ ./ost-survey.sh 10 /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301765" xreflabel=""/>Typical output is:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1290073" xreflabel=""/>Average read Speed: 6.73
-<anchor xml:id="dbdoclet.50438212_pgfId-1290074" xreflabel=""/>Average write Speed: 5.41
-<anchor xml:id="dbdoclet.50438212_pgfId-1290075" xreflabel=""/>read - Worst OST indx 0 5.84 MB/s
-<anchor xml:id="dbdoclet.50438212_pgfId-1290076" xreflabel=""/>write - Worst OST indx 0 3.77 MB/s
-<anchor xml:id="dbdoclet.50438212_pgfId-1290077" xreflabel=""/>read - Best OST indx 1 7.38 MB/s
-<anchor xml:id="dbdoclet.50438212_pgfId-1290078" xreflabel=""/>write - Best OST indx 1 6.31 MB/s
-<anchor xml:id="dbdoclet.50438212_pgfId-1290079" xreflabel=""/>3 OST devices found
-<anchor xml:id="dbdoclet.50438212_pgfId-1290080" xreflabel=""/>Ost index 0 Read speed 5.84 Write speed \
+ <para>Typical output is:</para>
+ <screen>Average read Speed: 6.73
+Average write Speed: 5.41
+read - Worst OST indx 0 5.84 MB/s
+write - Worst OST indx 0 3.77 MB/s
+read - Best OST indx 1 7.38 MB/s
+write - Best OST indx 1 6.31 MB/s
+3 OST devices found
+Ost index 0 Read speed 5.84 Write speed \
3.77
-<anchor xml:id="dbdoclet.50438212_pgfId-1290081" xreflabel=""/>Ost index 0 Read time 0.17 Write time \
+Ost index 0 Read time 0.17 Write time \
0.27
-<anchor xml:id="dbdoclet.50438212_pgfId-1290082" xreflabel=""/>Ost index 1 Read speed 7.38 Write speed \
+Ost index 1 Read speed 7.38 Write speed \
6.31
-<anchor xml:id="dbdoclet.50438212_pgfId-1290083" xreflabel=""/>Ost index 1 Read time 0.14 Write time \
+Ost index 1 Read time 0.14 Write time \
0.16
-<anchor xml:id="dbdoclet.50438212_pgfId-1290084" xreflabel=""/>Ost index 2 Read speed 6.98 Write speed \
+Ost index 2 Read speed 6.98 Write speed \
6.16
-<anchor xml:id="dbdoclet.50438212_pgfId-1290085" xreflabel=""/>Ost index 2 Read time 0.14 Write time \
+Ost index 2 Read time 0.14 Write time \
0.16
</screen>
</section>
<section xml:id="dbdoclet.50438212_58201">
<title>24.5 Collecting Application Profiling Information (stats-collect)</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1299523" xreflabel=""/>The stats-collect utility contains the following scripts used to collect application profiling information from Lustre clients and servers:</para>
+ <para>The stats-collect utility contains the following scripts used to collect application profiling information from Lustre clients and servers:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1299524" xreflabel=""/>lstat.sh - Script for a single node that is run on each profile node.</para>
+ <para>lstat.sh - Script for a single node that is run on each profile node.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1299525" xreflabel=""/>gather_stats_everywhere.sh - Script that collect statistics.</para>
+ <para>gather_stats_everywhere.sh - Script that collect statistics.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1299526" xreflabel=""/>config.sh - Script that contains customized configuration descriptions.</para>
+ <para>config.sh - Script that contains customized configuration descriptions.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1299527" xreflabel=""/>The stats-collect utility requires:</para>
+ <para>The stats-collect utility requires:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1299528" xreflabel=""/> Lustre to be installed and set up on your cluster</para>
+ <para> Lustre to be installed and set up on your cluster</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1299529" xreflabel=""/> SSH and SCP access to these nodes without requiring a password</para>
+ <para> SSH and SCP access to these nodes without requiring a password</para>
</listitem>
</itemizedlist>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438212_pgfId-1299531" xreflabel=""/>24.5.1 Using stats-collect</title>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301821" xreflabel=""/>The stats-collect utility is configured by including profiling configuration variables in the config.sh script. Each configuration variable takes the following form, where 0 indicates statistics are to be collected only when the script starts and stops and <emphasis>n</emphasis> indicates the interval in seconds at which statistics are to be collected:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1301853" xreflabel=""/><emphasis><statistic></emphasis>_INTERVAL=<emphasis>[</emphasis>0<emphasis>|n]</emphasis></screen>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301887" xreflabel=""/>Statistics that can be collected include:</para>
+ <title>24.5.1 Using stats-collect</title>
+ <para>The stats-collect utility is configured by including profiling configuration variables in the config.sh script. Each configuration variable takes the following form, where 0 indicates statistics are to be collected only when the script starts and stops and <emphasis>n</emphasis> indicates the interval in seconds at which statistics are to be collected:</para>
+ <screen><emphasis><statistic></emphasis>_INTERVAL=<emphasis>[</emphasis>0<emphasis>|n]</emphasis></screen>
+ <para>Statistics that can be collected include:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301893" xreflabel=""/>VMSTAT - Memory and CPU usage and aggregate read/write operations</para>
+ <para>VMSTAT - Memory and CPU usage and aggregate read/write operations</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301898" xreflabel=""/>SERVICE - Lustre OST and MDT RPC service statistics</para>
+ <para>SERVICE - Lustre OST and MDT RPC service statistics</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301899" xreflabel=""/>BRW - OST block read/write statistics (brw_stats)</para>
+ <para>BRW - OST block read/write statistics (brw_stats)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301900" xreflabel=""/>SDIO - SCSI disk IO statistics (sd_iostats)</para>
+ <para>SDIO - SCSI disk IO statistics (sd_iostats)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301901" xreflabel=""/>MBALLOC - ldiskfs block allocation statistics</para>
+ <para>MBALLOC - ldiskfs block allocation statistics</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301902" xreflabel=""/>IO - Lustre target operations statistics</para>
+ <para>IO - Lustre target operations statistics</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301903" xreflabel=""/>JBD - ldisfs journal statistics</para>
+ <para>JBD - ldisfs journal statistics</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301904" xreflabel=""/>CLIENT - Lustre OSC request statistics</para>
+ <para>CLIENT - Lustre OSC request statistics</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1301989" xreflabel=""/>To collect profile information:</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1302001" xreflabel=""/> 1. Begin collecting statistics on each node specified in the config.sh script.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304893" xreflabel=""/>Starting the collect profile daemon on each node by entering:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1302024" xreflabel=""/>sh gather_stats_everywhere.sh config.sh start
+ <para>To collect profile information:</para>
+ <para> 1. Begin collecting statistics on each node specified in the config.sh script.</para>
+ <para>Starting the collect profile daemon on each node by entering:</para>
+ <screen>sh gather_stats_everywhere.sh config.sh start
</screen>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1302184" xreflabel=""/> 2. Run the test.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1302047" xreflabel=""/> 3. Stop collecting statistics on each node, clean up the temporary file, and create a profiling tarball.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1304926" xreflabel=""/>Enter:</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1302051" xreflabel=""/>sh gather_stats_everywhere.sh config.sh stop <emphasis><log_name.tgz></emphasis></screen>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1305257" xreflabel=""/>When <emphasis><log_name.tgz></emphasis> is specified, a profile tarball /tmp/<emphasis><log_name.tgz></emphasis> is created.</para>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1302055" xreflabel=""/> 4. Analyze the collected statistics and create a csv tarball for the specified profiling data.</para>
- <screen><anchor xml:id="dbdoclet.50438212_pgfId-1302059" xreflabel=""/>sh gather_stats_everywhere.sh config.sh analyse log_tarball.tgz csv
+ <para> 2. Run the test.</para>
+ <para> 3. Stop collecting statistics on each node, clean up the temporary file, and create a profiling tarball.</para>
+ <para>Enter:</para>
+ <screen>sh gather_stats_everywhere.sh config.sh stop <emphasis><log_name.tgz></emphasis></screen>
+ <para>When <emphasis><log_name.tgz></emphasis> is specified, a profile tarball /tmp/<emphasis><log_name.tgz></emphasis> is created.</para>
+ <para> 4. Analyze the collected statistics and create a csv tarball for the specified profiling data.</para>
+ <screen>sh gather_stats_everywhere.sh config.sh analyse log_tarball.tgz csv
</screen>
- <para><anchor xml:id="dbdoclet.50438212_pgfId-1302020" xreflabel=""/> </para>
+ <para> </para>
</section>
</section>
</chapter>
<info>
<title xml:id='configurationfilesmoduleparameters.title'>Configuration Files and Module Parameters</title>
</info>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293284" xreflabel=""/>This section describes configuration files and module parameters and includes the following sections:</para>
+ <para>This section describes configuration files and module parameters and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438293_15350"/></para>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438293_15350">
<title>35.1 Introduction</title>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293295" xreflabel=""/>LNET network hardware and routing are now configured via module parameters. Parameters should be specified in the /etc/modprobe.conf file, for example:</para>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293296" xreflabel=""/>alias lustre llite
-<anchor xml:id="dbdoclet.50438293_pgfId-1293297" xreflabel=""/>options lnet networks=tcp0,elan0
+ <para>LNET network hardware and routing are now configured via module parameters. Parameters should be specified in the /etc/modprobe.conf file, for example:</para>
+ <screen>alias lustre llite
+options lnet networks=tcp0,elan0
</screen>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293298" xreflabel=""/>The above option specifies that this node should use all the available TCP and Elan interfaces.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293299" xreflabel=""/>Module parameters are read when the module is first loaded. Type-specific LND modules (for instance, ksocklnd) are loaded automatically by the LNET module when LNET starts (typically upon modprobe ptlrpc).</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293300" xreflabel=""/>Under Linux 2.6, LNET configuration parameters can be viewed under /sys/module/; generic and acceptor parameters under LNET, and LND-specific parameters under the name of the corresponding LND.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293301" xreflabel=""/>Under Linux 2.4, sysfs is not available, but the LND-specific parameters are accessible via equivalent paths under /proc.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293302" xreflabel=""/>Important: All old (pre v.1.4.6) Lustre configuration lines should be removed from the module configuration files and replaced with the following. Make sure that CONFIG_KMOD is set in your linux.config so LNET can load the following modules it needs. The basic module files are:</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293303" xreflabel=""/>modprobe.conf (for Linux 2.6)</para>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293304" xreflabel=""/>alias lustre llite
-<anchor xml:id="dbdoclet.50438293_pgfId-1293305" xreflabel=""/>options lnet networks=tcp0,elan0
+ <para>The above option specifies that this node should use all the available TCP and Elan interfaces.</para>
+ <para>Module parameters are read when the module is first loaded. Type-specific LND modules (for instance, ksocklnd) are loaded automatically by the LNET module when LNET starts (typically upon modprobe ptlrpc).</para>
+ <para>Under Linux 2.6, LNET configuration parameters can be viewed under /sys/module/; generic and acceptor parameters under LNET, and LND-specific parameters under the name of the corresponding LND.</para>
+ <para>Under Linux 2.4, sysfs is not available, but the LND-specific parameters are accessible via equivalent paths under /proc.</para>
+ <para>Important: All old (pre v.1.4.6) Lustre configuration lines should be removed from the module configuration files and replaced with the following. Make sure that CONFIG_KMOD is set in your linux.config so LNET can load the following modules it needs. The basic module files are:</para>
+ <para>modprobe.conf (for Linux 2.6)</para>
+ <screen>alias lustre llite
+options lnet networks=tcp0,elan0
</screen>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293306" xreflabel=""/>modules.conf (for Linux 2.4)</para>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293307" xreflabel=""/>alias lustre llite
-<anchor xml:id="dbdoclet.50438293_pgfId-1293308" xreflabel=""/>options lnet networks=tcp0,elan0
+ <para>modules.conf (for Linux 2.4)</para>
+ <screen>alias lustre llite
+options lnet networks=tcp0,elan0
</screen>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293309" xreflabel=""/>For the following parameters, default option settings are shown in parenthesis. Changes to parameters marked with a W affect running systems. (Unmarked parameters can only be set when LNET loads for the first time.) Changes to parameters marked with Wc only have effect when connections are established (existing connections are not affected by these changes.)</para>
+ <para>For the following parameters, default option settings are shown in parenthesis. Changes to parameters marked with a W affect running systems. (Unmarked parameters can only be set when LNET loads for the first time.) Changes to parameters marked with Wc only have effect when connections are established (existing connections are not affected by these changes.)</para>
</section>
<section xml:id="dbdoclet.50438293_78010">
<title>35.2 Module <anchor xml:id="dbdoclet.50438293_marker-1293311" xreflabel=""/>Options</title>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293313" xreflabel=""/> With routed or other multi-network configurations, use ip2nets rather than networks, so all nodes can use the same configuration.</para>
+ <para> With routed or other multi-network configurations, use ip2nets rather than networks, so all nodes can use the same configuration.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293314" xreflabel=""/> For a routed network, use the same 'routes†configuration everywhere. Nodes specified as routers automatically enable forwarding and any routes that are not relevant to a particular node are ignored. Keep a common configuration to guarantee that all nodes have consistent routing tables.</para>
+ <para> For a routed network, use the same 'routes†configuration everywhere. Nodes specified as routers automatically enable forwarding and any routes that are not relevant to a particular node are ignored. Keep a common configuration to guarantee that all nodes have consistent routing tables.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293315" xreflabel=""/> A separate modprobe.conf.lnet included from modprobe.conf makes distributing the configuration much easier.</para>
+ <para> A separate modprobe.conf.lnet included from modprobe.conf makes distributing the configuration much easier.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293316" xreflabel=""/> If you set config_on_load=1, LNET starts at modprobe time rather than waiting for Lustre to start. This ensures routers start working at module load time.</para>
+ <para> If you set config_on_load=1, LNET starts at modprobe time rather than waiting for Lustre to start. This ensures routers start working at module load time.</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293317" xreflabel=""/># lctl
-<anchor xml:id="dbdoclet.50438293_pgfId-1293318" xreflabel=""/># lctl> net down
+ <screen># lctl
+# lctl> net down
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293319" xreflabel=""/> Remember the lctl ping {nid} command - it is a handy way to check your LNET configuration.</para>
+ <para> Remember the lctl ping {nid} command - it is a handy way to check your LNET configuration.</para>
</listitem>
</itemizedlist>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438293_pgfId-1293321" xreflabel=""/>35.2.1 <anchor xml:id="dbdoclet.50438293_94707" xreflabel=""/>LNET <anchor xml:id="dbdoclet.50438293_marker-1293320" xreflabel=""/>Options</title>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293322" xreflabel=""/>This section describes LNET options.</para>
+ <title>35.2.1 <anchor xml:id="dbdoclet.50438293_94707" xreflabel=""/>LNET <anchor xml:id="dbdoclet.50438293_marker-1293320" xreflabel=""/>Options</title>
+ <para>This section describes LNET options.</para>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438293_pgfId-1293323" xreflabel=""/>35.2.1.1 Network Topology</title>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293324" xreflabel=""/>Network topology module parameters determine which networks a node should join, whether it should route between these networks, and how it communicates with non-local networks.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293362" xreflabel=""/>Here is a list of various networks and the supported software stacks:</para>
+ <title>35.2.1.1 Network Topology</title>
+ <para>Network topology module parameters determine which networks a node should join, whether it should route between these networks, and how it communicates with non-local networks.</para>
+ <para>Here is a list of various networks and the supported software stacks:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438293_pgfId-1293327" xreflabel=""/>Network</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438293_pgfId-1293329" xreflabel=""/>Software Stack</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Network</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Software Stack</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1293343" xreflabel=""/>o2ib</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1293345" xreflabel=""/>OFED Version 2</para></entry>
+ <entry><para> o2ib</para></entry>
+ <entry><para> OFED Version 2</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1293351" xreflabel=""/>mx</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1293353" xreflabel=""/>Myrinet MX</para></entry>
+ <entry><para> mx</para></entry>
+ <entry><para> Myrinet MX</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1293355" xreflabel=""/>gm</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1293357" xreflabel=""/>Myrinet GM-2</para></entry>
+ <entry><para> gm</para></entry>
+ <entry><para> Myrinet GM-2</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
<note><para>Lustre ignores the loopback interface (lo0), but Lustre use any IP addresses aliased to the loopback (by default). When in doubt, explicitly specify networks.</para></note>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293364" xreflabel=""/><emphasis role="bold">ip2nets</emphasis> ("") is a string that lists globally-available networks, each with a set of IP address ranges. LNET determines the locally-available networks from this list by matching the IP address ranges with the local IPs of a node. The purpose of this option is to be able to use the same modules.conf file across a variety of nodes on different networks. The string has the following syntax.</para>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293365" xreflabel=""/><ip2nets> :== <net-match> [ <comment> ] { <net-sep> <net-match> }
-<anchor xml:id="dbdoclet.50438293_pgfId-1293366" xreflabel=""/><net-match> :== [ <w> ] <net-spec> <w> <ip-range> { <w> <ip-range> }
-<anchor xml:id="dbdoclet.50438293_pgfId-1293367" xreflabel=""/>[ <w> ]
-<anchor xml:id="dbdoclet.50438293_pgfId-1293368" xreflabel=""/><net-spec> :== <network> [ "(" <interface-list> ")" ]
-<anchor xml:id="dbdoclet.50438293_pgfId-1293369" xreflabel=""/><network> :== <nettype> [ <number> ]
-<anchor xml:id="dbdoclet.50438293_pgfId-1293370" xreflabel=""/><nettype> :== "tcp" | "elan" | "openib" | ...
-<anchor xml:id="dbdoclet.50438293_pgfId-1293371" xreflabel=""/><iface-list> :== <interface> [ "," <iface-list> ]
-<anchor xml:id="dbdoclet.50438293_pgfId-1293372" xreflabel=""/><ip-range> :== <r-expr> "." <r-expr> "." <r-expr> "." <r-expr>
-<anchor xml:id="dbdoclet.50438293_pgfId-1293373" xreflabel=""/><r-expr> :== <number> | "*" | "[" <r-list> "]"
-<anchor xml:id="dbdoclet.50438293_pgfId-1293374" xreflabel=""/><r-list> :== <range> [ "," <r-list> ]
-<anchor xml:id="dbdoclet.50438293_pgfId-1293375" xreflabel=""/><range> :== <number> [ "-" <number> [ "/" <number> ] ]
-<anchor xml:id="dbdoclet.50438293_pgfId-1293376" xreflabel=""/><comment :== "#" { <non-net-sep-chars> }
-<anchor xml:id="dbdoclet.50438293_pgfId-1293377" xreflabel=""/><net-sep> :== ";" | "\n"
-<anchor xml:id="dbdoclet.50438293_pgfId-1293378" xreflabel=""/><w> :== <whitespace-chars> { <whitespace-chars> }
+ <para><emphasis role="bold">ip2nets</emphasis> ("") is a string that lists globally-available networks, each with a set of IP address ranges. LNET determines the locally-available networks from this list by matching the IP address ranges with the local IPs of a node. The purpose of this option is to be able to use the same modules.conf file across a variety of nodes on different networks. The string has the following syntax.</para>
+ <screen><ip2nets> :== <net-match> [ <comment> ] { <net-sep> <net-match> }
+<net-match> :== [ <w> ] <net-spec> <w> <ip-range> { <w> <ip-range> }
+[ <w> ]
+<net-spec> :== <network> [ "(" <interface-list> ")" ]
+<network> :== <nettype> [ <number> ]
+<nettype> :== "tcp" | "elan" | "openib" | ...
+<iface-list> :== <interface> [ "," <iface-list> ]
+<ip-range> :== <r-expr> "." <r-expr> "." <r-expr> "." <r-expr>
+<r-expr> :== <number> | "*" | "[" <r-list> "]"
+<r-list> :== <range> [ "," <r-list> ]
+<range> :== <number> [ "-" <number> [ "/" <number> ] ]
+<comment :== "#" { <non-net-sep-chars> }
+<net-sep> :== ";" | "\n"
+<w> :== <whitespace-chars> { <whitespace-chars> }
</screen>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293379" xreflabel=""/><<emphasis role="bold">net-spec</emphasis>> contains enough information to uniquely identify the network and load an appropriate LND. The LND determines the missing "address-within-network" part of the NID based on the interfaces it can use.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293380" xreflabel=""/><<emphasis role="bold">iface-list</emphasis>> specifies which hardware interface the network can use. If omitted, all interfaces are used. LNDs that do not support the <iface-list> syntax cannot be configured to use particular interfaces and just use what is there. Only a single instance of these LNDs can exist on a node at any time, and <iface-list> must be omitted.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293381" xreflabel=""/><<emphasis role="bold">net-match</emphasis>> entries are scanned in the order declared to see if one of the node's IP addresses matches one of the <ip-range> expressions. If there is a match, <net-spec> specifies the network to instantiate. Note that it is the first match for a particular network that counts. This can be used to simplify the match expression for the general case by placing it after the special cases. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293382" xreflabel=""/>ip2nets="tcp(eth1,eth2) 134.32.1.[4-10/2]; tcp(eth1) *.*.*.*"
+ <para><<emphasis role="bold">net-spec</emphasis>> contains enough information to uniquely identify the network and load an appropriate LND. The LND determines the missing "address-within-network" part of the NID based on the interfaces it can use.</para>
+ <para><<emphasis role="bold">iface-list</emphasis>> specifies which hardware interface the network can use. If omitted, all interfaces are used. LNDs that do not support the <iface-list> syntax cannot be configured to use particular interfaces and just use what is there. Only a single instance of these LNDs can exist on a node at any time, and <iface-list> must be omitted.</para>
+ <para><<emphasis role="bold">net-match</emphasis>> entries are scanned in the order declared to see if one of the node's IP addresses matches one of the <ip-range> expressions. If there is a match, <net-spec> specifies the network to instantiate. Note that it is the first match for a particular network that counts. This can be used to simplify the match expression for the general case by placing it after the special cases. For example:</para>
+ <screen>ip2nets="tcp(eth1,eth2) 134.32.1.[4-10/2]; tcp(eth1) *.*.*.*"
</screen>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293383" xreflabel=""/>4 nodes on the 134.32.1.* network have 2 interfaces (134.32.1.{4,6,8,10}) but all the rest have 1.</para>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293384" xreflabel=""/>ip2nets="<emphasis role="bold">vib</emphasis> 192.168.0.*; tcp(eth2) 192.168.0.[1,7,4,12]"
+ <para>4 nodes on the 134.32.1.* network have 2 interfaces (134.32.1.{4,6,8,10}) but all the rest have 1.</para>
+ <screen>ip2nets="<emphasis role="bold">vib</emphasis> 192.168.0.*; tcp(eth2) 192.168.0.[1,7,4,12]"
</screen>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293385" xreflabel=""/>This describes an IB cluster on 192.168.0.*. Four of these nodes also have IP interfaces; these four could be used as routers.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293386" xreflabel=""/>Note that match-all expressions (For instance, *.*.*.*) effectively mask all other</para>
+ <para>This describes an IB cluster on 192.168.0.*. Four of these nodes also have IP interfaces; these four could be used as routers.</para>
+ <para>Note that match-all expressions (For instance, *.*.*.*) effectively mask all other</para>
<para> <net-match> entries specified after them. They should be used with caution.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293387" xreflabel=""/>Here is a more complicated situation, the route parameter is explained below. We have:</para>
+ <para>Here is a more complicated situation, the route parameter is explained below. We have:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293388" xreflabel=""/> Two TCP subnets</para>
+ <para> Two TCP subnets</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293389" xreflabel=""/> One Elan subnet</para>
+ <para> One Elan subnet</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293390" xreflabel=""/> One machine set up as a router, with both TCP and Elan interfaces</para>
+ <para> One machine set up as a router, with both TCP and Elan interfaces</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293391" xreflabel=""/> IP over Elan configured, but only IP will be used to label the nodes.</para>
+ <para> IP over Elan configured, but only IP will be used to label the nodes.</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293392" xreflabel=""/>options lnet ip2nets=â€tcp 198.129.135.* 192.128.88.98; \
+ <screen>options lnet ip2nets=â€tcp 198.129.135.* 192.128.88.98; \
elan 198.128.88.98 198.129.135.3; \
routes='cp 1022@elan # Elan NID of router; \
elan 198.128.88.98@tcp # TCP NID of router '
</screen>
</section>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438293_pgfId-1293393" xreflabel=""/>35.2.1.2 networks ("tcp")</title>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293394" xreflabel=""/>This is an alternative to "ip2nets" which can be used to specify the networks to be instantiated explicitly. The syntax is a simple comma separated list of <net-spec>s (see above). The default is only used if neither 'ip2nets†nor 'networks†is specified.</para>
+ <title>35.2.1.2 networks ("tcp")</title>
+ <para>This is an alternative to "ip2nets" which can be used to specify the networks to be instantiated explicitly. The syntax is a simple comma separated list of <net-spec>s (see above). The default is only used if neither 'ip2nets†nor 'networks†is specified.</para>
</section>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438293_pgfId-1293395" xreflabel=""/>35.2.1.3 routes ("")</title>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293396" xreflabel=""/>This is a string that lists networks and the NIDs of routers that forward to them.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293397" xreflabel=""/>It has the following syntax (<w> is one or more whitespace characters):</para>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293398" xreflabel=""/><routes> :== <route>{ ; <route> }
-<anchor xml:id="dbdoclet.50438293_pgfId-1293399" xreflabel=""/><route> :== [<net>[<w><hopcount>]<w><nid>{<w><nid>}
+ <title>35.2.1.3 routes ("")</title>
+ <para>This is a string that lists networks and the NIDs of routers that forward to them.</para>
+ <para>It has the following syntax (<w> is one or more whitespace characters):</para>
+ <screen><routes> :== <route>{ ; <route> }
+<route> :== [<net>[<w><hopcount>]<w><nid>{<w><nid>}
</screen>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293400" xreflabel=""/>So a node on the network tcp1 that needs to go through a router to get to the Elan network:</para>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293401" xreflabel=""/>options lnet networks=tcp1 routes="elan 1 192.168.2.2@tcpA"
+ <para>So a node on the network tcp1 that needs to go through a router to get to the Elan network:</para>
+ <screen>options lnet networks=tcp1 routes="elan 1 192.168.2.2@tcpA"
</screen>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293402" xreflabel=""/>The hopcount is used to help choose the best path between multiply-routed configurations.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293403" xreflabel=""/>A simple but powerful expansion syntax is provided, both for target networks and router NIDs as follows.</para>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293404" xreflabel=""/><expansion> :== "[" <entry> { "," <entry> } "]"
-<anchor xml:id="dbdoclet.50438293_pgfId-1293405" xreflabel=""/><entry> :== <numeric range> | <non-numeric item>
-<anchor xml:id="dbdoclet.50438293_pgfId-1293406" xreflabel=""/><numeric range> :== <number> [ "-" <number> [ "/" <number> ] ]
+ <para>The hopcount is used to help choose the best path between multiply-routed configurations.</para>
+ <para>A simple but powerful expansion syntax is provided, both for target networks and router NIDs as follows.</para>
+ <screen><expansion> :== "[" <entry> { "," <entry> } "]"
+<entry> :== <numeric range> | <non-numeric item>
+<numeric range> :== <number> [ "-" <number> [ "/" <number> ] ]
</screen>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293407" xreflabel=""/>The expansion is a list enclosed in square brackets. Numeric items in the list may be a single number, a contiguous range of numbers, or a strided range of numbers. For example, routes="elan 192.168.1.[22-24]@tcp" says that network elan0 is adjacent (hopcount defaults to 1); and is accessible via 3 routers on the tcp0 network (192.168.1.22@tcp, 192.168.1.23@tcp and 192.168.1.24@tcp).</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293408" xreflabel=""/>routes="[tcp,<emphasis role="bold">vib</emphasis>] 2 [8-14/2]@elan" says that 2 networks (tcp0 and vib0) are accessible through 4 routers (8@elan, 10@elan, 12@elan and 14@elan). The hopcount of 2 means that traffic to both these networks will be traversed 2 routers - first one of the routers specified in this entry, then one more.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293409" xreflabel=""/>Duplicate entries, entries that route to a local network, and entries that specify routers on a non-local network are ignored.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293410" xreflabel=""/>Equivalent entries are resolved in favor of the route with the shorter hopcount. The hopcount, if omitted, defaults to 1 (the remote network is adjacent).</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293411" xreflabel=""/>It is an error to specify routes to the same destination with routers on different local networks.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293412" xreflabel=""/>If the target network string contains no expansions, then the hopcount defaults to 1 and may be omitted (that is, the remote network is adjacent). In practice, this is true for most multi-network configurations. It is an error to specify an inconsistent hop count for a given target network. This is why an explicit hopcount is required if the target network string specifies more than one network.</para>
+ <para>The expansion is a list enclosed in square brackets. Numeric items in the list may be a single number, a contiguous range of numbers, or a strided range of numbers. For example, routes="elan 192.168.1.[22-24]@tcp" says that network elan0 is adjacent (hopcount defaults to 1); and is accessible via 3 routers on the tcp0 network (192.168.1.22@tcp, 192.168.1.23@tcp and 192.168.1.24@tcp).</para>
+ <para>routes="[tcp,<emphasis role="bold">vib</emphasis>] 2 [8-14/2]@elan" says that 2 networks (tcp0 and vib0) are accessible through 4 routers (8@elan, 10@elan, 12@elan and 14@elan). The hopcount of 2 means that traffic to both these networks will be traversed 2 routers - first one of the routers specified in this entry, then one more.</para>
+ <para>Duplicate entries, entries that route to a local network, and entries that specify routers on a non-local network are ignored.</para>
+ <para>Equivalent entries are resolved in favor of the route with the shorter hopcount. The hopcount, if omitted, defaults to 1 (the remote network is adjacent).</para>
+ <para>It is an error to specify routes to the same destination with routers on different local networks.</para>
+ <para>If the target network string contains no expansions, then the hopcount defaults to 1 and may be omitted (that is, the remote network is adjacent). In practice, this is true for most multi-network configurations. It is an error to specify an inconsistent hop count for a given target network. This is why an explicit hopcount is required if the target network string specifies more than one network.</para>
</section>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438293_pgfId-1293413" xreflabel=""/>35.2.1.4 forwarding ("")</title>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293414" xreflabel=""/>This is a string that can be set either to "enabled" or "disabled" for explicit control of whether this node should act as a router, forwarding communications between all local networks.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293415" xreflabel=""/>A standalone router can be started by simply starting LNET ('modprobe ptlrpcâ€) with appropriate network topology options.</para>
+ <title>35.2.1.4 forwarding ("")</title>
+ <para>This is a string that can be set either to "enabled" or "disabled" for explicit control of whether this node should act as a router, forwarding communications between all local networks.</para>
+ <para>A standalone router can be started by simply starting LNET ('modprobe ptlrpcâ€) with appropriate network topology options.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438293_pgfId-1294462" xreflabel=""/>Variable</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438293_pgfId-1294464" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Variable</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294507" xreflabel=""/>acceptor</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294509" xreflabel=""/> The acceptor is a TCP/IP service that some LNDs use to establish communications. If a local network requires it and it has not been disabled, the acceptor listens on a single port for connection requests that it redirects to the appropriate local network. The acceptor is part of the LNET module and configured by the following options:</para><itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1294510" xreflabel=""/><emphasis role="bold">secure</emphasis> - Accept connections only from reserved TCP ports (< 1023).</para>
+ <entry><para> acceptor</para></entry>
+ <entry><para> The acceptor is a TCP/IP service that some LNDs use to establish communications. If a local network requires it and it has not been disabled, the acceptor listens on a single port for connection requests that it redirects to the appropriate local network. The acceptor is part of the LNET module and configured by the following options:</para><itemizedlist><listitem>
+ <para><emphasis role="bold">secure</emphasis> - Accept connections only from reserved TCP ports (< 1023).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1294511" xreflabel=""/><emphasis role="bold">all</emphasis> - Accept connections from any TCP port. NOTE: this is required for liblustre clients to allow connections on non-privileged ports.</para>
+ <para><emphasis role="bold">all</emphasis> - Accept connections from any TCP port. NOTE: this is required for liblustre clients to allow connections on non-privileged ports.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1294512" xreflabel=""/><emphasis role="bold">none</emphasis> - Do not run the acceptor.</para>
+ <para><emphasis role="bold">none</emphasis> - Do not run the acceptor.</para>
</listitem>
</itemizedlist></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294514" xreflabel=""/>accept_port</para><para> (988)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294516" xreflabel=""/> Port number on which the acceptor should listen for connection requests. All nodes in a site configuration that require an acceptor must use the same port.</para></entry>
+ <entry><para> accept_port</para><para> (988)</para></entry>
+ <entry><para> Port number on which the acceptor should listen for connection requests. All nodes in a site configuration that require an acceptor must use the same port.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294522" xreflabel=""/>accept_backlog</para><para> (127)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294524" xreflabel=""/>Maximum length that the queue of pending connections may grow to (see listen(2)).</para></entry>
+ <entry><para> accept_backlog</para><para> (127)</para></entry>
+ <entry><para> Maximum length that the queue of pending connections may grow to (see listen(2)).</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294526" xreflabel=""/>accept_timeout</para><para> (5, W)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294528" xreflabel=""/>Maximum time in seconds the acceptor is allowed to block while communicating with a peer.</para></entry>
+ <entry><para> accept_timeout</para><para> (5, W)</para></entry>
+ <entry><para> Maximum time in seconds the acceptor is allowed to block while communicating with a peer.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294530" xreflabel=""/>accept_proto_version</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294532" xreflabel=""/> Version of the acceptor protocol that should be used by outgoing connection requests. It defaults to the most recent acceptor protocol version, but it may be set to the previous version to allow the node to initiate connections with nodes that only understand that version of the acceptor protocol. The acceptor can, with some restrictions, handle either version (that is, it can accept connections from both 'old' and 'new' peers). For the current version of the acceptor protocol (version 1), the acceptor is compatible with old peers if it is only required by a single local network.</para></entry>
+ <entry><para> accept_proto_version</para></entry>
+ <entry><para> Version of the acceptor protocol that should be used by outgoing connection requests. It defaults to the most recent acceptor protocol version, but it may be set to the previous version to allow the node to initiate connections with nodes that only understand that version of the acceptor protocol. The acceptor can, with some restrictions, handle either version (that is, it can accept connections from both 'old' and 'new' peers). For the current version of the acceptor protocol (version 1), the acceptor is compatible with old peers if it is only required by a single local network.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293446" xreflabel=""/></para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293447" xreflabel=""/></para>
+ <para></para>
+ <para></para>
</section>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438293_pgfId-1293449" xreflabel=""/>35.2.2 SOCKLND <anchor xml:id="dbdoclet.50438293_marker-1293448" xreflabel=""/>Kernel TCP/IP LND</title>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293450" xreflabel=""/>The SOCKLND kernel TCP/IP LND (socklnd) is connection-based and uses the acceptor to establish communications via sockets with its peers.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293451" xreflabel=""/>It supports multiple instances and load balances dynamically over multiple interfaces. If no interfaces are specified by the ip2nets or networks module parameter, all non-loopback IP interfaces are used. The address-within-network is determined by the address of the first IP interface an instance of the socklnd encounters.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293452" xreflabel=""/>Consider a node on the 'edge†of an InfiniBand network, with a low-bandwidth management Ethernet (eth0), IP over IB configured (ipoib0), and a pair of GigE NICs (eth1,eth2) providing off-cluster connectivity. This node should be configured with "networks=<emphasis role="bold">vib</emphasis>,tcp(eth1,eth2)†to ensure that the socklnd ignores the management Ethernet and IPoIB.</para>
+ <title>35.2.2 SOCKLND <anchor xml:id="dbdoclet.50438293_marker-1293448" xreflabel=""/>Kernel TCP/IP LND</title>
+ <para>The SOCKLND kernel TCP/IP LND (socklnd) is connection-based and uses the acceptor to establish communications via sockets with its peers.</para>
+ <para>It supports multiple instances and load balances dynamically over multiple interfaces. If no interfaces are specified by the ip2nets or networks module parameter, all non-loopback IP interfaces are used. The address-within-network is determined by the address of the first IP interface an instance of the socklnd encounters.</para>
+ <para>Consider a node on the 'edge†of an InfiniBand network, with a low-bandwidth management Ethernet (eth0), IP over IB configured (ipoib0), and a pair of GigE NICs (eth1,eth2) providing off-cluster connectivity. This node should be configured with "networks=<emphasis role="bold">vib</emphasis>,tcp(eth1,eth2)†to ensure that the socklnd ignores the management Ethernet and IPoIB.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438293_pgfId-1294541" xreflabel=""/>Variable</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438293_pgfId-1294543" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Variable</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294621" xreflabel=""/>timeout</para><para> (50,W)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294623" xreflabel=""/> Time (in seconds) that communications may be stalled before the LND completes them with failure.</para></entry>
+ <entry><para> timeout</para><para> (50,W)</para></entry>
+ <entry><para> Time (in seconds) that communications may be stalled before the LND completes them with failure.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294625" xreflabel=""/>nconnds</para><para> (4)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294627" xreflabel=""/>Sets the number of connection daemons.</para></entry>
+ <entry><para> nconnds</para><para> (4)</para></entry>
+ <entry><para> Sets the number of connection daemons.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294629" xreflabel=""/>min_reconnectms</para><para> (1000,W)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294631" xreflabel=""/>Minimum connection retry interval (in milliseconds). After a failed connection attempt, this is the time that must elapse before the first retry. As connections attempts fail, this time is doubled on each successive retry up to a maximum of 'max_reconnectms'.</para></entry>
+ <entry><para> min_reconnectms</para><para> (1000,W)</para></entry>
+ <entry><para> Minimum connection retry interval (in milliseconds). After a failed connection attempt, this is the time that must elapse before the first retry. As connections attempts fail, this time is doubled on each successive retry up to a maximum of 'max_reconnectms'.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294633" xreflabel=""/>max_reconnectms</para><para> (6000,W)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294635" xreflabel=""/>Maximum connection retry interval (in milliseconds).</para></entry>
+ <entry><para> max_reconnectms</para><para> (6000,W)</para></entry>
+ <entry><para> Maximum connection retry interval (in milliseconds).</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294637" xreflabel=""/>eager_ack</para><para> (0 on linux,</para><para> 1 on darwin,W)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294639" xreflabel=""/>Boolean that determines whether the socklnd should attempt to flush sends on message boundaries.</para></entry>
+ <entry><para> eager_ack</para><para> (0 on linux,</para><para> 1 on darwin,W)</para></entry>
+ <entry><para> Boolean that determines whether the socklnd should attempt to flush sends on message boundaries.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294641" xreflabel=""/>typed_conns</para><para> (1,Wc)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294643" xreflabel=""/>Boolean that determines whether the socklnd should use different sockets for different types of messages. When clear, all communication with a particular peer takes place on the same socket. Otherwise, separate sockets are used for bulk sends, bulk receives and everything else.</para></entry>
+ <entry><para> typed_conns</para><para> (1,Wc)</para></entry>
+ <entry><para> Boolean that determines whether the socklnd should use different sockets for different types of messages. When clear, all communication with a particular peer takes place on the same socket. Otherwise, separate sockets are used for bulk sends, bulk receives and everything else.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294645" xreflabel=""/>min_bulk</para><para> (1024,W)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294647" xreflabel=""/>Determines when a message is considered "bulk".</para></entry>
+ <entry><para> min_bulk</para><para> (1024,W)</para></entry>
+ <entry><para> Determines when a message is considered "bulk".</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294649" xreflabel=""/>tx_buffer_size, rx_buffer_size</para><para> (8388608,Wc)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294651" xreflabel=""/> Socket buffer sizes. Setting this option to zero (0), allows the system to auto-tune buffer sizes. WARNING: Be very careful changing this value as improper sizing can harm performance.</para></entry>
+ <entry><para> tx_buffer_size, rx_buffer_size</para><para> (8388608,Wc)</para></entry>
+ <entry><para> Socket buffer sizes. Setting this option to zero (0), allows the system to auto-tune buffer sizes. WARNING: Be very careful changing this value as improper sizing can harm performance.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294653" xreflabel=""/>nagle</para><para> (0,Wc)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294655" xreflabel=""/>Boolean that determines if nagle should be enabled. It should never be set in production systems.</para></entry>
+ <entry><para> nagle</para><para> (0,Wc)</para></entry>
+ <entry><para> Boolean that determines if nagle should be enabled. It should never be set in production systems.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294657" xreflabel=""/>keepalive_idle</para><para> (30,Wc)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294659" xreflabel=""/>Time (in seconds) that a socket can remain idle before a keepalive probe is sent. Setting this value to zero (0) disables keepalives.</para></entry>
+ <entry><para> keepalive_idle</para><para> (30,Wc)</para></entry>
+ <entry><para> Time (in seconds) that a socket can remain idle before a keepalive probe is sent. Setting this value to zero (0) disables keepalives.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294661" xreflabel=""/>keepalive_intvl</para><para> (2,Wc)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294663" xreflabel=""/>Time (in seconds) to repeat unanswered keepalive probes. Setting this value to zero (0) disables keepalives.</para></entry>
+ <entry><para> keepalive_intvl</para><para> (2,Wc)</para></entry>
+ <entry><para> Time (in seconds) to repeat unanswered keepalive probes. Setting this value to zero (0) disables keepalives.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294665" xreflabel=""/>keepalive_count</para><para> (10,Wc)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294667" xreflabel=""/>Number of unanswered keepalive probes before pronouncing socket (hence peer) death.</para></entry>
+ <entry><para> keepalive_count</para><para> (10,Wc)</para></entry>
+ <entry><para> Number of unanswered keepalive probes before pronouncing socket (hence peer) death.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294669" xreflabel=""/>enable_irq_affinity</para><para> (0,Wc)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294671" xreflabel=""/>Boolean that determines whether to enable IRQ affinity. The default is zero (0).</para><para><anchor xml:id="dbdoclet.50438293_pgfId-1296138" xreflabel=""/>When set, socklnd attempts to maximize performance by handling device interrupts and data movement for particular (hardware) interfaces on particular CPUs. This option is not available on all platforms. This option requires an SMP system to exist and produces best performance with multiple NICs. Systems with multiple CPUs and a single NIC may see increase in the performance with this parameter disabled.</para></entry>
+ <entry><para> enable_irq_affinity</para><para> (0,Wc)</para></entry>
+ <entry><para> Boolean that determines whether to enable IRQ affinity. The default is zero (0).</para><para>When set, socklnd attempts to maximize performance by handling device interrupts and data movement for particular (hardware) interfaces on particular CPUs. This option is not available on all platforms. This option requires an SMP system to exist and produces best performance with multiple NICs. Systems with multiple CPUs and a single NIC may see increase in the performance with this parameter disabled.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1294673" xreflabel=""/>zc_min_frag</para><para> (2048,W)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1294675" xreflabel=""/>Determines the minimum message fragment that should be considered for zero-copy sends. Increasing it above the platform's PAGE_SIZE disables all zero copy sends. This option is not available on all platforms.</para></entry>
+ <entry><para> zc_min_frag</para><para> (2048,W)</para></entry>
+ <entry><para> Determines the minimum message fragment that should be considered for zero-copy sends. Increasing it above the platform's PAGE_SIZE disables all zero copy sends. This option is not available on all platforms.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438293_pgfId-1293720" xreflabel=""/>35.2.3 Portals <anchor xml:id="dbdoclet.50438293_marker-1293719" xreflabel=""/>LND (Linux)</title>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293721" xreflabel=""/>The Portals LND Linux (ptllnd) can be used as a interface layer to communicate with Sandia Portals networking devices. This version is intended to work on Cray XT3 Linux nodes that use Cray Portals as a network transport.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293722" xreflabel=""/><emphasis role="bold">Message Buffers</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293723" xreflabel=""/>When ptllnd starts up, it allocates and posts sufficient message buffers to allow all expected peers (set by concurrent_peers) to send one unsolicited message. The first message that a peer actually sends is a</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293724" xreflabel=""/>(so-called) "HELLO" message, used to negotiate how much additional buffering to setup (typically 8 messages). If 10000 peers actually exist, then enough buffers are posted for 80000 messages.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293725" xreflabel=""/>The maximum message size is set by the max_msg_size module parameter (default value is 512). This parameter sets the bulk transfer breakpoint. Below this breakpoint, payload data is sent in the message itself. Above this breakpoint, a buffer descriptor is sent and the receiver gets the actual payload.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293726" xreflabel=""/>The buffer size is set by the rxb_npages module parameter (default value is 1). The default conservatively avoids allocation problems due to kernel memory fragmentation. However, increasing this value to 2 is probably not risky.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293727" xreflabel=""/>The ptllnd also keeps an additional rxb_nspare buffers (default value is 8) posted to account for full buffers being handled.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293728" xreflabel=""/>Assuming a 4K page size with 10000 peers, 1258 buffers can be expected to be posted at startup, increasing to a maximum of 10008 as peers that are actually connected. By doubling rxb_npages halving max_msg_size, this number can be reduced by a factor of 4.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293729" xreflabel=""/><emphasis role="bold">ME/MD Queue Length</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293730" xreflabel=""/>The ptllnd uses a single portal set by the portal module parameter (default value of 9) for both message and bulk buffers. Message buffers are always attached with PTL_INS_AFTER and match anything sent with "message" matchbits. Bulk buffers are always attached with PTL_INS_BEFORE and match only specific matchbits for that particular bulk transfer.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293731" xreflabel=""/>This scheme assumes that the majority of ME / MDs posted are for "message" buffers, and that the overhead of searching through the preceding "bulk" buffers is acceptable. Since the number of "bulk" buffers posted at any time is also dependent on the bulk transfer breakpoint set by max_msg_size, this seems like an issue worth measuring at scale.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293732" xreflabel=""/><emphasis role="bold">TX Descriptors</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293733" xreflabel=""/>The ptllnd has a pool of so-called "tx descriptors", which it uses not only for outgoing messages, but also to hold state for bulk transfers requested by incoming messages. This pool should scale with the total number of peers.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293734" xreflabel=""/>To enable the building of the Portals LND (ptllnd.ko) configure with this option:</para>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1297547" xreflabel=""/>./configure --with-portals=<path-to-portals-headers></screen>
+ <title>35.2.3 Portals <anchor xml:id="dbdoclet.50438293_marker-1293719" xreflabel=""/>LND (Linux)</title>
+ <para>The Portals LND Linux (ptllnd) can be used as a interface layer to communicate with Sandia Portals networking devices. This version is intended to work on Cray XT3 Linux nodes that use Cray Portals as a network transport.</para>
+ <para><emphasis role="bold">Message Buffers</emphasis></para>
+ <para>When ptllnd starts up, it allocates and posts sufficient message buffers to allow all expected peers (set by concurrent_peers) to send one unsolicited message. The first message that a peer actually sends is a</para>
+ <para>(so-called) "HELLO" message, used to negotiate how much additional buffering to setup (typically 8 messages). If 10000 peers actually exist, then enough buffers are posted for 80000 messages.</para>
+ <para>The maximum message size is set by the max_msg_size module parameter (default value is 512). This parameter sets the bulk transfer breakpoint. Below this breakpoint, payload data is sent in the message itself. Above this breakpoint, a buffer descriptor is sent and the receiver gets the actual payload.</para>
+ <para>The buffer size is set by the rxb_npages module parameter (default value is 1). The default conservatively avoids allocation problems due to kernel memory fragmentation. However, increasing this value to 2 is probably not risky.</para>
+ <para>The ptllnd also keeps an additional rxb_nspare buffers (default value is 8) posted to account for full buffers being handled.</para>
+ <para>Assuming a 4K page size with 10000 peers, 1258 buffers can be expected to be posted at startup, increasing to a maximum of 10008 as peers that are actually connected. By doubling rxb_npages halving max_msg_size, this number can be reduced by a factor of 4.</para>
+ <para><emphasis role="bold">ME/MD Queue Length</emphasis></para>
+ <para>The ptllnd uses a single portal set by the portal module parameter (default value of 9) for both message and bulk buffers. Message buffers are always attached with PTL_INS_AFTER and match anything sent with "message" matchbits. Bulk buffers are always attached with PTL_INS_BEFORE and match only specific matchbits for that particular bulk transfer.</para>
+ <para>This scheme assumes that the majority of ME / MDs posted are for "message" buffers, and that the overhead of searching through the preceding "bulk" buffers is acceptable. Since the number of "bulk" buffers posted at any time is also dependent on the bulk transfer breakpoint set by max_msg_size, this seems like an issue worth measuring at scale.</para>
+ <para><emphasis role="bold">TX Descriptors</emphasis></para>
+ <para>The ptllnd has a pool of so-called "tx descriptors", which it uses not only for outgoing messages, but also to hold state for bulk transfers requested by incoming messages. This pool should scale with the total number of peers.</para>
+ <para>To enable the building of the Portals LND (ptllnd.ko) configure with this option:</para>
+ <screen>./configure --with-portals=<path-to-portals-headers></screen>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438293_pgfId-1297504" xreflabel=""/>Variable</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438293_pgfId-1297506" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Variable</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1297508" xreflabel=""/>ntx</para><para> (256)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1297510" xreflabel=""/>Total number of messaging descriptors.</para></entry>
+ <entry><para> ntx</para><para> (256)</para></entry>
+ <entry><para> Total number of messaging descriptors.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1297512" xreflabel=""/>concurrent_peers</para><para> (1152)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1297514" xreflabel=""/>Maximum number of concurrent peers. Peers that attempt to connect beyond the maximum are not allowed.</para></entry>
+ <entry><para> concurrent_peers</para><para> (1152)</para></entry>
+ <entry><para> Maximum number of concurrent peers. Peers that attempt to connect beyond the maximum are not allowed.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1297516" xreflabel=""/>peer_hash_table_size</para><para> (101)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1297518" xreflabel=""/>Number of hash table slots for the peers. This number should scale with concurrent_peers. The size of the peer hash table is set by the module parameter peer_hash_table_size which defaults to a value of 101. This number should be prime to ensure the peer hash table is populated evenly. It is advisable to increase this value to 1001 for ~10000 peers.</para></entry>
+ <entry><para> peer_hash_table_size</para><para> (101)</para></entry>
+ <entry><para> Number of hash table slots for the peers. This number should scale with concurrent_peers. The size of the peer hash table is set by the module parameter peer_hash_table_size which defaults to a value of 101. This number should be prime to ensure the peer hash table is populated evenly. It is advisable to increase this value to 1001 for ~10000 peers.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1297520" xreflabel=""/>cksum</para><para> (0)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1297522" xreflabel=""/>Set to non-zero to enable message (not RDMA) checksums for outgoing packets. Incoming packets are always check-summed if necessary, independent of this value.</para></entry>
+ <entry><para> cksum</para><para> (0)</para></entry>
+ <entry><para> Set to non-zero to enable message (not RDMA) checksums for outgoing packets. Incoming packets are always check-summed if necessary, independent of this value.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1297524" xreflabel=""/>timeout</para><para> (50)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1297526" xreflabel=""/>Amount of time (in seconds) that a request can linger in a peers-active queue before the peer is considered dead.</para></entry>
+ <entry><para> timeout</para><para> (50)</para></entry>
+ <entry><para> Amount of time (in seconds) that a request can linger in a peers-active queue before the peer is considered dead.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1297528" xreflabel=""/>portal</para><para> (9)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1297530" xreflabel=""/>Portal ID to use for the ptllnd traffic.</para></entry>
+ <entry><para> portal</para><para> (9)</para></entry>
+ <entry><para> Portal ID to use for the ptllnd traffic.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1297532" xreflabel=""/>rxb_npages</para><para> (64 * #cpus)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1297534" xreflabel=""/>Number of pages in an RX buffer.</para></entry>
+ <entry><para> rxb_npages</para><para> (64 * #cpus)</para></entry>
+ <entry><para> Number of pages in an RX buffer.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1297536" xreflabel=""/>credits</para><para> (128)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1297538" xreflabel=""/>Maximum total number of concurrent sends that are outstanding to a single peer at a given time.</para></entry>
+ <entry><para> credits</para><para> (128)</para></entry>
+ <entry><para> Maximum total number of concurrent sends that are outstanding to a single peer at a given time.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1297540" xreflabel=""/>peercredits</para><para> (8)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1297542" xreflabel=""/>Maximum number of concurrent sends that are outstanding to a single peer at a given time.</para></entry>
+ <entry><para> peercredits</para><para> (8)</para></entry>
+ <entry><para> Maximum number of concurrent sends that are outstanding to a single peer at a given time.</para></entry>
</row>
<row>
- <entry><para>
<anchor xml:id="dbdoclet.50438293_pgfId-1297544" xreflabel=""/>max_msg_size</para><para> (512)</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1297546" xreflabel=""/> Maximum immediate message size. This MUST be the same on all nodes in a cluster. A peer that connects with a different max_msg_size value will be rejected.</para></entry>
+ <entry><para> max_msg_size</para><para> (512)</para></entry>
+ <entry><para> Maximum immediate message size. This MUST be the same on all nodes in a cluster. A peer that connects with a different max_msg_size value will be rejected.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438293_pgfId-1295998" xreflabel=""/>35.2.4 MX <anchor xml:id="dbdoclet.50438293_marker-1295997" xreflabel=""/>LND</title>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293864" xreflabel=""/>MXLND supports a number of load-time parameters using Linux's module parameter system. The following variables are available:</para>
+ <title>35.2.4 MX <anchor xml:id="dbdoclet.50438293_marker-1295997" xreflabel=""/>LND</title>
+ <para>MXLND supports a number of load-time parameters using Linux's module parameter system. The following variables are available:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438293_pgfId-1296048" xreflabel=""/>Variable</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438293_pgfId-1296050" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Variable</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296093" xreflabel=""/>n_waitd</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296095" xreflabel=""/>Number of completion daemons.</para></entry>
+ <entry><para> n_waitd</para></entry>
+ <entry><para> Number of completion daemons.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296097" xreflabel=""/>max_peers</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296099" xreflabel=""/>Maximum number of peers that may connect.</para></entry>
+ <entry><para> max_peers</para></entry>
+ <entry><para> Maximum number of peers that may connect.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296101" xreflabel=""/>cksum</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296103" xreflabel=""/>Enables small message (< 4 KB) checksums if set to a non-zero value.</para></entry>
+ <entry><para> cksum</para></entry>
+ <entry><para> Enables small message (< 4 KB) checksums if set to a non-zero value.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296105" xreflabel=""/>ntx</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296107" xreflabel=""/>Number of total tx message descriptors.</para></entry>
+ <entry><para> ntx</para></entry>
+ <entry><para> Number of total tx message descriptors.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296109" xreflabel=""/>credits</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296111" xreflabel=""/>Number of concurrent sends to a single peer.</para></entry>
+ <entry><para> credits</para></entry>
+ <entry><para> Number of concurrent sends to a single peer.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296113" xreflabel=""/>board</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296115" xreflabel=""/>Index value of the Myrinet board (NIC).</para></entry>
+ <entry><para> board</para></entry>
+ <entry><para> Index value of the Myrinet board (NIC).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296117" xreflabel=""/>ep_id</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296119" xreflabel=""/>MX endpoint ID.</para></entry>
+ <entry><para> ep_id</para></entry>
+ <entry><para> MX endpoint ID.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296121" xreflabel=""/>polling</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296123" xreflabel=""/>Use zero (0) to block (wait). A value > 0 will poll that many times before blocking.</para></entry>
+ <entry><para> polling</para></entry>
+ <entry><para> Use zero (0) to block (wait). A value > 0 will poll that many times before blocking.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296129" xreflabel=""/>hosts</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438293_pgfId-1296131" xreflabel=""/>IP-to-hostname resolution file.</para></entry>
+ <entry><para> hosts</para></entry>
+ <entry><para> IP-to-hostname resolution file.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293906" xreflabel=""/>Of the described variables, only hosts is required. It must be the absolute path to the MXLND hosts file.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293907" xreflabel=""/>For example:</para>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293908" xreflabel=""/>options kmxlnd hosts=/etc/hosts.mxlnd
+ <para>Of the described variables, only hosts is required. It must be the absolute path to the MXLND hosts file.</para>
+ <para>For example:</para>
+ <screen>options kmxlnd hosts=/etc/hosts.mxlnd
</screen>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293909" xreflabel=""/>The file format for the hosts file is:</para>
- <screen><anchor xml:id="dbdoclet.50438293_pgfId-1293910" xreflabel=""/>IP HOST BOARD EP_ID
+ <para>The file format for the hosts file is:</para>
+ <screen>IP HOST BOARD EP_ID
</screen>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293911" xreflabel=""/>The values must be space and/or tab separated where:</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293912" xreflabel=""/>IP is a valid IPv4 address</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293913" xreflabel=""/>HOST is the name returned by `hostname` on that machine</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293914" xreflabel=""/>BOARD is the index of the Myricom NIC (0 for the first card, etc.)</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293915" xreflabel=""/>EP_ID is the MX endpoint ID</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293916" xreflabel=""/>To obtain the optimal performance for your platform, you may want to vary the remaining options.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293917" xreflabel=""/>n_waitd (1) sets the number of threads that process completed MX requests (sends and receives).</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293918" xreflabel=""/>max_peers (1024) tells MXLND the upper limit of machines that it will need to communicate with. This affects how many receives it will pre-post and each receive will use one page of memory. Ideally, on clients, this value will be equal to the total number of Lustre servers (MDS and OSS). On servers, it needs to equal the total number of machines in the storage system. cksum (0) turns on small message checksums. It can be used to aid in troubleshooting. MX also provides an optional checksumming feature which can check all messages (large and small). For details, see the MX README.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293919" xreflabel=""/>ntx (256) is the number of total sends in flight from this machine. In actuality, MXLND reserves half of them for connect messages so make this value twice as large as you want for the total number of sends in flight.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293920" xreflabel=""/>credits (8) is the number of in-flight messages for a specific peer. This is part of the flow-control system in Lustre. Increasing this value may improve performance but it requires more memory because each message requires at least one page.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293921" xreflabel=""/>board (0) is the index of the Myricom NIC. Hosts can have multiple Myricom NICs and this identifies which one MXLND should use. This value must match the board value in your MXLND hosts file for this host.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293922" xreflabel=""/>ep_id (3) is the MX endpoint ID. Each process that uses MX is required to have at least one MX endpoint to access the MX library and NIC. The ID is a simple index starting at zero (0). This value must match the endpoint ID value in your MXLND hosts file for this host.</para>
- <para><anchor xml:id="dbdoclet.50438293_pgfId-1293923" xreflabel=""/>polling (0) determines whether this host will poll or block for MX request completions. A value of 0 blocks and any positive value will poll that many times before blocking. Since polling increases CPU usage, we suggest that you set this to zero (0) on the client and experiment with different values for servers.</para>
+ <para>The values must be space and/or tab separated where:</para>
+ <para>IP is a valid IPv4 address</para>
+ <para>HOST is the name returned by `hostname` on that machine</para>
+ <para>BOARD is the index of the Myricom NIC (0 for the first card, etc.)</para>
+ <para>EP_ID is the MX endpoint ID</para>
+ <para>To obtain the optimal performance for your platform, you may want to vary the remaining options.</para>
+ <para>n_waitd (1) sets the number of threads that process completed MX requests (sends and receives).</para>
+ <para>max_peers (1024) tells MXLND the upper limit of machines that it will need to communicate with. This affects how many receives it will pre-post and each receive will use one page of memory. Ideally, on clients, this value will be equal to the total number of Lustre servers (MDS and OSS). On servers, it needs to equal the total number of machines in the storage system. cksum (0) turns on small message checksums. It can be used to aid in troubleshooting. MX also provides an optional checksumming feature which can check all messages (large and small). For details, see the MX README.</para>
+ <para>ntx (256) is the number of total sends in flight from this machine. In actuality, MXLND reserves half of them for connect messages so make this value twice as large as you want for the total number of sends in flight.</para>
+ <para>credits (8) is the number of in-flight messages for a specific peer. This is part of the flow-control system in Lustre. Increasing this value may improve performance but it requires more memory because each message requires at least one page.</para>
+ <para>board (0) is the index of the Myricom NIC. Hosts can have multiple Myricom NICs and this identifies which one MXLND should use. This value must match the board value in your MXLND hosts file for this host.</para>
+ <para>ep_id (3) is the MX endpoint ID. Each process that uses MX is required to have at least one MX endpoint to access the MX library and NIC. The ID is a simple index starting at zero (0). This value must match the endpoint ID value in your MXLND hosts file for this host.</para>
+ <para>polling (0) determines whether this host will poll or block for MX request completions. A value of 0 blocks and any positive value will poll that many times before blocking. Since polling increases CPU usage, we suggest that you set this to zero (0) on the client and experiment with different values for servers.</para>
</section>
</section>
</chapter>
<info>
<title xml:id='configuringfailover.title'>Configuring Lustre Failover</title>
</info>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1292189" xreflabel=""/>This chapter describes how to configure Lustre failover using the Heartbeat cluster infrastructure daemon. It includes:</para>
+ <para>This chapter describes how to configure Lustre failover using the Heartbeat cluster infrastructure daemon. It includes:</para>
<itemizedlist><listitem>
<para><xref linkend='dbdoclet.50438188_82389'/></para>
<section xml:id="dbdoclet.50438188_82389">
<title>11.1 Creating a Failover Environment</title>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1292209" xreflabel=""/>Lustre provides failover mechanisms only at the file system level. No failover functionality is provided for system-level components, such as node failure detection or power control, as would typically be provided in a complete failover solution. Additional tools are also needed to provide resource fencing, control and monitoring.</para>
+ <para>Lustre provides failover mechanisms only at the file system level. No failover functionality is provided for system-level components, such as node failure detection or power control, as would typically be provided in a complete failover solution. Additional tools are also needed to provide resource fencing, control and monitoring.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438188_pgfId-1292210" xreflabel=""/>11.1.1 Power Management Software</title>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1292211" xreflabel=""/>Lustre failover requires power control and management capability to verify that a failed node is shut down before I/O is directed to the failover node. This avoids double-mounting the two nodes, and the risk of unrecoverable data corruption. A variety of power management tools will work, but two packages that are commonly used with Lustre are STONITH and PowerMan.</para>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1292212" xreflabel=""/>Shoot The Other Node In The HEAD (STONITH), is a set of power management tools provided with the Linux-HA package. STONITH has native support for many power control devices and is extensible. It uses expect scripts to automate control.</para>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1292213" xreflabel=""/>PowerMan, available from the Lawrence Livermore National Laboratory (LLNL), is used to control remote power control (RPC) devices from a central location. PowerMan provides native support for several RPC varieties and expect-like configuration simplifies the addition of new devices.</para>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1292214" xreflabel=""/>The latest versions of PowerMan are available at:</para>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1292216" xreflabel=""/><link xl:href="http://sourceforge.net/projects/powerman">http://sourceforge.net/projects/powerman</link></para>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1292217" xreflabel=""/>For more information about PowerMan, go to:</para>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1292219" xreflabel=""/><link xl:href="https://computing.llnl.gov/linux/powerman.html">https://computing.llnl.gov/linux/powerman.html</link></para>
+ <title>11.1.1 Power Management Software</title>
+ <para>Lustre failover requires power control and management capability to verify that a failed node is shut down before I/O is directed to the failover node. This avoids double-mounting the two nodes, and the risk of unrecoverable data corruption. A variety of power management tools will work, but two packages that are commonly used with Lustre are STONITH and PowerMan.</para>
+ <para>Shoot The Other Node In The HEAD (STONITH), is a set of power management tools provided with the Linux-HA package. STONITH has native support for many power control devices and is extensible. It uses expect scripts to automate control.</para>
+ <para>PowerMan, available from the Lawrence Livermore National Laboratory (LLNL), is used to control remote power control (RPC) devices from a central location. PowerMan provides native support for several RPC varieties and expect-like configuration simplifies the addition of new devices.</para>
+ <para>The latest versions of PowerMan are available at:</para>
+ <para><link xl:href="http://sourceforge.net/projects/powerman">http://sourceforge.net/projects/powerman</link></para>
+ <para>For more information about PowerMan, go to:</para>
+ <para><link xl:href="https://computing.llnl.gov/linux/powerman.html">https://computing.llnl.gov/linux/powerman.html</link></para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438188_pgfId-1292220" xreflabel=""/>11.1.2 Power Equipment</title>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1292221" xreflabel=""/>Lustre failover also requires the use of RPC devices, which come in different configurations. Lustre server nodes may be equipped with some kind of service processor that allows remote power control. If a Lustre server node is not equipped with a service processor, then a multi-port, Ethernet-addressable RPC may be used as an alternative. For recommended products, refer to the list of supported RPC devices on the PowerMan website.</para>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1292223" xreflabel=""/><link xl:href="https://computing.llnl.gov/linux/powerman.html">https://computing.llnl.gov/linux/powerman.html</link></para>
+ <title>11.1.2 Power Equipment</title>
+ <para>Lustre failover also requires the use of RPC devices, which come in different configurations. Lustre server nodes may be equipped with some kind of service processor that allows remote power control. If a Lustre server node is not equipped with a service processor, then a multi-port, Ethernet-addressable RPC may be used as an alternative. For recommended products, refer to the list of supported RPC devices on the PowerMan website.</para>
+ <para><link xl:href="https://computing.llnl.gov/linux/powerman.html">https://computing.llnl.gov/linux/powerman.html</link></para>
</section>
</section>
<section xml:id="dbdoclet.50438188_92688">
<title>11.2 Setting up High-Availability (HA) Software with Lustre</title>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1292225" xreflabel=""/>Lustre must be combined with high-availability (HA) software to enable a complete Lustre failover solution. Lustre can be used with several HA packages including:</para>
+ <para>Lustre must be combined with high-availability (HA) software to enable a complete Lustre failover solution. Lustre can be used with several HA packages including:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438188_pgfId-1293083" xreflabel=""/><emphasis>Red Hat Cluster Manager</emphasis> - For more information about setting up Lustre failover with Red Hat Cluster Manager, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Red_Hat_Cluster_Manager_with_Lustre">Using Red Hat Cluster Manager with Lustre</link>.</para>
+ <para><emphasis>Red Hat Cluster Manager</emphasis> - For more information about setting up Lustre failover with Red Hat Cluster Manager, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Red_Hat_Cluster_Manager_with_Lustre">Using Red Hat Cluster Manager with Lustre</link>.</para>
</listitem>
<listitem>
- <para><
anchor xml:id="dbdoclet.50438188_pgfId-1293110" xreflabel=""/><emphasis>Pacemaker</emphasis> - For more information about setting up Lustre failover with Pacemaker, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre">Using Pacemaker with Lustre</link>.<anchor xml:id="dbdoclet.50438188_61775" xreflabel=""/></para>
+ <para><emphasis>Pacemaker</emphasis> - For more information about setting up Lustre failover with Pacemaker, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre">Using Pacemaker with Lustre</link>.<anchor xml:id="dbdoclet.50438188_61775" xreflabel=""/></para>
</listitem>
</itemizedlist>
</section>
<title xml:id='configuringlnet.title'>Configuring Lustre Networking (LNET)</title>
</info>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304688" xreflabel=""/>This chapter describes how to configure Lustre Networking (LNET). It includes the following sections:</para>
+ <para>This chapter describes how to configure Lustre Networking (LNET). It includes the following sections:</para>
<toc/>
<itemizedlist><listitem>
<section xml:id="dbdoclet.50438216_33148" >
<title>9.1 Overview of LNET Module Parameters</title>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304720" xreflabel=""/>LNET kernel module (lnet) parameters specify how LNET is to be configured to work with Lustre, including which NICs will be configured to work with Lustre and the routing to be used with Lustre.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304721" xreflabel=""/>Parameters for lnet are specified in the modprobe.conf or modules.conf file (depending on your Linux distribution) in one or more entries with the syntax:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304722" xreflabel=""/>options lnet <<emphasis>parameter</emphasis>>=<<emphasis>parameter value</emphasis>>
+ <para>LNET kernel module (lnet) parameters specify how LNET is to be configured to work with Lustre, including which NICs will be configured to work with Lustre and the routing to be used with Lustre.</para>
+ <para>Parameters for lnet are specified in the modprobe.conf or modules.conf file (depending on your Linux distribution) in one or more entries with the syntax:</para>
+ <screen>options lnet <<emphasis>parameter</emphasis>>=<<emphasis>parameter value</emphasis>>
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304723" xreflabel=""/>To specify the network interfaces that are to be used for Lustre, set either the networks parameter or the ip2nets parameter (only one of these parameters can be used at a time):</para>
+ <para>To specify the network interfaces that are to be used for Lustre, set either the networks parameter or the ip2nets parameter (only one of these parameters can be used at a time):</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304724" xreflabel=""/>networks - Specifies the networks to be used.</para>
+ <para>networks - Specifies the networks to be used.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304725" xreflabel=""/>ip2nets - Lists globally-available networks, each with a range of IP addresses. LNET then identifies locally-available networks through address list-matching lookup.</para>
+ <para>ip2nets - Lists globally-available networks, each with a range of IP addresses. LNET then identifies locally-available networks through address list-matching lookup.</para>
</listitem>
</itemizedlist>
-<para><anchor xml:id="dbdoclet.50438216_pgfId-1304731" xreflabel=""/>See <xref linkend='dbdoclet.50438216_46279'/> and <xref linkend="dbdoclet.50438216_31414"/>Setting the LNET Module ip2nets Parameter for more details.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304733" xreflabel=""/>To set up routing between networks, use:</para>
+<para>See <xref linkend='dbdoclet.50438216_46279'/> and <xref linkend="dbdoclet.50438216_31414"/>Setting the LNET Module ip2nets Parameter for more details.</para>
+ <para>To set up routing between networks, use:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304734" xreflabel=""/>routes - Lists networks and the NIDs of routers that forward to them.</para>
+ <para>routes - Lists networks and the NIDs of routers that forward to them.</para>
</listitem>
</itemizedlist>
-<para><anchor xml:id="dbdoclet.50438216_pgfId-1304738" xreflabel=""/>See <xref linkend="dbdoclet.50438216_71227"/> for more details.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304739" xreflabel=""/>A router checker can be configured to enable Lustre nodes to detect router health status, avoid routers that appear dead, and reuse those that restore service after failures. See <xref linkend="dbdoclet.50438216_35668"/> for more details.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304743" xreflabel=""/>For a complete reference to the LNET module parameters, see <emphasis><xref linkend="configurationfilesmoduleparameters"/>LNET Options</emphasis>.</para>
+<para>See <xref linkend="dbdoclet.50438216_71227"/> for more details.</para>
+ <para>A router checker can be configured to enable Lustre nodes to detect router health status, avoid routers that appear dead, and reuse those that restore service after failures. See <xref linkend="dbdoclet.50438216_35668"/> for more details.</para>
+ <para>For a complete reference to the LNET module parameters, see <emphasis><xref linkend="configurationfilesmoduleparameters"/>LNET Options</emphasis>.</para>
<note>
<para>We recommend that you use 'dotted-quad' notation for IP addresses rather than host names to make it easier to read debug logs and debug configurations with multiple interfaces.</para>
</note>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438216_pgfId-1304745" xreflabel=""/>9.1.1 Using a Lustre Network Identifier (NID) to Identify a Node</title>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304746" xreflabel=""/>A Lustre network identifier (NID) is used to uniquely identify a Lustre network endpoint by node ID and network type. The format of the NID is:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304747" xreflabel=""/><<emphasis>network</emphasis><emphasis>id</emphasis>>@<<emphasis>network</emphasis><emphasis>type</emphasis>></screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304748" xreflabel=""/>Examples are:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304749" xreflabel=""/>10.67.73.200@tcp0<anchor xml:id="dbdoclet.50438216_pgfId-1304750" xreflabel=""/>10.67.75.100@o2ib</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304751" xreflabel=""/>The first entry above identifes a TCP/IP node, while the second entry identifies an InfiniBand node.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304752" xreflabel=""/>When a mount command is run on a client, the client uses the NID of the MDS to retrieve configuration information. If an MDS has more than one NID, the client should use the appropriate NID for its local network.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304753" xreflabel=""/>To determine the appropriate NID to specify in the mount command, use the lctl command. To display MDS NIDs, run on the MDS :</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304754" xreflabel=""/>lctl list_nids
+ <title>9.1.1 Using a Lustre Network Identifier (NID) to Identify a Node</title>
+ <para>A Lustre network identifier (NID) is used to uniquely identify a Lustre network endpoint by node ID and network type. The format of the NID is:</para>
+ <screen><<emphasis>network</emphasis><emphasis>id</emphasis>>@<<emphasis>network</emphasis><emphasis>type</emphasis>></screen>
+ <para>Examples are:</para>
+ <screen>10.67.73.200@tcp010.67.75.100@o2ib</screen>
+ <para>The first entry above identifes a TCP/IP node, while the second entry identifies an InfiniBand node.</para>
+ <para>When a mount command is run on a client, the client uses the NID of the MDS to retrieve configuration information. If an MDS has more than one NID, the client should use the appropriate NID for its local network.</para>
+ <para>To determine the appropriate NID to specify in the mount command, use the lctl command. To display MDS NIDs, run on the MDS :</para>
+ <screen>lctl list_nids
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304755" xreflabel=""/>To determine if a client can reach the MDS using a particular NID, run on the client:</para>
- <screen>
<anchor xml:id="dbdoclet.50438216_pgfId-1304757" xreflabel=""/>lctl which_nid <<emphasis>MDS NID</emphasis>><anchor xml:id="dbdoclet.50438216_14112" xreflabel=""/></screen>
+ <para>To determine if a client can reach the MDS using a particular NID, run on the client:</para>
+ <screen>lctl which_nid <<emphasis>MDS NID</emphasis>><anchor xml:id="dbdoclet.50438216_14112" xreflabel=""/></screen>
</section>
</section>
<section xml:id="dbdoclet.50438216_46279">
<title>9.2 Setting the LNET Module networks Parameter</title>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304760" xreflabel=""/>If a node has more than one network interface, you'll typically want to dedicate a specific interface to Lustre. You can do this by including an entry in the modprobe.conf file on the node that sets the LNET module networks parameter:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304761" xreflabel=""/>options lnet networks=<<emphasis>comma-separated list of networks</emphasis>>
+ <para>If a node has more than one network interface, you'll typically want to dedicate a specific interface to Lustre. You can do this by including an entry in the modprobe.conf file on the node that sets the LNET module networks parameter:</para>
+ <screen>options lnet networks=<<emphasis>comma-separated list of networks</emphasis>>
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304762" xreflabel=""/>This example specifies that a Lustre node will use a TCP/IP interface and an InfiniBand interface:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304763" xreflabel=""/>options lnet networks=tcp0(eth0),o2ib(ib0)
+ <para>This example specifies that a Lustre node will use a TCP/IP interface and an InfiniBand interface:</para>
+ <screen>options lnet networks=tcp0(eth0),o2ib(ib0)
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304764" xreflabel=""/>This example specifies that the Lustre node will use the TCP/IP interface eth1:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304765" xreflabel=""/>options lnet networks=tcp0(eth1)
+ <para>This example specifies that the Lustre node will use the TCP/IP interface eth1:</para>
+ <screen>options lnet networks=tcp0(eth1)
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304766" xreflabel=""/>Depending on the network design, it may be necessary to specify explicit interfaces. To explicitly specify that interface eth2 be used for network tcp0 and eth3 be used for tcp1 , use this entry:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304767" xreflabel=""/>options lnet networks=tcp0(eth2),tcp1(eth3)
+ <para>Depending on the network design, it may be necessary to specify explicit interfaces. To explicitly specify that interface eth2 be used for network tcp0 and eth3 be used for tcp1 , use this entry:</para>
+ <screen>options lnet networks=tcp0(eth2),tcp1(eth3)
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304768" xreflabel=""/>When more than one interface is available during the network setup, Lustre chooses the best route based on the hop count. Once the network connection is established, Lustre expects the network to stay connected. In a Lustre network, connections do not fail over to another interface, even if multiple interfaces are available on the same node.</para>
+ <para>When more than one interface is available during the network setup, Lustre chooses the best route based on the hop count. Once the network connection is established, Lustre expects the network to stay connected. In a Lustre network, connections do not fail over to another interface, even if multiple interfaces are available on the same node.</para>
<note>
<para>LNET lines in modprobe.conf are only used by the local node to determine what to call its interfaces. They are not used for routing decisions.</para>
</note>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438216_pgfId-1304771" xreflabel=""/>9.2.1 <anchor xml:id="dbdoclet.50438216_74334" xreflabel=""/>Multihome Server Example</title>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304772" xreflabel=""/>If a server with multiple IP addresses (multihome server) is connected to a Lustre network, certain configuration setting are required. An example illustrating these setting consists of a network with the following nodes:</para>
+ <title>9.2.1 <anchor xml:id="dbdoclet.50438216_74334" xreflabel=""/>Multihome Server Example</title>
+ <para>If a server with multiple IP addresses (multihome server) is connected to a Lustre network, certain configuration setting are required. An example illustrating these setting consists of a network with the following nodes:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304773" xreflabel=""/> Server svr1 with three TCP NICs (eth0, eth1, and eth2) and an InfiniBand NIC.</para>
+ <para> Server svr1 with three TCP NICs (eth0, eth1, and eth2) and an InfiniBand NIC.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304774" xreflabel=""/> Server svr2 with three TCP NICs (eth0, eth1, and eth2) and an InfiniBand NIC. Interface eth2 will not be used for Lustre networking.</para>
+ <para> Server svr2 with three TCP NICs (eth0, eth1, and eth2) and an InfiniBand NIC. Interface eth2 will not be used for Lustre networking.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304775" xreflabel=""/> TCP clients, each with a single TCP interface.</para>
+ <para> TCP clients, each with a single TCP interface.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304776" xreflabel=""/> InfiniBand clients, each with a single Infiniband interface and a TCP/IP interface for administration.</para>
+ <para> InfiniBand clients, each with a single Infiniband interface and a TCP/IP interface for administration.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304777" xreflabel=""/>To set the networks option for this example:</para>
+ <para>To set the networks option for this example:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304778" xreflabel=""/> On each server, svr1 and svr2, include the following line in the modprobe.conf file:</para>
+ <para> On each server, svr1 and svr2, include the following line in the modprobe.conf file:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304779" xreflabel=""/>options lnet networks=tcp0(eth0),tcp1(eth1),o2ib
+ <screen>options lnet networks=tcp0(eth0),tcp1(eth1),o2ib
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304780" xreflabel=""/> For TCP-only clients, the first available non-loopback IP interface is used for tcp0. Thus, TCP clients with only one interface do not need to have options defined in the modprobe.conf file.</para>
+ <para> For TCP-only clients, the first available non-loopback IP interface is used for tcp0. Thus, TCP clients with only one interface do not need to have options defined in the modprobe.conf file.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304781" xreflabel=""/> On the InfiniBand clients, include the following line in the modprobe.conf file:</para>
+ <para> On the InfiniBand clients, include the following line in the modprobe.conf file:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304782" xreflabel=""/>options lnet networks=o2ib
+ <screen>options lnet networks=o2ib
</screen>
<note>
</section>
<section xml:id="dbdoclet.50438216_31414">
<title>9.3 Setting the LNET Module ip2nets Parameter</title>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304786" xreflabel=""/>The ip2nets option is typically used when a single, universal modprobe.conf file is run on all servers and clients. Each node identifies the locally available networks based on the listed IP address patterns that match the node's local IP addresses.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304787" xreflabel=""/>Note that the IP address patterns listed in the ip2nets option are <emphasis>only</emphasis> used to identify the networks that an individual node should instantiate. They are <emphasis>not</emphasis> used by LNET for any other communications purpose.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304788" xreflabel=""/>For the example below, the nodes in the network have these IP addresses:</para>
+ <para>The ip2nets option is typically used when a single, universal modprobe.conf file is run on all servers and clients. Each node identifies the locally available networks based on the listed IP address patterns that match the node's local IP addresses.</para>
+ <para>Note that the IP address patterns listed in the ip2nets option are <emphasis>only</emphasis> used to identify the networks that an individual node should instantiate. They are <emphasis>not</emphasis> used by LNET for any other communications purpose.</para>
+ <para>For the example below, the nodes in the network have these IP addresses:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304789" xreflabel=""/> Server svr1: eth0 IP address 192.168.0.2, IP over Infiniband (o2ib) address 132.6.1.2.</para>
+ <para> Server svr1: eth0 IP address 192.168.0.2, IP over Infiniband (o2ib) address 132.6.1.2.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304790" xreflabel=""/> Server svr2: eth0 IP address 192.168.0.4, IP over Infiniband (o2ib) address 132.6.1.4.</para>
+ <para> Server svr2: eth0 IP address 192.168.0.4, IP over Infiniband (o2ib) address 132.6.1.4.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304791" xreflabel=""/> TCP clients have IP addresses 192.168.0.5-255.</para>
+ <para> TCP clients have IP addresses 192.168.0.5-255.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304792" xreflabel=""/> Infiniband clients have IP over Infiniband (o2ib) addresses 132.6.[2-3].2, .4, .6, .8.</para>
+ <para> Infiniband clients have IP over Infiniband (o2ib) addresses 132.6.[2-3].2, .4, .6, .8.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304793" xreflabel=""/>The following entry is placed in the modprobe.conf file on each server and client:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304794" xreflabel=""/>options lnet 'ip2nets="tcp0(eth0) 192.168.0.[2,4]; \
-<anchor xml:id="dbdoclet.50438216_pgfId-1304795" xreflabel=""/>tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"'
+ <para>The following entry is placed in the modprobe.conf file on each server and client:</para>
+ <screen>options lnet 'ip2nets="tcp0(eth0) 192.168.0.[2,4]; \
+tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"'
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304796" xreflabel=""/>Each entry in ip2nets is referred to as a 'rule'.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304797" xreflabel=""/>The order of LNET entries is important when configuring servers. If a server node can be reached using more than one network, the first network specified in modprobe.conf will be used.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304798" xreflabel=""/>Because svr1 and svr2 match the first rule, LNET uses eth0 for tcp0 on those machines. (Although svr1 and svr2 also match the second rule, the first matching rule for a particular network is used).</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304799" xreflabel=""/>The [2-8/2] format indicates a range of 2-8 stepped by 2; that is 2,4,6,8. Thus, the clients at 132.6.3.5 will not find a matching o2ib network.</para>
+ <para>Each entry in ip2nets is referred to as a 'rule'.</para>
+ <para>The order of LNET entries is important when configuring servers. If a server node can be reached using more than one network, the first network specified in modprobe.conf will be used.</para>
+ <para>Because svr1 and svr2 match the first rule, LNET uses eth0 for tcp0 on those machines. (Although svr1 and svr2 also match the second rule, the first matching rule for a particular network is used).</para>
+ <para>The [2-8/2] format indicates a range of 2-8 stepped by 2; that is 2,4,6,8. Thus, the clients at 132.6.3.5 will not find a matching o2ib network.</para>
</section>
<section xml:id="dbdoclet.50438216_71227">
<title>9.4 Setting the LNET Module routes Parameter</title>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304802" xreflabel=""/>The LNET module routes parameter is used to identify routers in a Lustre configuration. These parameters are set in modprob.conf on each Lustre node.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304803" xreflabel=""/>The LNET routes parameter specifies a colon-separated list of router definitions. Each route is defined as a network number, followed by a list of routers:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304804" xreflabel=""/>routes=<<emphasis>net type</emphasis>> <<emphasis>router NID(s)</emphasis>>
+ <para>The LNET module routes parameter is used to identify routers in a Lustre configuration. These parameters are set in modprob.conf on each Lustre node.</para>
+ <para>The LNET routes parameter specifies a colon-separated list of router definitions. Each route is defined as a network number, followed by a list of routers:</para>
+ <screen>routes=<<emphasis>net type</emphasis>> <<emphasis>router NID(s)</emphasis>>
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304805" xreflabel=""/>This example specifies bi-directional routing in which TCP clients can reach Lustre resources on the IB networks and IB servers can access the TCP networks:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1306272" xreflabel=""/>options lnet 'ip2nets="tcp0 192.168.0.*; \
-<anchor xml:id="dbdoclet.50438216_pgfId-1306289" xreflabel=""/>o2ib0(ib0) 132.6.1.[1-128]"' 'routes="tcp0 132.6.1.[1-8]@o2ib0; \o2ib0 192.16\
+ <para>This example specifies bi-directional routing in which TCP clients can reach Lustre resources on the IB networks and IB servers can access the TCP networks:</para>
+ <screen>options lnet 'ip2nets="tcp0 192.168.0.*; \
+o2ib0(ib0) 132.6.1.[1-128]"' 'routes="tcp0 132.6.1.[1-8]@o2ib0; \o2ib0 192.16\
8.0.[1-8]@tcp0"'
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304807" xreflabel=""/>All LNET routers that bridge two networks are equivalent. They are not configured as primary or secondary, and the load is balanced across all available routers.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304808" xreflabel=""/>The number of LNET routers is not limited. Enough routers should be used to handle the required file serving bandwidth plus a 25 percent margin for headroom.</para>
+ <para>All LNET routers that bridge two networks are equivalent. They are not configured as primary or secondary, and the load is balanced across all available routers.</para>
+ <para>The number of LNET routers is not limited. Enough routers should be used to handle the required file serving bandwidth plus a 25 percent margin for headroom.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438216_pgfId-1304811" xreflabel=""/>9.4.1 <anchor xml:id="dbdoclet.50438216_34481" xreflabel=""/><anchor xml:id="dbdoclet.50438216_56435" xreflabel=""/>Routing Example</title>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304813" xreflabel=""/>On the clients, place the following entry in the modprobe.conf file</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1306296" xreflabel=""/>lnet networks="tcp" routes="o2ib0 192.168.0.[1-8]@tcp0"
+ <title>9.4.1 <anchor xml:id="dbdoclet.50438216_34481" xreflabel=""/><anchor xml:id="dbdoclet.50438216_56435" xreflabel=""/>Routing Example</title>
+ <para>On the clients, place the following entry in the modprobe.conf file</para>
+ <screen>lnet networks="tcp" routes="o2ib0 192.168.0.[1-8]@tcp0"
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304815" xreflabel=""/>On the router nodes, use:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304817" xreflabel=""/>lnet networks="tcp o2ib" forwarding=enabled
+ <para>On the router nodes, use:</para>
+ <screen>lnet networks="tcp o2ib" forwarding=enabled
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1306310" xreflabel=""/>On the MDS, use the reverse as shown below:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304820" xreflabel=""/>lnet networks="o2ib0" rountes="tcp0 132.6.1.[1-8]@o2ib0"
+ <para>On the MDS, use the reverse as shown below:</para>
+ <screen>lnet networks="o2ib0" rountes="tcp0 132.6.1.[1-8]@o2ib0"
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1305429" xreflabel=""/>To start the routers, run:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1305433" xreflabel=""/>modprobe lnet
-<anchor xml:id="dbdoclet.50438216_pgfId-1305434" xreflabel=""/>lctl network configure
+ <para>To start the routers, run:</para>
+ <screen>modprobe lnet
+lctl network configure
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438216_10523">
<title>9.5 Testing the LNET Configuration</title>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304852" xreflabel=""/>After configuring Lustre Networking, it is highly recommended that you test your LNET configuration using the LNET Self-Test provided with the Lustre software. For more information about using LNET Self-Test, see <xref linkend='lnetselftest'/>.</para>
+ <para>After configuring Lustre Networking, it is highly recommended that you test your LNET configuration using the LNET Self-Test provided with the Lustre software. For more information about using LNET Self-Test, see <xref linkend='lnetselftest'/>.</para>
</section>
<section xml:id="dbdoclet.50438216_35668">
<title>9.6 Configuring the Router Checker</title>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304858" xreflabel=""/>In a Lustre configuration in which different types of networks, such as a TCP/IP network and an Infiniband network, are connected by routers, a router checker can be run on the clients and servers in the routed configuration to monitor the status of the routers. In a multi-hop routing configuration, router checkers can be configured on routers to monitor the health of their next-hop routers.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1305791" xreflabel=""/>A router checker is configured by setting lnet parameters in modprobe.conf by including an entry in this form:</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304861" xreflabel=""/>options lnet <<emphasis>router checker parameter</emphasis>>=<<emphasis>parameter value</emphasis>>
+ <para>In a Lustre configuration in which different types of networks, such as a TCP/IP network and an Infiniband network, are connected by routers, a router checker can be run on the clients and servers in the routed configuration to monitor the status of the routers. In a multi-hop routing configuration, router checkers can be configured on routers to monitor the health of their next-hop routers.</para>
+ <para>A router checker is configured by setting lnet parameters in modprobe.conf by including an entry in this form:</para>
+ <screen>options lnet <<emphasis>router checker parameter</emphasis>>=<<emphasis>parameter value</emphasis>>
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304862" xreflabel=""/>The router checker parameters are:</para>
+ <para>The router checker parameters are:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304865" xreflabel=""/>live_router_check_interval - Specifies a time interval in seconds after which the router checker will ping the live routers. The default value is 0, meaning no checking is done. To set the value to 60, enter:</para>
+ <para>live_router_check_interval - Specifies a time interval in seconds after which the router checker will ping the live routers. The default value is 0, meaning no checking is done. To set the value to 60, enter:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304866" xreflabel=""/>options lnet live_router_check_interval=60
+ <screen>options lnet live_router_check_interval=60
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304867" xreflabel=""/>dead_router_check_interval - Specifies a time interval in seconds after which the router checker will check for dead routers. The default value is 0, meaning no checking is done. To set the value to 60, enter:</para>
+ <para>dead_router_check_interval - Specifies a time interval in seconds after which the router checker will check for dead routers. The default value is 0, meaning no checking is done. To set the value to 60, enter:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304868" xreflabel=""/>options lnet dead_router_check_interval=60
+ <screen>options lnet dead_router_check_interval=60
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1305950" xreflabel=""/>auto_down - Enables/disables (1/0) the automatic marking of router state as up or down. The default value is 1. To disable router marking, enter:</para>
+ <para>auto_down - Enables/disables (1/0) the automatic marking of router state as up or down. The default value is 1. To disable router marking, enter:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1305951" xreflabel=""/>options lnet auto_down=0
+ <screen>options lnet auto_down=0
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304869" xreflabel=""/>router_ping_timeout - Specifies a timeout for the router checker when it checks live or dead routers. The router checker sends a ping message to each dead or live router once every dead_router_check_interval or live_router_check_interval respectively. The default value is 50. To set the value to 60, enter:</para>
+ <para>router_ping_timeout - Specifies a timeout for the router checker when it checks live or dead routers. The router checker sends a ping message to each dead or live router once every dead_router_check_interval or live_router_check_interval respectively. The default value is 50. To set the value to 60, enter:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1306120" xreflabel=""/>options lnet router_ping_timeout=60
+ <screen>options lnet router_ping_timeout=60
</screen>
<note>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1306117" xreflabel=""/>check_routers_before_use - Specifies that routers are to be checked before use. Set to off by default. If this parameter is set to on, the dead_router_check_interval parameter must be given a positive integer value.</para>
+ <para>check_routers_before_use - Specifies that routers are to be checked before use. Set to off by default. If this parameter is set to on, the dead_router_check_interval parameter must be given a positive integer value.</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304872" xreflabel=""/>options lnet check_routers_before_use=on
+ <screen>options lnet check_routers_before_use=on
</screen>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304873" xreflabel=""/>The router checker obtains the following information from each router:</para>
+ <para>The router checker obtains the following information from each router:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304874" xreflabel=""/> Time the router was disabled</para>
+ <para> Time the router was disabled</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304875" xreflabel=""/> Elapsed disable time</para>
+ <para> Elapsed disable time</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304876" xreflabel=""/>If the router checker does not get a reply message from the router within router_ping_timeout seconds, it considers the router to be down.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304878" xreflabel=""/>If a router is marked 'up' and responds to a ping, the timeout is reset.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304881" xreflabel=""/>If 100 packets have been sent successfully through a router, the sent-packets counter for that router will have a value of 100.</para>
+ <para>If the router checker does not get a reply message from the router within router_ping_timeout seconds, it considers the router to be down.</para>
+ <para>If a router is marked 'up' and responds to a ping, the timeout is reset.</para>
+ <para>If 100 packets have been sent successfully through a router, the sent-packets counter for that router will have a value of 100.</para>
</section>
<section xml:id="dbdoclet.50438216_15200">
<title>9.7 Best Practices for LNET Options</title>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304887" xreflabel=""/>For the networks, ip2nets, and routes options, follow these best practices to avoid configuration errors.</para>
+ <para>For the networks, ip2nets, and routes options, follow these best practices to avoid configuration errors.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438216_pgfId-1304888" xreflabel=""/>Escaping commas with quotes</title>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304889" xreflabel=""/>Depending on the Linux distribution, commas may need to be escaped using single or double quotes. In the extreme case, the options entry would look like this:</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304890" xreflabel=""/>options lnet'networks="tcp0,elan0"' 'routes="tcp [2,10]@elan0"'</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304891" xreflabel=""/>Added quotes may confuse some distributions. Messages such as the following may indicate an issue related to added quotes:</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304892" xreflabel=""/>lnet: Unknown parameter 'networks'</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304893" xreflabel=""/>A 'Refusing connection - no matching NID' message generally points to an error in the LNET module configuration.</para>
+ <title>Escaping commas with quotes</title>
+ <para>Depending on the Linux distribution, commas may need to be escaped using single or double quotes. In the extreme case, the options entry would look like this:</para>
+ <para>options lnet'networks="tcp0,elan0"' 'routes="tcp [2,10]@elan0"'</para>
+ <para>Added quotes may confuse some distributions. Messages such as the following may indicate an issue related to added quotes:</para>
+ <para>lnet: Unknown parameter 'networks'</para>
+ <para>A 'Refusing connection - no matching NID' message generally points to an error in the LNET module configuration.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438216_pgfId-1304894" xreflabel=""/>Including comments</title>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304895" xreflabel=""/><emphasis>Place the semicolon terminating a comment immediately after the comment.</emphasis> LNET silently ignores everything between the # character at the beginning of the comment and the next semicolon.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304896" xreflabel=""/>In this <emphasis>incorrect</emphasis> example, LNET silently ignores pt11 192.168.0.[92,96], resulting in these nodes not being properly initialized. No error message is generated.</para>
- <screen><anchor xml:id="dbdoclet.50438216_pgfId-1304897" xreflabel=""/>options lnet ip2nets=</screen>
+ <title>Including comments</title>
+ <para><emphasis>Place the semicolon terminating a comment immediately after the comment.</emphasis> LNET silently ignores everything between the # character at the beginning of the comment and the next semicolon.</para>
+ <para>In this <emphasis>incorrect</emphasis> example, LNET silently ignores pt11 192.168.0.[92,96], resulting in these nodes not being properly initialized. No error message is generated.</para>
+ <screen>options lnet ip2nets=</screen>
</section>
</section>
</chapter>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1290861" xreflabel=""/>This chapter shows how to configure a simple Lustre system comprised of a combined MGS/MDT, an OST and a client. It includes:</para>
+ <para>This chapter shows how to configure a simple Lustre system comprised of a combined MGS/MDT, an OST and a client. It includes:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438267_50692"/>
</para>
<section xml:id="dbdoclet.50438267_50692">
<title>10.1 Configuring a Simple Lustre File System</title>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1290876" xreflabel=""/>A Lustre system can be set up in a variety of configurations by using the administrative utilities provided with Lustre. The procedure below shows how to to configure a simple Lustre file system consisting of a combined MGS/MDS, one OSS with two OSTs, and a client. For an overview of the entire Lustre installation procedure, see <xref linkend='installoverview'/>.</para>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291868" xreflabel=""/>This configuration procedure assumes you have completed the following:</para>
+ <para>A Lustre system can be set up in a variety of configurations by using the administrative utilities provided with Lustre. The procedure below shows how to to configure a simple Lustre file system consisting of a combined MGS/MDS, one OSS with two OSTs, and a client. For an overview of the entire Lustre installation procedure, see <xref linkend='installoverview'/>.</para>
+ <para>This configuration procedure assumes you have completed the following:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293115" xreflabel=""/><emphasis>Set up and configured your hardware</emphasis> . For more information about hardware requirements, see <xref linkend='settinguplustresystem'/>.</para>
+ <para><emphasis>Set up and configured your hardware</emphasis> . For more information about hardware requirements, see <xref linkend='settinguplustresystem'/>.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293121" xreflabel=""/><emphasis>Downloaded and installed the Lustre software.</emphasis> For more information about preparing for and installing the Lustre software, see <xref linkend='installinglustre'/>.</para>
+ <para><emphasis>Downloaded and installed the Lustre software.</emphasis> For more information about preparing for and installing the Lustre software, see <xref linkend='installinglustre'/>.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293288" xreflabel=""/>The following optional steps should also be completed, if needed, before the Lustre software is configured:</para>
+ <para>The following optional steps should also be completed, if needed, before the Lustre software is configured:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293289" xreflabel=""/><emphasis>Set up a hardware or software RAID on block devices to be used as OSTs or MDTs.</emphasis> For information about setting up RAID, see the documentation for your RAID controller or <xref linkend='configuringstorage'/>.</para>
+ <para><emphasis>Set up a hardware or software RAID on block devices to be used as OSTs or MDTs.</emphasis> For information about setting up RAID, see the documentation for your RAID controller or <xref linkend='configuringstorage'/>.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293297" xreflabel=""/><emphasis>Set up network interface bonding on Ethernet interfaces.</emphasis> For information about setting up network interface bonding, see <xref linkend='settingupbonding'/>.</para>
+ <para><emphasis>Set up network interface bonding on Ethernet interfaces.</emphasis> For information about setting up network interface bonding, see <xref linkend='settingupbonding'/>.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1294595" xreflabel=""/><emphasis>Set</emphasis>lnet<emphasis>module parameters to specify how Lustre Networking (LNET) is to be configured to work with Lustre and test the LNET configuration.</emphasis> LNET will, by default, use the first TCP/IP interface it discovers on a system. If this network configuration is sufficient, you do not need to configure LNET. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
+ <para><emphasis>Set</emphasis>lnet<emphasis>module parameters to specify how Lustre Networking (LNET) is to be configured to work with Lustre and test the LNET configuration.</emphasis> LNET will, by default, use the first TCP/IP interface it discovers on a system. If this network configuration is sufficient, you do not need to configure LNET. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
</listitem>
</itemizedlist>
-<para><anchor xml:id="dbdoclet.50438267_pgfId-1294840" xreflabel=""/>For information about configuring LNET, see <xref linkend='configuringlnet'/>. For information about testing LNET, see <xref linkend='lnetselftest'/>.</para>
+<para>For information about configuring LNET, see <xref linkend='configuringlnet'/>. For information about testing LNET, see <xref linkend='lnetselftest'/>.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1294602" xreflabel=""/><emphasis>Run the benchmark script sgpdd_survey to determine baseline performance of your hardware.</emphasis> Benchmarking your hardware will simplify debugging performance issues that are unrelated to Lustre and ensure you are getting the best possible performance with your installation. For information about running sgpdd_survey, see <xref linkend='benchmarkingtests'/>.</para>
+ <para><emphasis>Run the benchmark script sgpdd_survey to determine baseline performance of your hardware.</emphasis> Benchmarking your hardware will simplify debugging performance issues that are unrelated to Lustre and ensure you are getting the best possible performance with your installation. For information about running sgpdd_survey, see <xref linkend='benchmarkingtests'/>.</para>
</listitem>
</itemizedlist>
The sgpdd_survey script overwrites the device being tested so it must be run before the OSTs are configured.</para>
</note>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291941" xreflabel=""/>To configure a simple Lustre file system, complete these steps:</para>
+ <para>To configure a simple Lustre file system, complete these steps:</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1290894" xreflabel=""/><emphasis role="bold">Create</emphasis> a combined MGS/MDT file system on a block device. On the MDS node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1290905" xreflabel=""/>mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgs --mdt <<emphasis>block device name</emphasis>>
+ <para><emphasis role="bold">Create</emphasis> a combined MGS/MDT file system on a block device. On the MDS node, run:</para>
+ <screen>mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgs --mdt <<emphasis>block device name</emphasis>>
</screen>
<note>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1290909" xreflabel=""/>Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1290910" xreflabel=""/>mount -t lustre <<emphasis>block device name</emphasis>> <<emphasis>mount point</emphasis>>
+ <para>Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</para>
+ <screen>mount -t lustre <<emphasis>block device name</emphasis>> <<emphasis>mount point</emphasis>>
</screen>
<note><para>
</listitem>
- <listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1290915" xreflabel=""/>Create the OST. On the OSS node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1290916" xreflabel=""/>mkfs.lustre --ost --fsname=<<emphasis>fsname</emphasis>> --mgsnode=<<emphasis>NID</emphasis>> <<emphasis>block device name</emphasis>>
+ <listitem xml:id="dbdoclet.50438267_pgfId-1290915">
+ <para>Create the OST. On the OSS node, run:</para>
+ <screen>mkfs.lustre --ost --fsname=<<emphasis>fsname</emphasis>> --mgsnode=<<emphasis>NID</emphasis>> <<emphasis>block device name</emphasis>>
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293526" xreflabel=""/>When you create an OST, you are formatting a ldiskfs file system on a block storage device like you would with any local file system.</para>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291922" xreflabel=""/>You can have as many OSTs per OSS as the hardware or drivers allow. For more information about storage and memory requirements for a Lustre file system, see <xref linkend='settinguplustresystem'/>.</para>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1290918" xreflabel=""/>You can only configure one OST per block device. You should create an OST that uses the raw block device and does not use partitioning.</para>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293567" xreflabel=""/>If you are using block devices that are accessible from multiple OSS nodes, ensure that you mount the OSTs from only one OSS node at at time. It is strongly recommended that multiple-mount protection be enabled for such devices to prevent serious data corruption. For more information about multiple-mount protection, see <xref linkend='managingfailover'/>.</para>
+ <para>When you create an OST, you are formatting a ldiskfs file system on a block storage device like you would with any local file system.</para>
+ <para>You can have as many OSTs per OSS as the hardware or drivers allow. For more information about storage and memory requirements for a Lustre file system, see <xref linkend='settinguplustresystem'/>.</para>
+ <para>You can only configure one OST per block device. You should create an OST that uses the raw block device and does not use partitioning.</para>
+ <para>If you are using block devices that are accessible from multiple OSS nodes, ensure that you mount the OSTs from only one OSS node at at time. It is strongly recommended that multiple-mount protection be enabled for such devices to prevent serious data corruption. For more information about multiple-mount protection, see <xref linkend='managingfailover'/>.</para>
<note><para>
Lustre currently supports block devices up to 16 TB on OEL 5/RHEL 5 (up to 8 TB on other distributions). If the device size is only slightly larger that 16 TB, it is recommended that you limit the file system size to 16 TB at format time. If the size is significantly larger than 16 TB, you should reconfigure the storage into devices smaller than 16 TB. We recommend that you not place partitions on top of RAID 5/6 block devices due to negative impacts on performance.</para>
</listitem>
- <listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293955" xreflabel=""/>Mount the OST. On the OSS node where the OST was created, run:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1290925" xreflabel=""/>mount -t lustre <emphasis><block device name> <mount point></emphasis></screen>
+ <listitem xml:id="dbdoclet.50438267_pgfId-1293955">
+ <para>Mount the OST. On the OSS node where the OST was created, run:</para>
+ <screen>mount -t lustre <emphasis><block device name> <mount point></emphasis></screen>
<note><para>
</note>
</listitem>
- <listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1290934" xreflabel=""/>Mount the Lustre file system on the client. On the client node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1290935" xreflabel=""/>mount -t lustre <<emphasis>MGS node</emphasis>>:/<<emphasis>fsname</emphasis>> <<emphasis>mount point</emphasis>>
+ <listitem xml:id="dbdoclet.50438267_pgfId-1290934">
+ <para>Mount the Lustre file system on the client. On the client node, run:</para>
+ <screen>mount -t lustre <<emphasis>MGS node</emphasis>>:/<<emphasis>fsname</emphasis>> <<emphasis>mount point</emphasis>>
</screen>
<note><para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1290940" xreflabel=""/>Verify that the file system started and is working correctly. Do this by running the lfs df, dd and ls commands on the client node.</para>
+ <para>Verify that the file system started and is working correctly. Do this by running the lfs df, dd and ls commands on the client node.</para>
<note><para>
If you have a problem mounting the file system, check the syslogs on the client and all the servers for errors and also check the network settings. A common issue with newly-installed systems is that hosts.deny or firewall rules may prevent connections on port 988.</para>
<para><emphasis>(Optional) Run benchmarking to</emphasis>ols to validate the performance of hardware and software layers in the cluster. Available tools include:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1294743" xreflabel=""/>obdfilter_survey - Characterizes the storage performance of a Lustre file system. For details, see <xref linkend='benchmarkingtests'/><emphasis><link xl:href="BenchmarkingTests.html#50438212_40624">Testing OST Performance (obdfilter_survey)</link></emphasis>.</para>
+ <para>obdfilter_survey - Characterizes the storage performance of a Lustre file system. For details, see <xref linkend='benchmarkingtests'/><emphasis><link xl:href="BenchmarkingTests.html#50438212_40624">Testing OST Performance (obdfilter_survey)</link></emphasis>.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1294744" xreflabel=""/>ost_survey - Performs I/O against OSTs to detect anomalies between otherwise identical disk subsystems. For details, see <xref linkend='benchmarkingtests'/><emphasis><link xl:href="BenchmarkingTests.html#50438212_85136">Testing OST I/O Performance (ost_survey)</link></emphasis>.</para>
+ <para>ost_survey - Performs I/O against OSTs to detect anomalies between otherwise identical disk subsystems. For details, see <xref linkend='benchmarkingtests'/><emphasis><link xl:href="BenchmarkingTests.html#50438212_85136">Testing OST I/O Performance (ost_survey)</link></emphasis>.</para>
</listitem>
</itemizedlist>
</listitem>
</orderedlist>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438267_pgfId-1290956" xreflabel=""/>10.1.1 Simple Lustre <anchor xml:id="dbdoclet.50438267_marker-1290955" xreflabel=""/>Configuration Example</title>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1290959" xreflabel=""/>To see the steps in a simple Lustre configuration, follow this example in which a combined MGS/MDT and two OSTs are created. Three block devices are used, one for the combined MGS/MDS node and one for each OSS node. Common parameters used in the example are listed below, along with individual node parameters.</para>
+ <title>10.1.1 Simple Lustre <anchor xml:id="dbdoclet.50438267_marker-1290955" xreflabel=""/>Configuration Example</title>
+ <para>To see the steps in a simple Lustre configuration, follow this example in which a combined MGS/MDT and two OSTs are created. Three block devices are used, one for the combined MGS/MDS node and one for each OSS node. Common parameters used in the example are listed below, along with individual node parameters.</para>
<informaltable frame="all">
<tgroup cols="4">
<colspec colname="c1" colwidth="2*"/>
<colspec colname="c4" colwidth="25*"/>
<thead>
<row>
- <entry nameend="c2" namest="c1"><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438267_pgfId-1290962" xreflabel=""/>Common Parameters</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438267_pgfId-1290966" xreflabel=""/>Value</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438267_pgfId-1290968" xreflabel=""/>Description</emphasis></para></entry>
+ <entry nameend="c2" namest="c1"><para><emphasis role="bold">Common Parameters</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Value</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1290970" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1290972" xreflabel=""/><emphasis role="bold">MGS node</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1290974" xreflabel=""/>10.2.0.1@tcp0</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1290976" xreflabel=""/>Node for the combined MGS/MDS</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">MGS node</emphasis></para></entry>
+ <entry><para> 10.2.0.1@tcp0</para></entry>
+ <entry><para> Node for the combined MGS/MDS</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1290978" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1290980" xreflabel=""/><emphasis role="bold">file system</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1290982" xreflabel=""/>temp</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1290984" xreflabel=""/>Name of the Lustre file system</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">file system</emphasis></para></entry>
+ <entry><para> temp</para></entry>
+ <entry><para> Name of the Lustre file system</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1290986" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1290988" xreflabel=""/><emphasis role="bold">network type</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1290990" xreflabel=""/>TCP/IP</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1290992" xreflabel=""/>Network type used for Lustre file system temp</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">network type</emphasis></para></entry>
+ <entry><para> TCP/IP</para></entry>
+ <entry><para> Network type used for Lustre file system temp</para></entry>
</row>
</tbody>
</tgroup>
<colspec colname="c4" colwidth="25*"/>
<thead>
<row>
- <entry nameend="c2" namest="c1"><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438267_pgfId-1290995" xreflabel=""/>Node Parameters</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438267_pgfId-1290999" xreflabel=""/>Value</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438267_pgfId-1291001" xreflabel=""/>Description</emphasis></para></entry>
+ <entry nameend="c2" namest="c1"><para><emphasis role="bold">Node Parameters</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Value</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry nameend="c4" namest="c1"><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291003" xreflabel=""/>MGS/MDS node</para></entry>
+ <entry nameend="c4" namest="c1"><para> MGS/MDS node</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291011" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291013" xreflabel=""/><emphasis role="bold">MGS/MDS node</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291015" xreflabel=""/>mdt1</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291017" xreflabel=""/>MDS in Lustre file system temp</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">MGS/MDS node</emphasis></para></entry>
+ <entry><para> mdt1</para></entry>
+ <entry><para> MDS in Lustre file system temp</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291019" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291021" xreflabel=""/><emphasis role="bold">block device</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291023" xreflabel=""/>/dev/sdb</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291025" xreflabel=""/>Block device for the combined MGS/MDS node</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">block device</emphasis></para></entry>
+ <entry><para> /dev/sdb</para></entry>
+ <entry><para> Block device for the combined MGS/MDS node</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291027" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291029" xreflabel=""/><emphasis role="bold">mount point</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291031" xreflabel=""/>/mnt/mdt</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291033" xreflabel=""/>Mount point for the mdt1 block device (/dev/sdb) on the MGS/MDS node</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">mount point</emphasis></para></entry>
+ <entry><para> /mnt/mdt</para></entry>
+ <entry><para> Mount point for the mdt1 block device (/dev/sdb) on the MGS/MDS node</para></entry>
</row>
<row>
- <entry nameend="c4" namest="c1"><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291035" xreflabel=""/>First OSS node</para></entry>
+ <entry nameend="c4" namest="c1"><para> First OSS node</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291043" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291045" xreflabel=""/><emphasis role="bold">OSS node</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291047" xreflabel=""/>oss1</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291049" xreflabel=""/>First OSS node in Lustre file system temp</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">OSS node</emphasis></para></entry>
+ <entry><para> oss1</para></entry>
+ <entry><para> First OSS node in Lustre file system temp</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291051" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291053" xreflabel=""/><emphasis role="bold">OST</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291055" xreflabel=""/>ost1</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291057" xreflabel=""/>First OST in Lustre file system temp</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">OST</emphasis></para></entry>
+ <entry><para> ost1</para></entry>
+ <entry><para> First OST in Lustre file system temp</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291059" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291061" xreflabel=""/><emphasis role="bold">block device</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291063" xreflabel=""/>/dev/sdc</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291065" xreflabel=""/>Block device for the first OSS node (oss1)</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">block device</emphasis></para></entry>
+ <entry><para> /dev/sdc</para></entry>
+ <entry><para> Block device for the first OSS node (oss1)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291067" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291069" xreflabel=""/><emphasis role="bold">mount point</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291071" xreflabel=""/>/mnt/ost1</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291073" xreflabel=""/>Mount point for the ost1 block device (/dev/sdc) on the oss1 node</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">mount point</emphasis></para></entry>
+ <entry><para> /mnt/ost1</para></entry>
+ <entry><para> Mount point for the ost1 block device (/dev/sdc) on the oss1 node</para></entry>
</row>
<row>
- <entry nameend="c4" namest="c1"><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291075" xreflabel=""/>Second OSS node</para></entry>
+ <entry nameend="c4" namest="c1"><para> Second OSS node</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291083" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291085" xreflabel=""/><emphasis role="bold">OSS node</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291087" xreflabel=""/>oss2</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291089" xreflabel=""/>Second OSS node in Lustre file system temp</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">OSS node</emphasis></para></entry>
+ <entry><para> oss2</para></entry>
+ <entry><para> Second OSS node in Lustre file system temp</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291091" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291093" xreflabel=""/><emphasis role="bold">OST</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291095" xreflabel=""/>ost2</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291097" xreflabel=""/>Second OST in Lustre file system temp</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">OST</emphasis></para></entry>
+ <entry><para> ost2</para></entry>
+ <entry><para> Second OST in Lustre file system temp</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291099" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291101" xreflabel=""/><emphasis role="bold">block device</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291103" xreflabel=""/>/dev/sdd</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291105" xreflabel=""/>Block device for the second OSS node (oss2)</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">block device</emphasis></para></entry>
+ <entry><para> /dev/sdd</para></entry>
+ <entry><para> Block device for the second OSS node (oss2)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291107" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291109" xreflabel=""/><emphasis role="bold">mount point</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291111" xreflabel=""/>/mnt/ost2</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291113" xreflabel=""/>Mount point for the ost2 block device (/dev/sdd) on the oss2 node</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">mount point</emphasis></para></entry>
+ <entry><para> /mnt/ost2</para></entry>
+ <entry><para> Mount point for the ost2 block device (/dev/sdd) on the oss2 node</para></entry>
</row>
<row>
- <entry nameend="c4" namest="c1"><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291115" xreflabel=""/>Client node</para></entry>
+ <entry nameend="c4" namest="c1"><para> Client node</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291123" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291125" xreflabel=""/><emphasis role="bold">client node</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291127" xreflabel=""/>client1</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291129" xreflabel=""/>Client in Lustre file system temp</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">client node</emphasis></para></entry>
+ <entry><para> client1</para></entry>
+ <entry><para> Client in Lustre file system temp</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291131" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291133" xreflabel=""/><emphasis role="bold">mount point</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291135" xreflabel=""/>/lustre</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1291137" xreflabel=""/>Mount point for Lustre file system temp on the client1 node</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">mount point</emphasis></para></entry>
+ <entry><para> /lustre</para></entry>
+ <entry><para> Mount point for Lustre file system temp on the client1 node</para></entry>
</row>
</tbody>
</tgroup>
We recommend that you use 'dotted-quad' notation for IP addresses rather than host names to make it easier to read debug logs and debug configurations with multiple interfaces.</para>
</note>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291430" xreflabel=""/>For this example, complete the steps below:</para>
+ <para>For this example, complete the steps below:</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291140" xreflabel=""/> 1. Create a combined MGS/MDT file system on the block device. On the MDS node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291141" xreflabel=""/>[root@mds /]# mkfs.lustre --fsname=temp --mgs --mdt /dev/sdb
+ <para>Create a combined MGS/MDT file system on the block device. On the MDS node, run:</para>
+ <screen>[root@mds /]# mkfs.lustre --fsname=temp --mgs --mdt /dev/sdb
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291142" xreflabel=""/>This command generates this output:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291143" xreflabel=""/> Permanent disk data:
-<anchor xml:id="dbdoclet.50438267_pgfId-1291144" xreflabel=""/>Target: temp-MDTffff
-<anchor xml:id="dbdoclet.50438267_pgfId-1291145" xreflabel=""/>Index: unassigned
-<anchor xml:id="dbdoclet.50438267_pgfId-1291146" xreflabel=""/>Lustre FS: temp
-<anchor xml:id="dbdoclet.50438267_pgfId-1291147" xreflabel=""/>Mount type: ldiskfs
-<anchor xml:id="dbdoclet.50438267_pgfId-1291148" xreflabel=""/>Flags: 0x75
-<anchor xml:id="dbdoclet.50438267_pgfId-1291149" xreflabel=""/> (MDT MGS needs_index first_time update )
-<anchor xml:id="dbdoclet.50438267_pgfId-1291150" xreflabel=""/>Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
-<anchor xml:id="dbdoclet.50438267_pgfId-1291151" xreflabel=""/>Parameters: mdt.group_upcall=/usr/sbin/l_getgroups
-<anchor xml:id="dbdoclet.50438267_pgfId-1291152" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438267_pgfId-1291153" xreflabel=""/>checking for existing Lustre data: not found
-<anchor xml:id="dbdoclet.50438267_pgfId-1291154" xreflabel=""/>device size = 16MB
-<anchor xml:id="dbdoclet.50438267_pgfId-1291155" xreflabel=""/>2 6 18
-<anchor xml:id="dbdoclet.50438267_pgfId-1291156" xreflabel=""/>formatting backing filesystem ldiskfs on /dev/sdb
-<anchor xml:id="dbdoclet.50438267_pgfId-1291157" xreflabel=""/> target name temp-MDTffff
-<anchor xml:id="dbdoclet.50438267_pgfId-1291158" xreflabel=""/> 4k blocks 0
-<anchor xml:id="dbdoclet.50438267_pgfId-1291159" xreflabel=""/> options -i 4096 -I 512 -q -O dir_index,uninit_groups -F
-<anchor xml:id="dbdoclet.50438267_pgfId-1291160" xreflabel=""/>mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-MDTffff -i 4096 -I 512 -q -O
-<anchor xml:id="dbdoclet.50438267_pgfId-1291161" xreflabel=""/>dir_index,uninit_groups -F /dev/sdb
-<anchor xml:id="dbdoclet.50438267_pgfId-1291162" xreflabel=""/>Writing CONFIGS/mountdata
+ <para>This command generates this output:</para>
+ <screen> Permanent disk data:
+Target: temp-MDTffff
+Index: unassigned
+Lustre FS: temp
+Mount type: ldiskfs
+Flags: 0x75
+ (MDT MGS needs_index first_time update )
+Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
+Parameters: mdt.group_upcall=/usr/sbin/l_getgroups
+
+checking for existing Lustre data: not found
+device size = 16MB
+2 6 18
+formatting backing filesystem ldiskfs on /dev/sdb
+ target name temp-MDTffff
+ 4k blocks 0
+ options -i 4096 -I 512 -q -O dir_index,uninit_groups -F
+mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-MDTffff -i 4096 -I 512 -q -O
+dir_index,uninit_groups -F /dev/sdb
+Writing CONFIGS/mountdata
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291163" xreflabel=""/> 2. Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291164" xreflabel=""/>[root@mds /]# mount -t lustre /dev/sdb /mnt/mdt
+ <para>Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</para>
+ <screen>[root@mds /]# mount -t lustre /dev/sdb /mnt/mdt
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291165" xreflabel=""/>This command generates this output:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291166" xreflabel=""/>Lustre: temp-MDT0000: new disk, initializing
-<anchor xml:id="dbdoclet.50438267_pgfId-1291167" xreflabel=""/>Lustre: 3009:0:(lproc_mds.c:262:lprocfs_wr_group_upcall()) temp-MDT0000: gr\
+ <para>This command generates this output:</para>
+ <screen>Lustre: temp-MDT0000: new disk, initializing
+Lustre: 3009:0:(lproc_mds.c:262:lprocfs_wr_group_upcall()) temp-MDT0000: gr\
oup upcall set to /usr/sbin/l_getgroups
-<anchor xml:id="dbdoclet.50438267_pgfId-1291168" xreflabel=""/>Lustre: temp-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups
-<anchor xml:id="dbdoclet.50438267_pgfId-1291169" xreflabel=""/>Lustre: Server temp-MDT0000 on device /dev/sdb has started
+Lustre: temp-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups
+Lustre: Server temp-MDT0000 on device /dev/sdb has started
</screen>
</listitem>
- <listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291170" xreflabel=""/> 3. Create and mount ost1.</para>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291171" xreflabel=""/>In this example, the OSTs (ost1 and ost2) are being created on different OSSs (oss1 and oss2 respectively).</para>
+ <listitem xml:id="dbdoclet.50438267_pgfId-1291170">
+ <para>Create and mount ost1.</para>
+ <para>In this example, the OSTs (ost1 and ost2) are being created on different OSSs (oss1 and oss2 respectively).</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291172" xreflabel=""/>a. Create ost1. On oss1 node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291173" xreflabel=""/>[root@oss1 /]# mkfs.lustre --ost --fsname=temp --mgsnode=10.2.0.1@tcp0 /dev\
+ <para>a. Create ost1. On oss1 node, run:</para>
+ <screen>[root@oss1 /]# mkfs.lustre --ost --fsname=temp --mgsnode=10.2.0.1@tcp0 /dev\
/sdc
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291174" xreflabel=""/>The command generates this output:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291175" xreflabel=""/> Permanent disk data:
-<anchor xml:id="dbdoclet.50438267_pgfId-1291176" xreflabel=""/>Target: temp-OSTffff
-<anchor xml:id="dbdoclet.50438267_pgfId-1291177" xreflabel=""/>Index: unassigned
-<anchor xml:id="dbdoclet.50438267_pgfId-1291178" xreflabel=""/>Lustre FS: temp
-<anchor xml:id="dbdoclet.50438267_pgfId-1291179" xreflabel=""/>Mount type: ldiskfs
-<anchor xml:id="dbdoclet.50438267_pgfId-1291180" xreflabel=""/>Flags: 0x72
-<anchor xml:id="dbdoclet.50438267_pgfId-1291181" xreflabel=""/>(OST needs_index first_time update)
-<anchor xml:id="dbdoclet.50438267_pgfId-1291182" xreflabel=""/>Persistent mount opts: errors=remount-ro,extents,mballoc
-<anchor xml:id="dbdoclet.50438267_pgfId-1291183" xreflabel=""/>Parameters: mgsnode=10.2.0.1@tcp
-<anchor xml:id="dbdoclet.50438267_pgfId-1291184" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438267_pgfId-1291185" xreflabel=""/>checking for existing Lustre data: not found
-<anchor xml:id="dbdoclet.50438267_pgfId-1291186" xreflabel=""/>device size = 16MB
-<anchor xml:id="dbdoclet.50438267_pgfId-1291187" xreflabel=""/>2 6 18
-<anchor xml:id="dbdoclet.50438267_pgfId-1291188" xreflabel=""/>formatting backing filesystem ldiskfs on /dev/sdc
-<anchor xml:id="dbdoclet.50438267_pgfId-1291189" xreflabel=""/> target name temp-OSTffff
-<anchor xml:id="dbdoclet.50438267_pgfId-1291190" xreflabel=""/> 4k blocks 0
-<anchor xml:id="dbdoclet.50438267_pgfId-1291191" xreflabel=""/> options -I 256 -q -O dir_index,uninit_groups -F
-<anchor xml:id="dbdoclet.50438267_pgfId-1291192" xreflabel=""/>mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-OSTffff -I 256 -q -O
-<anchor xml:id="dbdoclet.50438267_pgfId-1291193" xreflabel=""/>dir_index,uninit_groups -F /dev/sdc
-<anchor xml:id="dbdoclet.50438267_pgfId-1291194" xreflabel=""/>Writing CONFIGS/mountdata
+ <para>The command generates this output:</para>
+ <screen> Permanent disk data:
+Target: temp-OSTffff
+Index: unassigned
+Lustre FS: temp
+Mount type: ldiskfs
+Flags: 0x72
+(OST needs_index first_time update)
+Persistent mount opts: errors=remount-ro,extents,mballoc
+Parameters: mgsnode=10.2.0.1@tcp
+
+checking for existing Lustre data: not found
+device size = 16MB
+2 6 18
+formatting backing filesystem ldiskfs on /dev/sdc
+ target name temp-OSTffff
+ 4k blocks 0
+ options -I 256 -q -O dir_index,uninit_groups -F
+mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-OSTffff -I 256 -q -O
+dir_index,uninit_groups -F /dev/sdc
+Writing CONFIGS/mountdata
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293613" xreflabel=""/>b. Mount ost1 on the OSS on which it was created. On oss1 node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1293614" xreflabel=""/>root@oss1 /] mount -t lustre /dev/sdc /mnt/ost1
+ <para>b. Mount ost1 on the OSS on which it was created. On oss1 node, run:</para>
+ <screen>root@oss1 /] mount -t lustre /dev/sdc /mnt/ost1
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293615" xreflabel=""/>The command generates this output:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1293616" xreflabel=""/>LDISKFS-fs: file extents enabled
-<anchor xml:id="dbdoclet.50438267_pgfId-1293617" xreflabel=""/>LDISKFS-fs: mballoc enabled
-<anchor xml:id="dbdoclet.50438267_pgfId-1293618" xreflabel=""/>Lustre: temp-OST0000: new disk, initializing
-<anchor xml:id="dbdoclet.50438267_pgfId-1293619" xreflabel=""/>Lustre: Server temp-OST0000 on device /dev/sdb has started
+ <para>The command generates this output:</para>
+ <screen>LDISKFS-fs: file extents enabled
+LDISKFS-fs: mballoc enabled
+Lustre: temp-OST0000: new disk, initializing
+Lustre: Server temp-OST0000 on device /dev/sdb has started
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293620" xreflabel=""/>Shortly afterwards, this output appears:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1293621" xreflabel=""/>Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0
-<anchor xml:id="dbdoclet.50438267_pgfId-1293643" xreflabel=""/>Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphans
+ <para>Shortly afterwards, this output appears:</para>
+ <screen>Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0
+Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphans
</screen>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293656" xreflabel=""/>Create and mount ost2.</para>
+ <para>Create and mount ost2.</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293645" xreflabel=""/>Create ost2. On oss2 node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291196" xreflabel=""/>[root@oss2 /]# mkfs.lustre --ost --fsname=temp --mgsnode=10.2.0.1@tcp0 /dev\
+ <para>Create ost2. On oss2 node, run:</para>
+ <screen>[root@oss2 /]# mkfs.lustre --ost --fsname=temp --mgsnode=10.2.0.1@tcp0 /dev\
/sdd
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291197" xreflabel=""/>The command generates this output:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291198" xreflabel=""/> Permanent disk data:
-<anchor xml:id="dbdoclet.50438267_pgfId-1291199" xreflabel=""/>Target: temp-OSTffff
-<anchor xml:id="dbdoclet.50438267_pgfId-1291200" xreflabel=""/>Index: unassigned
-<anchor xml:id="dbdoclet.50438267_pgfId-1291201" xreflabel=""/>Lustre FS: temp
-<anchor xml:id="dbdoclet.50438267_pgfId-1291202" xreflabel=""/>Mount type: ldiskfs
-<anchor xml:id="dbdoclet.50438267_pgfId-1291203" xreflabel=""/>Flags: 0x72
-<anchor xml:id="dbdoclet.50438267_pgfId-1291204" xreflabel=""/>(OST needs_index first_time update)
-<anchor xml:id="dbdoclet.50438267_pgfId-1291205" xreflabel=""/>Persistent mount opts: errors=remount-ro,extents,mballoc
-<anchor xml:id="dbdoclet.50438267_pgfId-1291206" xreflabel=""/>Parameters: mgsnode=10.2.0.1@tcp
-<anchor xml:id="dbdoclet.50438267_pgfId-1291209" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438267_pgfId-1291211" xreflabel=""/>checking for existing Lustre data: not found
-<anchor xml:id="dbdoclet.50438267_pgfId-1291212" xreflabel=""/>device size = 16MB
-<anchor xml:id="dbdoclet.50438267_pgfId-1291213" xreflabel=""/>2 6 18
-<anchor xml:id="dbdoclet.50438267_pgfId-1291214" xreflabel=""/>formatting backing filesystem ldiskfs on /dev/sdd
-<anchor xml:id="dbdoclet.50438267_pgfId-1291215" xreflabel=""/> target name temp-OSTffff
-<anchor xml:id="dbdoclet.50438267_pgfId-1291216" xreflabel=""/> 4k blocks 0
-<anchor xml:id="dbdoclet.50438267_pgfId-1291217" xreflabel=""/> options -I 256 -q -O dir_index,uninit_groups -F
-<anchor xml:id="dbdoclet.50438267_pgfId-1291218" xreflabel=""/>mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-OSTffff -I 256 -q -O
-<anchor xml:id="dbdoclet.50438267_pgfId-1291219" xreflabel=""/>dir_index,uninit_groups -F /dev/sdc
-<anchor xml:id="dbdoclet.50438267_pgfId-1291220" xreflabel=""/>Writing CONFIGS/mountdata
+ <para>The command generates this output:</para>
+ <screen> Permanent disk data:
+Target: temp-OSTffff
+Index: unassigned
+Lustre FS: temp
+Mount type: ldiskfs
+Flags: 0x72
+(OST needs_index first_time update)
+Persistent mount opts: errors=remount-ro,extents,mballoc
+Parameters: mgsnode=10.2.0.1@tcp
+
+checking for existing Lustre data: not found
+device size = 16MB
+2 6 18
+formatting backing filesystem ldiskfs on /dev/sdd
+ target name temp-OSTffff
+ 4k blocks 0
+ options -I 256 -q -O dir_index,uninit_groups -F
+mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-OSTffff -I 256 -q -O
+dir_index,uninit_groups -F /dev/sdc
+Writing CONFIGS/mountdata
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291233" xreflabel=""/>Mount ost2 on the OSS on which it was created. On oss2 node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291234" xreflabel=""/>root@oss2 /] mount -t lustre /dev/sdd /mnt/ost2
+ <para>Mount ost2 on the OSS on which it was created. On oss2 node, run:</para>
+ <screen>root@oss2 /] mount -t lustre /dev/sdd /mnt/ost2
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291235" xreflabel=""/>The command generates this output:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291236" xreflabel=""/>LDISKFS-fs: file extents enabled
-<anchor xml:id="dbdoclet.50438267_pgfId-1291237" xreflabel=""/>LDISKFS-fs: mballoc enabled
-<anchor xml:id="dbdoclet.50438267_pgfId-1291238" xreflabel=""/>Lustre: temp-OST0000: new disk, initializing
-<anchor xml:id="dbdoclet.50438267_pgfId-1291239" xreflabel=""/>Lustre: Server temp-OST0000 on device /dev/sdb has started
+ <para>The command generates this output:</para>
+ <screen>LDISKFS-fs: file extents enabled
+LDISKFS-fs: mballoc enabled
+Lustre: temp-OST0000: new disk, initializing
+Lustre: Server temp-OST0000 on device /dev/sdb has started
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291240" xreflabel=""/>Shortly afterwards, this output appears:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291241" xreflabel=""/>Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0
-<anchor xml:id="dbdoclet.50438267_pgfId-1291242" xreflabel=""/>Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphans
+ <para>Shortly afterwards, this output appears:</para>
+ <screen>Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0
+Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphans
</screen>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291243" xreflabel=""/>Mount the Lustre file system on the client. On the client node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291244" xreflabel=""/>root@client1 /] mount -t lustre 10.2.0.1@tcp0:/temp /lustre
+ <para>Mount the Lustre file system on the client. On the client node, run:</para>
+ <screen>root@client1 /] mount -t lustre 10.2.0.1@tcp0:/temp /lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291245" xreflabel=""/>This command generates this output:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291246" xreflabel=""/>Lustre: Client temp-client has started
+ <para>This command generates this output:</para>
+ <screen>Lustre: Client temp-client has started
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291247" xreflabel=""/>Verify that the file system started and is working by running the df, dd and ls commands on the client node.</para>
+ <para>Verify that the file system started and is working by running the df, dd and ls commands on the client node.</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291248" xreflabel=""/>Run the lfsdf -h command:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291249" xreflabel=""/>[root@client1 /] lfs df -h
+ <para>Run the lfsdf -h command:</para>
+ <screen>[root@client1 /] lfs df -h
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293698" xreflabel=""/>The lfsdf-h command lists space usage per OST and the MDT in human-readable format. This command generates output similar to this:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1294042" xreflabel=""/>UUID bytes Used Available \
+ <para>The lfsdf-h command lists space usage per OST and the MDT in human-readable format. This command generates output similar to this:</para>
+ <screen>UUID bytes Used Available \
Use% Mounted on
-<anchor xml:id="dbdoclet.50438267_pgfId-1294043" xreflabel=""/>temp-MDT0000_UUID 8.0G 400.0M 7.6G 0% \
+temp-MDT0000_UUID 8.0G 400.0M 7.6G 0% \
/lustre[MDT:0]
-<anchor xml:id="dbdoclet.50438267_pgfId-1294044" xreflabel=""/>temp-OST0000_UUID 800.0G 400.0M 799.6G 0% \
+temp-OST0000_UUID 800.0G 400.0M 799.6G 0% \
/lustre[OST:0]
-<anchor xml:id="dbdoclet.50438267_pgfId-1294045" xreflabel=""/>temp-OST0001_UUID 800.0G 400.0M 799.6G 0% \
+temp-OST0001_UUID 800.0G 400.0M 799.6G 0% \
/lustre[OST:1]
-<anchor xml:id="dbdoclet.50438267_pgfId-1294046" xreflabel=""/>filesystem summary: 1.6T 800.0M 1.6T \
+filesystem summary: 1.6T 800.0M 1.6T \
0% /lustre
-<anchor xml:id="dbdoclet.50438267_pgfId-1294039" xreflabel=""/>
+
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293715" xreflabel=""/>Run the lfsdf-ih command.</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1293716" xreflabel=""/>[root@client1 /] lfs df -ih
+ <para>Run the lfsdf-ih command.</para>
+ <screen>[root@client1 /] lfs df -ih
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293717" xreflabel=""/>The lfsdf-ih command lists inode usage per OST and the MDT. This command generates output similar to this:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1294079" xreflabel=""/>UUID Inodes IUsed IFree IUse% Mounted\
+ <para>The lfsdf-ih command lists inode usage per OST and the MDT. This command generates output similar to this:</para>
+ <screen>UUID Inodes IUsed IFree IUse% Mounted\
on
-<anchor xml:id="dbdoclet.50438267_pgfId-1294080" xreflabel=""/>temp-MDT0000_UUID 2.5M 32 2.5M 0% /lustre[MDT:0]
-<anchor xml:id="dbdoclet.50438267_pgfId-1294081" xreflabel=""/>temp-OST0000_UUID 5.5M 54 5.5M 0% /lustre[OST:0]
-<anchor xml:id="dbdoclet.50438267_pgfId-1294082" xreflabel=""/>temp-OST0001_UUID 5.5M 54 5.5M 0% /lustre[OST:1]
-<anchor xml:id="dbdoclet.50438267_pgfId-1294083" xreflabel=""/>filesystem summary: 2.5M 32 2.5M 0% /lustre
-<anchor xml:id="dbdoclet.50438267_pgfId-1294084" xreflabel=""/>
+temp-MDT0000_UUID 2.5M 32 2.5M 0% /lustre[MDT:0]
+temp-OST0000_UUID 5.5M 54 5.5M 0% /lustre[OST:0]
+temp-OST0001_UUID 5.5M 54 5.5M 0% /lustre[OST:1]
+filesystem summary: 2.5M 32 2.5M 0% /lustre
+
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291257" xreflabel=""/>c. Run the dd command:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291258" xreflabel=""/>[root@client1 /] cd /lustre
-<anchor xml:id="dbdoclet.50438267_pgfId-1291259" xreflabel=""/>[root@client1 /lustre] dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2
+ <para>c. Run the dd command:</para>
+ <screen>[root@client1 /] cd /lustre
+[root@client1 /lustre] dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293740" xreflabel=""/>The dd command verifies write functionality by creating a file containing all zeros (0s). In this command, an 8 MB file is created. This command generates output similar to this:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291261" xreflabel=""/>2+0 records in
-<anchor xml:id="dbdoclet.50438267_pgfId-1291262" xreflabel=""/>2+0 records out
-<anchor xml:id="dbdoclet.50438267_pgfId-1291263" xreflabel=""/>8388608 bytes (8.4 MB) copied, 0.159628 seconds, 52.6 MB/s
+ <para>The dd command verifies write functionality by creating a file containing all zeros (0s). In this command, an 8 MB file is created. This command generates output similar to this:</para>
+ <screen>2+0 records in
+2+0 records out
+8388608 bytes (8.4 MB) copied, 0.159628 seconds, 52.6 MB/s
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291264" xreflabel=""/>d. Run the ls command:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291265" xreflabel=""/>[root@client1 /lustre] ls -lsah
+ <para>d. Run the ls command:</para>
+ <screen>[root@client1 /lustre] ls -lsah
</screen>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291266" xreflabel=""/>The ls-lsah command lists files and directories in the current working directory. This command generates output similar to this:</para>
- <screen><anchor xml:id="dbdoclet.50438267_pgfId-1291267" xreflabel=""/>total 8.0M
-<anchor xml:id="dbdoclet.50438267_pgfId-1291268" xreflabel=""/>4.0K drwxr-xr-x 2 root root 4.0K Oct 16 15:27 .
-<anchor xml:id="dbdoclet.50438267_pgfId-1292416" xreflabel=""/>8.0K drwxr-xr-x 25 root root 4.0K Oct 16 15:27 ..
-<anchor xml:id="dbdoclet.50438267_pgfId-1292417" xreflabel=""/>8.0M -rw-r--r-- 1 root root 8.0M Oct 16 15:27 zero.dat
-<anchor xml:id="dbdoclet.50438267_pgfId-1292423" xreflabel=""/>
+ <para>The ls-lsah command lists files and directories in the current working directory. This command generates output similar to this:</para>
+ <screen>total 8.0M
+4.0K drwxr-xr-x 2 root root 4.0K Oct 16 15:27 .
+8.0K drwxr-xr-x 25 root root 4.0K Oct 16 15:27 ..
+8.0M -rw-r--r-- 1 root root 8.0M Oct 16 15:27 zero.dat
+
</screen>
</listitem>
</orderedlist>
</listitem>
</orderedlist>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1293200" xreflabel=""/>Once the Lustre file system is configured, it is ready for use.</para>
+ <para>Once the Lustre file system is configured, it is ready for use.</para>
</section>
</section>
<section xml:id="dbdoclet.50438267_76752">
<title>10.2 Additional Configuration Options</title>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1292458" xreflabel=""/>This section describes how to scale the Lustre file system or make configuration changes using the Lustre configuration utilities.</para>
+ <para>This section describes how to scale the Lustre file system or make configuration changes using the Lustre configuration utilities.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438267_pgfId-1292441" xreflabel=""/>10.2.1 Scaling the <anchor xml:id="dbdoclet.50438267_marker-1292440" xreflabel=""/>Lustre File System</title>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1292442" xreflabel=""/>A Lustre file system can be scaled by adding OSTs or clients. For instructions on creating additional OSTs repeat <xref linkend="dbdoclet.50438267_pgfId-1291170"/>Step 3 and <xref linkend="dbdoclet.50438267_pgfId-1293955"/>Step 4 above. For mounting additional clients, repeat <xref linkend="dbdoclet.50438267_pgfId-1290934"/>Step 5 for each client.</para>
+ <title>10.2.1 Scaling the <anchor xml:id="dbdoclet.50438267_marker-1292440" xreflabel=""/>Lustre File System</title>
+ <para>A Lustre file system can be scaled by adding OSTs or clients. For instructions on creating additional OSTs repeat <xref linkend="dbdoclet.50438267_pgfId-1291170"/>Step 3 and <xref linkend="dbdoclet.50438267_pgfId-1293955"/>Step 4 above. For mounting additional clients, repeat <xref linkend="dbdoclet.50438267_pgfId-1290934"/>Step 5 for each client.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438267_pgfId-1292798" xreflabel=""/>10.2.2 <anchor xml:id="dbdoclet.50438267_50212" xreflabel=""/>Changing Striping Defaults</title>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1292906" xreflabel=""/>The default settings for the file layout stripe pattern are shown in <xref linkend='configuringlustre.tab.stripe'/>.</para>
+ <title>10.2.2 <anchor xml:id="dbdoclet.50438267_50212" xreflabel=""/>Changing Striping Defaults</title>
+ <para>The default settings for the file layout stripe pattern are shown in <xref linkend='configuringlustre.tab.stripe'/>.</para>
<table frame="none" xml:id='configuringlustre.tab.stripe'>
<title>Default stripe pattern</title>
<tgroup cols="3">
<colspec colname="c3" colwidth="13*"/>
<tbody>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438267_pgfId-1292877" xreflabel=""/>File Layout Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438267_pgfId-1292879" xreflabel=""/>Default</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438267_pgfId-1292881" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">File Layout Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Default</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292883" xreflabel=""/>stripe_size</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292885" xreflabel=""/>1 MB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292887" xreflabel=""/>Amount of data to write to one OST before moving to the next OST.</para></entry>
+ <entry><para> stripe_size</para></entry>
+ <entry><para> 1 MB</para></entry>
+ <entry><para> Amount of data to write to one OST before moving to the next OST.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292889" xreflabel=""/>stripe_count</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292891" xreflabel=""/>1</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292893" xreflabel=""/>The number of OSTs to use for a single file.</para></entry>
+ <entry><para> stripe_count</para></entry>
+ <entry><para> 1</para></entry>
+ <entry><para> The number of OSTs to use for a single file.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292895" xreflabel=""/>start_ost</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292897" xreflabel=""/>-1</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292899" xreflabel=""/>The first OST where objects are created for each file. The default -1 allows the MDS to choose the starting index based on available space and load balancing. <emphasis>It's strongly recommended not to change the default for this parameter to a value other than -1.</emphasis></para></entry>
+ <entry><para> start_ost</para></entry>
+ <entry><para> -1</para></entry>
+ <entry><para> The first OST where objects are created for each file. The default -1 allows the MDS to choose the starting index based on available space and load balancing. <emphasis>It's strongly recommended not to change the default for this parameter to a value other than -1.</emphasis></para></entry>
</row>
</tbody>
</tgroup>
</table>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1292907" xreflabel=""/>Use the lfs setstripe command described in <xref linkend='managingstripingfreespace'/> to change the file layout configuration.</para>
+ <para>Use the lfs setstripe command described in <xref linkend='managingstripingfreespace'/> to change the file layout configuration.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438267_pgfId-1292908" xreflabel=""/>10.2.3 Using the Lustre Configuration Utilities</title>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1292436" xreflabel=""/>If additional configuration is necessary, several configuration utilities are available:</para>
+ <title>10.2.3 Using the Lustre Configuration Utilities</title>
+ <para>If additional configuration is necessary, several configuration utilities are available:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1292490" xreflabel=""/>mkfs.lustre - Use to format a disk for a Lustre service.</para>
+ <para>mkfs.lustre - Use to format a disk for a Lustre service.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1292492" xreflabel=""/>tunefs.lustre - Use to modify configuration information on a Lustre target disk.</para>
+ <para>tunefs.lustre - Use to modify configuration information on a Lustre target disk.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1292494" xreflabel=""/>lctl - Use to directly control Lustre via an ioctl interface, allowing various configuration, maintenance and debugging features to be accessed.</para>
+ <para>lctl - Use to directly control Lustre via an ioctl interface, allowing various configuration, maintenance and debugging features to be accessed.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1291315" xreflabel=""/>mount.lustre - Use to start a Lustre client or target service.</para>
+ <para>mount.lustre - Use to start a Lustre client or target service.</para>
</listitem>
</itemizedlist>
-<para><anchor xml:id="dbdoclet.50438267_pgfId-1295269" xreflabel=""/>For examples using these utilities, see the topic <xref linkend='systemconfigurationutilities'/></para>
- <para><anchor xml:id="dbdoclet.50438267_pgfId-1295265" xreflabel=""/>The lfs utility is usful for configuring and querying a variety of options related to files. For more information, see <xref linkend='userutilities'/>.</para>
+<para>For examples using these utilities, see the topic <xref linkend='systemconfigurationutilities'/></para>
+ <para>The lfs utility is usful for configuring and querying a variety of options related to files. For more information, see <xref linkend='userutilities'/>.</para>
<note><para>
Some sample scripts are included in the directory where Lustre is installed. If you have installed the Lustre source code, the scripts are located in the lustre/tests sub-directory. These scripts enable quick setup of some simple standard Lustre configurations.</para>
<title xml:id='configuringquotas.title'>Configuring and Managing Quotas</title>
</info>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290092" xreflabel=""/>This chapter describes how to configure quotas and includes the following sections:</para>
+ <para>This chapter describes how to configure quotas and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438217_54945"/></para>
<section xml:id="dbdoclet.50438217_54945">
<title>21.1 Working with <anchor xml:id="dbdoclet.50438217_marker-1290118" xreflabel=""/>Quotas</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290120" xreflabel=""/>Quotas allow a system administrator to limit the amount of disk space a user or group can use in a directory. Quotas are set by root, and can be specified for individual users and/or groups. Before a file is written to a partition where quotas are set, the quota of the creator's group is checked. If a quota exists, then the file size counts towards the group's quota. If no quota exists, then the owner's user quota is checked before the file is written. Similarly, inode usage for specific functions can be controlled if a user over-uses the allocated space.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290121" xreflabel=""/>Lustre quota enforcement differs from standard Linux quota enforcement in several ways:</para>
+ <para>Quotas allow a system administrator to limit the amount of disk space a user or group can use in a directory. Quotas are set by root, and can be specified for individual users and/or groups. Before a file is written to a partition where quotas are set, the quota of the creator's group is checked. If a quota exists, then the file size counts towards the group's quota. If no quota exists, then the owner's user quota is checked before the file is written. Similarly, inode usage for specific functions can be controlled if a user over-uses the allocated space.</para>
+ <para>Lustre quota enforcement differs from standard Linux quota enforcement in several ways:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290122" xreflabel=""/> Quotas are administered via the lfs command (post-mount).</para>
+ <para> Quotas are administered via the lfs command (post-mount).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290123" xreflabel=""/> Quotas are distributed (as Lustre is a distributed file system), which has several ramifications.</para>
+ <para> Quotas are distributed (as Lustre is a distributed file system), which has several ramifications.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290124" xreflabel=""/> Quotas are allocated and consumed in a quantized fashion.</para>
+ <para> Quotas are allocated and consumed in a quantized fashion.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290125" xreflabel=""/> Client does not set the usrquota or grpquota options to mount. When quota is enabled, it is enabled for all clients of the file system; started automatically using quota_type or started manually with lfs quotaon.</para>
+ <para> Client does not set the usrquota or grpquota options to mount. When quota is enabled, it is enabled for all clients of the file system; started automatically using quota_type or started manually with lfs quotaon.</para>
</listitem>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438217_31982">
<title>21.2 Enabling <anchor xml:id="dbdoclet.50438217_marker-1290128" xreflabel=""/>Disk Quotas</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290130" xreflabel=""/>Use this procedure to enable (configure) disk quotas in Lustre.</para>
+ <para>Use this procedure to enable (configure) disk quotas in Lustre.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290131" xreflabel=""/> 1. If you have re-complied your Linux kernel, be sure that CONFIG_QUOTA and CONFIG_QUOTACTL are enabled. Also, verify that CONFIG_QFMT_V1 and/or CONFIG_QFMT_V2 are enabled.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290132" xreflabel=""/>Quota is enabled in all Linux 2.6 kernels supplied for Lustre.</para>
+ <para> 1. If you have re-complied your Linux kernel, be sure that CONFIG_QUOTA and CONFIG_QUOTACTL are enabled. Also, verify that CONFIG_QFMT_V1 and/or CONFIG_QFMT_V2 are enabled.</para>
+ <para>Quota is enabled in all Linux 2.6 kernels supplied for Lustre.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290133" xreflabel=""/> 2. Start the server.</para>
+ <para> 2. Start the server.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290134" xreflabel=""/> 3. Mount the Lustre file system on the client and verify that the lquota module has loaded properly by using the lsmod command.</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290135" xreflabel=""/>$ lsmod
-<anchor xml:id="dbdoclet.50438217_pgfId-1290136" xreflabel=""/>[root@oss161 ~]# lsmod
-<anchor xml:id="dbdoclet.50438217_pgfId-1290137" xreflabel=""/>Module Size Used by
-<anchor xml:id="dbdoclet.50438217_pgfId-1290138" xreflabel=""/>obdfilter 220532 1
-<anchor xml:id="dbdoclet.50438217_pgfId-1290139" xreflabel=""/>fsfilt_ldiskfs 52228 1
-<anchor xml:id="dbdoclet.50438217_pgfId-1290140" xreflabel=""/>ost 96712 1
-<anchor xml:id="dbdoclet.50438217_pgfId-1290141" xreflabel=""/>mgc 60384 1
-<anchor xml:id="dbdoclet.50438217_pgfId-1290142" xreflabel=""/>ldiskfs 186896 2 fsfilt_ldiskfs
-<anchor xml:id="dbdoclet.50438217_pgfId-1290143" xreflabel=""/>lustre 401744 0
-<anchor xml:id="dbdoclet.50438217_pgfId-1290144" xreflabel=""/>lov 289064 1 lustre
-<anchor xml:id="dbdoclet.50438217_pgfId-1290145" xreflabel=""/>lquota 107048 4 obdfilter
-<anchor xml:id="dbdoclet.50438217_pgfId-1290146" xreflabel=""/>mdc 95016 1 lustre
-<anchor xml:id="dbdoclet.50438217_pgfId-1290147" xreflabel=""/>ksocklnd 111812 1
+ <para> 3. Mount the Lustre file system on the client and verify that the lquota module has loaded properly by using the lsmod command.</para>
+ <screen>$ lsmod
+[root@oss161 ~]# lsmod
+Module Size Used by
+obdfilter 220532 1
+fsfilt_ldiskfs 52228 1
+ost 96712 1
+mgc 60384 1
+ldiskfs 186896 2 fsfilt_ldiskfs
+lustre 401744 0
+lov 289064 1 lustre
+lquota 107048 4 obdfilter
+mdc 95016 1 lustre
+ksocklnd 111812 1
</screen>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290148" xreflabel=""/>The Lustre mount command no longer recognizes the usrquota and grpquota options. If they were previously specified, remove them from /etc/fstab.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290149" xreflabel=""/>When quota is enabled, it is enabled for all file system clients (started automatically using quota_type or manually with lfs quotaon).</para>
+ <para>The Lustre mount command no longer recognizes the usrquota and grpquota options. If they were previously specified, remove them from /etc/fstab.</para>
+ <para>When quota is enabled, it is enabled for all file system clients (started automatically using quota_type or manually with lfs quotaon).</para>
<note><para>Lustre with the Linux kernel 2.4 does <emphasis>not</emphasis> support quotas.</para></note>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290151" xreflabel=""/>To enable quotas automatically when the file system is started, you must set the mdt.quota_type and ost.quota_type parameters, respectively, on the MDT and OSTs. The parameters can be set to the string u (user), g (group) or ug for both users and groups.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290152" xreflabel=""/>You can enable quotas at mkfs time (mkfs.lustre --param mdt.quota_type=ug) or with tunefs.lustre. As an example:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290153" xreflabel=""/>tunefs.lustre --param ost.quota_type=ug $ost_dev
+ <para>To enable quotas automatically when the file system is started, you must set the mdt.quota_type and ost.quota_type parameters, respectively, on the MDT and OSTs. The parameters can be set to the string u (user), g (group) or ug for both users and groups.</para>
+ <para>You can enable quotas at mkfs time (mkfs.lustre --param mdt.quota_type=ug) or with tunefs.lustre. As an example:</para>
+ <screen>tunefs.lustre --param ost.quota_type=ug $ost_dev
</screen>
<caution><para>If you are using mkfs.lustre --param mdt.quota_type=ug or tunefs.lustre --param ost.quota_type=ug, be sure to run the command on all OSTs and the MDT. Otherwise, abnormal results may occur.</para></caution>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438217_pgfId-1290155" xreflabel=""/>21.2.0.1 Administrative and Operational Quotas</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290156" xreflabel=""/>Lustre has two kinds of quota files:</para>
+ <title>21.2.0.1 Administrative and Operational Quotas</title>
+ <para>Lustre has two kinds of quota files:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290157" xreflabel=""/> Administrative quotas (for the MDT), which contain limits for users/groups for the entire cluster.</para>
+ <para> Administrative quotas (for the MDT), which contain limits for users/groups for the entire cluster.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290158" xreflabel=""/> Operational quotas (for the MDT and OSTs), which contain quota information dedicated to a cluster node.</para>
+ <para> Operational quotas (for the MDT and OSTs), which contain quota information dedicated to a cluster node.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290159" xreflabel=""/>Lustre 1.6.5 introduced the v2 file format for administrative quota files, with continued support for the old file format (v1). The mdt.quota_type parameter also handles '1' and '2' options, to specify the Lustre quota versions that will be used. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290160" xreflabel=""/>--param mdt.quota_type=ug1
-<anchor xml:id="dbdoclet.50438217_pgfId-1290161" xreflabel=""/>--param mdt.quota_type=u2
+ <para>Lustre 1.6.5 introduced the v2 file format for administrative quota files, with continued support for the old file format (v1). The mdt.quota_type parameter also handles '1' and '2' options, to specify the Lustre quota versions that will be used. For example:</para>
+ <screen>--param mdt.quota_type=ug1
+--param mdt.quota_type=u2
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290162" xreflabel=""/>Lustre 1.6.6 introduced the v2 file format for operational quotas, with continued support for the old file format (v1). The ost.quota_type parameter handles '1' and '2' options, to specify the Lustre quota versions that will be used. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290163" xreflabel=""/>--param ost.quota_type=ug2
-<anchor xml:id="dbdoclet.50438217_pgfId-1290164" xreflabel=""/>--param ost.quota_type=u1
+ <para>Lustre 1.6.6 introduced the v2 file format for operational quotas, with continued support for the old file format (v1). The ost.quota_type parameter handles '1' and '2' options, to specify the Lustre quota versions that will be used. For example:</para>
+ <screen>--param ost.quota_type=ug2
+--param ost.quota_type=u1
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290168" xreflabel=""/>For more information about the v1 and v2 formats, see <link xl:href="ConfiguringQuotas.html#50438217_66360">Quota File Formats</link>.</para>
+ <para>For more information about the v1 and v2 formats, see <link xl:href="ConfiguringQuotas.html#50438217_66360">Quota File Formats</link>.</para>
</section>
</section>
<section xml:id="dbdoclet.50438217_49939">
<title>21.3 Creating Quota <anchor xml:id="dbdoclet.50438217_marker-1290170" xreflabel=""/>Files and Quota Administration</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290172" xreflabel=""/>Once each quota-enabled file system is remounted, it is capable of working with disk quotas. However, the file system is not yet ready to support quotas. If umount has been done regularly, run the lfs command with the quotaon option. If umount has not been done, perform these steps:</para>
+ <para>Once each quota-enabled file system is remounted, it is capable of working with disk quotas. However, the file system is not yet ready to support quotas. If umount has been done regularly, run the lfs command with the quotaon option. If umount has not been done, perform these steps:</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290173" xreflabel=""/> 1. Take Lustre ''offline''.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1291329" xreflabel=""/>That is, verify that no write operations (append, write, truncate, create or delete) are being performed (preparing to run lfs quotacheck). Operations that do not change Lustre files (such as read or mount) are okay to run.</para>
+ <para> 1. Take Lustre ''offline''.</para>
+ <para>That is, verify that no write operations (append, write, truncate, create or delete) are being performed (preparing to run lfs quotacheck). Operations that do not change Lustre files (such as read or mount) are okay to run.</para>
<caution><para>When lfsquotacheck is run, Lustre must NOT be performing any write operations. Failure to follow this caution may cause the statistic information of quota to be inaccurate. For example, the number of blocks used by OSTs for users or groups will be inaccurate, which can cause unexpected quota problems.</para></caution>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290175" xreflabel=""/> 2. Run the <emphasis role="bold">lfs</emphasis> command with the <emphasis role="bold">quotacheck</emphasis> option:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290176" xreflabel=""/># lfs quotacheck -ug /mnt/lustre
+ <para> 2. Run the <emphasis role="bold">lfs</emphasis> command with the <emphasis role="bold">quotacheck</emphasis> option:</para>
+ <screen># lfs quotacheck -ug /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290177" xreflabel=""/>By default, quota is turned on after quotacheck completes. Available options are:</para>
+ <para>By default, quota is turned on after quotacheck completes. Available options are:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290178" xreflabel=""/>u -- checks the user disk quota information</para>
+ <para>u -- checks the user disk quota information</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290179" xreflabel=""/>g -- checks the group disk quota information</para>
+ <para>g -- checks the group disk quota information</para>
</listitem>
</itemizedlist>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290180" xreflabel=""/>The lfsquotacheck command checks all objects on all OSTs and the MDS to sum up for every UID/GID. It reads all Lustre metadata and re-computes the number of blocks/inodes that each UID/GID has used. If there are many files in Lustre, it may take a long time to complete.</para>
+ <para>The lfsquotacheck command checks all objects on all OSTs and the MDS to sum up for every UID/GID. It reads all Lustre metadata and re-computes the number of blocks/inodes that each UID/GID has used. If there are many files in Lustre, it may take a long time to complete.</para>
<note><para>User and group quotas are separate. If either quota limit is reached, a process with the corresponding UID/GID cannot allocate more space on the file system.</para></note>
<note><para>For Lustre 1.6 releases before version 1.6.5, and 1.4 releases before version 1.4.12, if the underlying ldiskfs file system has not unmounted gracefully (due to a crash, for example), re-run quotacheck to obtain accurate quota information. Lustre 1.6.5 and 1.4.12 use journaled quota, so it is not necessary to run quotacheck after an unclean shutdown.</para><para> In certain failure situations (e.g., when a broken Lustre installation or build is used), re-run quotacheck after checking the server kernel logs and fixing the root problem.</para></note>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290184" xreflabel=""/>The lfs command includes several command options to work with quotas:</para>
+ <para>The lfs command includes several command options to work with quotas:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290185" xreflabel=""/><varname>quotaon</varname> -- enables disk quotas on the specified file system. The file system quota files must be present in the root directory of the file system.</para>
+ <para><varname>quotaon</varname> -- enables disk quotas on the specified file system. The file system quota files must be present in the root directory of the file system.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290186" xreflabel=""/><varname>quotaoff</varname> -- disables disk quotas on the specified file system.</para>
+ <para><varname>quotaoff</varname> -- disables disk quotas on the specified file system.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290187" xreflabel=""/><varname>quota</varname> -- displays general quota information (disk usage and limits)</para>
+ <para><varname>quota</varname> -- displays general quota information (disk usage and limits)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290188" xreflabel=""/><varname>setquota</varname> -- specifies quota limits and tunes the grace period. By default, the grace period is one week.</para>
+ <para><varname>setquota</varname> -- specifies quota limits and tunes the grace period. By default, the grace period is one week.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290189" xreflabel=""/> Usage:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290190" xreflabel=""/>lfs quotaon [-ugf] <filesystem>
-<anchor xml:id="dbdoclet.50438217_pgfId-1290191" xreflabel=""/>lfs quotaoff [-ug] <filesystem>
-<anchor xml:id="dbdoclet.50438217_pgfId-1290192" xreflabel=""/>lfs quota [-q] [-v] [-o obd_uuid] [-u|-g <uname>|uid|gname|gid>] <filesystem>
-<anchor xml:id="dbdoclet.50438217_pgfId-1290193" xreflabel=""/>lfs quota -t <-u|-g> <filesystem>
-<anchor xml:id="dbdoclet.50438217_pgfId-1290194" xreflabel=""/>lfs setquota <-u|--user|-g|--group> <username|groupname> [-b <block-softlimit>] [\
+ <para> Usage:</para>
+ <screen>lfs quotaon [-ugf] <filesystem>
+lfs quotaoff [-ug] <filesystem>
+lfs quota [-q] [-v] [-o obd_uuid] [-u|-g <uname>|uid|gname|gid>] <filesystem>
+lfs quota -t <-u|-g> <filesystem>
+lfs setquota <-u|--user|-g|--group> <username|groupname> [-b <block-softlimit>] [\
-B <block-hardlimit>] [-i <inode-softlimit>] [-I <inode-hardlimit>] <filesystem>
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290195" xreflabel=""/>Examples:</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290196" xreflabel=""/>In all of the examples below, the file system is /mnt lustre.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290197" xreflabel=""/>To turn on user and group quotas, run:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290198" xreflabel=""/>$ lfs quotaon -ug /mnt/lustre
+ <para>Examples:</para>
+ <para>In all of the examples below, the file system is /mnt lustre.</para>
+ <para>To turn on user and group quotas, run:</para>
+ <screen>$ lfs quotaon -ug /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290199" xreflabel=""/>To turn off user and group quotas, run:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290200" xreflabel=""/>$ lfs quotaoff -ug /mnt/lustre
+ <para>To turn off user and group quotas, run:</para>
+ <screen>$ lfs quotaoff -ug /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290201" xreflabel=""/>To display general quota information (disk usage and limits) for the user running the command and his primary group, run:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290202" xreflabel=""/>$ lfs quota /mnt/lustre
+ <para>To display general quota information (disk usage and limits) for the user running the command and his primary group, run:</para>
+ <screen>$ lfs quota /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290203" xreflabel=""/>To display general quota information for a specific user ("bob" in this example), run:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290204" xreflabel=""/>$ lfs quota -u bob /mnt/lustre
+ <para>To display general quota information for a specific user ("bob" in this example), run:</para>
+ <screen>$ lfs quota -u bob /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290205" xreflabel=""/>To display general quota information for a specific user ("bob" in this example) and detailed quota statistics for each MDT and OST, run:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290206" xreflabel=""/>$ lfs quota -u bob -v /mnt/lustre
+ <para>To display general quota information for a specific user ("bob" in this example) and detailed quota statistics for each MDT and OST, run:</para>
+ <screen>$ lfs quota -u bob -v /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290207" xreflabel=""/>To display general quota information for a specific group ("eng" in this example), run:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290208" xreflabel=""/>$ lfs quota -g eng /mnt/lustre
+ <para>To display general quota information for a specific group ("eng" in this example), run:</para>
+ <screen>$ lfs quota -g eng /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290209" xreflabel=""/>To display block and inode grace times for user quotas, run:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290210" xreflabel=""/>$ lfs quota -t -u /mnt/lustre
+ <para>To display block and inode grace times for user quotas, run:</para>
+ <screen>$ lfs quota -t -u /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290211" xreflabel=""/>To set user and group quotas for a specific user ("bob" in this example), run:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290212" xreflabel=""/>$ lfs setquota -u bob 307200 309200 10000 11000 /mnt/lustre
+ <para>To set user and group quotas for a specific user ("bob" in this example), run:</para>
+ <screen>$ lfs setquota -u bob 307200 309200 10000 11000 /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290213" xreflabel=""/>In this example, the quota for user "bob" is set to 300 MB (309200*1024) and the hard limit is 11,000 files. Therefore, the inode hard limit should be 11000.</para>
+ <para>In this example, the quota for user "bob" is set to 300 MB (309200*1024) and the hard limit is 11,000 files. Therefore, the inode hard limit should be 11000.</para>
<note><para>For the Lustre command $lfssetquota/quota ... the qunit for block is KB (1024) and the qunit for inode is 1.</para></note>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290215" xreflabel=""/>The quota command displays the quota allocated and consumed for each Lustre device. Using the previous setquota example, running this lfs quota command:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290216" xreflabel=""/>$ lfs quota -u bob -v /mnt/lustre
+ <para>The quota command displays the quota allocated and consumed for each Lustre device. Using the previous setquota example, running this lfs quota command:</para>
+ <screen>$ lfs quota -u bob -v /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290217" xreflabel=""/>displays this command output:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290218" xreflabel=""/>Disk quotas for user bob (uid 6000):
-<anchor xml:id="dbdoclet.50438217_pgfId-1290219" xreflabel=""/>Filesystem kbytes quota limit grace \
+ <para>displays this command output:</para>
+ <screen>Disk quotas for user bob (uid 6000):
+Filesystem kbytes quota limit grace \
files quota limit grace
-<anchor xml:id="dbdoclet.50438217_pgfId-1290220" xreflabel=""/>/mnt/lustre 0 30720 30920 \
+/mnt/lustre 0 30720 30920 \
- 0 10000 11000 -
-<anchor xml:id="dbdoclet.50438217_pgfId-1290221" xreflabel=""/>lustre-MDT0000_UUID 0 - 16384 \
+lustre-MDT0000_UUID 0 - 16384 \
- 0 - 2560 -
-<anchor xml:id="dbdoclet.50438217_pgfId-1290222" xreflabel=""/>lustre-OST0000_UUID 0 - 16384 \
+lustre-OST0000_UUID 0 - 16384 \
- 0 - 0 -
-<anchor xml:id="dbdoclet.50438217_pgfId-1290223" xreflabel=""/>lustre-OST0001_UUID 0 - 16384 \
+lustre-OST0001_UUID 0 - 16384 \
- 0 - 0 -
</screen>
</section>
<section xml:id="dbdoclet.50438217_15106">
<title>21.4 Quota<anchor xml:id="dbdoclet.50438217_marker-1290226" xreflabel=""/> Allocation</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290228" xreflabel=""/>In Lustre, quota must be properly allocated or users may experience unnecessary failures. The file system block quota is divided up among the OSTs within the file system. Each OST requests an allocation which is increased up to the quota limit. The quota allocation is then quantized to reduce the number of quota-related request traffic. By default, Lustre supports both user and group quotas to limit disk usage and file counts.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290229" xreflabel=""/>The quota system in Lustre is completely compatible with the quota systems used on other file systems. The Lustre quota system distributes quotas from the quota master. Generally, the MDS is the quota master for both inodes and blocks. All OSTs and the MDS are quota slaves to the OSS nodes. To reduce quota requests and get reasonably accurate quota distribution, the transfer quota unit (qunit) between quota master and quota slaves is changed dynamically by the lquota module. The default minimum value of qunit is 1 MB for blocks and 2 for inodes. The proc entries to set these values are: /proc/fs/lustre/mds/lustre-MDT*/quota_least_bunit and /proc/fs/lustre/mds/lustre-MDT*/quota_least_iunit. The default maximum value of qunit is 128 MB for blocks and 5120 for inodes. The proc entries to set these values are quota_bunit_sz and quota_iunit_sz in the MDT and OSTs.</para>
+ <para>In Lustre, quota must be properly allocated or users may experience unnecessary failures. The file system block quota is divided up among the OSTs within the file system. Each OST requests an allocation which is increased up to the quota limit. The quota allocation is then quantized to reduce the number of quota-related request traffic. By default, Lustre supports both user and group quotas to limit disk usage and file counts.</para>
+ <para>The quota system in Lustre is completely compatible with the quota systems used on other file systems. The Lustre quota system distributes quotas from the quota master. Generally, the MDS is the quota master for both inodes and blocks. All OSTs and the MDS are quota slaves to the OSS nodes. To reduce quota requests and get reasonably accurate quota distribution, the transfer quota unit (qunit) between quota master and quota slaves is changed dynamically by the lquota module. The default minimum value of qunit is 1 MB for blocks and 2 for inodes. The proc entries to set these values are: /proc/fs/lustre/mds/lustre-MDT*/quota_least_bunit and /proc/fs/lustre/mds/lustre-MDT*/quota_least_iunit. The default maximum value of qunit is 128 MB for blocks and 5120 for inodes. The proc entries to set these values are quota_bunit_sz and quota_iunit_sz in the MDT and OSTs.</para>
<note><para>In general, the quota_bunit_sz value should be larger than 1 MB. For testing purposes, it can be set to 4 KB, if necessary.</para></note>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290231" xreflabel=""/>The file system block quota is divided up among the OSTs and the MDS within the file system. Only the MDS uses the file system inode quota.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290232" xreflabel=""/>This means that the minimum quota for block is 1 MB* (the number of OSTs + the number of MDSs), which is 1 MB* (number of OSTs + 1). If you attempt to assign a smaller quota, users maybe not be able to create files. As noted, the default minimum quota for inodes is 2. The default is established at file system creation time, but can be tuned via /proc values (described below). The inode quota is also allocated in a quantized manner on the MDS.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290233" xreflabel=""/>If we look at the setquota example again, running this lfsquota command:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290234" xreflabel=""/># lfs quota -u bob -v /mnt/lustre
+ <para>The file system block quota is divided up among the OSTs and the MDS within the file system. Only the MDS uses the file system inode quota.</para>
+ <para>This means that the minimum quota for block is 1 MB* (the number of OSTs + the number of MDSs), which is 1 MB* (number of OSTs + 1). If you attempt to assign a smaller quota, users maybe not be able to create files. As noted, the default minimum quota for inodes is 2. The default is established at file system creation time, but can be tuned via /proc values (described below). The inode quota is also allocated in a quantized manner on the MDS.</para>
+ <para>If we look at the setquota example again, running this lfsquota command:</para>
+ <screen># lfs quota -u bob -v /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290235" xreflabel=""/>displays this command output:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290236" xreflabel=""/>Disk quotas for user bob (uid 500):
-<anchor xml:id="dbdoclet.50438217_pgfId-1290237" xreflabel=""/>Filesystem kbytes quota limit grace \
+ <para>displays this command output:</para>
+ <screen>Disk quotas for user bob (uid 500):
+Filesystem kbytes quota limit grace \
files quota limit grace
-<anchor xml:id="dbdoclet.50438217_pgfId-1290238" xreflabel=""/>/mnt/lustre 30720* 30720 30920 \
+/mnt/lustre 30720* 30720 30920 \
6d23h56m44s 10101* 10000 11000 6d23h59m50s
-<anchor xml:id="dbdoclet.50438217_pgfId-1290239" xreflabel=""/>lustre-MDT0000_UUID 0 - 1024 \
+lustre-MDT0000_UUID 0 - 1024 \
- 10101 - 10240
-<anchor xml:id="dbdoclet.50438217_pgfId-1290240" xreflabel=""/>lustre-OST0000_UUID 0 - 1024 \
+lustre-OST0000_UUID 0 - 1024 \
- - - -
-<anchor xml:id="dbdoclet.50438217_pgfId-1290241" xreflabel=""/>lustre-OST0001_UUID 30720* - 28872 \
+lustre-OST0001_UUID 30720* - 28872 \
- - - -
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290242" xreflabel=""/>The total quota limit of 30,920 is allotted to user bob, which is further distributed to two OSTs and one MDS.</para>
+ <para>The total quota limit of 30,920 is allotted to user bob, which is further distributed to two OSTs and one MDS.</para>
<note><para>Values appended with '*' show the limit that has been over-used (exceeding the quota), and receives this message Disk quota exceeded. For example:</para><para> \</para><para>$ cp: writing `/mnt/lustre/var/cache/fontconfig/ beeeeb3dfe132a8a0633a017c99ce0-x86.cache': Disk quota exceeded.</para></note>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290244" xreflabel=""/>The requested quota of 300 MB is divided across the OSTs.</para>
+ <para>The requested quota of 300 MB is divided across the OSTs.</para>
<note><para>It is very important to note that the block quota is consumed per OST and the MDS per block and inode (there is only one MDS for inodes). Therefore, when the quota is consumed on one OST, the client may not be able to create files regardless of the quota available on other OSTs.</para></note>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438217_pgfId-1290246" xreflabel=""/>Additional information:</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290247" xreflabel=""/><emphasis role="bold">Grace period</emphasis> -- The period of time (in seconds) within which users are allowed to exceed their soft limit. There are four types of grace periods:</para>
+ <title>Additional information:</title>
+ <para><emphasis role="bold">Grace period</emphasis> -- The period of time (in seconds) within which users are allowed to exceed their soft limit. There are four types of grace periods:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290248" xreflabel=""/> user block soft limit</para>
+ <para> user block soft limit</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290249" xreflabel=""/> user inode soft limit</para>
+ <para> user inode soft limit</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290250" xreflabel=""/> group block soft limit</para>
+ <para> group block soft limit</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290251" xreflabel=""/> group inode soft limit</para>
+ <para> group inode soft limit</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290252" xreflabel=""/>The grace periods are applied to all users. The user block soft limit is for all users who are using a blocks quota.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290253" xreflabel=""/><emphasis role="bold">Soft limit</emphasis> -- Once you are beyond the soft limit, the quota module begins to time, but you still can write block and inode. When you are always beyond the soft limit and use up your grace time, you get the same result as the hard limit. For inodes and blocks, it is the same. Usually, the soft limit MUST be less than the hard limit; if not, the quota module never triggers the timing. If the soft limit is not needed, leave it as zero (0).</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290254" xreflabel=""/><emphasis role="bold">Hard limit</emphasis> -- When you are beyond the hard limit, you get -EQUOTA and cannot write inode/block any more. The hard limit is the absolute limit. When a grace period is set, you can exceed the soft limit within the grace period if are under the hard limits.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290255" xreflabel=""/>Lustre quota allocation is controlled by two variables, quota_bunit_sz and quota_iunit_sz referring to KBs and inodes, respectively. These values can be accessed on the MDS as /proc/fs/lustre/mds/*/quota_* and on the OST as /proc/fs/lustre/obdfilter/*/quota_*. The quota_bunit_sz and quota_iunit_sz variables are the maximum qunit values for blocks and inodes, respectively. At any time, module lquota chooses a reasonable qunit between the minimum and maximum values.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290256" xreflabel=""/>The /proc values are bounded by two other variables quota_btune_sz and quota_itune_sz. By default, the *tune_sz variables are set at 1/2 the *unit_sz variables, and you cannot set *tune_sz larger than *unit_sz. You must set bunit_sz first if it is increasing by more than 2x, and btune_sz first if it is decreasing by more than 2x.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290257" xreflabel=""/><emphasis role="bold">Total number of inodes</emphasis> -- To determine the total number of inodes, use lfsdf-i (and also /proc/fs/lustre/*/*/filestotal). For more information on using the lfsdf-i command and the command output, see <link xl:href="ManagingStripingFreeSpace.html#50438209_35838">Checking File System Free Space</link>.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290261" xreflabel=""/>Unfortunately, the statfs interface does not report the free inode count directly, but instead reports the total inode and used inode counts. The free inode count is calculated for df from (total inodes - used inodes).</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290262" xreflabel=""/>It is not critical to know a file system's total inode count. Instead, you should know (accurately), the free inode count and the used inode count for a file system. Lustre manipulates the total inode count in order to accurately report the other two values.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290263" xreflabel=""/>The values set for the MDS must match the values set on the OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290264" xreflabel=""/>The quota_bunit_sz parameter displays bytes, however lfs setquota uses KBs. The quota_bunit_sz parameter must be a multiple of 1024. A proper minimum KB size for lfs setquota can be calculated as:</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290265" xreflabel=""/><emphasis role="bold">Size in KBs = minimum_quota_bunit_sz * (number of OSTS + 1) = 1024 * (number of OSTs +1)</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290266" xreflabel=""/>We add one (1) to the number of OSTs as the MDS also consumes KBs. As inodes are only consumed on the MDS, the minimum inode size for lfs setquota is equal to quota_iunit_sz.</para>
+ <para>The grace periods are applied to all users. The user block soft limit is for all users who are using a blocks quota.</para>
+ <para><emphasis role="bold">Soft limit</emphasis> -- Once you are beyond the soft limit, the quota module begins to time, but you still can write block and inode. When you are always beyond the soft limit and use up your grace time, you get the same result as the hard limit. For inodes and blocks, it is the same. Usually, the soft limit MUST be less than the hard limit; if not, the quota module never triggers the timing. If the soft limit is not needed, leave it as zero (0).</para>
+ <para><emphasis role="bold">Hard limit</emphasis> -- When you are beyond the hard limit, you get -EQUOTA and cannot write inode/block any more. The hard limit is the absolute limit. When a grace period is set, you can exceed the soft limit within the grace period if are under the hard limits.</para>
+ <para>Lustre quota allocation is controlled by two variables, quota_bunit_sz and quota_iunit_sz referring to KBs and inodes, respectively. These values can be accessed on the MDS as /proc/fs/lustre/mds/*/quota_* and on the OST as /proc/fs/lustre/obdfilter/*/quota_*. The quota_bunit_sz and quota_iunit_sz variables are the maximum qunit values for blocks and inodes, respectively. At any time, module lquota chooses a reasonable qunit between the minimum and maximum values.</para>
+ <para>The /proc values are bounded by two other variables quota_btune_sz and quota_itune_sz. By default, the *tune_sz variables are set at 1/2 the *unit_sz variables, and you cannot set *tune_sz larger than *unit_sz. You must set bunit_sz first if it is increasing by more than 2x, and btune_sz first if it is decreasing by more than 2x.</para>
+ <para><emphasis role="bold">Total number of inodes</emphasis> -- To determine the total number of inodes, use lfsdf-i (and also /proc/fs/lustre/*/*/filestotal). For more information on using the lfsdf-i command and the command output, see <link xl:href="ManagingStripingFreeSpace.html#50438209_35838">Checking File System Free Space</link>.</para>
+ <para>Unfortunately, the statfs interface does not report the free inode count directly, but instead reports the total inode and used inode counts. The free inode count is calculated for df from (total inodes - used inodes).</para>
+ <para>It is not critical to know a file system's total inode count. Instead, you should know (accurately), the free inode count and the used inode count for a file system. Lustre manipulates the total inode count in order to accurately report the other two values.</para>
+ <para>The values set for the MDS must match the values set on the OSTs.</para>
+ <para>The quota_bunit_sz parameter displays bytes, however lfs setquota uses KBs. The quota_bunit_sz parameter must be a multiple of 1024. A proper minimum KB size for lfs setquota can be calculated as:</para>
+ <para><emphasis role="bold">Size in KBs = minimum_quota_bunit_sz * (number of OSTS + 1) = 1024 * (number of OSTs +1)</emphasis></para>
+ <para>We add one (1) to the number of OSTs as the MDS also consumes KBs. As inodes are only consumed on the MDS, the minimum inode size for lfs setquota is equal to quota_iunit_sz.</para>
<note><para>Setting the quota below this limit may prevent the user from all file creation.</para></note>
</section>
</section>
<section xml:id="dbdoclet.50438217_27895">
<title>21.5 Known Issues <anchor xml:id="dbdoclet.50438217_marker-1290269" xreflabel=""/>with Quotas</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290271" xreflabel=""/>Using quotas in Lustre can be complex and there are several known issues.</para>
+ <para>Using quotas in Lustre can be complex and there are several known issues.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438217_pgfId-1290273" xreflabel=""/>21.5.1 Granted<anchor xml:id="dbdoclet.50438217_marker-1290272" xreflabel=""/> Cache and Quota Limits</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290274" xreflabel=""/>In Lustre, granted cache does not respect quota limits. In this situation, OSTs grant cache to Lustre client to accelerate I/O. Granting cache causes writes to be successful in OSTs, even if they exceed the quota limits, and will overwrite them.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290275" xreflabel=""/>The sequence is:</para>
+ <title>21.5.1 Granted<anchor xml:id="dbdoclet.50438217_marker-1290272" xreflabel=""/> Cache and Quota Limits</title>
+ <para>In Lustre, granted cache does not respect quota limits. In this situation, OSTs grant cache to Lustre client to accelerate I/O. Granting cache causes writes to be successful in OSTs, even if they exceed the quota limits, and will overwrite them.</para>
+ <para>The sequence is:</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290276" xreflabel=""/>A user writes files to Lustre.</para>
+ <para>A user writes files to Lustre.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290277" xreflabel=""/>If the Lustre client has enough granted cache, then it returns 'success' to users and arranges the writes to the OSTs.</para>
+ <para>If the Lustre client has enough granted cache, then it returns 'success' to users and arranges the writes to the OSTs.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290278" xreflabel=""/>Because Lustre clients have delivered success to users, the OSTs cannot fail these writes.</para>
+ <para>Because Lustre clients have delivered success to users, the OSTs cannot fail these writes.</para>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290279" xreflabel=""/>Because of granted cache, writes always overwrite quota limitations. For example, if you set a 400 GB quota on user A and use IOR to write for user A from a bundle of clients, you will write much more data than 400 GB, and cause an out-of-quota error (-EDQUOT).</para>
+ <para>Because of granted cache, writes always overwrite quota limitations. For example, if you set a 400 GB quota on user A and use IOR to write for user A from a bundle of clients, you will write much more data than 400 GB, and cause an out-of-quota error (-EDQUOT).</para>
<note><para>The effect of granted cache on quota limits can be mitigated, but not eradicated. Reduce the max_dirty_buffer in the clients (can be set from 0 to 512). To set max_dirty_buffer to 0:</para><para> * In releases after Lustre 1.6.5, lctl set_param osc.*.max_dirty_mb=0.</para><para> * In releases before Lustre 1.6.5, proc/fs/lustre/osc/*/max_dirty_mb; do echo 512 > $O</para></note>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438217_pgfId-1290283" xreflabel=""/>21.5.2 <anchor xml:id="dbdoclet.50438217_50442" xreflabel=""/>Quota <anchor xml:id="dbdoclet.50438217_marker-1290282" xreflabel=""/>Limits</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290284" xreflabel=""/>Available quota limits depend on the Lustre version you are using.</para>
+ <title>21.5.2 <anchor xml:id="dbdoclet.50438217_50442" xreflabel=""/>Quota <anchor xml:id="dbdoclet.50438217_marker-1290282" xreflabel=""/>Limits</title>
+ <para>Available quota limits depend on the Lustre version you are using.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290285" xreflabel=""/> Lustre version 1.4.11 and earlier (for 1.4.x releases) and Lustre version 1.6.4 and earlier (for 1.6.x releases) support quota limits less than 4 TB.</para>
+ <para> Lustre version 1.4.11 and earlier (for 1.4.x releases) and Lustre version 1.6.4 and earlier (for 1.6.x releases) support quota limits less than 4 TB.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290286" xreflabel=""/> Lustre versions 1.4.12, 1.6.5 and later support quota limits of 4 TB and greater in Lustre configurations with OST storage limits of 4 TB and less.</para>
+ <para> Lustre versions 1.4.12, 1.6.5 and later support quota limits of 4 TB and greater in Lustre configurations with OST storage limits of 4 TB and less.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290287" xreflabel=""/> Future Lustre versions are expected to support quota limits of 4 TB and greater with no OST storage limits.</para>
+ <para> Future Lustre versions are expected to support quota limits of 4 TB and greater with no OST storage limits.</para>
</listitem>
<listitem>
<informaltable frame="all">
<colspec colname="c3" colwidth="33*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438217_pgfId-1290290" xreflabel=""/>Lustre Version</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438217_pgfId-1290292" xreflabel=""/>Quota Limit Per User/Per Group</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438217_pgfId-1290294" xreflabel=""/>OST Storage Limit</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Lustre Version</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Quota Limit Per User/Per Group</emphasis></para></entry>
+ <entry><para><emphasis role="bold">OST Storage Limit</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290296" xreflabel=""/>1.4.11 and earlier</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290298" xreflabel=""/>< 4TB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290300" xreflabel=""/>n/a</para></entry>
+ <entry><para> 1.4.11 and earlier</para></entry>
+ <entry><para> < 4TB</para></entry>
+ <entry><para> n/a</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290302" xreflabel=""/>1.4.12</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290304" xreflabel=""/>=> 4TB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290306" xreflabel=""/><= 4TB of storage</para></entry>
+ <entry><para> 1.4.12</para></entry>
+ <entry><para> => 4TB</para></entry>
+ <entry><para> <= 4TB of storage</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290308" xreflabel=""/>1.6.4 and earlier</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290310" xreflabel=""/>< 4TB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290312" xreflabel=""/>n/a</para></entry>
+ <entry><para> 1.6.4 and earlier</para></entry>
+ <entry><para> < 4TB</para></entry>
+ <entry><para> n/a</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290314" xreflabel=""/>1.6.5</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290316" xreflabel=""/>=> 4TB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290318" xreflabel=""/><= 4TB of storage</para></entry>
+ <entry><para> 1.6.5</para></entry>
+ <entry><para> => 4TB</para></entry>
+ <entry><para> <= 4TB of storage</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290320" xreflabel=""/>Future Lustre versions</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290322" xreflabel=""/>=> 4TB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290324" xreflabel=""/>No storage limit</para></entry>
+ <entry><para> Future Lustre versions</para></entry>
+ <entry><para> => 4TB</para></entry>
+ <entry><para> No storage limit</para></entry>
</row>
</tbody>
</tgroup>
</itemizedlist>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438217_pgfId-1290327" xreflabel=""/>21.5.3 <anchor xml:id="dbdoclet.50438217_66360" xreflabel=""/>Quota <anchor xml:id="dbdoclet.50438217_marker-1290326" xreflabel=""/>File Formats</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290328" xreflabel=""/>Lustre 1.6.5 introduced the v2 file format for administrative quotas, with 64-bit limits that support large-limits handling. The old quota file format (v1), with 32-bit limits, is also supported. Lustre 1.6.6 introduced the v2 file format for operational quotas. A few notes regarding the current quota file formats:</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290329" xreflabel=""/>Lustre 1.6.5 and later use mdt.quota_type to force a specific administrative quota version (v2 or v1).</para>
+ <title>21.5.3 <anchor xml:id="dbdoclet.50438217_66360" xreflabel=""/>Quota <anchor xml:id="dbdoclet.50438217_marker-1290326" xreflabel=""/>File Formats</title>
+ <para>Lustre 1.6.5 introduced the v2 file format for administrative quotas, with 64-bit limits that support large-limits handling. The old quota file format (v1), with 32-bit limits, is also supported. Lustre 1.6.6 introduced the v2 file format for operational quotas. A few notes regarding the current quota file formats:</para>
+ <para>Lustre 1.6.5 and later use mdt.quota_type to force a specific administrative quota version (v2 or v1).</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290330" xreflabel=""/> For the v2 quota file format, (OBJECTS/admin_quotafile_v2.{usr,grp})</para>
+ <para> For the v2 quota file format, (OBJECTS/admin_quotafile_v2.{usr,grp})</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290331" xreflabel=""/> For the v1 quota file format, (OBJECTS/admin_quotafile.{usr,grp})</para>
+ <para> For the v1 quota file format, (OBJECTS/admin_quotafile.{usr,grp})</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290332" xreflabel=""/>Lustre 1.6.6 and later use ost.quota_type to force a specific operational quota version (v2 or v1).</para>
+ <para>Lustre 1.6.6 and later use ost.quota_type to force a specific operational quota version (v2 or v1).</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290333" xreflabel=""/> For the v2 quota file format, (lquota_v2.{user,group})</para>
+ <para> For the v2 quota file format, (lquota_v2.{user,group})</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290334" xreflabel=""/> For the v1 quota file format, (lquota.{user,group})</para>
+ <para> For the v1 quota file format, (lquota.{user,group})</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290335" xreflabel=""/>The quota_type specifier can be used to set different combinations of administrative/operational quota file versions on a Lustre node:</para>
+ <para>The quota_type specifier can be used to set different combinations of administrative/operational quota file versions on a Lustre node:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290336" xreflabel=""/> "1" - v1 (32-bit) administrative quota file, v1 (32-bit) operational quota file (default in releases before Lustre 1.6.5)</para>
+ <para> "1" - v1 (32-bit) administrative quota file, v1 (32-bit) operational quota file (default in releases before Lustre 1.6.5)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290337" xreflabel=""/> "2" - v2 (64-bit) administrative quota file, v1 (32-bit) operational quota file (default in Lustre 1.6.5)</para>
+ <para> "2" - v2 (64-bit) administrative quota file, v1 (32-bit) operational quota file (default in Lustre 1.6.5)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290338" xreflabel=""/> "3" - v2 (64-bit) administrative quota file, v2 (64-bit) operational quota file (default in releases after Lustre 1.6.5)</para>
+ <para> "3" - v2 (64-bit) administrative quota file, v2 (64-bit) operational quota file (default in releases after Lustre 1.6.5)</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290339" xreflabel=""/>If quotas do not exist or look broken, then quotacheck creates quota files of a required name and format.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290340" xreflabel=""/>If Lustre is using the v2 quota file format when only v1 quota files exist, then quotacheck converts old v1 quota files to new v2 quota files. This conversion is triggered automatically, and is transparent to users. If an old quota file does not exist or looks broken, then the new v2 quota file will be empty. In case of an error, details can be found in the kernel log of the corresponding MDS/OST. During conversion of a v1 quota file to a v2 quota file, the v2 quota file is marked as broken, to avoid it being used if a crash occurs. The quota module does not use broken quota files (keeping quota off).</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290341" xreflabel=""/>In most situations, Lustre administrators do not need to set specific versioning options. Upgrading Lustre without using quota_type to force specific quota file versions results in quota files being upgraded automatically to the latest version. The option ensures backward compatibility, preventing a quota file upgrade to a version which is not supported by earlier Lustre versions.</para>
+ <para>If quotas do not exist or look broken, then quotacheck creates quota files of a required name and format.</para>
+ <para>If Lustre is using the v2 quota file format when only v1 quota files exist, then quotacheck converts old v1 quota files to new v2 quota files. This conversion is triggered automatically, and is transparent to users. If an old quota file does not exist or looks broken, then the new v2 quota file will be empty. In case of an error, details can be found in the kernel log of the corresponding MDS/OST. During conversion of a v1 quota file to a v2 quota file, the v2 quota file is marked as broken, to avoid it being used if a crash occurs. The quota module does not use broken quota files (keeping quota off).</para>
+ <para>In most situations, Lustre administrators do not need to set specific versioning options. Upgrading Lustre without using quota_type to force specific quota file versions results in quota files being upgraded automatically to the latest version. The option ensures backward compatibility, preventing a quota file upgrade to a version which is not supported by earlier Lustre versions.</para>
</section>
</section>
<section xml:id="dbdoclet.50438217_20772">
<title>21.6 Lustre <anchor xml:id="dbdoclet.50438217_marker-1290343" xreflabel=""/>Quota Statistics</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290345" xreflabel=""/>Lustre includes statistics that monitor quota activity, such as the kinds of quota RPCs sent during a specific period, the average time to complete the RPCs, etc. These statistics are useful to measure performance of a Lustre file system.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290346" xreflabel=""/>Each quota statistic consists of a quota event and min_time, max_time and sum_time values for the event.</para>
+ <para>Lustre includes statistics that monitor quota activity, such as the kinds of quota RPCs sent during a specific period, the average time to complete the RPCs, etc. These statistics are useful to measure performance of a Lustre file system.</para>
+ <para>Each quota statistic consists of a quota event and min_time, max_time and sum_time values for the event.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438217_pgfId-1290349" xreflabel=""/>Quota Event</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438217_pgfId-1290351" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Quota Event</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290353" xreflabel=""/><emphasis role="bold">sync_acq_req</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290355" xreflabel=""/>Quota slaves send a acquiring_quota request and wait for its return.</para></entry>
+ <entry><para> <emphasis role="bold">sync_acq_req</emphasis></para></entry>
+ <entry><para> Quota slaves send a acquiring_quota request and wait for its return.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290357" xreflabel=""/><emphasis role="bold">sync_rel_req</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290359" xreflabel=""/>Quota slaves send a releasing_quota request and wait for its return.</para></entry>
+ <entry><para> <emphasis role="bold">sync_rel_req</emphasis></para></entry>
+ <entry><para> Quota slaves send a releasing_quota request and wait for its return.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290361" xreflabel=""/><emphasis role="bold">async_acq_req</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290363" xreflabel=""/>Quota slaves send an acquiring_quota request and do not wait for its return.</para></entry>
+ <entry><para> <emphasis role="bold">async_acq_req</emphasis></para></entry>
+ <entry><para> Quota slaves send an acquiring_quota request and do not wait for its return.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290365" xreflabel=""/><emphasis role="bold">async_rel_req</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290367" xreflabel=""/>Quota slaves send a releasing_quota request and do not wait for its return.</para></entry>
+ <entry><para> <emphasis role="bold">async_rel_req</emphasis></para></entry>
+ <entry><para> Quota slaves send a releasing_quota request and do not wait for its return.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290369" xreflabel=""/><emphasis role="bold">wait_for_blk_quota (lquota_chkquota)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290371" xreflabel=""/>Before data is written to OSTs, the OSTs check if the remaining block quota is sufficient. This is done in the lquota_chkquota function.</para></entry>
+ <entry><para> <emphasis role="bold">wait_for_blk_quota (lquota_chkquota)</emphasis></para></entry>
+ <entry><para> Before data is written to OSTs, the OSTs check if the remaining block quota is sufficient. This is done in the lquota_chkquota function.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290373" xreflabel=""/><emphasis role="bold">wait_for_ino_quota (lquota_chkquota)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290375" xreflabel=""/>Before files are created on the MDS, the MDS checks if the remaining inode quota is sufficient. This is done in the lquota_chkquota function.</para></entry>
+ <entry><para> <emphasis role="bold">wait_for_ino_quota (lquota_chkquota)</emphasis></para></entry>
+ <entry><para> Before files are created on the MDS, the MDS checks if the remaining inode quota is sufficient. This is done in the lquota_chkquota function.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290377" xreflabel=""/><emphasis role="bold">wait_for_blk_quota (lquota_pending_commit)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290379" xreflabel=""/>After blocks are written to OSTs, relative quota information is updated. This is done in the lquota_pending_commit function.</para></entry>
+ <entry><para> <emphasis role="bold">wait_for_blk_quota (lquota_pending_commit)</emphasis></para></entry>
+ <entry><para> After blocks are written to OSTs, relative quota information is updated. This is done in the lquota_pending_commit function.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290381" xreflabel=""/><emphasis role="bold">wait_for_ino_quota (lquota_pending_commit)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290383" xreflabel=""/>After files are created, relative quota information is updated. This is done in the lquota_pending_commit function.</para></entry>
+ <entry><para> <emphasis role="bold">wait_for_ino_quota (lquota_pending_commit)</emphasis></para></entry>
+ <entry><para> After files are created, relative quota information is updated. This is done in the lquota_pending_commit function.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290385" xreflabel=""/><emphasis role="bold">wait_for_pending_blk_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290387" xreflabel=""/>On the MDS or OSTs, there is one thread sending a quota request for a specific UID/GID for block quota at any time. At that time, if other threads need to do this too, they should wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
+ <entry><para> <emphasis role="bold">wait_for_pending_blk_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
+ <entry><para> On the MDS or OSTs, there is one thread sending a quota request for a specific UID/GID for block quota at any time. At that time, if other threads need to do this too, they should wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290389" xreflabel=""/><emphasis role="bold">wait_for_pending_ino_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290391" xreflabel=""/>On the MDS, there is one thread sending a quota request for a specific UID/GID for inode quota at any time. If other threads need to do this too, they should wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
+ <entry><para> <emphasis role="bold">wait_for_pending_ino_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
+ <entry><para> On the MDS, there is one thread sending a quota request for a specific UID/GID for inode quota at any time. If other threads need to do this too, they should wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290393" xreflabel=""/><emphasis role="bold">nowait_for_pending_blk_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290395" xreflabel=""/>On the MDS or OSTs, there is one thread sending a quota request for a specific UID/GID for block quota at any time. When threads enter qctxt_wait_pending_dqacq, they do not need to wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
+ <entry><para> <emphasis role="bold">nowait_for_pending_blk_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
+ <entry><para> On the MDS or OSTs, there is one thread sending a quota request for a specific UID/GID for block quota at any time. When threads enter qctxt_wait_pending_dqacq, they do not need to wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290397" xreflabel=""/><emphasis role="bold">nowait_for_pending_ino_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290399" xreflabel=""/>On the MDS, there is one thread sending a quota request for a specific UID/GID for inode quota at any time. When threads enter qctxt_wait_pending_dqacq, they do not need to wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
+ <entry><para> <emphasis role="bold">nowait_for_pending_ino_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
+ <entry><para> On the MDS, there is one thread sending a quota request for a specific UID/GID for inode quota at any time. When threads enter qctxt_wait_pending_dqacq, they do not need to wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290401" xreflabel=""/><emphasis role="bold">quota_ctl</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290403" xreflabel=""/>The quota_ctl statistic is generated when lfs setquota, lfs quota and so on, are issued.</para></entry>
+ <entry><para> <emphasis role="bold">quota_ctl</emphasis></para></entry>
+ <entry><para> The quota_ctl statistic is generated when lfs setquota, lfs quota and so on, are issued.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290405" xreflabel=""/><emphasis role="bold">adjust_qunit</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290407" xreflabel=""/>Each time qunit is adjusted, it is counted.</para></entry>
+ <entry><para> <emphasis role="bold">adjust_qunit</emphasis></para></entry>
+ <entry><para> Each time qunit is adjusted, it is counted.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438217_pgfId-1290408" xreflabel=""/>21.6.1 Interpreting Quota Statistics</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290409" xreflabel=""/>Quota statistics are an important measure of a Lustre file system's performance. Interpreting these statistics correctly can help you diagnose problems with quotas, and may indicate adjustments to improve system performance.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290410" xreflabel=""/>For example, if you run this command on the OSTs:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290411" xreflabel=""/>cat /proc/fs/lustre/lquota/lustre-OST0000/stats
+ <title>21.6.1 Interpreting Quota Statistics</title>
+ <para>Quota statistics are an important measure of a Lustre file system's performance. Interpreting these statistics correctly can help you diagnose problems with quotas, and may indicate adjustments to improve system performance.</para>
+ <para>For example, if you run this command on the OSTs:</para>
+ <screen>cat /proc/fs/lustre/lquota/lustre-OST0000/stats
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290412" xreflabel=""/>You will get a result similar to this:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290413" xreflabel=""/>snapshot_time 1219908615.506895 secs.usecs
-<anchor xml:id="dbdoclet.50438217_pgfId-1290414" xreflabel=""/>async_acq_req 1 samples [us] 32 32 32
-<anchor xml:id="dbdoclet.50438217_pgfId-1290415" xreflabel=""/>async_rel_req 1 samples [us] 5 5 5
-<anchor xml:id="dbdoclet.50438217_pgfId-1290416" xreflabel=""/>nowait_for_pending_blk_quota_req(qctxt_wait_pending_dqacq) 1 samples [us] 2\
+ <para>You will get a result similar to this:</para>
+ <screen>snapshot_time 1219908615.506895 secs.usecs
+async_acq_req 1 samples [us] 32 32 32
+async_rel_req 1 samples [us] 5 5 5
+nowait_for_pending_blk_quota_req(qctxt_wait_pending_dqacq) 1 samples [us] 2\
2 2
-<anchor xml:id="dbdoclet.50438217_pgfId-1290417" xreflabel=""/>quota_ctl 4 samples [us] 80 3470 4293
-<anchor xml:id="dbdoclet.50438217_pgfId-1290418" xreflabel=""/>adjust_qunit 1 samples [us] 70 70 70
-<anchor xml:id="dbdoclet.50438217_pgfId-1290419" xreflabel=""/>....
+quota_ctl 4 samples [us] 80 3470 4293
+adjust_qunit 1 samples [us] 70 70 70
+....
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290420" xreflabel=""/>In the first line, snapshot_time indicates when the statistics were taken. The remaining lines list the quota events and their associated data.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290421" xreflabel=""/>In the second line, the async_acq_req event occurs one time. The min_time, max_time and sum_time statistics for this event are 32, 32 and 32, respectively. The unit is microseconds (s).</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290422" xreflabel=""/>In the fifth line, the quota_ctl event occurs four times. The min_time, max_time and sum_time statistics for this event are 80, 3470 and 4293, respectively. The unit is microseconds (s).</para>
+ <para>In the first line, snapshot_time indicates when the statistics were taken. The remaining lines list the quota events and their associated data.</para>
+ <para>In the second line, the async_acq_req event occurs one time. The min_time, max_time and sum_time statistics for this event are 32, 32 and 32, respectively. The unit is microseconds (s).</para>
+ <para>In the fifth line, the quota_ctl event occurs four times. The min_time, max_time and sum_time statistics for this event are 80, 3470 and 4293, respectively. The unit is microseconds (s).</para>
</section>
</section>
</chapter>
<info>
<title xml:id='configuringstorage.title'>Configuring Storage on a Lustre File System</title>
</info>
-<para><anchor xml:id="dbdoclet.50438208_pgfId-1291547" xreflabel=""/>This chapter describes best practices for storage selection and file system options to optimize perforance on RAID, and includes the following sections:</para>
+<para>This chapter describes best practices for storage selection and file system options to optimize perforance on RAID, and includes the following sections:</para>
<itemizedlist>
<listitem>
<section xml:id='dbdoclet.50438208_60972'>
<title>6.1 Selecting Storage for the MDT and OSTs</title>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291569" xreflabel=""/>The Lustre architecture allows the use of any kind of block device as backend storage. The characteristics of such devices, particularly in the case of failures, vary significantly and have an impact on configuration choices.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291570" xreflabel=""/>This section describes issues and recommendations regarding backend storage.</para>
+ <para>The Lustre architecture allows the use of any kind of block device as backend storage. The characteristics of such devices, particularly in the case of failures, vary significantly and have an impact on configuration choices.</para>
+ <para>This section describes issues and recommendations regarding backend storage.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438208_pgfId-1291571" xreflabel=""/>6.1.1 Metadata Target (MDT)</title>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291572" xreflabel=""/>I/O on the MDT is typically mostly reads and writes of small amounts of data. For this reason, we recommend that you use RAID 1 for MDT storage. If you require more capacity for an MDT than one disk provides, we recommend RAID 1 + 0 or RAID 10.</para>
+ <title>6.1.1 Metadata Target (MDT)</title>
+ <para>I/O on the MDT is typically mostly reads and writes of small amounts of data. For this reason, we recommend that you use RAID 1 for MDT storage. If you require more capacity for an MDT than one disk provides, we recommend RAID 1 + 0 or RAID 10.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438208_pgfId-1291573" xreflabel=""/>6.1.2 Object Storage Server (OST)</title>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291574" xreflabel=""/>A quick calculation makes it clear that without further redundancy, RAID 6 is required for large clusters and RAID 5 is not acceptable:</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291575" xreflabel=""/>For a 2 PB file system (2,000 disks of 1 TB capacity) assume the mean time to failure (MTTF) of a disk is about 1,000 days. This means that the expected failure rate is 2000/1000 = 2 disks per day. Repair time at 10% of disk bandwidth is 1000 GB at 10MB/sec = 100,000 sec, or about 1 day.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291576" xreflabel=""/>For a RAID 5 stripe that is 10 disks wide, during 1 day of rebuilding, the chance that a second disk in the same array will fail is about 9/1000 or about 1% per day. After 50 days, you have a 50% chance of a double failure in a RAID 5 array leading to data loss.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291577" xreflabel=""/>Therefore, RAID 6 or another double parity algorithm is needed to provide sufficient redundancy for OST storage.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291578" xreflabel=""/>For better performance, we recommend that you create RAID sets with 4 or 8 data disks plus one or two parity disks. Using larger RAID sets will negatively impact performance compared to having multiple independent RAID sets.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291579" xreflabel=""/>To maximize performance for small I/O request sizes, storage configured as RAID 1+0 can yield much better results but will increase cost or reduce capacity.</para>
+ <title>6.1.2 Object Storage Server (OST)</title>
+ <para>A quick calculation makes it clear that without further redundancy, RAID 6 is required for large clusters and RAID 5 is not acceptable:</para>
+ <para>For a 2 PB file system (2,000 disks of 1 TB capacity) assume the mean time to failure (MTTF) of a disk is about 1,000 days. This means that the expected failure rate is 2000/1000 = 2 disks per day. Repair time at 10% of disk bandwidth is 1000 GB at 10MB/sec = 100,000 sec, or about 1 day.</para>
+ <para>For a RAID 5 stripe that is 10 disks wide, during 1 day of rebuilding, the chance that a second disk in the same array will fail is about 9/1000 or about 1% per day. After 50 days, you have a 50% chance of a double failure in a RAID 5 array leading to data loss.</para>
+ <para>Therefore, RAID 6 or another double parity algorithm is needed to provide sufficient redundancy for OST storage.</para>
+ <para>For better performance, we recommend that you create RAID sets with 4 or 8 data disks plus one or two parity disks. Using larger RAID sets will negatively impact performance compared to having multiple independent RAID sets.</para>
+ <para>To maximize performance for small I/O request sizes, storage configured as RAID 1+0 can yield much better results but will increase cost or reduce capacity.</para>
</section>
</section>
<section xml:id="dbdoclet.50438208_23285">
<title>6.2 Reliability Best Practices</title>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291583" xreflabel=""/>RAID monitoring software is recommended to quickly detect faulty disks and allow them to be replaced to avoid double failures and data loss. Hot spare disks are recommended so that rebuilds happen without delays.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291587" xreflabel=""/>Backups of the metadata file systems are recommended. For details, see <xref linkend='backupandrestore'/>.</para>
+ <para>RAID monitoring software is recommended to quickly detect faulty disks and allow them to be replaced to avoid double failures and data loss. Hot spare disks are recommended so that rebuilds happen without delays.</para>
+ <para>Backups of the metadata file systems are recommended. For details, see <xref linkend='backupandrestore'/>.</para>
</section>
<section xml:id="dbdoclet.50438208_40705">
<title>6.3 Performance Tradeoffs</title>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291595" xreflabel=""/>A writeback cache can dramatically increase write performance on many types of RAID arrays if the writes are not done at full stripe width. Unfortunately, unless the RAID array has battery-backed cache (a feature only found in some higher-priced hardware RAID arrays), interrupting the power to the array may result in out-of-sequence writes or corruption of RAID parity and future data loss.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291596" xreflabel=""/>If writeback cache is enabled, a file system check is required after the array loses power. Data may also be lost because of this.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291597" xreflabel=""/>Therefore, we recommend against the use of writeback cache when data integrity is critical. You should carefully consider whether the benefits of using writeback cache outweigh the risks.</para>
+ <para>A writeback cache can dramatically increase write performance on many types of RAID arrays if the writes are not done at full stripe width. Unfortunately, unless the RAID array has battery-backed cache (a feature only found in some higher-priced hardware RAID arrays), interrupting the power to the array may result in out-of-sequence writes or corruption of RAID parity and future data loss.</para>
+ <para>If writeback cache is enabled, a file system check is required after the array loses power. Data may also be lost because of this.</para>
+ <para>Therefore, we recommend against the use of writeback cache when data integrity is critical. You should carefully consider whether the benefits of using writeback cache outweigh the risks.</para>
</section>
<section xml:id="dbdoclet.50438208_51921">
<title>6.4 Formatting Options for RAID Devices</title>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1289920" xreflabel=""/>When formatting a file system on a RAID device, it is beneficial to ensure that I/O requests are aligned with the underlying RAID geometry. This ensures that the Lustre RPCs do not generate unnecessary disk operations which may reduce performance dramatically. Use the --mkfsoptions parameter to specify additional parameters when formatting the OST or MDT.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1289921" xreflabel=""/>For RAID 5, RAID 6, or RAID 1+0 storage, specifying the following option to the --mkfsoptions parameter option improves the layout of the file system metadata, ensuring that no single disk contains all of the allocation bitmaps:</para>
- <screen><anchor xml:id="dbdoclet.50438208_pgfId-1290699" xreflabel=""/>-Estride=<chunk_blocks>
+ <para>When formatting a file system on a RAID device, it is beneficial to ensure that I/O requests are aligned with the underlying RAID geometry. This ensures that the Lustre RPCs do not generate unnecessary disk operations which may reduce performance dramatically. Use the --mkfsoptions parameter to specify additional parameters when formatting the OST or MDT.</para>
+ <para>For RAID 5, RAID 6, or RAID 1+0 storage, specifying the following option to the --mkfsoptions parameter option improves the layout of the file system metadata, ensuring that no single disk contains all of the allocation bitmaps:</para>
+ <screen>-Estride=<chunk_blocks>
</screen>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1290704" xreflabel=""/>The <chunk_blocks> variable is in units of 4096-byte blocks and represents the amount of contiguous data written to a single disk before moving to the next disk. This is alternately referred to as the RAID stripe size. This is applicable to both MDT and OST file systems.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1289922" xreflabel=""/>For more information on how to override the defaults while formatting MDT or OST file systems, see <link xl:href="SettingUpLustreSystem.html#50438256_84701">Setting File System Formatting Options</link>.</para>
+ <para>The <chunk_blocks> variable is in units of 4096-byte blocks and represents the amount of contiguous data written to a single disk before moving to the next disk. This is alternately referred to as the RAID stripe size. This is applicable to both MDT and OST file systems.</para>
+ <para>For more information on how to override the defaults while formatting MDT or OST file systems, see <link xl:href="SettingUpLustreSystem.html#50438256_84701">Setting File System Formatting Options</link>.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438208_pgfId-1290843" xreflabel=""/>6.4.1 Computing file system parameters for mkfs</title>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291294" xreflabel=""/>For best results, use RAID 5 with 5 or 9 disks or RAID 6 with 6 or 10 disks, each on a different controller. The stripe width is the optimal minimum I/O size. Ideally, the RAID configuration should allow 1 MB Lustre RPCs to fit evenly on a single RAID stripe without an expensive read-modify-write cycle. Use this formula to determine the <emphasis><stripe_width></emphasis>, where <emphasis><number_of_data_disks></emphasis> does <emphasis>not</emphasis> include the RAID parity disks (1 for RAID 5 and 2 for RAID 6):</para>
- <screen><anchor xml:id="dbdoclet.50438208_pgfId-1290351" xreflabel=""/><emphasis><stripe_width_blocks></emphasis> = <emphasis><chunk_blocks></emphasis> * <emphasis><number_of_data_disks></emphasis> = 1 MB
+ <title>6.4.1 Computing file system parameters for mkfs</title>
+ <para>For best results, use RAID 5 with 5 or 9 disks or RAID 6 with 6 or 10 disks, each on a different controller. The stripe width is the optimal minimum I/O size. Ideally, the RAID configuration should allow 1 MB Lustre RPCs to fit evenly on a single RAID stripe without an expensive read-modify-write cycle. Use this formula to determine the <emphasis><stripe_width></emphasis>, where <emphasis><number_of_data_disks></emphasis> does <emphasis>not</emphasis> include the RAID parity disks (1 for RAID 5 and 2 for RAID 6):</para>
+ <screen><emphasis><stripe_width_blocks></emphasis> = <emphasis><chunk_blocks></emphasis> * <emphasis><number_of_data_disks></emphasis> = 1 MB
</screen>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1290352" xreflabel=""/>If the RAID configuration does not allow <emphasis><chunk_blocks></emphasis> to fit evenly into 1 MB, select <emphasis><chunkblocks></emphasis>, such that <emphasis><stripe_width_blocks></emphasis> is close to 1 MB, but not larger.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291369" xreflabel=""/>The <emphasis><stripe_width_blocks></emphasis>value must equal <emphasis><chunk_blocks></emphasis>*<emphasis><number_of_data_disks></emphasis>. Specifying the <emphasis><stripe_width_blocks></emphasis> parameter is only relevant for RAID 5 or RAID 6, and is not needed for RAID 1 plus 0.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291443" xreflabel=""/>Run --reformat on the file system device (/dev/sdc), specifying the RAID geometry to the underlying ldiskfs file system, where:</para>
- <screen><anchor xml:id="dbdoclet.50438208_pgfId-1291453" xreflabel=""/>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, <anchor xml:id="dbdoclet.50438208_pgfId-1291454" xreflabel=""/>stripe_width=<emphasis><stripe_width_blocks></emphasis>"
+ <para>If the RAID configuration does not allow <emphasis><chunk_blocks></emphasis> to fit evenly into 1 MB, select <emphasis><chunkblocks></emphasis>, such that <emphasis><stripe_width_blocks></emphasis> is close to 1 MB, but not larger.</para>
+ <para>The <emphasis><stripe_width_blocks></emphasis>value must equal <emphasis><chunk_blocks></emphasis>*<emphasis><number_of_data_disks></emphasis>. Specifying the <emphasis><stripe_width_blocks></emphasis> parameter is only relevant for RAID 5 or RAID 6, and is not needed for RAID 1 plus 0.</para>
+ <para>Run --reformat on the file system device (/dev/sdc), specifying the RAID geometry to the underlying ldiskfs file system, where:</para>
+ <screen>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, stripe_width=<emphasis><stripe_width_blocks></emphasis>"
</screen>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1290353" xreflabel=""/>Example:</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291464" xreflabel=""/>A RAID 6 configuration with 6 disks has 4 data and 2 parity disks. The <emphasis><chunk_blocks></emphasis> <= 1024KB/4 = 256KB.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291362" xreflabel=""/>Because the number of data disks is equal to the power of 2, the stripe width is equal to 1 MB.</para>
- <screen><anchor xml:id="dbdoclet.50438208_pgfId-1291335" xreflabel=""/>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, <anchor xml:id="dbdoclet.50438208_pgfId-1291486" xreflabel=""/>stripe_width=<emphasis><stripe_width_blocks></emphasis>"...
+ <para>Example:</para>
+ <para>A RAID 6 configuration with 6 disks has 4 data and 2 parity disks. The <emphasis><chunk_blocks></emphasis> <= 1024KB/4 = 256KB.</para>
+ <para>Because the number of data disks is equal to the power of 2, the stripe width is equal to 1 MB.</para>
+ <screen>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, stripe_width=<emphasis><stripe_width_blocks></emphasis>"...
</screen>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438208_pgfId-1291318" xreflabel=""/>6.4.2 Choosing Parameters for an External <anchor xml:id="dbdoclet.50438208_marker-1289927" xreflabel=""/>Journal</title>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1289929" xreflabel=""/>If you have configured a RAID array and use it directly as an OST, it contains both data and metadata. For better performance, we recommend putting the OST journal on a separate device, by creating a small RAID 1 array and using it as an external journal for the OST.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1290391" xreflabel=""/>Lustre's default journal size is 400 MB. A journal size of up to 1 GB has shown increased performance but diminishing returns are seen for larger journals. Additionally, a copy of the journal is kept in RAM. Therefore, make sure you have enough memory available to hold copies of all the journals.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291643" xreflabel=""/>The file system journal options are specified to mkfs.luster using the --mkfsoptions parameter. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438208_pgfId-1291786" xreflabel=""/>--mkfsoptions "<other options> -j -J device=/dev/mdJ"
+ <title>6.4.2 Choosing Parameters for an External <anchor xml:id="dbdoclet.50438208_marker-1289927" xreflabel=""/>Journal</title>
+ <para>If you have configured a RAID array and use it directly as an OST, it contains both data and metadata. For better performance, we recommend putting the OST journal on a separate device, by creating a small RAID 1 array and using it as an external journal for the OST.</para>
+ <para>Lustre's default journal size is 400 MB. A journal size of up to 1 GB has shown increased performance but diminishing returns are seen for larger journals. Additionally, a copy of the journal is kept in RAM. Therefore, make sure you have enough memory available to hold copies of all the journals.</para>
+ <para>The file system journal options are specified to mkfs.luster using the --mkfsoptions parameter. For example:</para>
+ <screen>--mkfsoptions "<other options> -j -J device=/dev/mdJ"
</screen>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1289935" xreflabel=""/>To create an external journal, perform these steps for each OST on the OSS:</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1289936" xreflabel=""/> 1. Create a 400 MB (or larger) journal partition (RAID 1 is recommended).</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1289937" xreflabel=""/>In this example, /dev/sdb is a RAID 1 device.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291721" xreflabel=""/> 2. Create a journal device on the partition. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438208_pgfId-1289942" xreflabel=""/>[oss#] mke2fs -b 4096 -O journal_dev /dev/sdb <emphasis><journal_size></emphasis></screen>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291739" xreflabel=""/>The value of <emphasis><journal_size></emphasis> is specified in units of 4096-byte blocks. For example, 262144 for a 1 GB journal size.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1289943" xreflabel=""/> 3. Create the OST.</para>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1289944" xreflabel=""/>In this example, /dev/sdc is the RAID 6 device to be used as the OST, run:</para>
- <screen><anchor xml:id="dbdoclet.50438208_pgfId-1289945" xreflabel=""/>[oss#] mkfs.lustre --ost --mgsnode=mds@osib --mkfsoptions="-J device=/dev/sd\
+ <para>To create an external journal, perform these steps for each OST on the OSS:</para>
+ <para> 1. Create a 400 MB (or larger) journal partition (RAID 1 is recommended).</para>
+ <para>In this example, /dev/sdb is a RAID 1 device.</para>
+ <para> 2. Create a journal device on the partition. Run:</para>
+ <screen>[oss#] mke2fs -b 4096 -O journal_dev /dev/sdb <emphasis><journal_size></emphasis></screen>
+ <para>The value of <emphasis><journal_size></emphasis> is specified in units of 4096-byte blocks. For example, 262144 for a 1 GB journal size.</para>
+ <para> 3. Create the OST.</para>
+ <para>In this example, /dev/sdc is the RAID 6 device to be used as the OST, run:</para>
+ <screen>[oss#] mkfs.lustre --ost --mgsnode=mds@osib --mkfsoptions="-J device=/dev/sd\
b1" /dev/sdc
</screen>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1289946" xreflabel=""/> 4. Mount the OST as usual.</para>
+ <para> 4. Mount the OST as usual.</para>
</section>
</section>
<section xml:id="dbdoclet.50438208_88516">
<title>6.5 Connecting a SAN to a Lustre File System</title>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291861" xreflabel=""/>Depending on your cluster size and workload, you may want to connect a SAN to a Lustre file system. Before making this connection, consider the following:</para>
+ <para>Depending on your cluster size and workload, you may want to connect a SAN to a Lustre file system. Before making this connection, consider the following:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291862" xreflabel=""/> In many SAN file systems without Lustre, clients allocate and lock blocks or inodes individually as they are updated. The Lustre design avoids the high contention that some of these blocks and inodes may have.</para>
+ <para> In many SAN file systems without Lustre, clients allocate and lock blocks or inodes individually as they are updated. The Lustre design avoids the high contention that some of these blocks and inodes may have.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291863" xreflabel=""/> Lustre is highly scalable and can have a very large number of clients. SAN switches do not scale to a large number of nodes, and the cost per port of a SAN is generally higher than other networking.</para>
+ <para> Lustre is highly scalable and can have a very large number of clients. SAN switches do not scale to a large number of nodes, and the cost per port of a SAN is generally higher than other networking.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438208_pgfId-1291864" xreflabel=""/> File systems that allow direct-to-SAN access from the clients have a security risk because clients can potentially read any data on the SAN disks, and misbehaving clients can corrupt the file system for many reasons like improper file system, network, or other kernel software, bad cabling, bad memory, and so on. The risk increases with increase in the number of clients directly accessing the storage.</para>
+ <para> File systems that allow direct-to-SAN access from the clients have a security risk because clients can potentially read any data on the SAN disks, and misbehaving clients can corrupt the file system for many reasons like improper file system, network, or other kernel software, bad cabling, bad memory, and so on. The risk increases with increase in the number of clients directly accessing the storage.</para>
</listitem>
</itemizedlist>
</section>
<glossentry xml:id="acl">
<glossterm>ACL</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-998686" xreflabel=""/>Access Control List - An extended attribute associated with a file which contains authorization directives.</para>
+ <para>Access Control List - An extended attribute associated with a file which contains authorization directives.</para>
</glossdef>
</glossentry>
<glossentry xml:id="ostfail">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-998687" xreflabel=""/>Administrative OST failure</glossterm>
+ <glossterm>Administrative OST failure</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-996783" xreflabel=""/>A configuration directive given to a cluster to declare that an OST has failed, so errors can be immediately returned.</para>
+ <para>A configuration directive given to a cluster to declare that an OST has failed, so errors can be immediately returned.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>C</title>
<glossentry xml:id="cmd">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000465" xreflabel=""/> CMD</glossterm>
+ <glossterm> CMD</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000466" xreflabel=""/>Clustered metadata, a collection of metadata targets implementing a single file system namespace.</para>
+ <para>Clustered metadata, a collection of metadata targets implementing a single file system namespace.</para>
</glossdef>
</glossentry>
<glossentry xml:id="completioncallback">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000489" xreflabel=""/>Completion Callback
+ <glossterm>Completion Callback
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000490" xreflabel=""/>An RPC made by an OST or MDT to another system, usually a client, to indicate that the lock request is now granted.</para>
+ <para>An RPC made by an OST or MDT to another system, usually a client, to indicate that the lock request is now granted.</para>
</glossdef>
</glossentry>
<glossentry xml:id="changelog">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000494" xreflabel=""/>Configlog
+ <glossterm>Configlog
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000495" xreflabel=""/>An llog file used in a node, or retrieved from a management server over the network with configuration instructions for Lustre systems at startup time.</para>
+ <para>An llog file used in a node, or retrieved from a management server over the network with configuration instructions for Lustre systems at startup time.</para>
</glossdef>
</glossentry>
<glossentry xml:id="configlock">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000499" xreflabel=""/>Configuration Lock
+ <glossterm>Configuration Lock
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000500" xreflabel=""/>A lock held by every node in the cluster to control configuration changes. When callbacks are received, the nodes quiesce their traffic, cancel the lock and await configuration changes after which they reacquire the lock before resuming normal operation.</para>
+ <para>A lock held by every node in the cluster to control configuration changes. When callbacks are received, the nodes quiesce their traffic, cancel the lock and await configuration changes after which they reacquire the lock before resuming normal operation.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>D</title>
<glossentry xml:id="defaultstrippattern">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000504" xreflabel=""/>Default stripe pattern
+ <glossterm>Default stripe pattern
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000505" xreflabel=""/>Information in the LOV descriptor that describes the default stripe count used for new files in a file system. This can be amended by using a directory stripe descriptor or a per-file stripe descriptor.</para>
+ <para>Information in the LOV descriptor that describes the default stripe count used for new files in a file system. This can be amended by using a directory stripe descriptor or a per-file stripe descriptor.</para>
</glossdef>
</glossentry>
<glossentry xml:id="directio">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000733" xreflabel=""/>Direct I/O
+ <glossterm>Direct I/O
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000739" xreflabel=""/>A mechanism which can be used during read and write system calls. It bypasses the kernel. I/O cache to memory copy of data between kernel and application memory address spaces.</para>
+ <para>A mechanism which can be used during read and write system calls. It bypasses the kernel. I/O cache to memory copy of data between kernel and application memory address spaces.</para>
</glossdef>
</glossentry>
<glossentry xml:id="dirstripdesc">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000735" xreflabel=""/>Directory stripe descriptor
+ <glossterm>Directory stripe descriptor
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000515" xreflabel=""/>An extended attribute that describes the default stripe pattern for files underneath that directory.</para>
+ <para>An extended attribute that describes the default stripe pattern for files underneath that directory.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>E</title>
<glossentry xml:id="ea">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000519" xreflabel=""/>EA
+ <glossterm>EA
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000520" xreflabel=""/>Extended Attribute. A small amount of data which can be retrieved through a name associated with a particular inode. Lustre uses EAa to store striping information (location of file data on OSTs). Examples of extended attributes are ACLs, striping information, and crypto keys.</para>
+ <para>Extended Attribute. A small amount of data which can be retrieved through a name associated with a particular inode. Lustre uses EAa to store striping information (location of file data on OSTs). Examples of extended attributes are ACLs, striping information, and crypto keys.</para>
</glossdef>
</glossentry>
<glossentry xml:id="eviction">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000524" xreflabel=""/>Eviction
+ <glossterm>Eviction
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000525" xreflabel=""/>The process of eliminating server state for a client that is not returning to the cluster after a timeout or if server failures have occurred.</para>
+ <para>The process of eliminating server state for a client that is not returning to the cluster after a timeout or if server failures have occurred.</para>
</glossdef>
</glossentry>
<glossentry xml:id="export">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000611" xreflabel=""/>Export
+ <glossterm>Export
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000612" xreflabel=""/>The state held by a server for a client that is sufficient to transparently recover all in-flight operations when a single failure occurs.</para>
+ <para>The state held by a server for a client that is sufficient to transparently recover all in-flight operations when a single failure occurs.</para>
</glossdef>
</glossentry>
<glossentry xml:id="extendloc">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000625" xreflabel=""/>Extent Lock
+ <glossterm>Extent Lock
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000793" xreflabel=""/>A lock used by the OSC to protect an extent in a storage object for concurrent control of read/write, file size acquisition and truncation operations.</para>
+ <para>A lock used by the OSC to protect an extent in a storage object for concurrent control of read/write, file size acquisition and truncation operations.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>F</title>
<glossentry xml:id="failback">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000630" xreflabel=""/>Failback
+ <glossterm>Failback
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000631" xreflabel=""/>The failover process in which the default active server regains control over the service.</para>
+ <para>The failover process in which the default active server regains control over the service.</para>
</glossdef>
</glossentry>
<glossentry xml:id="failoutost">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000635" xreflabel=""/>Failout OST
+ <glossterm>Failout OST
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000636" xreflabel=""/>An OST which is not expected to recover if it fails to answer client requests. A failout OST can be administratively failed, thereby enabling clients to return errors when accessing data on the failed OST without making additional network requests.</para>
+ <para>An OST which is not expected to recover if it fails to answer client requests. A failout OST can be administratively failed, thereby enabling clients to return errors when accessing data on the failed OST without making additional network requests.</para>
</glossdef>
</glossentry>
<glossentry xml:id="failover">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000640" xreflabel=""/>Failover
+ <glossterm>Failover
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000641" xreflabel=""/>The process by which a standby computer server system takes over for an active computer server after a failure of the active node. Typically, the standby computer server gains exclusive access to a shared storage device between the two servers.</para>
+ <para>The process by which a standby computer server system takes over for an active computer server after a failure of the active node. Typically, the standby computer server gains exclusive access to a shared storage device between the two servers.</para>
</glossdef>
</glossentry>
<glossentry xml:id="fid">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000645" xreflabel=""/>FID
+ <glossterm>FID
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000646" xreflabel=""/>Lustre File xml:identifier. A collection of integers which uniquely xml:identify a file or object. The FID structure contains a sequence, xml:identity and version number.</para>
+ <para>Lustre File xml:identifier. A collection of integers which uniquely xml:identify a file or object. The FID structure contains a sequence, xml:identity and version number.</para>
</glossdef>
</glossentry>
<glossentry xml:id="fileset">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000650" xreflabel=""/>Fileset
+ <glossterm>Fileset
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000651" xreflabel=""/>A group of files that are defined through a directory that represents a file system's start point.</para>
+ <para>A group of files that are defined through a directory that represents a file system's start point.</para>
</glossdef>
</glossentry>
<glossentry xml:id="fldb">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000655" xreflabel=""/>FLDB
+ <glossterm>FLDB
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000656" xreflabel=""/>FID Location Database. This database maps a sequence of FIDs to a server which is managing the objects in the sequence.</para>
+ <para>FID Location Database. This database maps a sequence of FIDs to a server which is managing the objects in the sequence.</para>
</glossdef>
</glossentry>
<glossentry xml:id="flightgroup">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000660" xreflabel=""/>Flight Group
+ <glossterm>Flight Group
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000661" xreflabel=""/>Group or I/O transfer operations initiated in the OSC, which is simultaneously going between two endpoints. Tuning the flight group size correctly leads to a full pipe.</para>
+ <para>Group or I/O transfer operations initiated in the OSC, which is simultaneously going between two endpoints. Tuning the flight group size correctly leads to a full pipe.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>G</title>
<glossentry xml:id="glimpsecallback">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000665" xreflabel=""/>Glimpse callback
+ <glossterm>Glimpse callback
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002313" xreflabel=""/>An RPC made by an OST or MDT to another system, usually a client, to indicate to tthat an extent lock it is holding should be surrendered if it is not in use. If the system is using the lock, then the system should report the object size in the reply to the glimpse callback. Glimpses are introduced to optimize the acquisition of file sizes.</para>
+ <para>An RPC made by an OST or MDT to another system, usually a client, to indicate to tthat an extent lock it is holding should be surrendered if it is not in use. If the system is using the lock, then the system should report the object size in the reply to the glimpse callback. Glimpses are introduced to optimize the acquisition of file sizes.</para>
</glossdef>
</glossentry>
<glossentry xml:id="grouplock">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1002316" xreflabel=""/>Group Lock
+ <glossterm>Group Lock
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000676" xreflabel=""/> To follow.</para>
+ <para> To follow.</para>
</glossdef>
</glossentry>
<glossentry xml:id="groupupcall">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000680" xreflabel=""/>Group upcall
+ <glossterm>Group upcall
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000681" xreflabel=""/> To follow.</para>
+ <para> To follow.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>I</title>
<glossentry xml:id="import">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000690" xreflabel=""/>Import
+ <glossterm>Import
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000691" xreflabel=""/>The state held by a client to fully recover a transaction sequence after a server failure and restart.</para>
+ <para>The state held by a client to fully recover a transaction sequence after a server failure and restart.</para>
</glossdef>
</glossentry>
<glossentry xml:id="intentlock">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000695" xreflabel=""/>Intent Lock
+ <glossterm>Intent Lock
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000696" xreflabel=""/>A special locking operation introduced by Lustre into the Linux kernel. An intent lock combines a request for a lock, with the full information to perform the operation(s) for which the lock was requested. This offers the server the option of granting the lock or performing the operation and informing the client of the operation result without granting a lock. The use of intent locks enables metadata operations (even complicated ones), to be implemented with a single RPC from the client to the server.</para>
+ <para>A special locking operation introduced by Lustre into the Linux kernel. An intent lock combines a request for a lock, with the full information to perform the operation(s) for which the lock was requested. This offers the server the option of granting the lock or performing the operation and informing the client of the operation result without granting a lock. The use of intent locks enables metadata operations (even complicated ones), to be implemented with a single RPC from the client to the server.</para>
</glossdef>
</glossentry>
<glossentry xml:id="iov">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000700" xreflabel=""/>IOV
+ <glossterm>IOV
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000701" xreflabel=""/>I/O vector. A buffer destined for transport across the network which contains a collection (a/k/a as a vector) of blocks with data.</para>
+ <para>I/O vector. A buffer destined for transport across the network which contains a collection (a/k/a as a vector) of blocks with data.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>K</title>
<glossentry xml:id="kerberos">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000975" xreflabel=""/>Kerberos
+ <glossterm>Kerberos
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000976" xreflabel=""/>An authentication mechanism, optionally available in an upcoming Lustre version as a GSS backend.</para>
+ <para>An authentication mechanism, optionally available in an upcoming Lustre version as a GSS backend.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>L</title>
<glossentry xml:id="lbug">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000710" xreflabel=""/>LBUG
+ <glossterm>LBUG
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000711" xreflabel=""/>A bug that Lustre writes into a log indicating a serious system failure.</para>
+ <para>A bug that Lustre writes into a log indicating a serious system failure.</para>
</glossdef>
</glossentry>
<glossentry xml:id="ldlm">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000433" xreflabel=""/>LDLM
+ <glossterm>LDLM
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-996800" xreflabel=""/>Lustre Distributed Lock Manager.</para>
+ <para>Lustre Distributed Lock Manager.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lfs">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001024" xreflabel=""/>lfs
+ <glossterm>lfs
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002332" xreflabel=""/>The Lustre File System configuration tool for end users to set/check file striping, etc. See <link xl:href="UserUtilities.html#50438206_94597">glossdiv 32.1, lfs</link>.</para>
+ <para>The Lustre File System configuration tool for end users to set/check file striping, etc. See <link xl:href="UserUtilities.html#50438206_94597">glossdiv 32.1, lfs</link>.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lfsck">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001033" xreflabel=""/>lfsck
+ <glossterm>lfsck
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001034" xreflabel=""/>Lustre File System Check. A distributed version of a disk file system checker. Normally, lfsck does not need to be run, except when file systems are damaged through multiple disk failures and other means that cannot be recovered using file system journal recovery.</para>
+ <para>Lustre File System Check. A distributed version of a disk file system checker. Normally, lfsck does not need to be run, except when file systems are damaged through multiple disk failures and other means that cannot be recovered using file system journal recovery.</para>
</glossdef>
</glossentry>
<glossentry xml:id="liblustre">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001036" xreflabel=""/>liblustre
+ <glossterm>liblustre
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001037" xreflabel=""/>Lustre library. A user-mode Lustre client linked into a user program for Lustre fs access. liblustre clients cache no data, do not need to give back locks on time, and can recover safely from an eviction. They should not participate in recovery.</para>
+ <para>Lustre library. A user-mode Lustre client linked into a user program for Lustre fs access. liblustre clients cache no data, do not need to give back locks on time, and can recover safely from an eviction. They should not participate in recovery.</para>
</glossdef>
</glossentry>
<glossentry xml:id="llite">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001077" xreflabel=""/>Llite
+ <glossterm>Llite
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001078" xreflabel=""/>Lustre lite. This term is in use inside the code and module names to indicate that code elements are related to the Lustre file system.</para>
+ <para>Lustre lite. This term is in use inside the code and module names to indicate that code elements are related to the Lustre file system.</para>
</glossdef>
</glossentry>
<glossentry xml:id="llog">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001086" xreflabel=""/>Llog
+ <glossterm>Llog
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001087" xreflabel=""/>Lustre log. A log of entries used internally by Lustre. An llog is suitable for rapid transactional appends of records and cheap cancellation of records through a bitmap.</para>
+ <para>Lustre log. A log of entries used internally by Lustre. An llog is suitable for rapid transactional appends of records and cheap cancellation of records through a bitmap.</para>
</glossdef>
</glossentry>
<glossentry xml:id="llogcatalog">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001091" xreflabel=""/>Llog Catalog
+ <glossterm>Llog Catalog
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001127" xreflabel=""/>Lustre log catalog. An llog with records that each point at an llog. Catalogs were introduced to give llogs almost infinite size. llogs have an originator which writes records and a replicator which cancels record (usually through an RPC), when the records are not needed.</para>
+ <para>Lustre log catalog. An llog with records that each point at an llog. Catalogs were introduced to give llogs almost infinite size. llogs have an originator which writes records and a replicator which cancels record (usually through an RPC), when the records are not needed.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lmv">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001159" xreflabel=""/>LMV
+ <glossterm>LMV
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001186" xreflabel=""/>Logical Metadata Volume. A driver to abstract in the Lustre client that it is working with a metadata cluster instead of a single metadata server.</para>
+ <para>Logical Metadata Volume. A driver to abstract in the Lustre client that it is working with a metadata cluster instead of a single metadata server.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lnd">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001197" xreflabel=""/>LND
+ <glossterm>LND
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001198" xreflabel=""/>Lustre Network Driver. A code module that enables LNET support over a particular transport, such as TCP and various kinds of InfiniBand.</para>
+ <para>Lustre Network Driver. A code module that enables LNET support over a particular transport, such as TCP and various kinds of InfiniBand.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lnet">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001202" xreflabel=""/>LNET
+ <glossterm>LNET
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001203" xreflabel=""/>Lustre Networking. A message passing network protocol capable of running and routing through various physical layers. LNET forms the underpinning of LNETrpc.</para>
+ <para>Lustre Networking. A message passing network protocol capable of running and routing through various physical layers. LNET forms the underpinning of LNETrpc.</para>
</glossdef>
</glossentry>
<glossentry xml:id="loadmds">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001212" xreflabel=""/>Load-balancing MDSs
+ <glossterm>Load-balancing MDSs
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001213" xreflabel=""/>A cluster of MDSs that perform load balancing of on system requests.</para>
+ <para>A cluster of MDSs that perform load balancing of on system requests.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lockclient">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001217" xreflabel=""/>Lock Client
+ <glossterm>Lock Client
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001218" xreflabel=""/>A module that makes lock RPCs to a lock server and handles revocations from the server.</para>
+ <para>A module that makes lock RPCs to a lock server and handles revocations from the server.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lockserver">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001274" xreflabel=""/>Lock Server
+ <glossterm>Lock Server
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001275" xreflabel=""/>A system that manages locks on certain objects. It also issues lock callback requests, calls while servicing or, for objects that are already locked, completes lock requests.</para>
+ <para>A system that manages locks on certain objects. It also issues lock callback requests, calls while servicing or, for objects that are already locked, completes lock requests.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lov">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001283" xreflabel=""/>LOV
+ <glossterm>LOV
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001284" xreflabel=""/>Logical Object Volume. The object storage analog of a logical volume in a block device volume management system, such as LVM or EVMS. The LOV is primarily used to present a collection of OSTs as a single device to the MDT and client file system drivers.</para>
+ <para>Logical Object Volume. The object storage analog of a logical volume in a block device volume management system, such as LVM or EVMS. The LOV is primarily used to present a collection of OSTs as a single device to the MDT and client file system drivers.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lovdes">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001288" xreflabel=""/>LOV descriptor
+ <glossterm>LOV descriptor
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001289" xreflabel=""/>A set of configuration directives which describes which nodes are OSS systems in the Lustre cluster, providing names for their OSTs.</para>
+ <para>A set of configuration directives which describes which nodes are OSS systems in the Lustre cluster, providing names for their OSTs.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lustre">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001293" xreflabel=""/>Lustre
+ <glossterm>Lustre
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001294" xreflabel=""/>The name of the project chosen by Peter Braam in 1999 for an object-based storage architecture. Now the name is commonly associated with the Lustre file system.</para>
+ <para>The name of the project chosen by Peter Braam in 1999 for an object-based storage architecture. Now the name is commonly associated with the Lustre file system.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lustreclient">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001298" xreflabel=""/>Lustre client
+ <glossterm>Lustre client
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001299" xreflabel=""/>An operating instance with a mounted Lustre file system.</para>
+ <para>An operating instance with a mounted Lustre file system.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lustrefile">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001303" xreflabel=""/>Lustre file
+ <glossterm>Lustre file
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001304" xreflabel=""/>A file in the Lustre file system. The implementation of a Lustre file is through an inode on a metadata server which contains references to a storage object on OSSs.</para>
+ <para>A file in the Lustre file system. The implementation of a Lustre file is through an inode on a metadata server which contains references to a storage object on OSSs.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lustrelite">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001308" xreflabel=""/>Lustre lite
+ <glossterm>Lustre lite
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001309" xreflabel=""/>A preliminary version of Lustre developed for LLNL in 2002. With the release of Lustre 1.0 in late 2003, Lustre Lite became obsolete.</para>
+ <para>A preliminary version of Lustre developed for LLNL in 2002. With the release of Lustre 1.0 in late 2003, Lustre Lite became obsolete.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lvfs">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001313" xreflabel=""/>Lvfs
+ <glossterm>Lvfs
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001314" xreflabel=""/>A library that provides an interface between Lustre OSD and MDD drivers and file systems; this avoids introducing file system-specific abstractions into the OSD and MDD drivers.</para>
+ <para>A library that provides an interface between Lustre OSD and MDD drivers and file systems; this avoids introducing file system-specific abstractions into the OSD and MDD drivers.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>M</title>
<glossentry xml:id="mballoc">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001403" xreflabel=""/>Mballoc
+ <glossterm>Mballoc
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002369" xreflabel=""/>Multi-Block-Allocate. Lustre functionality that enables the ldiskfs file system to allocate multiple blocks with a single request to the block allocator. Normally, an ldiskfs file system only allocates only one block per request.</para>
+ <para>Multi-Block-Allocate. Lustre functionality that enables the ldiskfs file system to allocate multiple blocks with a single request to the block allocator. Normally, an ldiskfs file system only allocates only one block per request.</para>
</glossdef>
</glossentry>
<glossentry xml:id="mdc">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001408" xreflabel=""/>MDC
+ <glossterm>MDC
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001417" xreflabel=""/>MetaData Client - Lustre client component that sends metadata requests via RPC over LNET to the Metadata Target (MDT).</para>
+ <para>MetaData Client - Lustre client component that sends metadata requests via RPC over LNET to the Metadata Target (MDT).</para>
</glossdef>
</glossentry>
<glossentry xml:id="mdd">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001426" xreflabel=""/>MDD
+ <glossterm>MDD
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002412" xreflabel=""/>MetaData Disk Device - Lustre server component that interfaces with the underlying Object Storage Device to manage the Lustre file system namespace (directories, file ownership, attributes).</para>
+ <para>MetaData Disk Device - Lustre server component that interfaces with the underlying Object Storage Device to manage the Lustre file system namespace (directories, file ownership, attributes).</para>
</glossdef>
</glossentry>
<glossentry xml:id="mds">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001431" xreflabel=""/>MDS
+ <glossterm>MDS
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002444" xreflabel=""/>MetaData Server - Server node that is hosting the Metadata Target (MDT).</para>
+ <para>MetaData Server - Server node that is hosting the Metadata Target (MDT).</para>
</glossdef>
</glossentry>
<glossentry xml:id="mdt">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001545" xreflabel=""/>MDT
+ <glossterm>MDT
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001546" xreflabel=""/>Metadata Target. A metadata device made available through the Lustre meta-data network protocol.</para>
+ <para>Metadata Target. A metadata device made available through the Lustre meta-data network protocol.</para>
</glossdef>
</glossentry>
<glossentry xml:id="metadatawriteback">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001547" xreflabel=""/>Metadata Write-back Cache
+ <glossterm>Metadata Write-back Cache
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001447" xreflabel=""/>A cache of metadata updates (mkdir, create, setattr, other operations) which an application has performed, but have not yet been flushed to a storage device or server.</para>
+ <para>A cache of metadata updates (mkdir, create, setattr, other operations) which an application has performed, but have not yet been flushed to a storage device or server.</para>
</glossdef>
</glossentry>
<glossentry xml:id="mgs">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001451" xreflabel=""/>MGS
+ <glossterm>MGS
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001452" xreflabel=""/>Management Service. A software module that manages the startup configuration and changes to the configuration. Also, the server node on which this system runs.</para>
+ <para>Management Service. A software module that manages the startup configuration and changes to the configuration. Also, the server node on which this system runs.</para>
</glossdef>
</glossentry>
<glossentry xml:id="mountconf">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001505" xreflabel=""/>Mountconf
+ <glossterm>Mountconf
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001582" xreflabel=""/>The Lustre configuration protocol (introduced in version 1.6) which formats disk file systems on servers with the mkfs.lustre program, and prepares them for automatic incorporation into a Lustre cluster.</para>
+ <para>The Lustre configuration protocol (introduced in version 1.6) which formats disk file systems on servers with the mkfs.lustre program, and prepares them for automatic incorporation into a Lustre cluster.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>N</title>
<glossentry xml:id="nal">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001602" xreflabel=""/>NAL
+ <glossterm>NAL
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001603" xreflabel=""/>An older, obsolete term for LND.</para>
+ <para>An older, obsolete term for LND.</para>
</glossdef>
</glossentry>
<glossentry xml:id="nid">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001607" xreflabel=""/>NID
+ <glossterm>NID
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001608" xreflabel=""/>Network xml:identifier. Encodes the type, network number and network address of a network interface on a node for use by Lustre.</para>
+ <para>Network xml:identifier. Encodes the type, network number and network address of a network interface on a node for use by Lustre.</para>
</glossdef>
</glossentry>
<glossentry xml:id="nioapi">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001612" xreflabel=""/>NIO API
+ <glossterm>NIO API
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001657" xreflabel=""/>A subset of the LNET RPC module that implements a library for sending large network requests, moving buffers with RDMA.</para>
+ <para>A subset of the LNET RPC module that implements a library for sending large network requests, moving buffers with RDMA.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>O</title>
<glossentry xml:id="odb">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001668" xreflabel=""/>OBD
+ <glossterm>OBD
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001742" xreflabel=""/>Object Device. The base class of layering software constructs that provides Lustre functionality.</para>
+ <para>Object Device. The base class of layering software constructs that provides Lustre functionality.</para>
</glossdef>
</glossentry>
<glossentry xml:id="odbapi">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001673" xreflabel=""/>OBD API
+ <glossterm>OBD API
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001674" xreflabel=""/>See Storage Object API.</para>
+ <para>See Storage Object API.</para>
</glossdef>
</glossentry>
<glossentry xml:id="odbtype">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001678" xreflabel=""/>OBD type
+ <glossterm>OBD type
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001753" xreflabel=""/>Module that can implement the Lustre object or metadata APIs. Examples of OBD types include the LOV, OSC and OSD.</para>
+ <para>Module that can implement the Lustre object or metadata APIs. Examples of OBD types include the LOV, OSC and OSD.</para>
</glossdef>
</glossentry>
<glossentry xml:id="odbfilter">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001683" xreflabel=""/>Obdfilter
+ <glossterm>Obdfilter
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001759" xreflabel=""/>An older name for the OSD device driver.</para>
+ <para>An older name for the OSD device driver.</para>
</glossdef>
</glossentry>
<glossentry xml:id="objectdev">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001693" xreflabel=""/>Object device
+ <glossterm>Object device
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001694" xreflabel=""/>An instance of an object that exports the OBD API.</para>
+ <para>An instance of an object that exports the OBD API.</para>
</glossdef>
</glossentry>
<glossentry xml:id="objectstorage">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001698" xreflabel=""/>Object storage
+ <glossterm>Object storage
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001699" xreflabel=""/>Refers to a storage-device API or protocol involving storage objects. The two most well known instances of object storage are the T10 iSCSI storage object protocol and the Lustre object storage protocol (a network implementation of the Lustre object API). The principal difference between the Lustre and T10 protocols is that Lustre includes locking and recovery control in the protocol and is not tied to a SCSI transport layer.</para>
+ <para>Refers to a storage-device API or protocol involving storage objects. The two most well known instances of object storage are the T10 iSCSI storage object protocol and the Lustre object storage protocol (a network implementation of the Lustre object API). The principal difference between the Lustre and T10 protocols is that Lustre includes locking and recovery control in the protocol and is not tied to a SCSI transport layer.</para>
</glossdef>
</glossentry>
<glossentry xml:id="opencache">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001703" xreflabel=""/>opencache
+ <glossterm>opencache
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001704" xreflabel=""/>A cache of open file handles. This is a performance enhancement for NFS.</para>
+ <para>A cache of open file handles. This is a performance enhancement for NFS.</para>
</glossdef>
</glossentry>
<glossentry xml:id="orphanobjects">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001708" xreflabel=""/>Orphan objects
+ <glossterm>Orphan objects
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001709" xreflabel=""/>Storage objects for which there is no Lustre file pointing at them. Orphan objects can arise from crashes and are automatically removed by an llog recovery. When a client deletes a file, the MDT gives back a cookie for each stripe. The client then sends the cookie and directs the OST to delete the stripe. Finally, the OST sends the cookie back to the MDT to cancel it.</para>
+ <para>Storage objects for which there is no Lustre file pointing at them. Orphan objects can arise from crashes and are automatically removed by an llog recovery. When a client deletes a file, the MDT gives back a cookie for each stripe. The client then sends the cookie and directs the OST to delete the stripe. Finally, the OST sends the cookie back to the MDT to cancel it.</para>
</glossdef>
</glossentry>
<glossentry xml:id="orphanhandling">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001802" xreflabel=""/>Orphan handling
+ <glossterm>Orphan handling
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001803" xreflabel=""/>A component of the metadata service which allows for recovery of open, unlinked files after a server crash. The implementation of this feature retains open, unlinked files as orphan objects until it is determined that no clients are using them.</para>
+ <para>A component of the metadata service which allows for recovery of open, unlinked files after a server crash. The implementation of this feature retains open, unlinked files as orphan objects until it is determined that no clients are using them.</para>
</glossdef>
</glossentry>
<glossentry xml:id="osc">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001713" xreflabel=""/>OSC
+ <glossterm>OSC
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001714" xreflabel=""/>Object Storage Client. The client unit talking to an OST (via an OSS).</para>
+ <para>Object Storage Client. The client unit talking to an OST (via an OSS).</para>
</glossdef>
</glossentry>
<glossentry xml:id="osd">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001818" xreflabel=""/>OSD
+ <glossterm>OSD
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001819" xreflabel=""/>Object Storage Device. A generic, industry term for storage devices with more extended interface than block-oriented devices, such as disks. Lustre uses this name to describe to a software module that implements an object storage API in the kernel. Lustre also uses this name to refer to an instance of an object storage device created by that driver. The OSD device is layered on a file system, with methods that mimic create, destroy and I/O operations on file inodes.</para>
+ <para>Object Storage Device. A generic, industry term for storage devices with more extended interface than block-oriented devices, such as disks. Lustre uses this name to describe to a software module that implements an object storage API in the kernel. Lustre also uses this name to refer to an instance of an object storage device created by that driver. The OSD device is layered on a file system, with methods that mimic create, destroy and I/O operations on file inodes.</para>
</glossdef>
</glossentry>
<glossentry xml:id="oss">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001834" xreflabel=""/>OSS
+ <glossterm>OSS
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001835" xreflabel=""/>Object Storage Server. A server OBD that provides access to local OSTs.</para>
+ <para>Object Storage Server. A server OBD that provides access to local OSTs.</para>
</glossdef>
</glossentry>
<glossentry xml:id="ost">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001879" xreflabel=""/>OST
+ <glossterm>OST
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001880" xreflabel=""/>Object Storage Target. An OSD made accessible through a network protocol. Typically, an OST is associated with a unique OSD which, in turn is associated with a formatted disk file system on the server containing the storage objects.</para>
+ <para>Object Storage Target. An OSD made accessible through a network protocol. Typically, an OST is associated with a unique OSD which, in turn is associated with a formatted disk file system on the server containing the storage objects.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>P</title>
<glossentry xml:id="pdirops">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001912" xreflabel=""/>Pdirops
+ <glossterm>Pdirops
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001889" xreflabel=""/>A locking protocol introduced in the VFS by CFS to allow for concurrent operations on a single directory inode.</para>
+ <para>A locking protocol introduced in the VFS by CFS to allow for concurrent operations on a single directory inode.</para>
</glossdef>
</glossentry>
<glossentry xml:id="pool">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001893" xreflabel=""/>pool
+ <glossterm>pool
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001894" xreflabel=""/>OST pools allows the administrator to associate a name with an arbitrary subset of OSTs in a Lustre cluster. A group of OSTs can be combined into a named pool with unique access permissions and stripe characteristics.</para>
+ <para>OST pools allows the administrator to associate a name with an arbitrary subset of OSTs in a Lustre cluster. A group of OSTs can be combined into a named pool with unique access permissions and stripe characteristics.</para>
</glossdef>
</glossentry>
<glossentry xml:id="portal">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001943" xreflabel=""/>Portal
+ <glossterm>Portal
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002017" xreflabel=""/>A concept used by LNET. LNET messages are sent to a portal on a NID. Portals can receive packets when a memory descriptor is attached to the portal. Portals are implemented as integers.</para>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001944" xreflabel=""/>Examples of portals are the portals on which certain groups of object, metadata, configuration and locking requests and replies are received.</para>
+ <para>A concept used by LNET. LNET messages are sent to a portal on a NID. Portals can receive packets when a memory descriptor is attached to the portal. Portals are implemented as integers.</para>
+ <para>Examples of portals are the portals on which certain groups of object, metadata, configuration and locking requests and replies are received.</para>
</glossdef>
</glossentry>
<glossentry xml:id="ptlrpc">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1002360" xreflabel=""/>PTLRPC
+ <glossterm>PTLRPC
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002361" xreflabel=""/>An RPC protocol layered on LNET. This protocol deals with stateful servers and has exactly-once semantics and built in support for recovery.</para>
+ <para>An RPC protocol layered on LNET. This protocol deals with stateful servers and has exactly-once semantics and built in support for recovery.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>R</title>
<glossentry xml:id="recovery">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001962" xreflabel=""/>Recovery
+ <glossterm>Recovery
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001963" xreflabel=""/>The process that re-establishes the connection state when a client that was previously connected to a server reconnects after the server restarts.</para>
+ <para>The process that re-establishes the connection state when a client that was previously connected to a server reconnects after the server restarts.</para>
</glossdef>
</glossentry>
<glossentry xml:id="reply">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001967" xreflabel=""/>Reply
+ <glossterm>Reply
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001968" xreflabel=""/>The concept of re-executing a server request after the server lost information in its memory caches and shut down. The replay requests are retained by clients until the server(s) have confirmed that the data is persistent on disk. Only requests for which a client has received a reply are replayed.</para>
+ <para>The concept of re-executing a server request after the server lost information in its memory caches and shut down. The replay requests are retained by clients until the server(s) have confirmed that the data is persistent on disk. Only requests for which a client has received a reply are replayed.</para>
</glossdef>
</glossentry>
<glossentry xml:id="resent">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001972" xreflabel=""/>Re-sent request
+ <glossterm>Re-sent request
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001973" xreflabel=""/>A request that has seen no reply can be re-sent after a server reboot.</para>
+ <para>A request that has seen no reply can be re-sent after a server reboot.</para>
</glossdef>
</glossentry>
<glossentry xml:id="revocation">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001977" xreflabel=""/>Revocation Callback
+ <glossterm>Revocation Callback
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001978" xreflabel=""/>An RPC made by an OST or MDT to another system, usually a client, to revoke a granted lock.</para>
+ <para>An RPC made by an OST or MDT to another system, usually a client, to revoke a granted lock.</para>
</glossdef>
</glossentry>
<glossentry xml:id="rollback">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001982" xreflabel=""/>Rollback
+ <glossterm>Rollback
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001983" xreflabel=""/>The concept that server state is in a crash lost because it was cached in memory and not yet persistent on disk.</para>
+ <para>The concept that server state is in a crash lost because it was cached in memory and not yet persistent on disk.</para>
</glossdef>
</glossentry>
<glossentry xml:id="rootsquash">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001987" xreflabel=""/>Root squash
+ <glossterm>Root squash
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001988" xreflabel=""/>A mechanism whereby the xml:identity of a root user on a client system is mapped to a different xml:identity on the server to avoid root users on clients gaining broad permissions on servers. Typically, for management purposes, at least one client system should not be subject to root squash.</para>
+ <para>A mechanism whereby the xml:identity of a root user on a client system is mapped to a different xml:identity on the server to avoid root users on clients gaining broad permissions on servers. Typically, for management purposes, at least one client system should not be subject to root squash.</para>
</glossdef>
</glossentry>
<glossentry xml:id="routing">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001992" xreflabel=""/>routing
+ <glossterm>routing
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001993" xreflabel=""/>LNET routing between different networks and LNDs.</para>
+ <para>LNET routing between different networks and LNDs.</para>
</glossdef>
</glossentry>
<glossentry xml:id="rpc">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1001997" xreflabel=""/>RPC
+ <glossterm>RPC
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1001998" xreflabel=""/>Remote Procedure Call. A network encoding of a request.</para>
+ <para>Remote Procedure Call. A network encoding of a request.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>S</title>
<glossentry xml:id="storageobjapi">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1002002" xreflabel=""/>Storage Object API
+ <glossterm>Storage Object API
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002003" xreflabel=""/>The API that manipulates storage objects. This API is richer than that of block devices and includes the create/delete of storage objects, read/write of buffers from and to certain offsets, set attributes and other storage object metadata.</para>
+ <para>The API that manipulates storage objects. This API is richer than that of block devices and includes the create/delete of storage objects, read/write of buffers from and to certain offsets, set attributes and other storage object metadata.</para>
</glossdef>
</glossentry>
<glossentry xml:id="storageobjects">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1002153" xreflabel=""/>Storage Objects
+ <glossterm>Storage Objects
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002154" xreflabel=""/>A generic concept referring to data containers, similar/identical to file inodes.</para>
+ <para>A generic concept referring to data containers, similar/identical to file inodes.</para>
</glossdef>
</glossentry>
<glossentry xml:id="stride">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1002162" xreflabel=""/>Stride
+ <glossterm>Stride
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002163" xreflabel=""/>A contiguous, logical extent of a Lustre file written to a single OST.</para>
+ <para>A contiguous, logical extent of a Lustre file written to a single OST.</para>
</glossdef>
</glossentry>
<glossentry xml:id="stridesize">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1002167" xreflabel=""/>Stride size
+ <glossterm>Stride size
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002168" xreflabel=""/>The maximum size of a stride, typically 4 MB.</para>
+ <para>The maximum size of a stride, typically 4 MB.</para>
</glossdef>
</glossentry>
<glossentry xml:id="stripcount">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1002172" xreflabel=""/>Stripe count
+ <glossterm>Stripe count
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002173" xreflabel=""/>The number of OSTs holding objects for a RAID0-striped Lustre file.</para>
+ <para>The number of OSTs holding objects for a RAID0-striped Lustre file.</para>
</glossdef>
</glossentry>
<glossentry xml:id="stripingmetadata">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1002177" xreflabel=""/>Striping metadata
+ <glossterm>Striping metadata
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002178" xreflabel=""/>The extended attribute associated with a file that describes how its data is distributed over storage objects. See also default stripe pattern.</para>
+ <para>The extended attribute associated with a file that describes how its data is distributed over storage objects. See also default stripe pattern.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>T</title>
<glossentry xml:id="t10">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1002214" xreflabel=""/>T10 object protocol
+ <glossterm>T10 object protocol
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002215" xreflabel=""/>An object storage protocol tied to the SCSI transport layer. Lustre does not use T10.</para>
+ <para>An object storage protocol tied to the SCSI transport layer. Lustre does not use T10.</para>
</glossdef>
</glossentry>
</glossdiv>
<glossdiv>
<title>W</title>
<glossentry xml:id="widestriping">
- <glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1002229" xreflabel=""/>Wide striping
+ <glossterm>Wide striping
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1002230" xreflabel=""/>Strategy of using many OSTs to store stripes of a single file. This obtains maximum bandwidth to a single file through parallel utilization of many OSTs.</para>
+ <para>Strategy of using many OSTs to store stripes of a single file. This obtains maximum bandwidth to a single file through parallel utilization of many OSTs.</para>
</glossdef>
</glossentry>
</glossdiv>
<para>Part III provides information about tools and procedures to use to administer a Lustre file system. You will find information in this section about:</para>
<tip>
- <para><anchor xml:id="dbdoclet.50438269_pgfId-369891" xreflabel=""/>The starting point for administering Lustre is to monitor all logs and console logs for system health:</para>
+ <para>The starting point for administering Lustre is to monitor all logs and console logs for system health:</para>
<para> - Monitor logs on all servers and all clients.</para>
<para> - Invest in tools that allow you to condense logs from multiple systems.</para>
<para> - Use the logging resources provided by Linux.</para>
<title>Tuning Lustre for Performance</title>
</info>
<partintro>
- <para><anchor xml:id="dbdoclet.50438187_pgfId-369853" xreflabel=""/>Part IV describes tools and procedures used to tune a Lustre file system for optimum performance. You will find information in this section about:</para>
+ <para>Part IV describes tools and procedures used to tune a Lustre file system for optimum performance. You will find information in this section about:</para>
<itemizedlist>
<listitem>
<part version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='part.intro'>
<title>Introducing Lustre</title>
<partintro>
- <para><anchor xml:id="dbdoclet.50438249_pgfId-369630" xreflabel=""/>Part I provides background information to help you understand the Lustre architecture and how the major components fit together. You will find information in this section about:
+ <para>Part I provides background information to help you understand the Lustre architecture and how the major components fit together. You will find information in this section about:
</para>
<itemizedlist>
<info>
<title xml:id="installoverview.title">Installation Overview</title>
</info>
- <para><anchor xml:id="dbdoclet.50438192_pgfId-1306234" xreflabel=""/>This chapter provides on overview of the procedures required to set up, install and configure a Lustre file system.</para>
+ <para>This chapter provides on overview of the procedures required to set up, install and configure a Lustre file system.</para>
<note>
<para>
If you are new to Lustre, you may find it helpful to refer to <xref linkend="part.intro"/> for a description of the Lustre architecture, file system components and terminology before proceeding with the installation procedure.</para>
</note>
<section remap="h2">
<title>4.1 Steps to Installing Lustre</title>
- <para><anchor xml:id="dbdoclet.50438192_pgfId-1306237" xreflabel=""/>To set up Lustre file system hardware and install and configure the Lustre software, refer the the chapters below in the order listed:</para>
+ <para>To set up Lustre file system hardware and install and configure the Lustre software, refer the the chapters below in the order listed:</para>
<orderedlist>
<listitem>
<para> <emphasis>(Required)</emphasis> <emphasis role="bold">Set up your Lustre file System hardware.</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438192_pgfId-1307046" xreflabel=""/>See <xref linkend="settinguplustresystem"/> - Provides guidelines for configuring hardware for a Lustre file system including storage, memory, and networking requirements.</para>
+ <para>See <xref linkend="settinguplustresystem"/> - Provides guidelines for configuring hardware for a Lustre file system including storage, memory, and networking requirements.</para>
</listitem>
<listitem>
<para> <emphasis>(Optional - Highly Recommended)</emphasis> <emphasis role="bold">Configure storage on Lustre storage devices.</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438192_pgfId-1307007" xreflabel=""/>See <xref linkend="configuringstorage"/> - Provides instructions for setting up hardware RAID on Lustre storage devices.</para>
+ <para>See <xref linkend="configuringstorage"/> - Provides instructions for setting up hardware RAID on Lustre storage devices.</para>
</listitem>
<listitem>
<para> <emphasis>(Optional)</emphasis> <emphasis role="bold">Set up network interface bonding.</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438192_pgfId-1307015" xreflabel=""/>See <xref linkend="settingupbonding"/> - Describes setting up network interface bonding to allow multiple network interfaces to be used in parallel to increase bandwidth or redundancy.</para>
+ <para>See <xref linkend="settingupbonding"/> - Describes setting up network interface bonding to allow multiple network interfaces to be used in parallel to increase bandwidth or redundancy.</para>
</listitem>
<listitem>
<para><emphasis>(Required)</emphasis> <emphasis role="bold">Install Lustre software.</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438192_pgfId-1307022" xreflabel=""/>See <xref linkend="installinglustre"/> - Describes preparation steps and a procedure for installing the Lustre software.</para>
+ <para>See <xref linkend="installinglustre"/> - Describes preparation steps and a procedure for installing the Lustre software.</para>
</listitem>
<listitem>
<para><emphasis>(Optional)</emphasis> <emphasis role="bold">Configure Lustre Networking (LNET).</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438192_pgfId-1307025" xreflabel=""/>See <xref linkend="configuringlnet"/> - Describes how to configure LNET if the default configuration is not sufficient. By default, LNET will use the first TCP/IP interface it discovers on a system. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
+ <para>See <xref linkend="configuringlnet"/> - Describes how to configure LNET if the default configuration is not sufficient. By default, LNET will use the first TCP/IP interface it discovers on a system. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
</listitem>
<listitem>
<para><emphasis>(Required)</emphasis> <emphasis role="bold">Configure Lustre.</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438192_pgfId-1307120" xreflabel=""/>See <xref linkend="configuringlustre"/> - Provides an example of a simple Lustre configuration procedure and points to tools for completing more complex configurations.</para>
+ <para>See <xref linkend="configuringlustre"/> - Provides an example of a simple Lustre configuration procedure and points to tools for completing more complex configurations.</para>
</listitem>
<listitem>
<para><emphasis>(Optional)</emphasis> <emphasis role="bold">Configure Lustre Failover.</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438192_pgfId-1307031" xreflabel=""/>See <xref linkend="configuringfailover"/> - Describes how to configure Lustre failover using the Heartbeat cluster infrastructure daemon.</para>
+ <para>See <xref linkend="configuringfailover"/> - Describes how to configure Lustre failover using the Heartbeat cluster infrastructure daemon.</para>
</listitem>
</orderedlist>
</section>
<title xml:id='installinglustre.title'>Installing the Lustre Software</title>
</info>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292576" xreflabel=""/>This chapter describes how to install the Lustre software. It includes:</para>
+ <para>This chapter describes how to install the Lustre software. It includes:</para>
<itemizedlist>
<listitem>
<para>A list of supported Lustre LNET drivers can be found in <xref linkend='understandinglustrenetworking'/>.</para>
<para>If you are not using a supported configuration, you can install Lustre directly from the source code. For more information on this installation method, see <xref linkend='installinglustrefromsourcecode'/>.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438261_pgfId-1292590" xreflabel=""/>8.1.1 <anchor xml:id="dbdoclet.50438261_37079" xreflabel=""/>Required Software</title>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292591" xreflabel=""/>To install Lustre, the following are required:</para>
+ <title>8.1.1 <anchor xml:id="dbdoclet.50438261_37079" xreflabel=""/>Required Software</title>
+ <para>To install Lustre, the following are required:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292592" xreflabel=""/><emphasis>(On Linux servers only)</emphasis> Linux kernel patched with Lustre-specific patches for your platform and architecture. A Linux patched kernel can be installed on a client if it is desireable to use the same kernel on all nodes, but this is not required.</para>
+ <para><emphasis>(On Linux servers only)</emphasis> Linux kernel patched with Lustre-specific patches for your platform and architecture. A Linux patched kernel can be installed on a client if it is desireable to use the same kernel on all nodes, but this is not required.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292593" xreflabel=""/> Lustre modules compiled for the Linux kernel (see <xref linkend='installinglustre.tab.req'/> for which packages are required for servers and clients in your configuration).</para>
+ <para> Lustre modules compiled for the Linux kernel (see <xref linkend='installinglustre.tab.req'/> for which packages are required for servers and clients in your configuration).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292594" xreflabel=""/> Lustre utilities required for configuring Lustre (see <xref linkend='installinglustre.tab.req'/> for which packages are required for servers and clients in your configuration).</para>
+ <para> Lustre utilities required for configuring Lustre (see <xref linkend='installinglustre.tab.req'/> for which packages are required for servers and clients in your configuration).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294336" xreflabel=""/><emphasis>(On Linux servers only)</emphasis>e2fsprogs package containing Lustre-specific tools (e2fsck and lfsck) used to repair a backing file system. This replaces the existing e2fsprogs package but provides complete e2fsprogs functionality</para>
+ <para><emphasis>(On Linux servers only)</emphasis>e2fsprogs package containing Lustre-specific tools (e2fsck and lfsck) used to repair a backing file system. This replaces the existing e2fsprogs package but provides complete e2fsprogs functionality</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294337" xreflabel=""/><emphasis>(Optional)</emphasis> Network-specific kernel modules and libraries such as the Lustre-specific OFED package required for an InfiniBand network interconnect.</para>
+ <para><emphasis>(Optional)</emphasis> Network-specific kernel modules and libraries such as the Lustre-specific OFED package required for an InfiniBand network interconnect.</para>
</listitem>
</itemizedlist>
-<para><anchor xml:id="dbdoclet.50438261_pgfId-1294922" xreflabel=""/>At least one Lustre RPM must be installed on each server and on each client in a Lustre file system. <xref linkend='installinglustre.tab.req'/> lists required Lustre packages and indicates where they are to be installed. Some Lustre packages are installed on Lustre servers (MGS, MDS, and OSSs), some are installed on Lustre clients, and some are installed on all Lustre nodes.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1296420" xreflabel=""/>.</para>
+<para>At least one Lustre RPM must be installed on each server and on each client in a Lustre file system. <xref linkend='installinglustre.tab.req'/> lists required Lustre packages and indicates where they are to be installed. Some Lustre packages are installed on Lustre servers (MGS, MDS, and OSSs), some are installed on Lustre clients, and some are installed on all Lustre nodes.</para>
+ <para>.</para>
<table xml:id='installinglustre.tab.req' frame="all">
<title>Lustre required packages, descriptions and installation guidance</title>
<colspec colname="c5" colwidth="6*"/>
<thead>
<row>
- <entry nameend="c2" namest="c1"><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438261_pgfId-1295330" xreflabel=""/><anchor xml:id="dbdoclet.50438261_97803" xreflabel=""/>Lustre Package</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438261_pgfId-1295334" xreflabel=""/>Description</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438261_pgfId-1295336" xreflabel=""/>Install on servers</emphasis>
- <footnote><para><anchor xml:id="dbdoclet.50438261_pgfId-1295562" xreflabel=""/>Installing a patched kernel on a client node is not required. However, if a client node will be used as both a client and a server, or if you want to install the same kernel on all nodes for any reason, install the server packages designated with an asterisk (*) on the client node.</para></footnote></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438261_pgfId-1295338" xreflabel=""/>Install on clients</emphasis></para></entry>
+ <entry nameend="c2" namest="c1"><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438261_97803" xreflabel=""/>Lustre Package</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Install on servers</emphasis>
+ <footnote><para>Installing a patched kernel on a client node is not required. However, if a client node will be used as both a client and a server, or if you want to install the same kernel on all nodes for any reason, install the server packages designated with an asterisk (*) on the client node.</para></footnote></para></entry>
+ <entry><para><emphasis role="bold">Install on clients</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry nameend="c3" namest="c1"><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295340" xreflabel=""/><emphasis role="bold"> Lustre patched kernel RPMs:</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295346" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295348" xreflabel=""/> </para></entry>
+ <entry nameend="c3" namest="c1"><para> <emphasis role="bold"> Lustre patched kernel RPMs:</emphasis></para></entry>
+ <entry><para> </para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295350" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295352" xreflabel=""/>kernel-<ver>_lustre.<ver></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295355" xreflabel=""/>For OEL 5 or RHEL 5 server platform</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295357" xreflabel=""/>X*</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295359" xreflabel=""/> </para></entry>
+ <entry><para> </para></entry>
+ <entry><para> kernel-<ver>_lustre.<ver></para></entry>
+ <entry><para> For OEL 5 or RHEL 5 server platform</para></entry>
+ <entry><para> X*</para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295415" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295417" xreflabel=""/>kernel-ib-<ver></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295419" xreflabel=""/>Lustre OFED package. Required only if the network interconnect is InfiniBand.</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295421" xreflabel=""/>X*</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295424" xreflabel=""/>X</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> kernel-ib-<ver></para></entry>
+ <entry><para> Lustre OFED package. Required only if the network interconnect is InfiniBand.</para></entry>
+ <entry><para> X*</para></entry>
+ <entry><para> X</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295427" xreflabel=""/><emphasis> Lustre module RPMs:</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295431" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295433" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295435" xreflabel=""/> </para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis> Lustre module RPMs:</emphasis></para></entry>
+ <entry><para> </para></entry>
+ <entry><para> </para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295437" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295439" xreflabel=""/>lustre-modules-<ver></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295441" xreflabel=""/>For Lustre-patched kernel.</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295443" xreflabel=""/>X*</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295445" xreflabel=""/> </para></entry>
+ <entry><para> </para></entry>
+ <entry><para> lustre-modules-<ver></para></entry>
+ <entry><para> For Lustre-patched kernel.</para></entry>
+ <entry><para> X*</para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295447" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295449" xreflabel=""/>lustre-client-modules-<ver></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295451" xreflabel=""/>For clients.</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295453" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295455" xreflabel=""/>X</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> lustre-client-modules-<ver></para></entry>
+ <entry><para> For clients.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> X</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295457" xreflabel=""/><emphasis> Lustre utilities:</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295461" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295463" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295465" xreflabel=""/> </para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis> Lustre utilities:</emphasis></para></entry>
+ <entry><para> </para></entry>
+ <entry><para> </para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295467" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295469" xreflabel=""/>lustre-<ver></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295471" xreflabel=""/>Lustre utilities package. This includes userspace utilities to configure and run Lustre.</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295473" xreflabel=""/> </para><para><anchor xml:id="dbdoclet.50438261_pgfId-1295474" xreflabel=""/> </para><para><anchor xml:id="dbdoclet.50438261_pgfId-1295475" xreflabel=""/>X*</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295477" xreflabel=""/> </para><para><anchor xml:id="dbdoclet.50438261_pgfId-1295478" xreflabel=""/> </para></entry>
+ <entry><para> </para></entry>
+ <entry><para> lustre-<ver></para></entry>
+ <entry><para> Lustre utilities package. This includes userspace utilities to configure and run Lustre.</para></entry>
+ <entry><para> </para><para> </para><para>X*</para></entry>
+ <entry><para> </para><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295480" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295482" xreflabel=""/>lustre-client-<ver></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295484" xreflabel=""/>Lustre utilities for clients.</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295486" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295488" xreflabel=""/>X</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> lustre-client-<ver></para></entry>
+ <entry><para> Lustre utilities for clients.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> X</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295490" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295492" xreflabel=""/>lustre-ldiskfs-<ver></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295494" xreflabel=""/>Lustre-patched backing file system kernel module package for the ldiskfs file system.</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295496" xreflabel=""/> </para><para><anchor xml:id="dbdoclet.50438261_pgfId-1295497" xreflabel=""/>X</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295499" xreflabel=""/> </para></entry>
+ <entry><para> </para></entry>
+ <entry><para> lustre-ldiskfs-<ver></para></entry>
+ <entry><para> Lustre-patched backing file system kernel module package for the ldiskfs file system.</para></entry>
+ <entry><para> </para><para>X</para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295501" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295503" xreflabel=""/>e2fsprogs-<ver></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295505" xreflabel=""/>Utilities package used to maintain the ldiskfs backing file system.</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295507" xreflabel=""/> </para><para><anchor xml:id="dbdoclet.50438261_pgfId-1295508" xreflabel=""/>X</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438261_pgfId-1295510" xreflabel=""/> </para></entry>
+ <entry><para> </para></entry>
+ <entry><para> e2fsprogs-<ver></para></entry>
+ <entry><para> Utilities package used to maintain the ldiskfs backing file system.</para></entry>
+ <entry><para> </para><para>X</para></entry>
+ <entry><para> </para></entry>
</row>
</tbody>
</tgroup>
</table>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1295527" xreflabel=""/>In all supported Lustre installations, a patched kernel must be run on each server, including the the MGS, the MDS, and all OSSs. Running a patched kernel on a Lustre client is only required if the client will be used for multiple purposes, such as running as both a client and an OST or if you want to use the same kernel on all nodes.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1293056" xreflabel=""/>Lustre RPM packages are available on the <link xl:href="http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L">Lustre download site</link>. They must be installed in the order described in the <link xl:href="InstallingLustre.html#50438261_41900">Lustre Installation Procedure</link>.</para>
+ <para>In all supported Lustre installations, a patched kernel must be run on each server, including the the MGS, the MDS, and all OSSs. Running a patched kernel on a Lustre client is only required if the client will be used for multiple purposes, such as running as both a client and an OST or if you want to use the same kernel on all nodes.</para>
+ <para>Lustre RPM packages are available on the <link xl:href="http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L">Lustre download site</link>. They must be installed in the order described in the <link xl:href="InstallingLustre.html#50438261_41900">Lustre Installation Procedure</link>.</para>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438261_pgfId-1292891" xreflabel=""/>8.1.1.1 Network-specific Kernel Modules and Libraries</title>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294344" xreflabel=""/>Network-specific kernel modules and libraries may be needed such as the Lustre-specific OFED package required for an InfiniBand network interconnect.</para>
+ <title>8.1.1.1 Network-specific Kernel Modules and Libraries</title>
+ <para>Network-specific kernel modules and libraries may be needed such as the Lustre-specific OFED package required for an InfiniBand network interconnect.</para>
</section>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438261_pgfId-1293747" xreflabel=""/>8.1.1.2 Lustre-Specific Tools <anchor xml:id="dbdoclet.50438261_marker-1292890" xreflabel=""/>and Utilities</title>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292892" xreflabel=""/>Several third-party utilities are must be installed on servers:</para>
+ <title>8.1.1.2 Lustre-Specific Tools <anchor xml:id="dbdoclet.50438261_marker-1292890" xreflabel=""/>and Utilities</title>
+ <para>Several third-party utilities are must be installed on servers:</para>
<itemizedlist><listitem>
- <para><
anchor xml:id="dbdoclet.50438261_pgfId-1292894" xreflabel=""/><emphasis role="bold">e2fsprogs</emphasis><anchor xml:id="dbdoclet.50438261_marker-1292893" xreflabel=""/> : Lustre requires a recent version of e2fsprogs that understands extents. Use e2fsprogs-1.41-10-sun2 or later, available at:</para>
+ <para><emphasis role="bold">e2fsprogs</emphasis><anchor xml:id="dbdoclet.50438261_marker-1292893" xreflabel=""/> : Lustre requires a recent version of e2fsprogs that understands extents. Use e2fsprogs-1.41-10-sun2 or later, available at:</para>
<para><link xl:href="http://downloads.lustre.org/public/tools/e2fsprogs/">http://downloads.lustre.org/public/tools/e2fsprogs/</link></para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292897" xreflabel=""/>A quilt patchset of all changes to the vanilla e2fsprogs is available in e2fsprogs-{version}-patches.tgz.</para>
+ <para>A quilt patchset of all changes to the vanilla e2fsprogs is available in e2fsprogs-{version}-patches.tgz.</para>
<note><para>The Lustre-patched e2fsprogs utility is only required on machines that mount backend (ldiskfs) file systems, such as the OSS, MDS and MGS nodes. It does not need to be loaded on clients.</para>
</note>
</listitem>
<listitem>
- <para><
anchor xml:id="dbdoclet.50438261_pgfId-1292900" xreflabel=""/><emphasis role="bold">Perl</emphasis><anchor xml:id="dbdoclet.50438261_marker-1292899" xreflabel=""/> - Various userspace utilities are written in Perl. Any recent version of Perl will work with Lustre.</para>
+ <para><emphasis role="bold">Perl</emphasis><anchor xml:id="dbdoclet.50438261_marker-1292899" xreflabel=""/> - Various userspace utilities are written in Perl. Any recent version of Perl will work with Lustre.</para>
</listitem>
</itemizedlist>
</section>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438261_pgfId-1292903" xreflabel=""/>8.1.1.3 <anchor xml:id="dbdoclet.50438261_29838" xreflabel=""/>(Optional) <anchor xml:id="dbdoclet.50438261_marker-1292902" xreflabel=""/>High-Availability Software</title>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292904" xreflabel=""/>If you plan to enable failover server functionality with Lustre (either on an OSS or the MDS), you must add high-availability (HA) software to your cluster software. For more information, see <xref linkend='configuringfailover'/>Setting up High-Availability (HA) Software with Lustre.</para>
+ <title>8.1.1.3 <anchor xml:id="dbdoclet.50438261_29838" xreflabel=""/>(Optional) <anchor xml:id="dbdoclet.50438261_marker-1292902" xreflabel=""/>High-Availability Software</title>
+ <para>If you plan to enable failover server functionality with Lustre (either on an OSS or the MDS), you must add high-availability (HA) software to your cluster software. For more information, see <xref linkend='configuringfailover'/>Setting up High-Availability (HA) Software with Lustre.</para>
</section>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438261_pgfId-1292911" xreflabel=""/>8.1.1.4 (Optional) Debugging Tools and Other Optional Packages</title>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292912" xreflabel=""/>A variety of optional packages are provided on the <emphasis><link xl:href="http://downloads.whamcloud.com">Whamcloud download site</link></emphasis>. These include debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294094" xreflabel=""/>For more information about debugging tools, see the topic <emphasis><link xl:href="http://wiki.lustre.org/index.php/Debugging_Lustre">Debugging Lustre</link></emphasis> on the Lustre wiki.</para>
+ <title>8.1.1.4 (Optional) Debugging Tools and Other Optional Packages</title>
+ <para>A variety of optional packages are provided on the <emphasis><link xl:href="http://downloads.whamcloud.com">Whamcloud download site</link></emphasis>. These include debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages.</para>
+ <para>For more information about debugging tools, see the topic <emphasis><link xl:href="http://wiki.lustre.org/index.php/Debugging_Lustre">Debugging Lustre</link></emphasis> on the Lustre wiki.</para>
</section>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438261_pgfId-1292915" xreflabel=""/>8.1.2 <anchor xml:id="dbdoclet.50438261_19503" xreflabel=""/>Environmental <anchor xml:id="dbdoclet.50438261_marker-1292914" xreflabel=""/>Requirements</title>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292916" xreflabel=""/>Make sure the following environmental requirements are met before installing Lustre:</para>
+ <title>8.1.2 <anchor xml:id="dbdoclet.50438261_19503" xreflabel=""/>Environmental <anchor xml:id="dbdoclet.50438261_marker-1292914" xreflabel=""/>Requirements</title>
+ <para>Make sure the following environmental requirements are met before installing Lustre:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294402" xreflabel=""/><emphasis>(Required)</emphasis><emphasis role="bold">Disable Security-Enhanced Linux (SELinux) on servers and clients</emphasis> . Lustre does not support SELinux. Therefore, disable the SELinux system extension on all Lustre nodes and make sure other security extensions, like Novell AppArmorand network packet filtering tools (such as iptables) do not interfere with Lustre. See <link xl:href="InstallingLustre.html#50438261_88296">Step 3</link> in the <link xl:href="InstallingLustre.html#50438261_41900">Lustre Installation Procedure</link> below.</para>
+ <para><emphasis>(Required)</emphasis><emphasis role="bold">Disable Security-Enhanced Linux (SELinux) on servers and clients</emphasis> . Lustre does not support SELinux. Therefore, disable the SELinux system extension on all Lustre nodes and make sure other security extensions, like Novell AppArmorand network packet filtering tools (such as iptables) do not interfere with Lustre. See <link xl:href="InstallingLustre.html#50438261_88296">Step 3</link> in the <link xl:href="InstallingLustre.html#50438261_41900">Lustre Installation Procedure</link> below.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294954" xreflabel=""/><emphasis>(Required)</emphasis><emphasis role="bold">Maintain uniform user and group databases on all cluster nodes</emphasis> . Use the same user IDs (UID) and group IDs (GID) on all clients. If use of supplemental groups is required, verify that the group_upcall requirements have been met. See <link xl:href="LustreProgrammingInterfaces.html#50438291_32926">User/Group Cache Upcall</link>.</para>
+ <para><emphasis>(Required)</emphasis><emphasis role="bold">Maintain uniform user and group databases on all cluster nodes</emphasis> . Use the same user IDs (UID) and group IDs (GID) on all clients. If use of supplemental groups is required, verify that the group_upcall requirements have been met. See <link xl:href="LustreProgrammingInterfaces.html#50438291_32926">User/Group Cache Upcall</link>.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292917" xreflabel=""/><emphasis>(Recommended)</emphasis><emphasis role="bold">Provide remote shell access to clients</emphasis> . Although not strictly required to run Lustre, we recommend that all cluster nodes have remote shell client access to facilitate the use of Lustre configuration and monitoring scripts. Parallel Distributed SHell (pdsh) is preferable, although Secure SHell (SSH) is acceptable.</para>
+ <para><emphasis>(Recommended)</emphasis><emphasis role="bold">Provide remote shell access to clients</emphasis> . Although not strictly required to run Lustre, we recommend that all cluster nodes have remote shell client access to facilitate the use of Lustre configuration and monitoring scripts. Parallel Distributed SHell (pdsh) is preferable, although Secure SHell (SSH) is acceptable.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292918" xreflabel=""/><emphasis>(Recommended)</emphasis><emphasis role="bold">Ensure client clocks are synchronized</emphasis> . Lustre uses client clocks for timestamps. If clocks are out of sync between clients, files will appear with different time stamps when accessed by different clients. Drifting clocks can also cause problems, for example, by making it difficult to debug multi-node issues or correlate logs, which depend on timestamps. We recommend that you use Network Time Protocol (NTP) to keep client and server clocks in sync with each other. For more information about NTP, see: <link xl:href="http://www.ntp.org/">http://www.ntp.org</link>.</para>
+ <para><emphasis>(Recommended)</emphasis><emphasis role="bold">Ensure client clocks are synchronized</emphasis> . Lustre uses client clocks for timestamps. If clocks are out of sync between clients, files will appear with different time stamps when accessed by different clients. Drifting clocks can also cause problems, for example, by making it difficult to debug multi-node issues or correlate logs, which depend on timestamps. We recommend that you use Network Time Protocol (NTP) to keep client and server clocks in sync with each other. For more information about NTP, see: <link xl:href="http://www.ntp.org/">http://www.ntp.org</link>.</para>
</listitem>
</itemizedlist>
</section>
<para>Before installing Lustre, back up ALL data. Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured or administered properly.</para>
</caution>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292927" xreflabel=""/>Use this procedure to install Lustre from RPMs.</para>
+ <para>Use this procedure to install Lustre from RPMs.</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292928" xreflabel=""/>Verify that all Lustre installation requirements have been met.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292929" xreflabel=""/>For more information on these prerequisites, see:</para>
+ <para>Verify that all Lustre installation requirements have been met.</para>
+ <para>For more information on these prerequisites, see:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292930" xreflabel=""/> Hardware requirements in <xref linkend='settinguplustresystem'/></para>
+ <para> Hardware requirements in <xref linkend='settinguplustresystem'/></para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292931" xreflabel=""/> Software and environmental requirements in <xref linkend='installinglustre'/></para>
+ <para> Software and environmental requirements in <xref linkend='installinglustre'/></para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292934" xreflabel=""/>Download the Lustre RPMs.</para>
+ <para>Download the Lustre RPMs.</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292936" xreflabel=""/>On the <link xl:href='http://build.whamcloud.com/'>Lustre download site</link>, select your platform.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292937" xreflabel=""/>The files required to install Lustre (kernels, modules and utilities RPMs) are listed for the selected platform.</para>
+ <para>On the <link xl:href='http://build.whamcloud.com/'>Lustre download site</link>, select your platform.</para>
+ <para>The files required to install Lustre (kernels, modules and utilities RPMs) are listed for the selected platform.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292938" xreflabel=""/>Download the required files.</para>
+ <para>Download the required files.</para>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292940" xreflabel=""/>Install the Lustre packages on the servers.</para>
+ <para>Install the Lustre packages on the servers.</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292942" xreflabel=""/>Refer to <xref linkend='installinglustre.tab.req'/> to determine which Lustre packages are to be installed on servers for your platform and architecture.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1293257" xreflabel=""/>Some Lustre packages are installed on the Lustre servers (MGS, MDS, and OSSs). Others are installed on Lustre clients.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1293274" xreflabel=""/>Lustre packages must be installed in the order specified in the following steps.</para>
+ <para>Refer to <xref linkend='installinglustre.tab.req'/> to determine which Lustre packages are to be installed on servers for your platform and architecture.</para>
+ <para>Some Lustre packages are installed on the Lustre servers (MGS, MDS, and OSSs). Others are installed on Lustre clients.</para>
+ <para>Lustre packages must be installed in the order specified in the following steps.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292946" xreflabel=""/>Install the kernel, modules and ldiskfs packages.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292947" xreflabel=""/>Use the rpm -ivh command to install the kernel, module and ldiskfs packages.</para>
+ <para>Install the kernel, modules and ldiskfs packages.</para>
+ <para>Use the rpm -ivh command to install the kernel, module and ldiskfs packages.</para>
<note>
<para>It is not recommended that you use the rpm -Uvh command to install a kernel, because this may leave you with an unbootable system if the new kernel doesn't work for some reason.</para>
</note>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294565" xreflabel=""/>For example, the command in the following example would install required packages on a server with Infiniband networking</para>
- <screen><anchor xml:id="dbdoclet.50438261_pgfId-1292948" xreflabel=""/>$ rpm -ivh kernel-<ver>_lustre-<ver> kernel-ib-<ver> \
-<anchor xml:id="dbdoclet.50438261_pgfId-1292950" xreflabel=""/>lustre-modules-<ver> lustre-ldiskfs-<ver>
+ <para>For example, the command in the following example would install required packages on a server with Infiniband networking</para>
+ <screen>$ rpm -ivh kernel-<ver>_lustre-<ver> kernel-ib-<ver> \
+lustre-modules-<ver> lustre-ldiskfs-<ver>
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1295656" xreflabel=""/>Update the bootloader (grub.conf or lilo.conf) configuration file as needed.</para>
+ <para>Update the bootloader (grub.conf or lilo.conf) configuration file as needed.</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1295665" xreflabel=""/><emphasis role="bold">Verify that the bootloader configuration file has been updated with an entry for the patched kernel.</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1295636" xreflabel=""/>Before you can boot to a new distribution or kernel, there must be an entry for it in the bootloader configuration file. Often this is added automatically when the kernel RPM is installed.</para>
+ <para><emphasis role="bold">Verify that the bootloader configuration file has been updated with an entry for the patched kernel.</emphasis></para>
+ <para>Before you can boot to a new distribution or kernel, there must be an entry for it in the bootloader configuration file. Often this is added automatically when the kernel RPM is installed.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1295650" xreflabel=""/><emphasis role="bold">Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438261_pgfId-1295651" xreflabel=""/>selinux=0
+ <para><emphasis role="bold">Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below:</emphasis></para>
+ <screen>selinux=0
</screen>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292952" xreflabel=""/>Install the utilities/userspace packages.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292953" xreflabel=""/>Use the rpm -ivh command to install the utilities packages. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438261_pgfId-1292954" xreflabel=""/>$ rpm -ivh lustre-<ver>
+ <para>Install the utilities/userspace packages.</para>
+ <para>Use the rpm -ivh command to install the utilities packages. For example:</para>
+ <screen>$ rpm -ivh lustre-<ver>
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292955" xreflabel=""/>Install the e2fsprogs package.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292956" xreflabel=""/>Use the rpm -ivh command to install the e2fsprogs package. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438261_pgfId-1292957" xreflabel=""/>$ rpm -ivh e2fsprogs-<ver>
-<anchor xml:id="dbdoclet.50438261_pgfId-1292958" xreflabel=""/>
+ <para>Install the e2fsprogs package.</para>
+ <para>Use the rpm -ivh command to install the e2fsprogs package. For example:</para>
+ <screen>$ rpm -ivh e2fsprogs-<ver>
+
</screen>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292959" xreflabel=""/>If e2fsprogs is already installed on your Linux system, install the Lustre-specific e2fsprogs version by using rpm -Uvh to upgrade the existing e2fsprogs package. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438261_pgfId-1295617" xreflabel=""/>$ rpm -Uvh e2fsprogs-<ver> </screen>
+ <para>If e2fsprogs is already installed on your Linux system, install the Lustre-specific e2fsprogs version by using rpm -Uvh to upgrade the existing e2fsprogs package. For example:</para>
+ <screen>$ rpm -Uvh e2fsprogs-<ver> </screen>
<informaltable frame="none">
<tgroup cols="1">
<colspec colname="c1" colwidth="100*"/>
<tbody>
<row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438261_pgfId-1293343" xreflabel=""/>The rpm command options --force or --nodeps should not be used to install or update the Lustre-specific e2fsprogs package. If errors are reported, file a bug (for instructions see the topic <emphasis><link xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</link></emphasis> on the Lustre wiki).</para></entry>
+ <entry><para><emphasis role="bold">Note -</emphasis>The rpm command options --force or --nodeps should not be used to install or update the Lustre-specific e2fsprogs package. If errors are reported, file a bug (for instructions see the topic <emphasis><link xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</link></emphasis> on the Lustre wiki).</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292960" xreflabel=""/>(Optional) To add optional packages to your Lustre file system, install them now.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292961" xreflabel=""/>Optional packages include file system creation and repair tools, debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages. A complete list of optional packages for your platform is provided on the <link xl:href="http://wiki.whamcloud.com">Whamcloud wiki</link>.</para>
+ <para>(Optional) To add optional packages to your Lustre file system, install them now.</para>
+ <para>Optional packages include file system creation and repair tools, debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages. A complete list of optional packages for your platform is provided on the <link xl:href="http://wiki.whamcloud.com">Whamcloud wiki</link>.</para>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294441" xreflabel=""/>Install the Lustre packages on the clients.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294445" xreflabel=""/>Refer to <xref linkend='installinglustre.tab.req'/> to determine which Lustre packages are to be installed on clients for your platform and architecture.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294458" xreflabel=""/>Install the module packages for clients.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294459" xreflabel=""/>Use the rpm -ivh command to install the lustre-client and lustre-client-modules-<ver> packages. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438261_pgfId-1294468" xreflabel=""/>$ rpm -ivh lustre-client-modules-<ver> kernel-ib-<ver>
+ <para>Install the Lustre packages on the clients.</para>
+ <para>Refer to <xref linkend='installinglustre.tab.req'/> to determine which Lustre packages are to be installed on clients for your platform and architecture.</para>
+ <para>Install the module packages for clients.</para>
+ <para>Use the rpm -ivh command to install the lustre-client and lustre-client-modules-<ver> packages. For example:</para>
+ <screen>$ rpm -ivh lustre-client-modules-<ver> kernel-ib-<ver>
</screen>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294521" xreflabel=""/>Install the utilities/userspace packages for clients.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1294522" xreflabel=""/>Use the rpm -ivh command to install the utilities packages. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438261_pgfId-1294532" xreflabel=""/>$ rpm -ivh lustre-client
+ <para>Install the utilities/userspace packages for clients.</para>
+ <para>Use the rpm -ivh command to install the utilities packages. For example:</para>
+ <screen>$ rpm -ivh lustre-client
</screen>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1295693" xreflabel=""/>Update the bootloader (grub.conf or lilo.conf) configuration file as needed.</para>
+ <para>Update the bootloader (grub.conf or lilo.conf) configuration file as needed.</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1295694" xreflabel=""/><emphasis role="bold">Verify that the bootloader configuration file has been updated with an entry for the patched kernel.</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1295695" xreflabel=""/>Before you can boot to a new distribution or kernel, there must be an entry for it in the bootloader configuration file. Often this is added automatically when the kernel RPM is installed.</para>
+ <para><emphasis role="bold">Verify that the bootloader configuration file has been updated with an entry for the patched kernel.</emphasis></para>
+ <para>Before you can boot to a new distribution or kernel, there must be an entry for it in the bootloader configuration file. Often this is added automatically when the kernel RPM is installed.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1295696" xreflabel=""/><emphasis role="bold">Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438261_pgfId-1295697" xreflabel=""/>selinux=0
+ <para><emphasis role="bold">Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below:</emphasis></para>
+ <screen>selinux=0
</screen>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292964" xreflabel=""/>Reboot the patched clients and the servers.</para>
+ <para>Reboot the patched clients and the servers.</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292965" xreflabel=""/>If you applied the patched kernel to any clients, reboot them.</para>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292966" xreflabel=""/>Unpatched clients do not need to be rebooted.</para>
+ <para>If you applied the patched kernel to any clients, reboot them.</para>
+ <para>Unpatched clients do not need to be rebooted.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292967" xreflabel=""/>Reboot the servers.</para>
+ <para>Reboot the servers.</para>
</listitem>
</orderedlist>
- <para><anchor xml:id="dbdoclet.50438261_pgfId-1292971" xreflabel=""/>To configure LNET, go next to <xref linkend='configuringlnet'/>. If default settings will be used for LNET, go to <xref linkend='configuringlustre'/>.</para>
+ <para>To configure LNET, go next to <xref linkend='configuringlnet'/>. If default settings will be used for LNET, go to <xref linkend='configuringlustre'/>.</para>
</listitem>
</orderedlist>
</section>
<info>
<title xml:id='installinglustrefromsourcecode.title'>Installing Lustre from Source Code</title>
</info>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1304664" xreflabel=""/>If you need to build a customized Lustre server kernel or are using a Linux kernel that has not been tested with the version of Lustre you are installing, you may need to build and install Lustre from source code. This chapter describes:</para>
+ <para>If you need to build a customized Lustre server kernel or are using a Linux kernel that has not been tested with the version of Lustre you are installing, you may need to build and install Lustre from source code. This chapter describes:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438210_69313"/></para>
<section xml:id="dbdoclet.50438210_69313">
<title>29.1 Overview and Prerequisites</title>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1305142" xreflabel=""/>Lustre can be installed from either pre-built binary packages (RPMs) or freely-available source code. Installing from the package release is recommended unless you need to customize the Lustre server kernel or will be using an Linux kernel that has not been tested with Lustre. For a list of supported Linux distributions and architectures, see the topic <link xl:href="http://wiki.lustre.org/index.php/Lustre_2.0">Lustre_2.0</link> on the Lustre wiki. The procedure for installing Lustre from RPMs is describe in <link xl:href="InstallingLustre.html#50438261_81829">Chapter 8</link>: <link xl:href="InstallingLustre.html#50438261_62973">Installing the Lustre Software</link>.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297213" xreflabel=""/>To install Lustre from source code, the following are required:</para>
+ <para>Lustre can be installed from either pre-built binary packages (RPMs) or freely-available source code. Installing from the package release is recommended unless you need to customize the Lustre server kernel or will be using an Linux kernel that has not been tested with Lustre. For a list of supported Linux distributions and architectures, see the topic <link xl:href="http://wiki.lustre.org/index.php/Lustre_2.0">Lustre_2.0</link> on the Lustre wiki. The procedure for installing Lustre from RPMs is describe in <link xl:href="InstallingLustre.html#50438261_81829">Chapter 8</link>: <link xl:href="InstallingLustre.html#50438261_62973">Installing the Lustre Software</link>.</para>
+ <para>To install Lustre from source code, the following are required:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297214" xreflabel=""/> Linux kernel patched with Lustre-specific patches</para>
+ <para> Linux kernel patched with Lustre-specific patches</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297215" xreflabel=""/> Lustre modules compiled for the Linux kernel</para>
+ <para> Lustre modules compiled for the Linux kernel</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297216" xreflabel=""/> Lustre utilities required for Lustre configuration</para>
+ <para> Lustre utilities required for Lustre configuration</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1304577" xreflabel=""/>The installation procedure involves several steps:</para>
+ <para>The installation procedure involves several steps:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1302403" xreflabel=""/> Patching the core kernel</para>
+ <para> Patching the core kernel</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1302413" xreflabel=""/> Configuring the kernel to work with Lustre</para>
+ <para> Configuring the kernel to work with Lustre</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1302420" xreflabel=""/> Creating Lustre and kernel RPMs from source code.</para>
+ <para> Creating Lustre and kernel RPMs from source code.</para>
</listitem>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438210_65411">
<title>29.2 Patching the Kernel</title>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296830" xreflabel=""/>If you are using non-standard hardware, plan to apply a Lustre patch, or have another reason not to use packaged Lustre binaries, you have to apply several Lustre patches to the core kernel and run the Lustre configure script against the kernel.</para>
+ <para>If you are using non-standard hardware, plan to apply a Lustre patch, or have another reason not to use packaged Lustre binaries, you have to apply several Lustre patches to the core kernel and run the Lustre configure script against the kernel.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438210_pgfId-1296831" xreflabel=""/>29.2.1 Introducing the Quilt Utility</title>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296832" xreflabel=""/>To simplify the process of applying Lustre patches to the kernel, we recommend that you use the Quilt utility.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296833" xreflabel=""/>Quilt manages a stack of patches on a single source tree. A series file lists the patch files and the order in which they are applied. Patches are applied, incrementally, on the base tree and all preceding patches. You can:</para>
+ <title>29.2.1 Introducing the Quilt Utility</title>
+ <para>To simplify the process of applying Lustre patches to the kernel, we recommend that you use the Quilt utility.</para>
+ <para>Quilt manages a stack of patches on a single source tree. A series file lists the patch files and the order in which they are applied. Patches are applied, incrementally, on the base tree and all preceding patches. You can:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1302485" xreflabel=""/> Apply patches from the stack (quilt push)</para>
+ <para> Apply patches from the stack (quilt push)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1302495" xreflabel=""/> Remove patches from the stack (quilt pop)</para>
+ <para> Remove patches from the stack (quilt pop)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1302502" xreflabel=""/> Query the contents of the series file (quilt series), the contents of the stack (quilt applied, quilt previous, quilt top), and the patches that are not applied at a particular moment (quilt next, quilt unapplied).</para>
+ <para> Query the contents of the series file (quilt series), the contents of the stack (quilt applied, quilt previous, quilt top), and the patches that are not applied at a particular moment (quilt next, quilt unapplied).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1302509" xreflabel=""/> Edit and refresh (update) patches with Quilt, as well as revert inadvertent changes, and fork or clone patches and show the diffs before and after work.</para>
+ <para> Edit and refresh (update) patches with Quilt, as well as revert inadvertent changes, and fork or clone patches and show the diffs before and after work.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1302510" xreflabel=""/>A variety of Quilt packages (RPMs, SRPMs and tarballs) are available from various sources. Use the most recent version you can find. Quilt depends on several other utilities, e.g., the coreutils RPM that is only available in RedHat 9. For other RedHat kernels, you have to get the required packages to successfully install Quilt. If you cannot locate a Quilt package or fulfill its dependencies, you can build Quilt from a tarball, available at the Quilt project website:</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296837" xreflabel=""/><link xl:href="http://savannah.nongnu.org/projects/quilt">http://savannah.nongnu.org/projects/quilt</link></para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296838" xreflabel=""/>For additional information on using Quilt, including its commands, see <link xl:href="http://www.suse.de/~agruen/quilt.pdf">Introduction to Quilt</link> and the <link xl:href="http://linux.die.net/man/1/quilt">quilt(1) man page.</link></para>
+ <para>A variety of Quilt packages (RPMs, SRPMs and tarballs) are available from various sources. Use the most recent version you can find. Quilt depends on several other utilities, e.g., the coreutils RPM that is only available in RedHat 9. For other RedHat kernels, you have to get the required packages to successfully install Quilt. If you cannot locate a Quilt package or fulfill its dependencies, you can build Quilt from a tarball, available at the Quilt project website:</para>
+ <para><link xl:href="http://savannah.nongnu.org/projects/quilt">http://savannah.nongnu.org/projects/quilt</link></para>
+ <para>For additional information on using Quilt, including its commands, see <link xl:href="http://www.suse.de/~agruen/quilt.pdf">Introduction to Quilt</link> and the <link xl:href="http://linux.die.net/man/1/quilt">quilt(1) man page.</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438210_pgfId-1296842" xreflabel=""/>29.2.2 <anchor xml:id="dbdoclet.50438210_44148" xreflabel=""/>Get the Lustre Source and Unpatched Kernel</title>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1302564" xreflabel=""/>The Lustre Engineering Team has targeted several Linux kernels for use with Lustre servers (MDS/OSS) and provides a series of patches for each one. The Lustre patches are maintained in the kernel_patch directory bundled with the Lustre source code.</para>
+ <title>29.2.2 <anchor xml:id="dbdoclet.50438210_44148" xreflabel=""/>Get the Lustre Source and Unpatched Kernel</title>
+ <para>The Lustre Engineering Team has targeted several Linux kernels for use with Lustre servers (MDS/OSS) and provides a series of patches for each one. The Lustre patches are maintained in the kernel_patch directory bundled with the Lustre source code.</para>
<caution><para>Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured and administered correctly. Before installing Lustre, be cautious and back up ALL data.</para></caution>
<note><para>Each patch series has been tailored to a specific kernel version, and may or may not apply cleanly to other versions of the kernel.</para></note>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1305199" xreflabel=""/>To obtain the Lustre source and unpatched kernel:</para>
+ <para>To obtain the Lustre source and unpatched kernel:</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1305200" xreflabel=""/>Verify that all of the Lustre installation requirements have been met.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1305210" xreflabel=""/>For more information on these prerequisites, see:</para>
+ <para>Verify that all of the Lustre installation requirements have been met.</para>
+ <para>For more information on these prerequisites, see:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1305217" xreflabel=""/> Hardware requirements in <link xl:href="SettingUpLustreSystem.html#50438256_38751">Chapter 5</link>: <link xl:href="SettingUpLustreSystem.html#50438256_66186">Setting Up a Lustre File System</link></para>
+ <para> Hardware requirements in <link xl:href="SettingUpLustreSystem.html#50438256_38751">Chapter 5</link>: <link xl:href="SettingUpLustreSystem.html#50438256_66186">Setting Up a Lustre File System</link></para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1305220" xreflabel=""/> Software and environmental requirements in <link xl:href="InstallingLustre.html#50438261_99193">Preparing to Install the Lustre Software</link></para>
+ <para> Software and environmental requirements in <link xl:href="InstallingLustre.html#50438261_99193">Preparing to Install the Lustre Software</link></para>
</listitem>
</itemizedlist>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1305206" xreflabel=""/>Download the Lustre source code.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1305253" xreflabel=""/>On the <link xl:href="http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L">Lustre download site</link>, select a version of Lustre to download and then select 'Source' as the platform.</para>
+ <para>Download the Lustre source code.</para>
+ <para>On the <link xl:href="http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L">Lustre download site</link>, select a version of Lustre to download and then select 'Source' as the platform.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296849" xreflabel=""/>Download the unpatched kernel.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296857" xreflabel=""/>For convenience, Oracle maintains an archive of unpatched kernel sources at:</para>
- <para><
anchor xml:id="dbdoclet.50438210_pgfId-1305277" xreflabel=""/><link xl:href="http://downloads.lustre.org/public/kernels/">http://downloads.lustre.org/public/kernels/</link><anchor xml:id="dbdoclet.50438210_28487" xreflabel=""/></para>
+ <para>Download the unpatched kernel.</para>
+ <para>For convenience, Oracle maintains an archive of unpatched kernel sources at:</para>
+ <para><link xl:href="http://downloads.lustre.org/public/kernels/">http://downloads.lustre.org/public/kernels/</link><anchor xml:id="dbdoclet.50438210_28487" xreflabel=""/></para>
</listitem></orderedlist>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438210_pgfId-1305280" xreflabel=""/>29.2.3 Patch the Kernel</title>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1305429" xreflabel=""/>This procedure describes how to use Quilt to apply the Lustre patches to the kernel. To illustrate the steps in this procedure, a RHEL 5 kernel is patched for Lustre 1.6.5.1.</para>
+ <title>29.2.3 Patch the Kernel</title>
+ <para>This procedure describes how to use Quilt to apply the Lustre patches to the kernel. To illustrate the steps in this procedure, a RHEL 5 kernel is patched for Lustre 1.6.5.1.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1305430" xreflabel=""/>Unpack the Lustre source and kernel to separate source trees.</para>
+ <para>Unpack the Lustre source and kernel to separate source trees.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296865" xreflabel=""/>Unpack the Lustre source.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296866" xreflabel=""/>For this procedure, we assume that the resulting source tree is in /tmp/lustre-1.6.5.1</para>
+ <para>Unpack the Lustre source.</para>
+ <para>For this procedure, we assume that the resulting source tree is in /tmp/lustre-1.6.5.1</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296867" xreflabel=""/>Unpack the kernel.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296868" xreflabel=""/>For this procedure, we assume that the resulting source tree (also known as the destination tree) is in /tmp/kernels/linux-2.6.18</para>
+ <para>Unpack the kernel.</para>
+ <para>For this procedure, we assume that the resulting source tree (also known as the destination tree) is in /tmp/kernels/linux-2.6.18</para>
</listitem></orderedlist>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296869" xreflabel=""/>Select a config file for your kernel, located in the kernel_configs directory (lustre/kernel_patches/kernel_config).</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296870" xreflabel=""/>The kernel_config directory contains the .config files, which are named to indicate the kernel and architecture with which they are associated. For example, the configuration file for the 2.6.18 kernel shipped with RHEL 5 (suitable for i686 SMP systems) is kernel-2.6.18-2.6-rhel5-i686-smp.config.</para>
+ <para>Select a config file for your kernel, located in the kernel_configs directory (lustre/kernel_patches/kernel_config).</para>
+ <para>The kernel_config directory contains the .config files, which are named to indicate the kernel and architecture with which they are associated. For example, the configuration file for the 2.6.18 kernel shipped with RHEL 5 (suitable for i686 SMP systems) is kernel-2.6.18-2.6-rhel5-i686-smp.config.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296871" xreflabel=""/>Select the series file for your kernel, located in the series directory (lustre/kernel_patches/series).</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296872" xreflabel=""/>The series file contains the patches that need to be applied to the kernel.</para>
+ <para>Select the series file for your kernel, located in the series directory (lustre/kernel_patches/series).</para>
+ <para>The series file contains the patches that need to be applied to the kernel.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296873" xreflabel=""/>Set up the necessary symlinks between the kernel patches and the Lustre source.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296874" xreflabel=""/>This example assumes that the Lustre source files are unpacked under /tmp/lustre-1.6.5.1 and you have chosen the 2.6-rhel5.series file). Run:</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1296875" xreflabel=""/>$ cd /tmp/kernels/linux-2.6.18
-<anchor xml:id="dbdoclet.50438210_pgfId-1296876" xreflabel=""/>$ rm -f patches series
-<anchor xml:id="dbdoclet.50438210_pgfId-1296877" xreflabel=""/>$ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/series/2.6-\ rhel5.series\
+ <para>Set up the necessary symlinks between the kernel patches and the Lustre source.</para>
+ <para>This example assumes that the Lustre source files are unpacked under /tmp/lustre-1.6.5.1 and you have chosen the 2.6-rhel5.series file). Run:</para>
+ <screen>$ cd /tmp/kernels/linux-2.6.18
+$ rm -f patches series
+$ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/series/2.6-\ rhel5.series\
./series
-<anchor xml:id="dbdoclet.50438210_pgfId-1296878" xreflabel=""/>$ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/patches .
+$ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/patches .
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296879" xreflabel=""/>Use Quilt to apply the patches in the selected series file to the unpatched kernel. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1296880" xreflabel=""/>$ cd /tmp/kernels/linux-2.6.18
-<anchor xml:id="dbdoclet.50438210_pgfId-1296881" xreflabel=""/>$ quilt push -av
+ <para>Use Quilt to apply the patches in the selected series file to the unpatched kernel. Run:</para>
+ <screen>$ cd /tmp/kernels/linux-2.6.18
+$ quilt push -av
</screen>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296882" xreflabel=""/>The patched destination tree acts as a base Linux source tree for Lustre.</para>
+ <para>The patched destination tree acts as a base Linux source tree for Lustre.</para>
</listitem></orderedlist>
</section>
</section>
<section xml:id="dbdoclet.50438210_47529">
<title>29.3 Creating and Installing the Lustre Packages</title>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1296886" xreflabel=""/>After patching the kernel, configure it to work with Lustre, create the Lustre packages (RPMs) and install them.</para>
+ <para>After patching the kernel, configure it to work with Lustre, create the Lustre packages (RPMs) and install them.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297162" xreflabel=""/>Configure the patched kernel to run with Lustre. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1297163" xreflabel=""/>$ cd <path to kernel tree>
-<anchor xml:id="dbdoclet.50438210_pgfId-1297079" xreflabel=""/>$ cp /boot/config-`uname -r` .config
-<anchor xml:id="dbdoclet.50438210_pgfId-1297080" xreflabel=""/>$ make oldconfig || make menuconfig
-<anchor xml:id="dbdoclet.50438210_pgfId-1297083" xreflabel=""/>$ make include/asm
-<anchor xml:id="dbdoclet.50438210_pgfId-1297084" xreflabel=""/>$ make include/linux/version.h
-<anchor xml:id="dbdoclet.50438210_pgfId-1297085" xreflabel=""/>$ make SUBDIRS=scripts
-<anchor xml:id="dbdoclet.50438210_pgfId-1297086" xreflabel=""/>$ make include/linux/utsrelease.h
+ <para>Configure the patched kernel to run with Lustre. Run:</para>
+ <screen>$ cd <path to kernel tree>
+$ cp /boot/config-`uname -r` .config
+$ make oldconfig || make menuconfig
+$ make include/asm
+$ make include/linux/version.h
+$ make SUBDIRS=scripts
+$ make include/linux/utsrelease.h
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297087" xreflabel=""/>Run the Lustre configure script against the patched kernel and create the Lustre packages.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1297088" xreflabel=""/>$ cd <path to lustre source tree>
-<anchor xml:id="dbdoclet.50438210_pgfId-1297089" xreflabel=""/>$ ./configure --with-linux=<path to kernel tree>
-<anchor xml:id="dbdoclet.50438210_pgfId-1297090" xreflabel=""/>$ make rpms
+ <para>Run the Lustre configure script against the patched kernel and create the Lustre packages.</para>
+ <screen>$ cd <path to lustre source tree>
+$ ./configure --with-linux=<path to kernel tree>
+$ make rpms
</screen>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297091" xreflabel=""/>This creates a set of .rpms in /usr/src/redhat/RPMS/<arch> with an appended date-stamp. The SuSE path is /usr/src/packages.</para>
+ <para>This creates a set of .rpms in /usr/src/redhat/RPMS/<arch> with an appended date-stamp. The SuSE path is /usr/src/packages.</para>
<note><para>You do not need to run the Lustre configure script against an unpatched kernel.</para></note>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297092" xreflabel=""/><emphasis role="bold">Example set of RPMs:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1297093" xreflabel=""/>lustre-1.6.5.1-\2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i686.rpm
-<anchor xml:id="dbdoclet.50438210_pgfId-1297094" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438210_pgfId-1297095" xreflabel=""/>lustre-debuginfo-1.6.5.1-\2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_2008102\
+ <para><emphasis role="bold">Example set of RPMs:</emphasis></para>
+ <screen>lustre-1.6.5.1-\2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i686.rpm
+
+lustre-debuginfo-1.6.5.1-\2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_2008102\
1.i686.rpm
-<anchor xml:id="dbdoclet.50438210_pgfId-1297096" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438210_pgfId-1297097" xreflabel=""/>lustre-modules-1.6.5.1-\2.6.18_53.xx.xxel5_lustre.1.6.5.1.custom_20081021.i\
+
+lustre-modules-1.6.5.1-\2.6.18_53.xx.xxel5_lustre.1.6.5.1.custom_20081021.i\
686.rpm
-<anchor xml:id="dbdoclet.50438210_pgfId-1297098" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438210_pgfId-1297099" xreflabel=""/>lustre-source-1.6.5.1-\2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i\
+
+lustre-source-1.6.5.1-\2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i\
686.rpm
</screen>
<note><para>If the steps to create the RPMs fail, contact Lustre Support by reporting a bug. See <link xl:href="LustreTroubleshooting.html#50438198_30989">Reporting a Lustre Bug</link>.</para></note>
<note><para>Several features and packages are available that extend the core functionality of Lustre. These features/packages can be enabled at the build time by issuing appropriate arguments to the configure command. For a list of these features and packages, run ./configure -help in the Lustre source tree. The configs/ directory of the kernel source contains the config files matching each the kernel version. Copy one to .config at the root of the kernel tree.</para></note>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297103" xreflabel=""/><anchor xml:id="dbdoclet.50438210_41207" xreflabel=""/>Create the kernel package. Navigate to the kernel source directory and run:</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1297104" xreflabel=""/>$ make rpm
+ <para><anchor xml:id="dbdoclet.50438210_41207" xreflabel=""/>Create the kernel package. Navigate to the kernel source directory and run:</para>
+ <screen>$ make rpm
</screen>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297106" xreflabel=""/>Example result:</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1297107" xreflabel=""/>kernel-2.6.95.0.3.EL_lustre.1.6.5.1custom-1.i686.rpm
+ <para>Example result:</para>
+ <screen>kernel-2.6.95.0.3.EL_lustre.1.6.5.1custom-1.i686.rpm
</screen>
<note><para><link xl:href="InstallingLustrefrSourceCode.html#50438210_41207">Step 3</link> is only valid for RedHat and SuSE kernels. If you are using a stock Linux kernel, you need to get a script to create the kernel RPM.</para></note>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297755" xreflabel=""/>Install the Lustre packages.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1302732" xreflabel=""/>Some Lustre packages are installed on servers (MDS and OSSs), and others are installed on Lustre clients. For guidance on where to install specific packages, see <link xl:href="InstallingLustre.html#50438261_21654">TABLE 8-1</link> in <link xl:href="InstallingLustre.html#50438261_99193">Preparing to Install the Lustre Software</link>. which lists required packages and for each package, where to install it. Depending on the selected platform, not all of the packages listed in <link xl:href="InstallingLustre.html#50438261_21654">TABLE 8-1</link> need to be installed.</para>
+ <para>Install the Lustre packages.</para>
+ <para>Some Lustre packages are installed on servers (MDS and OSSs), and others are installed on Lustre clients. For guidance on where to install specific packages, see <link xl:href="InstallingLustre.html#50438261_21654">TABLE 8-1</link> in <link xl:href="InstallingLustre.html#50438261_99193">Preparing to Install the Lustre Software</link>. which lists required packages and for each package, where to install it. Depending on the selected platform, not all of the packages listed in <link xl:href="InstallingLustre.html#50438261_21654">TABLE 8-1</link> need to be installed.</para>
<note><para>Running the patched server kernel on the clients is optional. It is not necessary unless the clients will be used for multiple purposes, for example, to run as a client and an OST.</para></note>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1302716" xreflabel=""/>Lustre packages should be installed in this order:</para>
+ <para>Lustre packages should be installed in this order:</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297644" xreflabel=""/>Install the kernel, modules and ldiskfs packages.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297645" xreflabel=""/>Navigate to the directory where the RPMs are stored, and use the rpm -ivh command to install the kernel, module and ldiskfs packages.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1299031" xreflabel=""/>$ rpm -ivh kernel-lustre-smp-<ver> \
-<anchor xml:id="dbdoclet.50438210_pgfId-1299032" xreflabel=""/>kernel-ib-<ver> \
-<anchor xml:id="dbdoclet.50438210_pgfId-1299033" xreflabel=""/>lustre-modules-<ver> \
-<anchor xml:id="dbdoclet.50438210_pgfId-1299034" xreflabel=""/>lustre-ldiskfs-<ver>
+ <para>Install the kernel, modules and ldiskfs packages.</para>
+ <para>Navigate to the directory where the RPMs are stored, and use the rpm -ivh command to install the kernel, module and ldiskfs packages.</para>
+ <screen>$ rpm -ivh kernel-lustre-smp-<ver> \
+kernel-ib-<ver> \
+lustre-modules-<ver> \
+lustre-ldiskfs-<ver>
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297650" xreflabel=""/>Install the utilities/userspace packages.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297651" xreflabel=""/>Use the rpm -ivh command to install the utilities packages. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1297652" xreflabel=""/>$ rpm -ivh lustre-<ver>
+ <para>Install the utilities/userspace packages.</para>
+ <para>Use the rpm -ivh command to install the utilities packages. For example:</para>
+ <screen>$ rpm -ivh lustre-<ver>
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297653" xreflabel=""/>Install the e2fsprogs package.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297654" xreflabel=""/>Make sure the e2fsprogs package is unpacked, and use the rpm -i command to install it. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1297655" xreflabel=""/>$ rpm -i e2fsprogs-<ver>
+ <para>Install the e2fsprogs package.</para>
+ <para>Make sure the e2fsprogs package is unpacked, and use the rpm -i command to install it. For example:</para>
+ <screen>$ rpm -i e2fsprogs-<ver>
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1297656" xreflabel=""/>(Optional) If you want to add optional packages to your Lustre system, install them now.</para>
+ <para>(Optional) If you want to add optional packages to your Lustre system, install them now.</para>
</listitem></orderedlist>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1298263" xreflabel=""/>Verify that the boot loader (grub.conf or lilo.conf) has been updated to load the patched kernel.</para>
+ <para>Verify that the boot loader (grub.conf or lilo.conf) has been updated to load the patched kernel.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1298264" xreflabel=""/>Reboot the patched clients and the servers.</para>
+ <para>Reboot the patched clients and the servers.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1298265" xreflabel=""/>If you applied the patched kernel to any clients, reboot them.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1298266" xreflabel=""/>Unpatched clients do not need to be rebooted.</para>
+ <para>If you applied the patched kernel to any clients, reboot them.</para>
+ <para>Unpatched clients do not need to be rebooted.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1298267" xreflabel=""/>Reboot the servers.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1298268" xreflabel=""/>Once all the machines have rebooted, the next steps are to configure Lustre Networking (LNET) and the Lustre file system. See <link xl:href="ConfiguringLustre.html#50438267_88428">Chapter 10</link>: <link xl:href="ConfiguringLustre.html#50438267_66186">Configuring Lustre</link>.</para>
+ <para>Reboot the servers.</para>
+ <para>Once all the machines have rebooted, the next steps are to configure Lustre Networking (LNET) and the Lustre file system. See <link xl:href="ConfiguringLustre.html#50438267_88428">Chapter 10</link>: <link xl:href="ConfiguringLustre.html#50438267_66186">Configuring Lustre</link>.</para>
</listitem></orderedlist>
</listitem></orderedlist>
</section>
<section xml:id="dbdoclet.50438210_27248">
<title>29.4 Installing Lustre with a Third-Party Network Stack</title>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299781" xreflabel=""/>When using third-party network hardware, you must follow a specific process to install and recompile Lustre. This section provides an installation example, describing how to install Lustre 1.6.6 while using the Myricom MX 1.2.7 driver. The same process is used for other third-party network stacks, by replacing MX-specific references in <xref linkend="dbdoclet.50438210_12366"/> (Step 2) with the stack-specific build and using the proper --with option when configuring the Lustre source code.</para>
+ <para>When using third-party network hardware, you must follow a specific process to install and recompile Lustre. This section provides an installation example, describing how to install Lustre 1.6.6 while using the Myricom MX 1.2.7 driver. The same process is used for other third-party network stacks, by replacing MX-specific references in <xref linkend="dbdoclet.50438210_12366"/> (Step 2) with the stack-specific build and using the proper --with option when configuring the Lustre source code.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299638" xreflabel=""/>Compile and install the Lustre kernel.</para>
+ <para>Compile and install the Lustre kernel.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299642" xreflabel=""/>Install the necessary build tools.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1305395" xreflabel=""/>GCC and related tools must be installed. For more information, see <link xl:href="InstallingLustre.html#50438261_37079">Required Software</link>.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1299645" xreflabel=""/>$ yum install rpm-build redhat-rpm-config
-<anchor xml:id="dbdoclet.50438210_pgfId-1300055" xreflabel=""/>$ mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
-<anchor xml:id="dbdoclet.50438210_pgfId-1300058" xreflabel=""/>$ echo '%_topdir %(echo $HOME)/rpmbuild' > .rpmmacros
+ <para>Install the necessary build tools.</para>
+ <para>GCC and related tools must be installed. For more information, see <link xl:href="InstallingLustre.html#50438261_37079">Required Software</link>.</para>
+ <screen>$ yum install rpm-build redhat-rpm-config
+$ mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
+$ echo '%_topdir %(echo $HOME)/rpmbuild' > .rpmmacros
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299650" xreflabel=""/>Install the patched Lustre source code.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1300116" xreflabel=""/>This RPM is available at the <link xl:href="http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L">Lustre download site</link>.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1300294" xreflabel=""/>$ rpm -ivh \
-<anchor xml:id="dbdoclet.50438210_pgfId-1305349" xreflabel=""/>kernel-lustre-source-2.6.18-92.1.10.el5_lustre.1.6.6.x86_64.rpm
+ <para>Install the patched Lustre source code.</para>
+ <para>This RPM is available at the <link xl:href="http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L">Lustre download site</link>.</para>
+ <screen>$ rpm -ivh \
+kernel-lustre-source-2.6.18-92.1.10.el5_lustre.1.6.6.x86_64.rpm
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1300117" xreflabel=""/>Build the Linux kernel RPM.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1299659" xreflabel=""/>$ cd /usr/src/linux-2.6.18-92.1.10.el5_lustre.1.6.6
-<anchor xml:id="dbdoclet.50438210_pgfId-1300080" xreflabel=""/>$ make distclean
-<anchor xml:id="dbdoclet.50438210_pgfId-1300083" xreflabel=""/>$ make oldconfig dep bzImage modules
-<anchor xml:id="dbdoclet.50438210_pgfId-1300086" xreflabel=""/>$ cp /boot/config-`uname -r` .config
-<anchor xml:id="dbdoclet.50438210_pgfId-1300089" xreflabel=""/>$ make oldconfig || make menuconfig
-<anchor xml:id="dbdoclet.50438210_pgfId-1300092" xreflabel=""/>$ make include/asm
-<anchor xml:id="dbdoclet.50438210_pgfId-1300095" xreflabel=""/>$ make include/linux/version.h
-<anchor xml:id="dbdoclet.50438210_pgfId-1300098" xreflabel=""/>$ make SUBDIRS=scripts
-<anchor xml:id="dbdoclet.50438210_pgfId-1300101" xreflabel=""/>$ make rpm
+ <para>Build the Linux kernel RPM.</para>
+ <screen>$ cd /usr/src/linux-2.6.18-92.1.10.el5_lustre.1.6.6
+$ make distclean
+$ make oldconfig dep bzImage modules
+$ cp /boot/config-`uname -r` .config
+$ make oldconfig || make menuconfig
+$ make include/asm
+$ make include/linux/version.h
+$ make SUBDIRS=scripts
+$ make rpm
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299670" xreflabel=""/>Install the Linux kernel RPM.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1300130" xreflabel=""/>If you are building a set of RPMs for a cluster installation, this step is not necessary. Source RPMs are only needed on the build machine.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1299674" xreflabel=""/>$ rpm -ivh \
-<anchor xml:id="dbdoclet.50438210_pgfId-1305534" xreflabel=""/>~/rpmbuild/kernel-lustre-2.6.18-92.1.10.el5_\lustre.1.6.6.x86_64.rpm
-<anchor xml:id="dbdoclet.50438210_pgfId-1300142" xreflabel=""/>$ mkinitrd /boot/2.6.18-92.1.10.el5_lustre.1.6.6
+ <para>Install the Linux kernel RPM.</para>
+ <para>If you are building a set of RPMs for a cluster installation, this step is not necessary. Source RPMs are only needed on the build machine.</para>
+ <screen>$ rpm -ivh \
+~/rpmbuild/kernel-lustre-2.6.18-92.1.10.el5_\lustre.1.6.6.x86_64.rpm
+$ mkinitrd /boot/2.6.18-92.1.10.el5_lustre.1.6.6
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299677" xreflabel=""/>Update the boot loader (/etc/grub.conf) with the new kernel boot information.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1299679" xreflabel=""/>$ /sbin/shutdown 0 -r
+ <para>Update the boot loader (/etc/grub.conf) with the new kernel boot information.</para>
+ <screen>$ /sbin/shutdown 0 -r
</screen>
</listitem></orderedlist>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299681" xreflabel=""/><anchor xml:id="dbdoclet.50438210_12366" xreflabel=""/>Compile and install the MX stack.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1299686" xreflabel=""/>$ cd /usr/src/
-<anchor xml:id="dbdoclet.50438210_pgfId-1299907" xreflabel=""/>$ gunzip mx_1.2.7.tar.gz (can be obtained from www.myri.com/scs/)
-<anchor xml:id="dbdoclet.50438210_pgfId-1299910" xreflabel=""/>$ tar -xvf mx_1.2.7.tar
-<anchor xml:id="dbdoclet.50438210_pgfId-1299913" xreflabel=""/>$ cd mx-1.2.7
-<anchor xml:id="dbdoclet.50438210_pgfId-1299916" xreflabel=""/>$ ln -s common include
-<anchor xml:id="dbdoclet.50438210_pgfId-1299919" xreflabel=""/>$ ./configure --with-kernel-lib
-<anchor xml:id="dbdoclet.50438210_pgfId-1299922" xreflabel=""/>$ make
-<anchor xml:id="dbdoclet.50438210_pgfId-1299925" xreflabel=""/>$ make install
+ <para><anchor xml:id="dbdoclet.50438210_12366" xreflabel=""/>Compile and install the MX stack.</para>
+ <screen>$ cd /usr/src/
+$ gunzip mx_1.2.7.tar.gz (can be obtained from www.myri.com/scs/)
+$ tar -xvf mx_1.2.7.tar
+$ cd mx-1.2.7
+$ ln -s common include
+$ ./configure --with-kernel-lib
+$ make
+$ make install
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299696" xreflabel=""/>Compile and install the Lustre source code.</para>
+ <para>Compile and install the Lustre source code.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299698" xreflabel=""/>Install the Lustre source (this can be done via RPM or tarball). The source file is available at the <link xl:href="http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L">Lustre download site</link>. This example shows installation via the tarball.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1299700" xreflabel=""/>$ cd /usr/src/
-<anchor xml:id="dbdoclet.50438210_pgfId-1300181" xreflabel=""/>$ gunzip lustre-1.6.6.tar.gz
-<anchor xml:id="dbdoclet.50438210_pgfId-1300184" xreflabel=""/>$ tar -xvf lustre-1.6.6.tar
+ <para>Install the Lustre source (this can be done via RPM or tarball). The source file is available at the <link xl:href="http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L">Lustre download site</link>. This example shows installation via the tarball.</para>
+ <screen>$ cd /usr/src/
+$ gunzip lustre-1.6.6.tar.gz
+$ tar -xvf lustre-1.6.6.tar
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299706" xreflabel=""/>Configure and build the Lustre source code.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1300191" xreflabel=""/>The ./configure --help command shows a list of all of the --with options. All third-party network stacks are built in this manner.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1299711" xreflabel=""/>$ cd lustre-1.6.6
-<anchor xml:id="dbdoclet.50438210_pgfId-1300366" xreflabel=""/>$ ./configure --with-linux=/usr/src/linux \
-<anchor xml:id="dbdoclet.50438210_pgfId-1305509" xreflabel=""/>--with-mx=/usr/src/mx-1.2.7
-<anchor xml:id="dbdoclet.50438210_pgfId-1300369" xreflabel=""/>$ make
-<anchor xml:id="dbdoclet.50438210_pgfId-1300372" xreflabel=""/>$ make rpms
+ <para>Configure and build the Lustre source code.</para>
+ <para>The ./configure --help command shows a list of all of the --with options. All third-party network stacks are built in this manner.</para>
+ <screen>$ cd lustre-1.6.6
+$ ./configure --with-linux=/usr/src/linux \
+--with-mx=/usr/src/mx-1.2.7
+$ make
+$ make rpms
</screen>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299716" xreflabel=""/>The make rpms command output shows the location of the generated RPMs</para>
+ <para>The make rpms command output shows the location of the generated RPMs</para>
</listitem></orderedlist>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299718" xreflabel=""/>Use the rpm -ivh command to install the RPMS.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1299720" xreflabel=""/>$ rpm -ivh \
-<anchor xml:id="dbdoclet.50438210_pgfId-1305550" xreflabel=""/>lustre-1.6.6-2.6.18_92.1.10.el5_lustre.1.6.6smp.x86_64.rpm
-<anchor xml:id="dbdoclet.50438210_pgfId-1300246" xreflabel=""/>$ rpm -ivh \
-<anchor xml:id="dbdoclet.50438210_pgfId-1305560" xreflabel=""/>lustre-modules-1.6.6-2.6.18_92.1.10.el5_lustre.1.6.6\
-<anchor xml:id="dbdoclet.50438210_pgfId-1305564" xreflabel=""/>smp.x86_64.rpm
-<anchor xml:id="dbdoclet.50438210_pgfId-1300249" xreflabel=""/>$ rpm -ivh \
-<anchor xml:id="dbdoclet.50438210_pgfId-1305568" xreflabel=""/>lustre-ldiskfs-3.0.6-2.6.18_92.1.10.el5_lustre.1.6.6\
-<anchor xml:id="dbdoclet.50438210_pgfId-1305572" xreflabel=""/>smp.x86_64.rpm
+ <para>Use the rpm -ivh command to install the RPMS.</para>
+ <screen>$ rpm -ivh \
+lustre-1.6.6-2.6.18_92.1.10.el5_lustre.1.6.6smp.x86_64.rpm
+$ rpm -ivh \
+lustre-modules-1.6.6-2.6.18_92.1.10.el5_lustre.1.6.6\
+smp.x86_64.rpm
+$ rpm -ivh \
+lustre-ldiskfs-3.0.6-2.6.18_92.1.10.el5_lustre.1.6.6\
+smp.x86_64.rpm
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299726" xreflabel=""/>Add the following lines to the /etc/modprobe.conf file.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1299727" xreflabel=""/>options kmxlnd hosts=/etc/hosts.mxlnd
-<anchor xml:id="dbdoclet.50438210_pgfId-1300264" xreflabel=""/>options lnet networks=mx0(myri0),tcp0(eth0)
+ <para>Add the following lines to the /etc/modprobe.conf file.</para>
+ <screen>options kmxlnd hosts=/etc/hosts.mxlnd
+options lnet networks=mx0(myri0),tcp0(eth0)
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299730" xreflabel=""/>Populate the myri0 configuration with the proper IP addresses.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1299731" xreflabel=""/>vim /etc/sysconfig/network-scripts/myri0
+ <para>Populate the myri0 configuration with the proper IP addresses.</para>
+ <screen>vim /etc/sysconfig/network-scripts/myri0
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299733" xreflabel=""/>Add the following line to the /etc/hosts.mxlnd file.</para>
- <screen><anchor xml:id="dbdoclet.50438210_pgfId-1299734" xreflabel=""/>$ IP HOST BOARD EP_ID
+ <para>Add the following line to the /etc/hosts.mxlnd file.</para>
+ <screen>$ IP HOST BOARD EP_ID
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1299736" xreflabel=""/>Start Lustre.</para>
- <para><anchor xml:id="dbdoclet.50438210_pgfId-1300394" xreflabel=""/>Once all the machines have rebooted, the next steps are to configure Lustre Networking (LNET) and the Lustre file system. See <link xl:href="ConfiguringLustre.html#50438267_88428">Chapter 10</link>: <link xl:href="InstallingLustrefrSourceCode.html#50438210_67391">Installing Lustre from Source Code</link>.</para>
+ <para>Start Lustre.</para>
+ <para>Once all the machines have rebooted, the next steps are to configure Lustre Networking (LNET) and the Lustre file system. See <link xl:href="ConfiguringLustre.html#50438267_88428">Chapter 10</link>: <link xl:href="InstallingLustrefrSourceCode.html#50438210_67391">Installing Lustre from Source Code</link>.</para>
</listitem></orderedlist>
</section>
</chapter>
<info>
<title xml:id='lnetselftest.title'>Testing Lustre Network Performance (LNET Self-Test)</title>
</info>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1290818" xreflabel=""/><anchor xml:id="dbdoclet.50438223_36273" xreflabel=""/>This chapter describes the LNET self-test, which is used by site administrators to confirm that Lustre Networking (LNET) has been properly installed and configured, and that underlying network software and hardware are performing according to expectations. The chapter includes:</para>
+ <para><anchor xml:id="dbdoclet.50438223_36273" xreflabel=""/>This chapter describes the LNET self-test, which is used by site administrators to confirm that Lustre Networking (LNET) has been properly installed and configured, and that underlying network software and hardware are performing according to expectations. The chapter includes:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438223_91742"/></para>
<section xml:id="dbdoclet.50438223_91742">
<title>23.1 LNET Self-Test Overview</title>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300619" xreflabel=""/>LNET self-test is a kernel module that runs over LNET and the Lustre network drivers (LNDs. It is designed to:</para>
+ <para>LNET self-test is a kernel module that runs over LNET and the Lustre network drivers (LNDs. It is designed to:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300620" xreflabel=""/> Test the connection ability of the Lustre network</para>
+ <para> Test the connection ability of the Lustre network</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300621" xreflabel=""/> Run regression tests of the Lustre network</para>
+ <para> Run regression tests of the Lustre network</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300622" xreflabel=""/> Test performance of the Lustre network</para>
+ <para> Test performance of the Lustre network</para>
</listitem>
</itemizedlist>
-<para><anchor xml:id="dbdoclet.50438223_pgfId-1304396" xreflabel=""/>After you have obtained performance results for your Lustre network, refer to <xref linkend='lustretuning'/> for information about parameters that can be used to tune LNET for optimum performance.</para>
+<para>After you have obtained performance results for your Lustre network, refer to <xref linkend='lustretuning'/> for information about parameters that can be used to tune LNET for optimum performance.</para>
<note><para>Apart from the performance impact, LNET self-test is invisible to Lustre.</para></note>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1293959" xreflabel=""/>An LNET self-test cluster includes two types of nodes:</para>
+ <para>An LNET self-test cluster includes two types of nodes:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300652" xreflabel=""/><emphasis role="bold">Console node</emphasis> - A node used to control and monitor an LNET self-test cluster. The console node serves as the user interface of the LNET self-test system and can be any node in the test cluster. All self-test commands are entered from the console node. From the console node, a user can control and monitor the status of the entire LNET self-test cluster (session). The console node is exclusive in that a user cannot control two different sessions from one console node.</para>
+ <para><emphasis role="bold">Console node</emphasis> - A node used to control and monitor an LNET self-test cluster. The console node serves as the user interface of the LNET self-test system and can be any node in the test cluster. All self-test commands are entered from the console node. From the console node, a user can control and monitor the status of the entire LNET self-test cluster (session). The console node is exclusive in that a user cannot control two different sessions from one console node.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1296586" xreflabel=""/><emphasis role="bold">Test nodes</emphasis> - The nodes on which the tests are run. Test nodes are controlled by the user from the console node; the user does not need to log into them directly.</para>
+ <para><emphasis role="bold">Test nodes</emphasis> - The nodes on which the tests are run. Test nodes are controlled by the user from the console node; the user does not need to log into them directly.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300734" xreflabel=""/>LNET self-test has two user utilities:</para>
+ <para>LNET self-test has two user utilities:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1290849" xreflabel=""/><emphasis role="bold">lst</emphasis> - The user interface for the self-test console (run on the console node). It provides a list of commands to control the entire test system, including commands to create a session, create test groups, etc.</para>
+ <para><emphasis role="bold">lst</emphasis> - The user interface for the self-test console (run on the console node). It provides a list of commands to control the entire test system, including commands to create a session, create test groups, etc.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291488" xreflabel=""/><emphasis role="bold">lstclient</emphasis> - The userspace LNET self-test program (run on a test node). The lstclient utility is linked with userspace LNDs and LNET. This utility is not needed if only kernel space LNET and LNDs are used.</para>
+ <para><emphasis role="bold">lstclient</emphasis> - The userspace LNET self-test program (run on a test node). The lstclient utility is linked with userspace LNDs and LNET. This utility is not needed if only kernel space LNET and LNDs are used.</para>
</listitem>
</itemizedlist>
<note><para>Test nodes can be in either kernel or userspace. A console user can invite a kernel test node to join the test session by running lstadd_groupNID, but the console user cannot actively add a userspace test node to the test-session. However, the console user can passively accept a test node to the test session while the test node is running lstclient to connect to the console.</para></note>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438223_pgfId-1300634" xreflabel=""/>23.1.1 Prerequisites</title>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300640" xreflabel=""/>To run LNET self-test, these modules must be loaded:</para>
+ <title>23.1.1 Prerequisites</title>
+ <para>To run LNET self-test, these modules must be loaded:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300832" xreflabel=""/>libcfs</para>
+ <para>libcfs</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300836" xreflabel=""/>net</para>
+ <para>net</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300837" xreflabel=""/>lnet_selftest</para>
+ <para>lnet_selftest</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300838" xreflabel=""/> One of the klnds (i.e, ksocklnd, ko2iblnd...) as needed by your network configuration</para>
+ <para> One of the klnds (i.e, ksocklnd, ko2iblnd...) as needed by your network configuration</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300839" xreflabel=""/>To load the required modules, run:</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1300871" xreflabel=""/>modprobe lnet_selftest
+ <para>To load the required modules, run:</para>
+ <screen>modprobe lnet_selftest
</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300873" xreflabel=""/>This command recursively loads the modules on which LNET self-test depends.</para>
+ <para>This command recursively loads the modules on which LNET self-test depends.</para>
<note><para>While the console and test nodes require all the prerequisite modules to be loaded, userspace test nodes do not require these modules.</para></note>
</section>
</section>
<section xml:id="dbdoclet.50438223_48138">
<title>23.2 Using LNET Self-Test</title>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300891" xreflabel=""/>This section describes how to create and run an LNET self-test. The examples shown are for a test that simulates the traffic pattern of a set of Lustre servers on a TCP network accessed by Lustre clients on an InfiniBand network connected via LNET routers. In this example, half the clients are reading and half the clients are writing.</para>
+ <para>This section describes how to create and run an LNET self-test. The examples shown are for a test that simulates the traffic pattern of a set of Lustre servers on a TCP network accessed by Lustre clients on an InfiniBand network connected via LNET routers. In this example, half the clients are reading and half the clients are writing.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438223_pgfId-1300917" xreflabel=""/>23.2.1 Creating a Session</title>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300889" xreflabel=""/>A <emphasis>session</emphasis> is a set of processes that run on a test node. Only one session can be run at a time on a test node to ensure that the session has exclusive use of the node. The console node is used to create, change or destroy a session (new_session, end_session, show_session). For more about session parameters, see <link xl:href="LNETSelfTest.html#50438223_91247">Session Commands</link>.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1300956" xreflabel=""/>Almost all operations should be performed within the context of a session. From the console node, a user can only operate nodes in his own session. If a session ends, the session context in all test nodes is stopped.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301000" xreflabel=""/>The following commands set the LST_SESSION environment variable to identify the session on the console node and create a session called read_write:</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1301013" xreflabel=""/>export LST_SESSION=$$
-<anchor xml:id="dbdoclet.50438223_pgfId-1301014" xreflabel=""/>lst new_session read_write
+ <title>23.2.1 Creating a Session</title>
+ <para>A <emphasis>session</emphasis> is a set of processes that run on a test node. Only one session can be run at a time on a test node to ensure that the session has exclusive use of the node. The console node is used to create, change or destroy a session (new_session, end_session, show_session). For more about session parameters, see <link xl:href="LNETSelfTest.html#50438223_91247">Session Commands</link>.</para>
+ <para>Almost all operations should be performed within the context of a session. From the console node, a user can only operate nodes in his own session. If a session ends, the session context in all test nodes is stopped.</para>
+ <para>The following commands set the LST_SESSION environment variable to identify the session on the console node and create a session called read_write:</para>
+ <screen>export LST_SESSION=$$
+lst new_session read_write
</screen>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438223_pgfId-1296644" xreflabel=""/>23.2.2 Setting Up Groups</title>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301055" xreflabel=""/>A <emphasis>group</emphasis> is a named collection of nodes. Any number of groups can exist in a single LNET self-test session. Group membership is not restricted in that a node can be included in any number of groups.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301056" xreflabel=""/>Each node in a group has a rank, determined by the order in which it was added to the group. The rank is used to establish test traffic patterns.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301051" xreflabel=""/>A user can only control nodes in his/her session. To allocate nodes to the session, the user needs to add nodes to a group (of the session). All nodes in a group can be referenced by the group name. A node can be allocated to multiple groups of a session.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301084" xreflabel=""/>In the following example, three groups are established:</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1301100" xreflabel=""/>lst add_group servers 192.168.10.[8,10,12-16]@tcp
-<anchor xml:id="dbdoclet.50438223_pgfId-1301101" xreflabel=""/>lst add_group readers 192.168.1.[1-253/2]@o2ib
-<anchor xml:id="dbdoclet.50438223_pgfId-1301102" xreflabel=""/>lst add_group writers 192.168.1.[2-254/2]@o2ib
+ <title>23.2.2 Setting Up Groups</title>
+ <para>A <emphasis>group</emphasis> is a named collection of nodes. Any number of groups can exist in a single LNET self-test session. Group membership is not restricted in that a node can be included in any number of groups.</para>
+ <para>Each node in a group has a rank, determined by the order in which it was added to the group. The rank is used to establish test traffic patterns.</para>
+ <para>A user can only control nodes in his/her session. To allocate nodes to the session, the user needs to add nodes to a group (of the session). All nodes in a group can be referenced by the group name. A node can be allocated to multiple groups of a session.</para>
+ <para>In the following example, three groups are established:</para>
+ <screen>lst add_group servers 192.168.10.[8,10,12-16]@tcp
+lst add_group readers 192.168.1.[1-253/2]@o2ib
+lst add_group writers 192.168.1.[2-254/2]@o2ib
</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301147" xreflabel=""/>These three groups include:</para>
+ <para>These three groups include:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301153" xreflabel=""/> Nodes that will function as 'servers' to be accessed by 'clients' during the LNET self-test session</para>
+ <para> Nodes that will function as 'servers' to be accessed by 'clients' during the LNET self-test session</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301154" xreflabel=""/> Nodes that will function as 'clients' that will simulate <emphasis>reading</emphasis> data from the 'servers'</para>
+ <para> Nodes that will function as 'clients' that will simulate <emphasis>reading</emphasis> data from the 'servers'</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301155" xreflabel=""/> Nodes that will function as 'clients' that will simulate <emphasis>writing</emphasis> data to the 'servers'</para>
+ <para> Nodes that will function as 'clients' that will simulate <emphasis>writing</emphasis> data to the 'servers'</para>
</listitem>
</itemizedlist>
<note><para>A console user can associate kernel space test nodes with the session by running lst add_group NIDs, but a userspace test node cannot be actively added to the session. However, the console user can passively "accept" a test node to associate with a test session while the test node running lstclient connects to the console node, i.e: lstclient --sesid CONSOLE_NID --group NAME).</para></note>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438223_pgfId-1296646" xreflabel=""/>23.2.3 <anchor xml:id="dbdoclet.50438223_42848" xreflabel=""/>Defining and Running the Tests</title>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1298665" xreflabel=""/>A <emphasis>test</emphasis> generates a network load between two groups of nodes, a source group identified using the --from parameter and a target group identified using the --to parameter. When a test is running, each node in the --from<emphasis><group></emphasis> simulates a client by sending requests to nodes in the --to<emphasis><group></emphasis>, which are simulating a set of servers, and then receives responses in return. This activity is designed to mimic Lustre RPC traffic.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301259" xreflabel=""/>A <emphasis>batch</emphasis> is a collection of tests that are started and stopped together and run in parallel. A test must always be run as part of a batch, even if it is just a single test. Users can only run or stop a test batch, not individual tests.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301286" xreflabel=""/>Tests in a batch are non-destructive to the file system, and can be run in a normal Lustre environment (provided the performance impact is acceptable).</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301260" xreflabel=""/>A simple batch might contain a single test, for example, to determine whether the network bandwidth presents an I/O bottleneck. In this example, the --to<emphasis><group></emphasis> could be comprised of Lustre OSSs and --from<emphasis><group></emphasis> the compute nodes. A second test could be added to perform pings from a login node to the MDS to see how checkpointing affects the ls -l process.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301333" xreflabel=""/>Two types of tests are available:</para>
+ <title>23.2.3 <anchor xml:id="dbdoclet.50438223_42848" xreflabel=""/>Defining and Running the Tests</title>
+ <para>A <emphasis>test</emphasis> generates a network load between two groups of nodes, a source group identified using the --from parameter and a target group identified using the --to parameter. When a test is running, each node in the --from<emphasis><group></emphasis> simulates a client by sending requests to nodes in the --to<emphasis><group></emphasis>, which are simulating a set of servers, and then receives responses in return. This activity is designed to mimic Lustre RPC traffic.</para>
+ <para>A <emphasis>batch</emphasis> is a collection of tests that are started and stopped together and run in parallel. A test must always be run as part of a batch, even if it is just a single test. Users can only run or stop a test batch, not individual tests.</para>
+ <para>Tests in a batch are non-destructive to the file system, and can be run in a normal Lustre environment (provided the performance impact is acceptable).</para>
+ <para>A simple batch might contain a single test, for example, to determine whether the network bandwidth presents an I/O bottleneck. In this example, the --to<emphasis><group></emphasis> could be comprised of Lustre OSSs and --from<emphasis><group></emphasis> the compute nodes. A second test could be added to perform pings from a login node to the MDS to see how checkpointing affects the ls -l process.</para>
+ <para>Two types of tests are available:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301338" xreflabel=""/><emphasis role="bold">ping -</emphasis> A ping generates a short request message, which results in a short response. Pings are useful to determine latency and small message overhead and to simulate Lustre metadata traffic.</para>
+ <para><emphasis role="bold">ping -</emphasis> A ping generates a short request message, which results in a short response. Pings are useful to determine latency and small message overhead and to simulate Lustre metadata traffic.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301407" xreflabel=""/><emphasis role="bold">brw -</emphasis> In a brw ('bulk read write') test, data is transferred from the target to the source (brwread) or data is transferred from the source to the target (brwwrite). The size of the bulk transfer is set using the size parameter. A brw test is useful to determine network bandwidth and to simulate Lustre I/O traffic.</para>
+ <para><emphasis role="bold">brw -</emphasis> In a brw ('bulk read write') test, data is transferred from the target to the source (brwread) or data is transferred from the source to the target (brwwrite). The size of the bulk transfer is set using the size parameter. A brw test is useful to determine network bandwidth and to simulate Lustre I/O traffic.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301257" xreflabel=""/>In the example below, a batch is created called bulk_rw. Then two brw tests are added. In the first test, 1M of data is sent from the servers to the clients as a simulated read operation with a simple data validation check. In the second test, 4K of data is sent from the clients to the servers as a simulated write operation with a full data validation check.</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1301178" xreflabel=""/>lst add_batch bulk_rw
-<anchor xml:id="dbdoclet.50438223_pgfId-1301179" xreflabel=""/>lst add_test --batch bulk_rw --from readers --to servers \
-<anchor xml:id="dbdoclet.50438223_pgfId-1303584" xreflabel=""/>brw read check=simple size=1M
-<anchor xml:id="dbdoclet.50438223_pgfId-1303588" xreflabel=""/>lst add_test --batch bulk_rw --from writers --to servers \
-<anchor xml:id="dbdoclet.50438223_pgfId-1303589" xreflabel=""/>brw write check=full size=4K
+ <para>In the example below, a batch is created called bulk_rw. Then two brw tests are added. In the first test, 1M of data is sent from the servers to the clients as a simulated read operation with a simple data validation check. In the second test, 4K of data is sent from the clients to the servers as a simulated write operation with a full data validation check.</para>
+ <screen>lst add_batch bulk_rw
+lst add_test --batch bulk_rw --from readers --to servers \
+brw read check=simple size=1M
+lst add_test --batch bulk_rw --from writers --to servers \
+brw write check=full size=4K
</screen>
-<para><anchor xml:id="dbdoclet.50438223_pgfId-1303590" xreflabel=""/>The traffic pattern and test intensity is determined by several properties such as test type, distribution of test nodes, concurrency of test, and RDMA operation type. For more details, see <xref linkend='dbdoclet.50438223_36860'/>.</para>
+<para>The traffic pattern and test intensity is determined by several properties such as test type, distribution of test nodes, concurrency of test, and RDMA operation type. For more details, see <xref linkend='dbdoclet.50438223_36860'/>.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438223_pgfId-1290855" xreflabel=""/>23.2.4 Sample Script</title>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1290856" xreflabel=""/>This sample LNET self-test script simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an InfiniBand network (connected via LNET routers). In this example, half the clients are reading and half the clients are writing.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291527" xreflabel=""/>Run this script on the console node:</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1290863" xreflabel=""/>#!/bin/bash
-<anchor xml:id="dbdoclet.50438223_pgfId-1290864" xreflabel=""/>export LST_SESSION=$$
-<anchor xml:id="dbdoclet.50438223_pgfId-1290865" xreflabel=""/>lst new_session read/write
-<anchor xml:id="dbdoclet.50438223_pgfId-1290866" xreflabel=""/>lst add_group servers 192.168.10.[8,10,12-16]@tcp
-<anchor xml:id="dbdoclet.50438223_pgfId-1290867" xreflabel=""/>lst add_group readers 192.168.1.[1-253/2]@o2ib
-<anchor xml:id="dbdoclet.50438223_pgfId-1290868" xreflabel=""/>lst add_group writers 192.168.1.[2-254/2]@o2ib
-<anchor xml:id="dbdoclet.50438223_pgfId-1290869" xreflabel=""/>lst add_batch bulk_rw
-<anchor xml:id="dbdoclet.50438223_pgfId-1290870" xreflabel=""/>lst add_test --batch bulk_rw --from readers --to servers \
-<anchor xml:id="dbdoclet.50438223_pgfId-1290871" xreflabel=""/>brw read check=simple size=1M
-<anchor xml:id="dbdoclet.50438223_pgfId-1290872" xreflabel=""/>lst add_test --batch bulk_rw --from writers --to servers \
-<anchor xml:id="dbdoclet.50438223_pgfId-1290873" xreflabel=""/>brw write check=full size=4K
-<anchor xml:id="dbdoclet.50438223_pgfId-1290874" xreflabel=""/># start running
-<anchor xml:id="dbdoclet.50438223_pgfId-1290875" xreflabel=""/>lst run bulk_rw
-<anchor xml:id="dbdoclet.50438223_pgfId-1290876" xreflabel=""/># display server stats for 30 seconds
-<anchor xml:id="dbdoclet.50438223_pgfId-1290877" xreflabel=""/>lst stat servers & sleep 30; kill $!
-<anchor xml:id="dbdoclet.50438223_pgfId-1290878" xreflabel=""/># tear down
-<anchor xml:id="dbdoclet.50438223_pgfId-1290879" xreflabel=""/>lst end_session
+ <title>23.2.4 Sample Script</title>
+ <para>This sample LNET self-test script simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an InfiniBand network (connected via LNET routers). In this example, half the clients are reading and half the clients are writing.</para>
+ <para>Run this script on the console node:</para>
+ <screen>#!/bin/bash
+export LST_SESSION=$$
+lst new_session read/write
+lst add_group servers 192.168.10.[8,10,12-16]@tcp
+lst add_group readers 192.168.1.[1-253/2]@o2ib
+lst add_group writers 192.168.1.[2-254/2]@o2ib
+lst add_batch bulk_rw
+lst add_test --batch bulk_rw --from readers --to servers \
+brw read check=simple size=1M
+lst add_test --batch bulk_rw --from writers --to servers \
+brw write check=full size=4K
+# start running
+lst run bulk_rw
+# display server stats for 30 seconds
+lst stat servers & sleep 30; kill $!
+# tear down
+lst end_session
</screen>
<note><para>This script can be easily adapted to pass the group NIDs by shell variables or command line arguments (making it good for general-purpose use).</para></note>
</section>
<section xml:id="dbdoclet.50438223_27277">
<title>23.3 LNET Self-Test <anchor xml:id="dbdoclet.50438223_marker-1298562" xreflabel=""/>Command Reference</title>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1296018" xreflabel=""/>The LNET self-test (lst) utility is used to issue LNET self-test commands. The lst utility takes a number of command line arguments. The first argument is the command name and subsequent arguments are command-specific.</para>
+ <para>The LNET self-test (lst) utility is used to issue LNET self-test commands. The lst utility takes a number of command line arguments. The first argument is the command name and subsequent arguments are command-specific.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438223_pgfId-1290916" xreflabel=""/>23.3.1 <anchor xml:id="dbdoclet.50438223_91247" xreflabel=""/>Session Commands</title>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1296368" xreflabel=""/>This section describes lst session commands.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1290917" xreflabel=""/><emphasis role="bold">LST_SESSION</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1297000" xreflabel=""/>The lst utility uses the LST_SESSION environmental variable to identify the session locally on the self-test console node. This should be a numeric value that uniquely identifies all session processes on the node. It is convenient to set this to the process ID of the shell both for interactive use and in shell scripts. Almost all lst commands require LST_SESSION to be set.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301596" xreflabel=""/>Example:</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1301601" xreflabel=""/>export LST_SESSION=$$
+ <title>23.3.1 <anchor xml:id="dbdoclet.50438223_91247" xreflabel=""/>Session Commands</title>
+ <para>This section describes lst session commands.</para>
+ <para><emphasis role="bold">LST_SESSION</emphasis></para>
+ <para>The lst utility uses the LST_SESSION environmental variable to identify the session locally on the self-test console node. This should be a numeric value that uniquely identifies all session processes on the node. It is convenient to set this to the process ID of the shell both for interactive use and in shell scripts. Almost all lst commands require LST_SESSION to be set.</para>
+ <para>Example:</para>
+ <screen>export LST_SESSION=$$
</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1290923" xreflabel=""/><emphasis role="bold">new_session [--timeout SECONDS] [--force] NAME</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1290924" xreflabel=""/>Creates a new session.</para>
+ <para><emphasis role="bold">new_session [--timeout SECONDS] [--force] NAME</emphasis></para>
+ <para>Creates a new session.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1301517" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1301519" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1301521" xreflabel=""/>--timeout<emphasis><seconds></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1301523" xreflabel=""/>Console timeout value of the session. The session ends automatically if it remains idle (i.e., no commands are issued) for this period.</para></entry>
+ <entry><para> --timeout<emphasis><seconds></emphasis></para></entry>
+ <entry><para> Console timeout value of the session. The session ends automatically if it remains idle (i.e., no commands are issued) for this period.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1301525" xreflabel=""/>--force</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1301527" xreflabel=""/>Ends conflicting sessions. This determines who 'wins' when one session conflicts with another. For example, if there is already an active session on this node, then the attempt to create a new session fails unless the -force flag is specified. If the -force flag is specified, then the active session is ended. Similarly, if a session attempts to add a node that is already 'owned' by another session, the -force flag allows this session to 'steal' the node.</para></entry>
+ <entry><para> --force</para></entry>
+ <entry><para> Ends conflicting sessions. This determines who 'wins' when one session conflicts with another. For example, if there is already an active session on this node, then the attempt to create a new session fails unless the -force flag is specified. If the -force flag is specified, then the active session is ended. Similarly, if a session attempts to add a node that is already 'owned' by another session, the -force flag allows this session to 'steal' the node.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1301529" xreflabel=""/><emphasis><name></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1301531" xreflabel=""/>A human-readable string to print when listing sessions or reporting session conflicts.</para></entry>
+ <entry><para> <emphasis><name></emphasis></para></entry>
+ <entry><para> A human-readable string to print when listing sessions or reporting session conflicts.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1301575" xreflabel=""/><emphasis role="bold">Example:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1301576" xreflabel=""/>$ lst new_session --force read_write
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst new_session --force read_write
</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303652" xreflabel=""/><emphasis role="bold">end_session</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291942" xreflabel=""/>Stops all operations and tests in the current session and clears the session'™s status.</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1303653" xreflabel=""/>$ lst end_session
+ <para><emphasis role="bold">end_session</emphasis></para>
+ <para>Stops all operations and tests in the current session and clears the session'™s status.</para>
+ <screen>$ lst end_session
</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303656" xreflabel=""/><emphasis role="bold">show_session</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1290942" xreflabel=""/>Shows the session information. This command prints information about the current session. It does not require LST_SESSION to be defined in the process environment.</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1290945" xreflabel=""/>$ lst show_session
+ <para><emphasis role="bold">show_session</emphasis></para>
+ <para>Shows the session information. This command prints information about the current session. It does not require LST_SESSION to be defined in the process environment.</para>
+ <screen>$ lst show_session
</screen>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438223_pgfId-1290947" xreflabel=""/>23.3.2 Group Commands</title>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1296396" xreflabel=""/>This section describes lst group commands.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1290948" xreflabel=""/><emphasis role="bold">add_group</emphasis><emphasis><name> <NIDS> [<NIDs>...]</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1290949" xreflabel=""/>Creates the group and adds a list of test nodes to the group.</para>
+ <title>23.3.2 Group Commands</title>
+ <para>This section describes lst group commands.</para>
+ <para><emphasis role="bold">add_group</emphasis><emphasis><name> <NIDS> [<NIDs>...]</emphasis></para>
+ <para>Creates the group and adds a list of test nodes to the group.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1301542" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1301544" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294606" xreflabel=""/><emphasis><name></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294608" xreflabel=""/>Name of the group.</para></entry>
+ <entry><para> <emphasis><name></emphasis></para></entry>
+ <entry><para> Name of the group.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294610" xreflabel=""/><emphasis><NIDs></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294612" xreflabel=""/>A string that may be expanded to include one or more LNET NIDs.</para></entry>
+ <entry><para> <emphasis><NIDs></emphasis></para></entry>
+ <entry><para> A string that may be expanded to include one or more LNET NIDs.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303377" xreflabel=""/><emphasis role="bold">Example:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1303383" xreflabel=""/>$ lst add_group servers 192.168.10.[35,40-45]@tcp<anchor xml:id="dbdoclet.50438223_pgfId-1303381" xreflabel=""/>$ lst add_group clients 192.168.1.[10-100]@tcp 192.168.[2,4].\<anchor xml:id="dbdoclet.50438223_pgfId-1303393" xreflabel=""/>[10-20]@tcp</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303378" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1290954" xreflabel=""/><emphasis role="bold">update_group</emphasis><emphasis><name></emphasis><emphasis role="bold">[--refresh] [--clean</emphasis><emphasis><status></emphasis><emphasis role="bold">] [--remove</emphasis><emphasis><NIDs></emphasis><emphasis role="bold">]</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1290955" xreflabel=""/>Updates the state of nodes in a group or adjusts a group'™s membership. This command is useful if some nodes have crashed and should be excluded from the group.</para>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst add_group servers 192.168.10.[35,40-45]@tcp$ lst add_group clients 192.168.1.[10-100]@tcp 192.168.[2,4].\[10-20]@tcp</screen>
+ <para> </para>
+ <para><emphasis role="bold">update_group</emphasis><emphasis><name></emphasis><emphasis role="bold">[--refresh] [--clean</emphasis><emphasis><status></emphasis><emphasis role="bold">] [--remove</emphasis><emphasis><NIDs></emphasis><emphasis role="bold">]</emphasis></para>
+ <para>Updates the state of nodes in a group or adjusts a group'™s membership. This command is useful if some nodes have crashed and should be excluded from the group.</para>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
<colspec colname="c3" colwidth="33*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1301546" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1301548" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry nameend="c3" namest="c2"><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294758" xreflabel=""/>--refresh</para></entry>
- <entry nameend="c3" namest="c2"><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294760" xreflabel=""/>Refreshes the state of all inactive nodes in the group.</para></entry>
+ <entry><para> --refresh</para></entry>
+ <entry nameend="c3" namest="c2"><para> Refreshes the state of all inactive nodes in the group.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294764" xreflabel=""/>--clean<emphasis><status></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294766" xreflabel=""/>Removes nodes with a specified status from the group. Status may be:</para></entry>
+ <entry><para> --clean<emphasis><status></emphasis></para></entry>
+ <entry nameend="c3" namest="c2"><para> Removes nodes with a specified status from the group. Status may be:</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294736" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294774" xreflabel=""/>active</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294788" xreflabel=""/>The node is in the current session.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> active</para></entry>
+ <entry><para> The node is in the current session.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294732" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294776" xreflabel=""/>busy</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294790" xreflabel=""/>The node is now owned by another session.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> busy</para></entry>
+ <entry><para> The node is now owned by another session.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294728" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294778" xreflabel=""/>down</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294792" xreflabel=""/>The node has been marked down.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> down</para></entry>
+ <entry><para> The node has been marked down.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294724" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294780" xreflabel=""/>unknown</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294794" xreflabel=""/>The node'™s status has yet to be determined.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> unknown</para></entry>
+ <entry><para> The node'™s status has yet to be determined.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294720" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294782" xreflabel=""/>invalid</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294796" xreflabel=""/>Any state but active.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> invalid</para></entry>
+ <entry><para> Any state but active.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294802" xreflabel=""/>--remove<emphasis><NIDs></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294804" xreflabel=""/>Removes specified nodes from the group.</para></entry>
+ <entry><para> --remove<emphasis><NIDs></emphasis></para></entry>
+ <entry nameend="c3" namest="c2"><para> Removes specified nodes from the group.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303342" xreflabel=""/><emphasis role="bold">Example:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1303346" xreflabel=""/>$ lst update_group clients --refresh<anchor xml:id="dbdoclet.50438223_pgfId-1303358" xreflabel=""/>$ lst update_group clients --clean busy<anchor xml:id="dbdoclet.50438223_pgfId-1303359" xreflabel=""/>$ lst update_group clients --clean invalid // \<anchor xml:id="dbdoclet.50438223_pgfId-1303360" xreflabel=""/>invalid == busy || down || unknown<anchor xml:id="dbdoclet.50438223_pgfId-1303361" xreflabel=""/>$ lst update_group clients --remove \192.168.1.[10-20]@tcp</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303343" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1292226" xreflabel=""/><emphasis role="bold">list_group [</emphasis><emphasis><name></emphasis><emphasis role="bold">] [--active] [--busy] [--down] [--unknown] [--all]</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1294616" xreflabel=""/>Prints information about a group or lists all groups in the current session if no group is specified.</para>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst update_group clients --refresh$ lst update_group clients --clean busy$ lst update_group clients --clean invalid // \invalid == busy || down || unknown$ lst update_group clients --remove \192.168.1.[10-20]@tcp</screen>
+ <para> </para>
+ <para><emphasis role="bold">list_group [</emphasis><emphasis><name></emphasis><emphasis role="bold">] [--active] [--busy] [--down] [--unknown] [--all]</emphasis></para>
+ <para>Prints information about a group or lists all groups in the current session if no group is specified.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1301755" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1301757" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294824" xreflabel=""/><emphasis><name></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294826" xreflabel=""/>The name of the group.</para></entry>
+ <entry><para> <emphasis><name></emphasis></para></entry>
+ <entry><para> The name of the group.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294828" xreflabel=""/>--active</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294830" xreflabel=""/>Lists the active nodes.</para></entry>
+ <entry><para> --active</para></entry>
+ <entry><para> Lists the active nodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294832" xreflabel=""/>--busy</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294834" xreflabel=""/>Lists the busy nodes.</para></entry>
+ <entry><para> --busy</para></entry>
+ <entry><para> Lists the busy nodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294836" xreflabel=""/>--down</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294838" xreflabel=""/>Lists the down nodes.</para></entry>
+ <entry><para> --down</para></entry>
+ <entry><para> Lists the down nodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294840" xreflabel=""/>--unknown</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294842" xreflabel=""/>Lists unknown nodes.</para></entry>
+ <entry><para> --unknown</para></entry>
+ <entry><para> Lists unknown nodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294844" xreflabel=""/>--all</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1294846" xreflabel=""/>Lists all nodes.</para></entry>
+ <entry><para> --all</para></entry>
+ <entry><para> Lists all nodes.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1302415" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1304733" xreflabel=""/>Example:</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1302418" xreflabel=""/>$ lst list_group<anchor xml:id="dbdoclet.50438223_pgfId-1302419" xreflabel=""/>1) clients<anchor xml:id="dbdoclet.50438223_pgfId-1302420" xreflabel=""/>2) servers<anchor xml:id="dbdoclet.50438223_pgfId-1302421" xreflabel=""/>Total 2 groups<anchor xml:id="dbdoclet.50438223_pgfId-1302422" xreflabel=""/>$ lst list_group clients<anchor xml:id="dbdoclet.50438223_pgfId-1302423" xreflabel=""/>ACTIVE BUSY DOWN UNKNOWN TOTAL<anchor xml:id="dbdoclet.50438223_pgfId-1302424" xreflabel=""/>3 1 2 0 6<anchor xml:id="dbdoclet.50438223_pgfId-1302425" xreflabel=""/>$ lst list_group clients --all<anchor xml:id="dbdoclet.50438223_pgfId-1302426" xreflabel=""/>192.168.1.10@tcp Active<anchor xml:id="dbdoclet.50438223_pgfId-1302427" xreflabel=""/>192.168.1.11@tcp Active<anchor xml:id="dbdoclet.50438223_pgfId-1302428" xreflabel=""/>192.168.1.12@tcp Busy<anchor xml:id="dbdoclet.50438223_pgfId-1302429" xreflabel=""/>192.168.1.13@tcp Active<anchor xml:id="dbdoclet.50438223_pgfId-1302430" xreflabel=""/>192.168.1.14@tcp DOWN<anchor xml:id="dbdoclet.50438223_pgfId-1302431" xreflabel=""/>192.168.1.15@tcp DOWN<anchor xml:id="dbdoclet.50438223_pgfId-1302432" xreflabel=""/>Total 6 nodes<anchor xml:id="dbdoclet.50438223_pgfId-1302433" xreflabel=""/>$ lst list_group clients --busy<anchor xml:id="dbdoclet.50438223_pgfId-1302434" xreflabel=""/>192.168.1.12@tcp Busy<anchor xml:id="dbdoclet.50438223_pgfId-1302435" xreflabel=""/>Total 1 node</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1294659" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303729" xreflabel=""/><emphasis role="bold">del_group</emphasis><emphasis><name></emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291000" xreflabel=""/>Removes a group from the session. If the group is referred to by any test, then the operation fails. If nodes in the group are referred to only by this group, then they are kicked out from the current session; otherwise, they are still in the current session.</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1291003" xreflabel=""/>$ lst del_group clients
+ <para> </para>
+ <para>Example:</para>
+ <screen>$ lst list_group1) clients2) serversTotal 2 groups$ lst list_group clientsACTIVE BUSY DOWN UNKNOWN TOTAL3 1 2 0 6$ lst list_group clients --all192.168.1.10@tcp Active192.168.1.11@tcp Active192.168.1.12@tcp Busy192.168.1.13@tcp Active192.168.1.14@tcp DOWN192.168.1.15@tcp DOWNTotal 6 nodes$ lst list_group clients --busy192.168.1.12@tcp BusyTotal 1 node</screen>
+ <para> </para>
+ <para><emphasis role="bold">del_group</emphasis><emphasis><name></emphasis></para>
+ <para>Removes a group from the session. If the group is referred to by any test, then the operation fails. If nodes in the group are referred to only by this group, then they are kicked out from the current session; otherwise, they are still in the current session.</para>
+ <screen>$ lst del_group clients
</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291004" xreflabel=""/><emphasis role="bold">lstclient --sesid</emphasis><emphasis><NID></emphasis><emphasis role="bold">--group</emphasis><emphasis><name></emphasis> [--server_mode]</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291005" xreflabel=""/>Use lstclient to run the userland self-test client. The lstclient command should be executed after creating a session on the console. There are only two mandatory options for lstclient:</para>
+ <para><emphasis role="bold">lstclient --sesid</emphasis><emphasis><NID></emphasis><emphasis role="bold">--group</emphasis><emphasis><name></emphasis> [--server_mode]</para>
+ <para>Use lstclient to run the userland self-test client. The lstclient command should be executed after creating a session on the console. There are only two mandatory options for lstclient:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1301783" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1301785" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1301787" xreflabel=""/>--sesid<emphasis><NID></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1301789" xreflabel=""/>The first console'™s NID.</para></entry>
+ <entry><para> --sesid<emphasis><NID></emphasis></para></entry>
+ <entry><para> The first console'™s NID.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1301791" xreflabel=""/>--group<emphasis><name></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1301793" xreflabel=""/>The test group to join.</para></entry>
+ <entry><para> --group<emphasis><name></emphasis></para></entry>
+ <entry><para> The test group to join.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303207" xreflabel=""/>--server_mode</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303209" xreflabel=""/>When included, forces LNET to behave as a server, such as starting an acceptor if the underlying NID needs it or using privileged ports. Only root is allowed to use the --server_mode option.</para></entry>
+ <entry><para> --server_mode</para></entry>
+ <entry><para> When included, forces LNET to behave as a server, such as starting an acceptor if the underlying NID needs it or using privileged ports. Only root is allowed to use the --server_mode option.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1304734" xreflabel=""/><emphasis role="bold">Example:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1301816" xreflabel=""/>Console $ lst new_session testsession
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>Console $ lst new_session testsession
Client1 $ lstclient --sesid 192.168.1.52@tcp --group clients</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1296196" xreflabel=""/><emphasis role="bold">Example:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1296197" xreflabel=""/>Client1 $ lstclient --sesid 192.168.1.52@tcp |--group clients --server_mode
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>Client1 $ lstclient --sesid 192.168.1.52@tcp |--group clients --server_mode
</screen>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438223_pgfId-1291013" xreflabel=""/>23.3.3 <anchor xml:id="dbdoclet.50438223_36860" xreflabel=""/>Batch and Test Commands</title>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1296409" xreflabel=""/>This section describes lst batch and test commands.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291014" xreflabel=""/><emphasis role="bold">add_batch NAME</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291015" xreflabel=""/>A default batch test set named batch is created when the session is started. You can specify a batch name by using add_batch:</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1291018" xreflabel=""/>$ lst add_batch bulkperf
+ <title>23.3.3 <anchor xml:id="dbdoclet.50438223_36860" xreflabel=""/>Batch and Test Commands</title>
+ <para>This section describes lst batch and test commands.</para>
+ <para><emphasis role="bold">add_batch NAME</emphasis></para>
+ <para>A default batch test set named batch is created when the session is started. You can specify a batch name by using add_batch:</para>
+ <screen>$ lst add_batch bulkperf
</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291019" xreflabel=""/>Creates a batch test called bulkperf.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303289" xreflabel=""/>add_test --batch<emphasis><batchname></emphasis> [--loop <emphasis><#>]</emphasis><emphasis role="bold">[--concurrency</emphasis><emphasis><#></emphasis><emphasis role="bold">] [--distribute</emphasis><emphasis><#:#></emphasis><emphasis role="bold">]</emphasis></para>
+ <para>Creates a batch test called bulkperf.</para>
+ <para>add_test --batch<emphasis><batchname></emphasis> [--loop <emphasis><#>]</emphasis><emphasis role="bold">[--concurrency</emphasis><emphasis><#></emphasis><emphasis role="bold">] [--distribute</emphasis><emphasis><#:#></emphasis><emphasis role="bold">]</emphasis></para>
<para>--from<emphasis><group></emphasis> --to <emphasis><group></emphasis> {brw|ping} <test options></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303713" xreflabel=""/>Adds a test to a batch. The parameters are described below.</para>
+ <para>Adds a test to a batch. The parameters are described below.</para>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
<colspec colname="c3" colwidth="33*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1303903" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1303905" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry nameend="c3" namest="c2"><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303909" xreflabel=""/>--batch<emphasis><batchname></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303911" xreflabel=""/>Names a group of tests for later execution.</para></entry>
+ <entry><para> --batch<emphasis><batchname></emphasis></para></entry>
+ <entry nameend="c3" namest="c2"><para> Names a group of tests for later execution.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303915" xreflabel=""/>--loop<emphasis><#></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303917" xreflabel=""/>Number of times to run the test.</para></entry>
+ <entry><para> --loop<emphasis><#></emphasis></para></entry>
+ <entry nameend="c3" namest="c2"><para> Number of times to run the test.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303921" xreflabel=""/>--concurrency<emphasis><#></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303923" xreflabel=""/>The number of requests that are active at one time.</para></entry>
+ <entry><para> --concurrency<emphasis><#></emphasis></para></entry>
+ <entry nameend="c3" namest="c2"><para> The number of requests that are active at one time.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303981" xreflabel=""/>--distribute<emphasis><#:#></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303983" xreflabel=""/>Determines the ratio of client nodes to server nodes for the specified test. This allows you to specify a wide range of topologies, including one-to-one and all-to-all. Distribution divides the source group into subsets, which are paired with equivalent subsets from the target group so only nodes in matching subsets communicate.</para></entry>
+ <entry><para> --distribute<emphasis><#:#></emphasis></para></entry>
+ <entry nameend="c3" namest="c2"><para> Determines the ratio of client nodes to server nodes for the specified test. This allows you to specify a wide range of topologies, including one-to-one and all-to-all. Distribution divides the source group into subsets, which are paired with equivalent subsets from the target group so only nodes in matching subsets communicate.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304062" xreflabel=""/>--from<emphasis><group></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304064" xreflabel=""/>The source group (test client).</para></entry>
+ <entry><para> --from<emphasis><group></emphasis></para></entry>
+ <entry nameend="c3" namest="c2"><para> The source group (test client).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304068" xreflabel=""/>--to<emphasis><group></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304070" xreflabel=""/>The target group (test server).</para></entry>
+ <entry><para> --to<emphasis><group></emphasis></para></entry>
+ <entry nameend="c3" namest="c2"><para> The target group (test server).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303989" xreflabel=""/>ping</para></entry>
- <entry nameend="c3" namest="c2"><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303991" xreflabel=""/>Sends a small request message, resulting in a small reply message. For more details, see <link xl:href="LNETSelfTest.html#50438223_42848">Defining and Running the Tests</link></para></entry>
+ <entry><para> ping</para></entry>
+ <entry nameend="c3" namest="c2"><para> Sends a small request message, resulting in a small reply message. For more details, see <link xl:href="LNETSelfTest.html#50438223_42848">Defining and Running the Tests</link></para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303995" xreflabel=""/>brw</para></entry>
- <entry nameend="c3" namest="c2"><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303997" xreflabel=""/>Sends a small request message followed by a bulk data transfer, resulting in a small reply message. <link xl:href="LNETSelfTest.html#50438223_42848">Defining and Running the Tests</link>. Options are:</para></entry>
+ <entry><para> brw</para></entry>
+ <entry nameend="c3" namest="c2"><para> Sends a small request message followed by a bulk data transfer, resulting in a small reply message. <link xl:href="LNETSelfTest.html#50438223_42848">Defining and Running the Tests</link>. Options are:</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303957" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304001" xreflabel=""/>read | write</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304003" xreflabel=""/>Read or write. The default is read.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> read | write</para></entry>
+ <entry><para> Read or write. The default is read.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303963" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304005" xreflabel=""/>size=<#>| <#>K | <#>M</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304007" xreflabel=""/>I/O size in bytes, KB or MB (i.e., size=1024, size=4K, size=1M). The default is 4K bytes.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> size=<#>| <#>K | <#>M</para></entry>
+ <entry><para> I/O size in bytes, KB or MB (i.e., size=1024, size=4K, size=1M). The default is 4K bytes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1303969" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304009" xreflabel=""/>check=full|simple</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304011" xreflabel=""/>A data validation check (checksum of data). The default is that no check is done.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> check=full|simple</para></entry>
+ <entry><para> A data validation check (checksum of data). The default is that no check is done.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1302599" xreflabel=""/><emphasis role="bold">Examples showing use of the distribute parameter:</emphasis></para>
+ <para><emphasis role="bold">Examples showing use of the distribute parameter:</emphasis></para>
<screen>
Clients: (C1, C2, C3, C4, C5, C6)
Server: (S1, S2, S3)
--distribute 4:1 (C1,C2,C3,C4->S1), (C5,C6->S2), (NULL->S3)
--distribute 4:2 (C1,C2,C3,C4->S1,S2), (C5, C6->S3, S1)
--distribute 6:3 (C1,C2,C3,C4,C5,C6->S1,S2,S3)</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303015" xreflabel=""/>The setting --distribute 1:1 is the default setting where each source node communicates with one target node.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303016" xreflabel=""/>When the setting --distribute 1:<emphasis><n></emphasis> (where <emphasis><n></emphasis> is the size of the target group) is used, each source node communicates with every node in the target group.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303017" xreflabel=""/>Note that if there are more source nodes than target nodes, some source nodes may share the same target nodes. Also, if there are more target nodes than source nodes, some higher-ranked target nodes will be idle.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1304018" xreflabel=""/><emphasis role="bold">Example showing a brw test:</emphasis></para>
+ <para>The setting --distribute 1:1 is the default setting where each source node communicates with one target node.</para>
+ <para>When the setting --distribute 1:<emphasis><n></emphasis> (where <emphasis><n></emphasis> is the size of the target group) is used, each source node communicates with every node in the target group.</para>
+ <para>Note that if there are more source nodes than target nodes, some source nodes may share the same target nodes. Also, if there are more target nodes than source nodes, some higher-ranked target nodes will be idle.</para>
+ <para><emphasis role="bold">Example showing a brw test:</emphasis></para>
<screen>
$ lst add_group clients 192.168.1.[10-17]@tcp
$ lst add_group servers 192.168.10.[100-103]@tcp
$ lst add_batch bulkperf
$ lst add_test --batch bulkperf --loop 100 --concurrency 4 \
--distribute 4:2 --from clients brw WRITE size=16K</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1304104" xreflabel=""/>In the example above, a batch test called bulkperf that will do a 16 kbyte bulk write request. In this test, two groups of four clients (sources) write to each of four servers (targets) as shown below:</para>
+ <para>In the example above, a batch test called bulkperf that will do a 16 kbyte bulk write request. In this test, two groups of four clients (sources) write to each of four servers (targets) as shown below:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1302964" xreflabel=""/> 192.168.1.[10-13] will write to 192.168.10.[100,101]</para>
+ <para> 192.168.1.[10-13] will write to 192.168.10.[100,101]</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1302048" xreflabel=""/> 192.168.1.[14-17] will write to 192.168.10.[102,103]</para>
+ <para> 192.168.1.[14-17] will write to 192.168.10.[102,103]</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1302046" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1302047" xreflabel=""/><emphasis role="bold">list_batch [</emphasis><emphasis><name></emphasis><emphasis role="bold">] [--test</emphasis><emphasis><index></emphasis><emphasis role="bold">] [--active] [--invalid]</emphasis> [--server | client<emphasis role="bold">]</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1304185" xreflabel=""/>Lists batches in the current session or lists client and server nodes in a batch or a test.</para>
+ <para> </para>
+ <para><emphasis role="bold">list_batch [</emphasis><emphasis><name></emphasis><emphasis role="bold">] [--test</emphasis><emphasis><index></emphasis><emphasis role="bold">] [--active] [--invalid]</emphasis> [--server | client<emphasis role="bold">]</emphasis></para>
+ <para>Lists batches in the current session or lists client and server nodes in a batch or a test.</para>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
<colspec colname="c3" colwidth="33*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1304132" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1304134" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry nameend="c3" namest="c2"><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304200" xreflabel=""/>--test<emphasis><index></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304202" xreflabel=""/>Lists tests in a batch. If no option is used, all tests in the batch are listed. IIf one of these options are used, only specified tests in the batch are listed:</para></entry>
+ <entry><para> --test<emphasis><index></emphasis></para></entry>
+ <entry nameend="c3" namest="c2"><para> Lists tests in a batch. If no option is used, all tests in the batch are listed. IIf one of these options are used, only specified tests in the batch are listed:</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304150" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304152" xreflabel=""/>active</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304154" xreflabel=""/>Lists only active batch tests.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> active</para></entry>
+ <entry><para> Lists only active batch tests.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304156" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304158" xreflabel=""/>invalid</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304160" xreflabel=""/>Lists only invalid batch tests.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> invalid</para></entry>
+ <entry><para> Lists only invalid batch tests.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304162" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304164" xreflabel=""/>server | client</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1304166" xreflabel=""/>Lists client and server nodes in a batch test.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> server | client</para></entry>
+ <entry><para> Lists client and server nodes in a batch test.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291091" xreflabel=""/><emphasis role="bold">Example:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1302124" xreflabel=""/>$ lst list_batch<anchor xml:id="dbdoclet.50438223_pgfId-1302125" xreflabel=""/>bulkperf<anchor xml:id="dbdoclet.50438223_pgfId-1302126" xreflabel=""/>$ lst list_batch bulkperf<anchor xml:id="dbdoclet.50438223_pgfId-1302127" xreflabel=""/>Batch: bulkperf Tests: 1 State: Idle<anchor xml:id="dbdoclet.50438223_pgfId-1302128" xreflabel=""/>ACTIVE BUSY DOWN UNKNOWN TOTAL<anchor xml:id="dbdoclet.50438223_pgfId-1302129" xreflabel=""/>client 8 0 0 0 8<anchor xml:id="dbdoclet.50438223_pgfId-1302130" xreflabel=""/>server 4 0 0 0 4<anchor xml:id="dbdoclet.50438223_pgfId-1302131" xreflabel=""/>Test 1(brw) (loop: 100, concurrency: 4)<anchor xml:id="dbdoclet.50438223_pgfId-1302132" xreflabel=""/>ACTIVE BUSY DOWN UNKNOWN TOTAL<anchor xml:id="dbdoclet.50438223_pgfId-1302133" xreflabel=""/>client 8 0 0 0 8<anchor xml:id="dbdoclet.50438223_pgfId-1302134" xreflabel=""/>server 4 0 0 0 4<anchor xml:id="dbdoclet.50438223_pgfId-1302135" xreflabel=""/>$ lst list_batch bulkperf --server --active<anchor xml:id="dbdoclet.50438223_pgfId-1302136" xreflabel=""/>192.168.10.100@tcp Active<anchor xml:id="dbdoclet.50438223_pgfId-1302137" xreflabel=""/>192.168.10.101@tcp Active<anchor xml:id="dbdoclet.50438223_pgfId-1302138" xreflabel=""/>192.168.10.102@tcp Active<anchor xml:id="dbdoclet.50438223_pgfId-1302139" xreflabel=""/>192.168.10.103@tcp Active</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1302115" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1304261" xreflabel=""/><emphasis role="bold">run</emphasis><emphasis><name></emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1292856" xreflabel=""/>Runs the batch.</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1293830" xreflabel=""/>$ lst run bulkperf
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst list_batchbulkperf$ lst list_batch bulkperfBatch: bulkperf Tests: 1 State: IdleACTIVE BUSY DOWN UNKNOWN TOTALclient 8 0 0 0 8server 4 0 0 0 4Test 1(brw) (loop: 100, concurrency: 4)ACTIVE BUSY DOWN UNKNOWN TOTALclient 8 0 0 0 8server 4 0 0 0 4$ lst list_batch bulkperf --server --active192.168.10.100@tcp Active192.168.10.101@tcp Active192.168.10.102@tcp Active192.168.10.103@tcp Active</screen>
+ <para> </para>
+ <para><emphasis role="bold">run</emphasis><emphasis><name></emphasis></para>
+ <para>Runs the batch.</para>
+ <screen>$ lst run bulkperf
</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1304255" xreflabel=""/><emphasis role="bold">stop</emphasis><emphasis><name></emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1292852" xreflabel=""/>Stops the batch.</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1293837" xreflabel=""/>$ lst stop bulkperf
+ <para><emphasis role="bold">stop</emphasis><emphasis><name></emphasis></para>
+ <para>Stops the batch.</para>
+ <screen>$ lst stop bulkperf
</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1304256" xreflabel=""/><emphasis role="bold">query</emphasis><emphasis><name></emphasis><emphasis role="bold">[--test</emphasis><emphasis><index></emphasis><emphasis role="bold">] [--timeout</emphasis><emphasis><seconds></emphasis><emphasis role="bold">[--loop</emphasis><emphasis><#></emphasis><emphasis role="bold">] [--delay</emphasis><emphasis><seconds></emphasis><emphasis role="bold">] [--all]</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1292848" xreflabel=""/>Queries the batch status.</para>
+ <para><emphasis role="bold">query</emphasis><emphasis><name></emphasis><emphasis role="bold">[--test</emphasis><emphasis><index></emphasis><emphasis role="bold">] [--timeout</emphasis><emphasis><seconds></emphasis><emphasis role="bold">[--loop</emphasis><emphasis><#></emphasis><emphasis role="bold">] [--delay</emphasis><emphasis><seconds></emphasis><emphasis role="bold">] [--all]</emphasis></para>
+ <para>Queries the batch status.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1302405" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1302407" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295384" xreflabel=""/>--test<emphasis><index></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295386" xreflabel=""/>Only queries the specified test. The test index starts from 1.</para></entry>
+ <entry><para> --test<emphasis><index></emphasis></para></entry>
+ <entry><para> Only queries the specified test. The test index starts from 1.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295388" xreflabel=""/>--timeout<emphasis><seconds></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295390" xreflabel=""/>The timeout value to wait for RPC. The default is 5 seconds.</para></entry>
+ <entry><para> --timeout<emphasis><seconds></emphasis></para></entry>
+ <entry><para> The timeout value to wait for RPC. The default is 5 seconds.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295392" xreflabel=""/>--loop<emphasis><#></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295394" xreflabel=""/>The loop count of the query.</para></entry>
+ <entry><para> --loop<emphasis><#></emphasis></para></entry>
+ <entry><para> The loop count of the query.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295396" xreflabel=""/>--delay<emphasis><seconds></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295398" xreflabel=""/>The interval of each query. The default is 5 seconds.</para></entry>
+ <entry><para> --delay<emphasis><seconds></emphasis></para></entry>
+ <entry><para> The interval of each query. The default is 5 seconds.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295400" xreflabel=""/>--all</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295402" xreflabel=""/>The list status of all nodes in a batch or a test.</para></entry>
+ <entry><para> --all</para></entry>
+ <entry><para> The list status of all nodes in a batch or a test.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1302191" xreflabel=""/><emphasis role="bold">Example:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1302197" xreflabel=""/>$ lst run bulkperf<anchor xml:id="dbdoclet.50438223_pgfId-1303540" xreflabel=""/>$ lst query bulkperf --loop 5 --delay 3<anchor xml:id="dbdoclet.50438223_pgfId-1303541" xreflabel=""/>Batch is running<anchor xml:id="dbdoclet.50438223_pgfId-1303542" xreflabel=""/>Batch is running<anchor xml:id="dbdoclet.50438223_pgfId-1303543" xreflabel=""/>Batch is running<anchor xml:id="dbdoclet.50438223_pgfId-1303544" xreflabel=""/>Batch is running<anchor xml:id="dbdoclet.50438223_pgfId-1303545" xreflabel=""/>Batch is running<anchor xml:id="dbdoclet.50438223_pgfId-1303546" xreflabel=""/>$ lst query bulkperf --all<anchor xml:id="dbdoclet.50438223_pgfId-1303547" xreflabel=""/>192.168.1.10@tcp Running<anchor xml:id="dbdoclet.50438223_pgfId-1303548" xreflabel=""/>192.168.1.11@tcp Running<anchor xml:id="dbdoclet.50438223_pgfId-1303549" xreflabel=""/>192.168.1.12@tcp Running<anchor xml:id="dbdoclet.50438223_pgfId-1303550" xreflabel=""/>192.168.1.13@tcp Running<anchor xml:id="dbdoclet.50438223_pgfId-1303551" xreflabel=""/>192.168.1.14@tcp Running<anchor xml:id="dbdoclet.50438223_pgfId-1303555" xreflabel=""/>192.168.1.15@tcp Running<anchor xml:id="dbdoclet.50438223_pgfId-1303556" xreflabel=""/>192.168.1.16@tcp Running<anchor xml:id="dbdoclet.50438223_pgfId-1303561" xreflabel=""/>192.168.1.17@tcp Running<anchor xml:id="dbdoclet.50438223_pgfId-1303562" xreflabel=""/>$ lst stop bulkperf<anchor xml:id="dbdoclet.50438223_pgfId-1303563" xreflabel=""/>$ lst query bulkperf<anchor xml:id="dbdoclet.50438223_pgfId-1303565" xreflabel=""/>Batch is idle</screen>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst run bulkperf$ lst query bulkperf --loop 5 --delay 3Batch is runningBatch is runningBatch is runningBatch is runningBatch is running$ lst query bulkperf --all192.168.1.10@tcp Running192.168.1.11@tcp Running192.168.1.12@tcp Running192.168.1.13@tcp Running192.168.1.14@tcp Running192.168.1.15@tcp Running192.168.1.16@tcp Running192.168.1.17@tcp Running$ lst stop bulkperf$ lst query bulkperfBatch is idle</screen>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438223_pgfId-1291130" xreflabel=""/>23.3.4 Other Commands</title>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1293485" xreflabel=""/>This section describes other lst commands.</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291131" xreflabel=""/><emphasis role="bold">ping [-session] [--group</emphasis><emphasis><name></emphasis><emphasis role="bold">] [--nodes</emphasis><emphasis><NIDs></emphasis><emphasis role="bold">] [--batch</emphasis><emphasis><name></emphasis><emphasis role="bold">] [--server] [--timeout</emphasis><emphasis><seconds></emphasis><emphasis role="bold">]</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1295479" xreflabel=""/>Sends a 'hello' query to the nodes.</para>
+ <title>23.3.4 Other Commands</title>
+ <para>This section describes other lst commands.</para>
+ <para><emphasis role="bold">ping [-session] [--group</emphasis><emphasis><name></emphasis><emphasis role="bold">] [--nodes</emphasis><emphasis><NIDs></emphasis><emphasis role="bold">] [--batch</emphasis><emphasis><name></emphasis><emphasis role="bold">] [--server] [--timeout</emphasis><emphasis><seconds></emphasis><emphasis role="bold">]</emphasis></para>
+ <para>Sends a 'hello' query to the nodes.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1302286" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1302288" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295566" xreflabel=""/>--session</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295568" xreflabel=""/>Pings all nodes in the current session.</para></entry>
+ <entry><para> --session</para></entry>
+ <entry><para> Pings all nodes in the current session.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295570" xreflabel=""/>--group<emphasis><name></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295572" xreflabel=""/>Pings all nodes in a specified group.</para></entry>
+ <entry><para> --group<emphasis><name></emphasis></para></entry>
+ <entry><para> Pings all nodes in a specified group.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295574" xreflabel=""/>--nodes<emphasis><NIDs></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295576" xreflabel=""/>Pings all specified nodes.</para></entry>
+ <entry><para> --nodes<emphasis><NIDs></emphasis></para></entry>
+ <entry><para> Pings all specified nodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295578" xreflabel=""/>--batch<emphasis><name></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295580" xreflabel=""/>Pings all client nodes in a batch.</para></entry>
+ <entry><para> --batch<emphasis><name></emphasis></para></entry>
+ <entry><para> Pings all client nodes in a batch.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295582" xreflabel=""/>--server</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295584" xreflabel=""/>Sends RPC to all server nodes instead of client nodes. This option is only used with --batch<emphasis><name></emphasis>.</para></entry>
+ <entry><para> --server</para></entry>
+ <entry><para> Sends RPC to all server nodes instead of client nodes. This option is only used with --batch<emphasis><name></emphasis>.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295586" xreflabel=""/>--timeout<emphasis><seconds></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295588" xreflabel=""/>The RPC timeout value.</para></entry>
+ <entry><para> --timeout<emphasis><seconds></emphasis></para></entry>
+ <entry><para> The RPC timeout value.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1295515" xreflabel=""/><emphasis role="bold">Example:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1302267" xreflabel=""/>$ lst ping 192.168.10.[15-20]@tcp<anchor xml:id="dbdoclet.50438223_pgfId-1302268" xreflabel=""/>192.168.1.15@tcp Active [session: liang id: 192.168.1.3@tcp]<anchor xml:id="dbdoclet.50438223_pgfId-1302269" xreflabel=""/>192.168.1.16@tcp Active [session: liang id: 192.168.1.3@tcp]<anchor xml:id="dbdoclet.50438223_pgfId-1302270" xreflabel=""/>192.168.1.17@tcp Active [session: liang id: 192.168.1.3@tcp]<anchor xml:id="dbdoclet.50438223_pgfId-1302271" xreflabel=""/>192.168.1.18@tcp Busy [session: Isaac id: 192.168.10.10@tcp]<anchor xml:id="dbdoclet.50438223_pgfId-1302272" xreflabel=""/>192.168.1.19@tcp Down [session: <NULL> id: LNET_NID_ANY]<anchor xml:id="dbdoclet.50438223_pgfId-1302273" xreflabel=""/>192.168.1.20@tcp Down [session: <NULL> id: LNET_NID_ANY]</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1302262" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291155" xreflabel=""/><emphasis role="bold">stat [--bw] [--rate] [--read] [--write] [--max] [--min] [--avg] " " [--timeout</emphasis><emphasis><seconds></emphasis><emphasis role="bold">] [--delay</emphasis><emphasis><seconds></emphasis><emphasis role="bold">]</emphasis><emphasis><group></emphasis><emphasis role="bold">|<</emphasis><emphasis>NIDs></emphasis><emphasis role="bold">[</emphasis><emphasis><group></emphasis><emphasis role="bold">|</emphasis><emphasis><NIDs></emphasis><emphasis role="bold">]</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1298213" xreflabel=""/>The collection performance and RPC statistics of one or more nodes.</para>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst ping 192.168.10.[15-20]@tcp192.168.1.15@tcp Active [session: liang id: 192.168.1.3@tcp]192.168.1.16@tcp Active [session: liang id: 192.168.1.3@tcp]192.168.1.17@tcp Active [session: liang id: 192.168.1.3@tcp]192.168.1.18@tcp Busy [session: Isaac id: 192.168.10.10@tcp]192.168.1.19@tcp Down [session: <NULL> id: LNET_NID_ANY]192.168.1.20@tcp Down [session: <NULL> id: LNET_NID_ANY]</screen>
+ <para> </para>
+ <para><emphasis role="bold">stat [--bw] [--rate] [--read] [--write] [--max] [--min] [--avg] " " [--timeout</emphasis><emphasis><seconds></emphasis><emphasis role="bold">] [--delay</emphasis><emphasis><seconds></emphasis><emphasis role="bold">]</emphasis><emphasis><group></emphasis><emphasis role="bold">|<</emphasis><emphasis>NIDs></emphasis><emphasis role="bold">[</emphasis><emphasis><group></emphasis><emphasis role="bold">|</emphasis><emphasis><NIDs></emphasis><emphasis role="bold">]</emphasis></para>
+ <para>The collection performance and RPC statistics of one or more nodes.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1302370" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1302372" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295659" xreflabel=""/>--bw</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295661" xreflabel=""/>Displays the bandwidth of the specified group/nodes.</para></entry>
+ <entry><para> --bw</para></entry>
+ <entry><para> Displays the bandwidth of the specified group/nodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295663" xreflabel=""/>--rate</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295665" xreflabel=""/>Displays the rate of RPCs of the specified group/nodes.</para></entry>
+ <entry><para> --rate</para></entry>
+ <entry><para> Displays the rate of RPCs of the specified group/nodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295667" xreflabel=""/>--read</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295669" xreflabel=""/>Displays the read statistics of the specified group/nodes.</para></entry>
+ <entry><para> --read</para></entry>
+ <entry><para> Displays the read statistics of the specified group/nodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295671" xreflabel=""/>--write</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295673" xreflabel=""/>Displays the write statistics of the specified group/nodes.</para></entry>
+ <entry><para> --write</para></entry>
+ <entry><para> Displays the write statistics of the specified group/nodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295675" xreflabel=""/>--max</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295677" xreflabel=""/>Displays the maximum value of the statistics.</para></entry>
+ <entry><para> --max</para></entry>
+ <entry><para> Displays the maximum value of the statistics.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295679" xreflabel=""/>--min</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295681" xreflabel=""/>Displays the minimum value of the statistics.</para></entry>
+ <entry><para> --min</para></entry>
+ <entry><para> Displays the minimum value of the statistics.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295683" xreflabel=""/>--avg</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295685" xreflabel=""/>Displays the average of the statistics.</para></entry>
+ <entry><para> --avg</para></entry>
+ <entry><para> Displays the average of the statistics.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295687" xreflabel=""/>--timeout<emphasis><seconds></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295689" xreflabel=""/>The timeout of the statistics RPC. The default is 5 seconds.</para></entry>
+ <entry><para> --timeout<emphasis><seconds></emphasis></para></entry>
+ <entry><para> The timeout of the statistics RPC. The default is 5 seconds.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295691" xreflabel=""/>--delay<emphasis><seconds></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295693" xreflabel=""/>The interval of the statistics (in seconds).</para></entry>
+ <entry><para> --delay<emphasis><seconds></emphasis></para></entry>
+ <entry><para> The interval of the statistics (in seconds).</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1302348" xreflabel=""/><emphasis role="bold">Example:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1302353" xreflabel=""/>$ lst run bulkperf<anchor xml:id="dbdoclet.50438223_pgfId-1302354" xreflabel=""/>$ lst stat clients<anchor xml:id="dbdoclet.50438223_pgfId-1302355" xreflabel=""/>[LNet Rates of clients]<anchor xml:id="dbdoclet.50438223_pgfId-1302356" xreflabel=""/>[W] Avg: 1108 RPC/s Min: 1060 RPC/s Max: 1155 RPC/s<anchor xml:id="dbdoclet.50438223_pgfId-1302357" xreflabel=""/>[R] Avg: 2215 RPC/s Min: 2121 RPC/s Max: 2310 RPC/s<anchor xml:id="dbdoclet.50438223_pgfId-1302358" xreflabel=""/>[LNet Bandwidth of clients]<anchor xml:id="dbdoclet.50438223_pgfId-1302359" xreflabel=""/>[W] Avg: 16.60 MB/s Min: 16.10 MB/s Max: 17.1 MB/s<anchor xml:id="dbdoclet.50438223_pgfId-1302349" xreflabel=""/>[R] Avg: 40.49 MB/s Min: 40.30 MB/s Max: 40.68 MB/s</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303074" xreflabel=""/>Specifying a group name (<emphasis><group></emphasis>) causes statistics to be gathered for all nodes in a test group. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1303075" xreflabel=""/>$ lst stat servers
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst run bulkperf$ lst stat clients[LNet Rates of clients][W] Avg: 1108 RPC/s Min: 1060 RPC/s Max: 1155 RPC/s[R] Avg: 2215 RPC/s Min: 2121 RPC/s Max: 2310 RPC/s[LNet Bandwidth of clients][W] Avg: 16.60 MB/s Min: 16.10 MB/s Max: 17.1 MB/s[R] Avg: 40.49 MB/s Min: 40.30 MB/s Max: 40.68 MB/s</screen>
+ <para>Specifying a group name (<emphasis><group></emphasis>) causes statistics to be gathered for all nodes in a test group. For example:</para>
+ <screen>$ lst stat servers
</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303076" xreflabel=""/>where servers is the name of a test group created by lst add_group</para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1303077" xreflabel=""/>Specifying a NID range (<emphasis><NIDs></emphasis>) causes statistics to be gathered for selected nodes. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1303078" xreflabel=""/>$ lst stat 192.168.0.[1-100/2]@tcp
-<anchor xml:id="dbdoclet.50438223_pgfId-1303072" xreflabel=""/>Only LNET performance statistics are available. By default, all statistics \
+ <para>where servers is the name of a test group created by lst add_group</para>
+ <para>Specifying a NID range (<emphasis><NIDs></emphasis>) causes statistics to be gathered for selected nodes. For example:</para>
+ <screen>$ lst stat 192.168.0.[1-100/2]@tcp
+Only LNET performance statistics are available. By default, all statistics \
information is displayed. Users can specify additional information with the\
se options.
-<anchor xml:id="dbdoclet.50438223_pgfId-1303090" xreflabel=""/>
+
</screen>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1304374" xreflabel=""/><emphasis role="bold">show_error [--session] [<group>|<NIDs>]...</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1291189" xreflabel=""/>Lists the number of failed RPCs on test nodes.</para>
+ <para><emphasis role="bold">show_error [--session] [<group>|<NIDs>]...</emphasis></para>
+ <para>Lists the number of failed RPCs on test nodes.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1302397" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438223_pgfId-1302399" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295739" xreflabel=""/>--session</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438223_pgfId-1295741" xreflabel=""/>Lists errors in the current test session. With this option, historical RPC errors are not listed.</para></entry>
+ <entry><para> --session</para></entry>
+ <entry><para> Lists errors in the current test session. With this option, historical RPC errors are not listed.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438223_pgfId-1302373" xreflabel=""/><emphasis role="bold">Example:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438223_pgfId-1302379" xreflabel=""/>$ lst show_error clientsclients<anchor xml:id="dbdoclet.50438223_pgfId-1302380" xreflabel=""/>12345-192.168.1.15@tcp: [Session: 1 brw errors, 0 ping errors] \<anchor xml:id="dbdoclet.50438223_pgfId-1303569" xreflabel=""/>[RPC: 20 errors, 0 dropped,<anchor xml:id="dbdoclet.50438223_pgfId-1303575" xreflabel=""/>12345-192.168.1.16@tcp: [Session: 0 brw errors, 0 ping errors] \<anchor xml:id="dbdoclet.50438223_pgfId-1303570" xreflabel=""/>[RPC: 1 errors, 0 dropped, Total 2 error nodes in clients<anchor xml:id="dbdoclet.50438223_pgfId-1303571" xreflabel=""/>$ lst show_error --session clients<anchor xml:id="dbdoclet.50438223_pgfId-1302382" xreflabel=""/>clients<anchor xml:id="dbdoclet.50438223_pgfId-1302383" xreflabel=""/>12345-192.168.1.15@tcp: [Session: 1 brw errors, 0 ping errors]<anchor xml:id="dbdoclet.50438223_pgfId-1302384" xreflabel=""/>Total 1 error nodes in clients</screen>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst show_error clientsclients12345-192.168.1.15@tcp: [Session: 1 brw errors, 0 ping errors] \[RPC: 20 errors, 0 dropped,12345-192.168.1.16@tcp: [Session: 0 brw errors, 0 ping errors] \[RPC: 1 errors, 0 dropped, Total 2 error nodes in clients$ lst show_error --session clientsclients12345-192.168.1.15@tcp: [Session: 1 brw errors, 0 ping errors]Total 1 error nodes in clients</screen>
</section>
</section>
</chapter>
<title xml:id='lustredebugging.title'>Lustre Debugging</title>
</info>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295651" xreflabel=""/>This chapter describes tips and information to debug Lustre, and includes the following sections:</para>
+ <para>This chapter describes tips and information to debug Lustre, and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438274_15874"/></para>
</listitem>
<section xml:id="dbdoclet.50438274_15874">
<title>28.1 Diagnostic and Debugging Tools</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295666" xreflabel=""/>A variety of diagnostic and analysis tools are available to debug issues with the Lustre software. Some of these are provided in Linux distributions, while others have been developed and are made available by the Lustre project.</para>
+ <para>A variety of diagnostic and analysis tools are available to debug issues with the Lustre software. Some of these are provided in Linux distributions, while others have been developed and are made available by the Lustre project.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438274_pgfId-1295667" xreflabel=""/>28.1.1 Lustre Debugging Tools</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295668" xreflabel=""/>The following in-kernel debug mechanisms are incorporated into the Lustre software:</para>
+ <title>28.1.1 Lustre Debugging Tools</title>
+ <para>The following in-kernel debug mechanisms are incorporated into the Lustre software:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295669" xreflabel=""/><emphasis role="bold">Debug logs</emphasis> - A circular debug buffer to which Lustre internal debug messages are written (in contrast to error messages, which are printed to the syslog or console). Entries to the Lustre debug log are controlled by the mask set by /proc/sys/lnet/debug. The log size defaults to 5 MB per CPU but can be increased as a busy system will quickly overwrite 5 MB. When the buffer fills, the oldest information is discarded.</para>
+ <para><emphasis role="bold">Debug logs</emphasis> - A circular debug buffer to which Lustre internal debug messages are written (in contrast to error messages, which are printed to the syslog or console). Entries to the Lustre debug log are controlled by the mask set by /proc/sys/lnet/debug. The log size defaults to 5 MB per CPU but can be increased as a busy system will quickly overwrite 5 MB. When the buffer fills, the oldest information is discarded.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295670" xreflabel=""/><emphasis role="bold">Debug daemon</emphasis> - The debug daemon controls logging of debug messages.</para>
+ <para><emphasis role="bold">Debug daemon</emphasis> - The debug daemon controls logging of debug messages.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295671" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/debug</emphasis> - This file contains a mask that can be used to delimit the debugging information written out to the kernel debug logs.</para>
+ <para><emphasis role="bold">/proc/sys/lnet/debug</emphasis> - This file contains a mask that can be used to delimit the debugging information written out to the kernel debug logs.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295672" xreflabel=""/>The following tools are also provided with the Lustre software:</para>
+ <para>The following tools are also provided with the Lustre software:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295673" xreflabel=""/><emphasis role="bold">lctl</emphasis> - This tool is used with the debug_kernel option to manually dump the Lustre debugging log or post-process debugging logs that are dumped automatically. For more information about the lctl tool, see <xref linkend="dbdoclet.50438274_62472"/> and <xref linkend="systemconfigurationutilities"/>(lctl).</para>
+ <para><emphasis role="bold">lctl</emphasis> - This tool is used with the debug_kernel option to manually dump the Lustre debugging log or post-process debugging logs that are dumped automatically. For more information about the lctl tool, see <xref linkend="dbdoclet.50438274_62472"/> and <xref linkend="systemconfigurationutilities"/>(lctl).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295680" xreflabel=""/><emphasis role="bold">Lustre subsystem asserts</emphasis> - A panic-style assertion (LBUG) in the kernel causes Lustre to dump the debug log to the file /tmp/lustre-log.<emphasis><timestamp></emphasis> where it can be retrieved after a reboot. For more information, see <link xl:href="LustreTroubleshooting.html#50438198_40669">Viewing Error Messages</link>.</para>
+ <para><emphasis role="bold">Lustre subsystem asserts</emphasis> - A panic-style assertion (LBUG) in the kernel causes Lustre to dump the debug log to the file /tmp/lustre-log.<emphasis><timestamp></emphasis> where it can be retrieved after a reboot. For more information, see <link xl:href="LustreTroubleshooting.html#50438198_40669">Viewing Error Messages</link>.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295684" xreflabel=""/><emphasis role="bold">lfs</emphasis> - This utility provides access to the extended attributes (EAs) of a Lustre file (along with other information). For more inforamtion about lfs, see <link xl:href="UserUtilities.html#50438206_94597">lfs</link>.</para>
+ <para><emphasis role="bold">lfs</emphasis> - This utility provides access to the extended attributes (EAs) of a Lustre file (along with other information). For more inforamtion about lfs, see <link xl:href="UserUtilities.html#50438206_94597">lfs</link>.</para>
</listitem>
</itemizedlist>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438274_pgfId-1295688" xreflabel=""/>28.1.2 External Debugging Tools</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295689" xreflabel=""/>The tools described in this section are provided in the Linux kernel or are available at an external website. For information about using some of these tools for Lustre debugging, see <link xl:href="LustreDebugging.html#50438274_23607">Lustre Debugging Procedures</link> and <link xl:href="LustreDebugging.html#50438274_80443">Lustre Debugging for Developers</link>.</para>
+ <title>28.1.2 External Debugging Tools</title>
+ <para>The tools described in this section are provided in the Linux kernel or are available at an external website. For information about using some of these tools for Lustre debugging, see <link xl:href="LustreDebugging.html#50438274_23607">Lustre Debugging Procedures</link> and <link xl:href="LustreDebugging.html#50438274_80443">Lustre Debugging for Developers</link>.</para>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438274_pgfId-1295696" xreflabel=""/>28.1.2.1 Tools for Administrators and Developers</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295697" xreflabel=""/>Some general debugging tools provided as a part of the standard Linux distro are:</para>
+ <title>28.1.2.1 Tools for Administrators and Developers</title>
+ <para>Some general debugging tools provided as a part of the standard Linux distro are:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295698" xreflabel=""/><emphasis role="bold">strace</emphasis> . This tool allows a system call to be traced.</para>
+ <para><emphasis role="bold">strace</emphasis> . This tool allows a system call to be traced.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295699" xreflabel=""/><emphasis role="bold">/var/log/messages</emphasis> . syslogd prints fatal or serious messages at this log.</para>
+ <para><emphasis role="bold">/var/log/messages</emphasis> . syslogd prints fatal or serious messages at this log.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295700" xreflabel=""/><emphasis role="bold">Crash dumps</emphasis> . On crash-dump enabled kernels, sysrq c produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64 KB of the log) to the console.</para>
+ <para><emphasis role="bold">Crash dumps</emphasis> . On crash-dump enabled kernels, sysrq c produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64 KB of the log) to the console.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295701" xreflabel=""/><emphasis role="bold">debugfs</emphasis> . Interactive file system debugger.</para>
+ <para><emphasis role="bold">debugfs</emphasis> . Interactive file system debugger.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295702" xreflabel=""/>The following logging and data collection tools can be used to collect information for debugging Lustre kernel issues:</para>
+ <para>The following logging and data collection tools can be used to collect information for debugging Lustre kernel issues:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295703" xreflabel=""/><emphasis role="bold">kdump</emphasis> . A Linux kernel crash utility useful for debugging a system running Red Hat Enterprise Linux. For more information about kdump, see the Red Hat knowledge base article <link xl:href="http://kbase.redhat.com/faq/docs/DOC-6039">How do I configure kexec/kdump on Red Hat Enterprise Linux 5?</link>. To download kdump, go to the <link xl:href="http://fedoraproject.org/wiki/SystemConfig/kdump#Download">Fedora Project Download</link> site.</para>
+ <para><emphasis role="bold">kdump</emphasis> . A Linux kernel crash utility useful for debugging a system running Red Hat Enterprise Linux. For more information about kdump, see the Red Hat knowledge base article <link xl:href="http://kbase.redhat.com/faq/docs/DOC-6039">How do I configure kexec/kdump on Red Hat Enterprise Linux 5?</link>. To download kdump, go to the <link xl:href="http://fedoraproject.org/wiki/SystemConfig/kdump#Download">Fedora Project Download</link> site.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295706" xreflabel=""/><emphasis role="bold">netconsole</emphasis> . Enables kernel-level network logging over UDP. A system requires (SysRq) allows users to collect relevant data through netconsole.</para>
+ <para><emphasis role="bold">netconsole</emphasis> . Enables kernel-level network logging over UDP. A system requires (SysRq) allows users to collect relevant data through netconsole.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295707" xreflabel=""/><emphasis role="bold">netdump</emphasis> . A crash dump utility from Red Hat that allows memory images to be dumped over a network to a central server for analysis. The netdump utility was replaced by kdump in RHEL 5. For more information about netdump, see <link xl:href="http://www.redhat.com/support/wpapers/redhat/netdump/">Red Hat, Inc.'s Network Console and Crash Dump Facility</link>.</para>
+ <para><emphasis role="bold">netdump</emphasis> . A crash dump utility from Red Hat that allows memory images to be dumped over a network to a central server for analysis. The netdump utility was replaced by kdump in RHEL 5. For more information about netdump, see <link xl:href="http://www.redhat.com/support/wpapers/redhat/netdump/">Red Hat, Inc.'s Network Console and Crash Dump Facility</link>.</para>
</listitem>
</itemizedlist>
</section>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438274_pgfId-1295709" xreflabel=""/>28.1.2.2 Tools for Developers</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295710" xreflabel=""/>The tools described in this section may be useful for debugging Lustre in a development environment.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295711" xreflabel=""/>Of general interest is:</para>
+ <title>28.1.2.2 Tools for Developers</title>
+ <para>The tools described in this section may be useful for debugging Lustre in a development environment.</para>
+ <para>Of general interest is:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295712" xreflabel=""/><emphasis role="bold">leak_finder.pl</emphasis> . This program provided with Lustre is useful for finding memory leaks in the code.</para>
+ <para><emphasis role="bold">leak_finder.pl</emphasis> . This program provided with Lustre is useful for finding memory leaks in the code.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295713" xreflabel=""/>A virtual machine is often used to create an isolated development and test environment. Some commonly-used virtual machines are:</para>
+ <para>A virtual machine is often used to create an isolated development and test environment. Some commonly-used virtual machines are:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295714" xreflabel=""/><emphasis role="bold">VirtualBox Open Source Edition</emphasis> . Provides enterprise-class virtualization capability for all major platforms and is available free at <link xl:href="http://www.sun.com/software/products/virtualbox/get.jsp?intcmp=2945">Get Sun VirtualBox</link>.</para>
+ <para><emphasis role="bold">VirtualBox Open Source Edition</emphasis> . Provides enterprise-class virtualization capability for all major platforms and is available free at <link xl:href="http://www.sun.com/software/products/virtualbox/get.jsp?intcmp=2945">Get Sun VirtualBox</link>.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295716" xreflabel=""/><emphasis role="bold">VMware Server</emphasis> . Virtualization platform available as free introductory software at <link xl:href="http://downloads.vmware.com/d/info/datacenter_downloads/vmware_server/2_0">Download VMware Server</link>.</para>
+ <para><emphasis role="bold">VMware Server</emphasis> . Virtualization platform available as free introductory software at <link xl:href="http://downloads.vmware.com/d/info/datacenter_downloads/vmware_server/2_0">Download VMware Server</link>.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295718" xreflabel=""/><emphasis role="bold">Xen</emphasis> . A para-virtualized environment with virtualization capabilities similar to VMware Server and Virtual Box. However, Xen allows the use of modified kernels to provide near-native performance and the ability to emulate shared storage. For more information, go to <link xl:href="http://xen.org/">xen.org</link>.</para>
+ <para><emphasis role="bold">Xen</emphasis> . A para-virtualized environment with virtualization capabilities similar to VMware Server and Virtual Box. However, Xen allows the use of modified kernels to provide near-native performance and the ability to emulate shared storage. For more information, go to <link xl:href="http://xen.org/">xen.org</link>.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295720" xreflabel=""/>A variety of debuggers and analysis tools are available including:</para>
+ <para>A variety of debuggers and analysis tools are available including:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295721" xreflabel=""/><emphasis role="bold">kgdb</emphasis> . The Linux Kernel Source Level Debugger kgdb is used in conjunction with the GNU Debugger gdb for debugging the Linux kernel. For more information about using kgdb with gdb, see <link xl:href="http://www.linuxtopia.org/online_books/redhat_linux_debugging_with_gdb/running.html">Chapter 6. Running Programs Under gdb</link> in the <emphasis>Red Hat Linux 4 Debugging with GDB</emphasis> guide.</para>
+ <para><emphasis role="bold">kgdb</emphasis> . The Linux Kernel Source Level Debugger kgdb is used in conjunction with the GNU Debugger gdb for debugging the Linux kernel. For more information about using kgdb with gdb, see <link xl:href="http://www.linuxtopia.org/online_books/redhat_linux_debugging_with_gdb/running.html">Chapter 6. Running Programs Under gdb</link> in the <emphasis>Red Hat Linux 4 Debugging with GDB</emphasis> guide.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295723" xreflabel=""/><emphasis role="bold">crash</emphasis> . Used to analyze saved crash dump data when a system had panicked or locked up or appears unresponsive. For more information about using crash to analyze a crash dump, see:</para>
+ <para><emphasis role="bold">crash</emphasis> . Used to analyze saved crash dump data when a system had panicked or locked up or appears unresponsive. For more information about using crash to analyze a crash dump, see:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295725" xreflabel=""/> Red Hat Magazine article: <link xl:href="http://magazine.redhat.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/">A quick overview of Linux kernel crash dump analysis</link></para>
+ <para> Red Hat Magazine article: <link xl:href="http://magazine.redhat.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/">A quick overview of Linux kernel crash dump analysis</link></para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295727" xreflabel=""/><link xl:href="http://people.redhat.com/anderson/crash_whitepaper/#EXAMPLES">Crash Usage: A Case Study</link> from the white paper <emphasis>Red Hat Crash Utility</emphasis> by David Anderson</para>
+ <para><link xl:href="http://people.redhat.com/anderson/crash_whitepaper/#EXAMPLES">Crash Usage: A Case Study</link> from the white paper <emphasis>Red Hat Crash Utility</emphasis> by David Anderson</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295729" xreflabel=""/> Kernel Trap forum entry: <link xl:href="http://kerneltrap.org/node/5758">Linux: Kernel Crash Dumps</link></para>
+ <para> Kernel Trap forum entry: <link xl:href="http://kerneltrap.org/node/5758">Linux: Kernel Crash Dumps</link></para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295731" xreflabel=""/> White paper: <link xl:href="http://www.google.com/url?sa=t&source=web&ct=res&cd=8&ved=0CCUQFjAH&url=http%3A%2F%2Fwww.kernel.sg%2Fpapers%2Fcrash-dump-analysis.pdf&rct=j&q=redhat+crash+dump&ei=6aQBS-ifK4T8tAPcjdiHCw&usg=AFQjCNEk03E3GDtAsawG3gfpwc1gGNELAg">A Quick Overview of Linux Kernel Crash Dump Analysis</link></para>
+ <para> White paper: <link xl:href="http://www.google.com/url?sa=t&source=web&ct=res&cd=8&ved=0CCUQFjAH&url=http%3A%2F%2Fwww.kernel.sg%2Fpapers%2Fcrash-dump-analysis.pdf&rct=j&q=redhat+crash+dump&ei=6aQBS-ifK4T8tAPcjdiHCw&usg=AFQjCNEk03E3GDtAsawG3gfpwc1gGNELAg">A Quick Overview of Linux Kernel Crash Dump Analysis</link></para>
</listitem>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438274_23607">
<title>28.2 Lustre Debugging Procedures</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295734" xreflabel=""/>The procedures below may be useful to administrators or developers debugging a Lustre files system.</para>
+ <para>The procedures below may be useful to administrators or developers debugging a Lustre files system.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438274_pgfId-1295735" xreflabel=""/>28.2.1 Understanding the Lustre Debug Messaging Format</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295736" xreflabel=""/>Lustre debug messages are categorized by originating sybsystem, message type, and locaton in the source code. For a list of subsystems and message types, see <xref linkend="dbdoclet.50438274_57603"/>.</para>
+ <title>28.2.1 Understanding the Lustre Debug Messaging Format</title>
+ <para>Lustre debug messages are categorized by originating sybsystem, message type, and locaton in the source code. For a list of subsystems and message types, see <xref linkend="dbdoclet.50438274_57603"/>.</para>
<note><para>For a current list of subsystems and debug message types, see lnet/include/libcfs/libcfs.h in the Lustre tree</para></note>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295743" xreflabel=""/>The elements of a Lustre debug message are described in <xref linkend="dbdoclet.50438274_57177"/>Format of Lustre Debug Messages.</para>
+ <para>The elements of a Lustre debug message are described in <xref linkend="dbdoclet.50438274_57177"/>Format of Lustre Debug Messages.</para>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1295747" xreflabel=""/>28.2.1.1 <anchor xml:id="dbdoclet.50438274_57603" xreflabel=""/>Lustre <anchor xml:id="dbdoclet.50438274_marker-1295746" xreflabel=""/>Debug Messages</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295748" xreflabel=""/>Each Lustre debug message has the tag of the subsystem it originated in, the message type, and the location in the source code. The subsystems and debug types used in Lustre are as follows:</para>
+ <title>28.2.1.1 <anchor xml:id="dbdoclet.50438274_57603" xreflabel=""/>Lustre <anchor xml:id="dbdoclet.50438274_marker-1295746" xreflabel=""/>Debug Messages</title>
+ <para>Each Lustre debug message has the tag of the subsystem it originated in, the message type, and the location in the source code. The subsystems and debug types used in Lustre are as follows:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295749" xreflabel=""/> Standard Subsystems:</para>
+ <para> Standard Subsystems:</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295750" xreflabel=""/> mdc, mds, osc, ost, obdclass, obdfilter, llite, ptlrpc, portals, lnd, ldlm, lov</para>
+ <para> mdc, mds, osc, ost, obdclass, obdfilter, llite, ptlrpc, portals, lnd, ldlm, lov</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295840" xreflabel=""/> Debug Types:</para>
+ <para> Debug Types:</para>
</listitem>
<listitem>
<para><informaltable frame="all">
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1295753" xreflabel=""/>Types</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1295755" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Types</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295757" xreflabel=""/><emphasis role="bold">trace</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295759" xreflabel=""/>Entry/Exit markers</para></entry>
+ <entry><para> <emphasis role="bold">trace</emphasis></para></entry>
+ <entry><para> Entry/Exit markers</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295761" xreflabel=""/><emphasis role="bold">dlmtrace</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295763" xreflabel=""/>Locking-related information</para></entry>
+ <entry><para> <emphasis role="bold">dlmtrace</emphasis></para></entry>
+ <entry><para> Locking-related information</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295765" xreflabel=""/><emphasis role="bold">inode</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295767" xreflabel=""/> </para></entry>
+ <entry><para> <emphasis role="bold">inode</emphasis></para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295769" xreflabel=""/><emphasis role="bold">super</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295771" xreflabel=""/> </para></entry>
+ <entry><para> <emphasis role="bold">super</emphasis></para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295773" xreflabel=""/><emphasis role="bold">ext2</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295775" xreflabel=""/>Anything from the ext2_debug</para></entry>
+ <entry><para> <emphasis role="bold">ext2</emphasis></para></entry>
+ <entry><para> Anything from the ext2_debug</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295777" xreflabel=""/><emphasis role="bold">malloc</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295779" xreflabel=""/>Print malloc or free information</para></entry>
+ <entry><para> <emphasis role="bold">malloc</emphasis></para></entry>
+ <entry><para> Print malloc or free information</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295781" xreflabel=""/><emphasis role="bold">cache</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295783" xreflabel=""/>Cache-related information</para></entry>
+ <entry><para> <emphasis role="bold">cache</emphasis></para></entry>
+ <entry><para> Cache-related information</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295785" xreflabel=""/><emphasis role="bold">info</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295787" xreflabel=""/>General information</para></entry>
+ <entry><para> <emphasis role="bold">info</emphasis></para></entry>
+ <entry><para> General information</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295789" xreflabel=""/><emphasis role="bold">ioctl</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295791" xreflabel=""/>IOCTL-related information</para></entry>
+ <entry><para> <emphasis role="bold">ioctl</emphasis></para></entry>
+ <entry><para> IOCTL-related information</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295793" xreflabel=""/><emphasis role="bold">blocks</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295795" xreflabel=""/>Ext2 block allocation information</para></entry>
+ <entry><para> <emphasis role="bold">blocks</emphasis></para></entry>
+ <entry><para> Ext2 block allocation information</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295797" xreflabel=""/><emphasis role="bold">net</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295799" xreflabel=""/>Networking</para></entry>
+ <entry><para> <emphasis role="bold">net</emphasis></para></entry>
+ <entry><para> Networking</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295801" xreflabel=""/><emphasis role="bold">warning</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295803" xreflabel=""/> </para></entry>
+ <entry><para> <emphasis role="bold">warning</emphasis></para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295805" xreflabel=""/><emphasis role="bold">buffs</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295807" xreflabel=""/> </para></entry>
+ <entry><para> <emphasis role="bold">buffs</emphasis></para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295809" xreflabel=""/><emphasis role="bold">other</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295811" xreflabel=""/> </para></entry>
+ <entry><para> <emphasis role="bold">other</emphasis></para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295813" xreflabel=""/><emphasis role="bold">dentry</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295815" xreflabel=""/> </para></entry>
+ <entry><para> <emphasis role="bold">dentry</emphasis></para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295817" xreflabel=""/><emphasis role="bold">portals</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295819" xreflabel=""/>Entry/Exit markers</para></entry>
+ <entry><para> <emphasis role="bold">portals</emphasis></para></entry>
+ <entry><para> Entry/Exit markers</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295821" xreflabel=""/><emphasis role="bold">page</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295823" xreflabel=""/>Bulk page handling</para></entry>
+ <entry><para> <emphasis role="bold">page</emphasis></para></entry>
+ <entry><para> Bulk page handling</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295825" xreflabel=""/><emphasis role="bold">error</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295827" xreflabel=""/>Error messages</para></entry>
+ <entry><para> <emphasis role="bold">error</emphasis></para></entry>
+ <entry><para> Error messages</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295829" xreflabel=""/><emphasis role="bold">emerg</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295831" xreflabel=""/> </para></entry>
+ <entry><para> <emphasis role="bold">emerg</emphasis></para></entry>
+ <entry><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295833" xreflabel=""/><emphasis role="bold">rpctrace</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295835" xreflabel=""/>For distributed debugging</para></entry>
+ <entry><para> <emphasis role="bold">rpctrace</emphasis></para></entry>
+ <entry><para> For distributed debugging</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295837" xreflabel=""/><emphasis role="bold">ha</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295839" xreflabel=""/>Failover and recovery-related information</para></entry>
+ <entry><para> <emphasis role="bold">ha</emphasis></para></entry>
+ <entry><para> Failover and recovery-related information</para></entry>
</row>
</tbody>
</tgroup>
</itemizedlist>
</section>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1295842" xreflabel=""/>28.2.1.2 <anchor xml:id="dbdoclet.50438274_57177" xreflabel=""/>Format of Lustre Debug Messages</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295843" xreflabel=""/>Lustre uses the CDEBUG and CERROR macros to print the debug or error messages. To print the message, the CDEBUG macro uses portals_debug_msg (portals/linux/oslib/debug.c). The message format is described below, along with an example.</para>
+ <title>28.2.1.2 <anchor xml:id="dbdoclet.50438274_57177" xreflabel=""/>Format of Lustre Debug Messages</title>
+ <para>Lustre uses the CDEBUG and CERROR macros to print the debug or error messages. To print the message, the CDEBUG macro uses portals_debug_msg (portals/linux/oslib/debug.c). The message format is described below, along with an example.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1295846" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1295848" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295850" xreflabel=""/><emphasis role="bold">subsystem</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295852" xreflabel=""/>800000</para></entry>
+ <entry><para> <emphasis role="bold">subsystem</emphasis></para></entry>
+ <entry><para> 800000</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295854" xreflabel=""/><emphasis role="bold">debug mask</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295856" xreflabel=""/>000010</para></entry>
+ <entry><para> <emphasis role="bold">debug mask</emphasis></para></entry>
+ <entry><para> 000010</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295858" xreflabel=""/><emphasis role="bold">smp_processor_id</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295860" xreflabel=""/>0</para></entry>
+ <entry><para> <emphasis role="bold">smp_processor_id</emphasis></para></entry>
+ <entry><para> 0</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295862" xreflabel=""/><emphasis role="bold">sec.used</emphasis></para></entry>
- <entry><para>
<anchor xml:id="dbdoclet.50438274_pgfId-1295864" xreflabel=""/>10818808</para><para> 47.677302</para></entry>
+ <entry><para> <emphasis role="bold">sec.used</emphasis></para></entry>
+ <entry><para> 10818808</para><para> 47.677302</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295866" xreflabel=""/><emphasis role="bold">stack size</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295868" xreflabel=""/>1204:</para></entry>
+ <entry><para> <emphasis role="bold">stack size</emphasis></para></entry>
+ <entry><para> 1204:</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295870" xreflabel=""/><emphasis role="bold">pid</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295872" xreflabel=""/>2973:</para></entry>
+ <entry><para> <emphasis role="bold">pid</emphasis></para></entry>
+ <entry><para> 2973:</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295874" xreflabel=""/><emphasis role="bold">host pid (if uml) or zero</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295876" xreflabel=""/>31070:</para></entry>
+ <entry><para> <emphasis role="bold">host pid (if uml) or zero</emphasis></para></entry>
+ <entry><para> 31070:</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295878" xreflabel=""/><emphasis role="bold">(file:line #:functional())</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295880" xreflabel=""/>(as_dev.c:144:create_write_buffers())</para></entry>
+ <entry><para> <emphasis role="bold">(file:line #:functional())</emphasis></para></entry>
+ <entry><para> (as_dev.c:144:create_write_buffers())</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295882" xreflabel=""/><emphasis role="bold">debug message</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295884" xreflabel=""/>kmalloced '*obj': 24 at a375571c (tot 17447717)</para></entry>
+ <entry><para> <emphasis role="bold">debug message</emphasis></para></entry>
+ <entry><para> kmalloced '*obj': 24 at a375571c (tot 17447717)</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1295886" xreflabel=""/>28.2.1.3 Lustre <anchor xml:id="dbdoclet.50438274_marker-1295885" xreflabel=""/>Debug Messages Buffer</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295887" xreflabel=""/>Lustre debug messages are maintained in a buffer, with the maximum buffer size specified (in MBs) by the debug_mb parameter (/proc/sys/lnet/debug_mb). The buffer is circular, so debug messages are kept until the allocated buffer limit is reached, and then the first messages are overwritten.</para>
+ <title>28.2.1.3 Lustre <anchor xml:id="dbdoclet.50438274_marker-1295885" xreflabel=""/>Debug Messages Buffer</title>
+ <para>Lustre debug messages are maintained in a buffer, with the maximum buffer size specified (in MBs) by the debug_mb parameter (/proc/sys/lnet/debug_mb). The buffer is circular, so debug messages are kept until the allocated buffer limit is reached, and then the first messages are overwritten.</para>
</section>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1295889" xreflabel=""/>28.2.2 <anchor xml:id="dbdoclet.50438274_62472" xreflabel=""/>Using the lctl Tool to View Debug Messages</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295890" xreflabel=""/>The lctl tool allows debug messages to be filtered based on subsystems and message types to extract information useful for troubleshooting from a kernel debug log. For a command reference, see <link xl:href="SystemConfigurationUtilities.html#50438219_38274">lctl</link>.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295894" xreflabel=""/>You can use lctl to:</para>
+ <title>28.2.2 <anchor xml:id="dbdoclet.50438274_62472" xreflabel=""/>Using the lctl Tool to View Debug Messages</title>
+ <para>The lctl tool allows debug messages to be filtered based on subsystems and message types to extract information useful for troubleshooting from a kernel debug log. For a command reference, see <link xl:href="SystemConfigurationUtilities.html#50438219_38274">lctl</link>.</para>
+ <para>You can use lctl to:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295895" xreflabel=""/> Obtain a list of all the types and subsystems:</para>
+ <para> Obtain a list of all the types and subsystems:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295896" xreflabel=""/>lctl > debug_list <emphasis><subs | types></emphasis></screen>
+ <screen>lctl > debug_list <emphasis><subs | types></emphasis></screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295897" xreflabel=""/> Filter the debug log:</para>
+ <para> Filter the debug log:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295898" xreflabel=""/>lctl > filter <emphasis><subsystem name | debug type></emphasis></screen>
+ <screen>lctl > filter <emphasis><subsystem name | debug type></emphasis></screen>
<note><para>When lctl filters, it removes unwanted lines from the displayed output. This does not affect the contents of the debug log in the kernel's memory. As a result, you can print the log many times with different filtering levels without worrying about losing data.</para></note>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295900" xreflabel=""/> Show debug messages belonging to certain subsystem or type:</para>
+ <para> Show debug messages belonging to certain subsystem or type:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295901" xreflabel=""/>lctl > show <emphasis><subsystem name | debug type></emphasis></screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295902" xreflabel=""/>debug_kernel pulls the data from the kernel logs, filters it appropriately, and displays or saves it as per the specified options</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295903" xreflabel=""/>lctl > debug_kernel [<emphasis>output filename</emphasis>]
+ <screen>lctl > show <emphasis><subsystem name | debug type></emphasis></screen>
+ <para>debug_kernel pulls the data from the kernel logs, filters it appropriately, and displays or saves it as per the specified options</para>
+ <screen>lctl > debug_kernel [<emphasis>output filename</emphasis>]
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295904" xreflabel=""/>If the debugging is being done on User Mode Linux (UML), it might be useful to save the logs on the host machine so that they can be used at a later time.</para>
+ <para>If the debugging is being done on User Mode Linux (UML), it might be useful to save the logs on the host machine so that they can be used at a later time.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295905" xreflabel=""/> Filter a log on disk, if you already have a debug log saved to disk (likely from a crash):</para>
+ <para> Filter a log on disk, if you already have a debug log saved to disk (likely from a crash):</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295906" xreflabel=""/>lctl > debug_file <emphasis><input filename></emphasis> [<emphasis>output filename</emphasis>]
+ <screen>lctl > debug_file <emphasis><input filename></emphasis> [<emphasis>output filename</emphasis>]
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295907" xreflabel=""/>During the debug session, you can add markers or breaks to the log for any reason:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295908" xreflabel=""/>lctl > mark [marker text]
+ <para>During the debug session, you can add markers or breaks to the log for any reason:</para>
+ <screen>lctl > mark [marker text]
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295909" xreflabel=""/>The marker text defaults to the current date and time in the debug log (similar to the example shown below):</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295910" xreflabel=""/>DEBUG MARKER: Tue Mar 5 16:06:44 EST 2002
+ <para>The marker text defaults to the current date and time in the debug log (similar to the example shown below):</para>
+ <screen>DEBUG MARKER: Tue Mar 5 16:06:44 EST 2002
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295911" xreflabel=""/> Completely flush the kernel debug buffer:</para>
+ <para> Completely flush the kernel debug buffer:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295912" xreflabel=""/>lctl > clear
+ <screen>lctl > clear
</screen>
<note><para>Debug messages displayed with lctl are also subject to the kernel debug masks; the filters are additive.</para></note>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1295915" xreflabel=""/>28.2.2.1 Sample lctl<anchor xml:id="dbdoclet.50438274_marker-1295914" xreflabel=""/>Run</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295916" xreflabel=""/>Below is a sample run using the lctl command.</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295917" xreflabel=""/>bash-2.04# ./lctl
-<anchor xml:id="dbdoclet.50438274_pgfId-1295918" xreflabel=""/>lctl > debug_kernel /tmp/lustre_logs/log_all
-<anchor xml:id="dbdoclet.50438274_pgfId-1295919" xreflabel=""/>Debug log: 324 lines, 324 kept, 0 dropped.
-<anchor xml:id="dbdoclet.50438274_pgfId-1295920" xreflabel=""/>lctl > filter trace
-<anchor xml:id="dbdoclet.50438274_pgfId-1295921" xreflabel=""/>Disabling output of type "trace"
-<anchor xml:id="dbdoclet.50438274_pgfId-1295922" xreflabel=""/>lctl > debug_kernel /tmp/lustre_logs/log_notrace
-<anchor xml:id="dbdoclet.50438274_pgfId-1295923" xreflabel=""/>Debug log: 324 lines, 282 kept, 42 dropped.
-<anchor xml:id="dbdoclet.50438274_pgfId-1295924" xreflabel=""/>lctl > show trace
-<anchor xml:id="dbdoclet.50438274_pgfId-1295925" xreflabel=""/>Enabling output of type "trace"
-<anchor xml:id="dbdoclet.50438274_pgfId-1295926" xreflabel=""/>lctl > filter portals
-<anchor xml:id="dbdoclet.50438274_pgfId-1295927" xreflabel=""/>Disabling output from subsystem "portals"
-<anchor xml:id="dbdoclet.50438274_pgfId-1295928" xreflabel=""/>lctl > debug_kernel /tmp/lustre_logs/log_noportals
-<anchor xml:id="dbdoclet.50438274_pgfId-1295929" xreflabel=""/>Debug log: 324 lines, 258 kept, 66 dropped.
+ <title>28.2.2.1 Sample lctl<anchor xml:id="dbdoclet.50438274_marker-1295914" xreflabel=""/>Run</title>
+ <para>Below is a sample run using the lctl command.</para>
+ <screen>bash-2.04# ./lctl
+lctl > debug_kernel /tmp/lustre_logs/log_all
+Debug log: 324 lines, 324 kept, 0 dropped.
+lctl > filter trace
+Disabling output of type "trace"
+lctl > debug_kernel /tmp/lustre_logs/log_notrace
+Debug log: 324 lines, 282 kept, 42 dropped.
+lctl > show trace
+Enabling output of type "trace"
+lctl > filter portals
+Disabling output from subsystem "portals"
+lctl > debug_kernel /tmp/lustre_logs/log_noportals
+Debug log: 324 lines, 258 kept, 66 dropped.
</screen>
</section>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438274_pgfId-1295930" xreflabel=""/>28.2.3 Dumping the Buffer to a File (debug_daemon)</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295931" xreflabel=""/>The debug_daemon option is used by lctl to control the dumping of the debug_kernel buffer to a user-specified file. This functionality uses a kernel thread on top of debug_kernel, which works in parallel with the debug_daemon command.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295932" xreflabel=""/>The debug_daemon is highly dependent on file system write speed. File system write operations may not be fast enough to flush out all of the debug_buffer if the Lustre file system is under heavy system load and continues to CDEBUG to the debug_buffer. The debug_daemon will write the message DEBUG MARKER: Trace buffer full into the debug_buffer to indicate the debug_buffer contents are overlapping before the debug_daemon flushes data to a file.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295933" xreflabel=""/>Users can use lctlcontrol to start or stop the Lustre daemon from dumping the debug_buffer to a file. Users can also temporarily hold daemon from dumping the file. Use of the debug_daemon sub-command to lctl can provide the same function.</para>
+ <title>28.2.3 Dumping the Buffer to a File (debug_daemon)</title>
+ <para>The debug_daemon option is used by lctl to control the dumping of the debug_kernel buffer to a user-specified file. This functionality uses a kernel thread on top of debug_kernel, which works in parallel with the debug_daemon command.</para>
+ <para>The debug_daemon is highly dependent on file system write speed. File system write operations may not be fast enough to flush out all of the debug_buffer if the Lustre file system is under heavy system load and continues to CDEBUG to the debug_buffer. The debug_daemon will write the message DEBUG MARKER: Trace buffer full into the debug_buffer to indicate the debug_buffer contents are overlapping before the debug_daemon flushes data to a file.</para>
+ <para>Users can use lctlcontrol to start or stop the Lustre daemon from dumping the debug_buffer to a file. Users can also temporarily hold daemon from dumping the file. Use of the debug_daemon sub-command to lctl can provide the same function.</para>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438274_pgfId-1295934" xreflabel=""/>28.2.3.1 lctldebug_daemon Commands</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295935" xreflabel=""/>This section describes lctldebug_daemon commands.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295936" xreflabel=""/>To initiate the debug_daemon to start dumping debug_buffer into a file., enter</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295937" xreflabel=""/>$ lctl debug_daemon start [{file} {megabytes}]
+ <title>28.2.3.1 lctldebug_daemon Commands</title>
+ <para>This section describes lctldebug_daemon commands.</para>
+ <para>To initiate the debug_daemon to start dumping debug_buffer into a file., enter</para>
+ <screen>$ lctl debug_daemon start [{file} {megabytes}]
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295938" xreflabel=""/>The file can be a system default file, as shown in /proc/sys/lnet/debug_path. After Lustre starts, the default path is /tmp/lustre-log-$HOSTNAME. Users can specify a new filename for debug_daemon to output debug_buffer. The new file name shows up in /proc/sys/lnet/debug_path. Megabytes is the limitation of the file size in MBs.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295939" xreflabel=""/>The daemon wraps around and dumps data to the beginning of the file when the output file size is over the limit of the user-specified file size. To decode the dumped file to ASCII and order the log entries by time, run:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295940" xreflabel=""/>lctl debug_file {file} > {newfile}
+ <para>The file can be a system default file, as shown in /proc/sys/lnet/debug_path. After Lustre starts, the default path is /tmp/lustre-log-$HOSTNAME. Users can specify a new filename for debug_daemon to output debug_buffer. The new file name shows up in /proc/sys/lnet/debug_path. Megabytes is the limitation of the file size in MBs.</para>
+ <para>The daemon wraps around and dumps data to the beginning of the file when the output file size is over the limit of the user-specified file size. To decode the dumped file to ASCII and order the log entries by time, run:</para>
+ <screen>lctl debug_file {file} > {newfile}
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295941" xreflabel=""/>The output is internally sorted by the lctl command using quicksort.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295942" xreflabel=""/>To completely shut down the debug_daemon operation and flush the file output, enter:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295943" xreflabel=""/>debug_daemon stop
+ <para>The output is internally sorted by the lctl command using quicksort.</para>
+ <para>To completely shut down the debug_daemon operation and flush the file output, enter:</para>
+ <screen>debug_daemon stop
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295944" xreflabel=""/>Otherwise, debug_daemon is shut down as part of the Lustre file system shutdown process. Users can restart debug_daemon by using start command after each stop command issued.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295945" xreflabel=""/>This is an example using debug_daemon with the interactive mode of lctl to dump debug logs to a 10 MB file.</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295946" xreflabel=""/>#~/utils/lctl
+ <para>Otherwise, debug_daemon is shut down as part of the Lustre file system shutdown process. Users can restart debug_daemon by using start command after each stop command issued.</para>
+ <para>This is an example using debug_daemon with the interactive mode of lctl to dump debug logs to a 10 MB file.</para>
+ <screen>#~/utils/lctl
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295947" xreflabel=""/>To start the daemon to dump debug_buffer into a 40 MB /tmp/dump file, enter:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295948" xreflabel=""/>lctl > debug_daemon start /trace/log 40
+ <para>To start the daemon to dump debug_buffer into a 40 MB /tmp/dump file, enter:</para>
+ <screen>lctl > debug_daemon start /trace/log 40
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295949" xreflabel=""/>To completely shut down the daemon, enter:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295950" xreflabel=""/>lctl > debug_daemon stop
+ <para>To completely shut down the daemon, enter:</para>
+ <screen>lctl > debug_daemon stop
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295951" xreflabel=""/>To start another daemon with an unlimited file size, enter:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295952" xreflabel=""/>lctl > debug_daemon start /tmp/unlimited
+ <para>To start another daemon with an unlimited file size, enter:</para>
+ <screen>lctl > debug_daemon start /tmp/unlimited
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295953" xreflabel=""/>The text message *** End of debug_daemon trace log *** appears at the end of each output file.</para>
+ <para>The text message *** End of debug_daemon trace log *** appears at the end of each output file.</para>
</section>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1295954" xreflabel=""/>28.2.4 Controlling Information Written to the Kernel <anchor xml:id="dbdoclet.50438274_marker-1295955" xreflabel=""/>Debug Log</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295956" xreflabel=""/>Masks are provided in /proc/sys/lnet/subsystem_debug and /proc/sys/lnet/debug to be used with the systctl command to determine what information is to be written to the debug log. The subsystem_debug mask determines the information written to the log based on the subsystem (such as iobdfilter, net, portals, or OSC). The debug mask controls information based on debug type (such as info, error, trace, or alloc).</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295957" xreflabel=""/>To turn off Lustre debugging completely:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295958" xreflabel=""/>sysctl -w lnet.debug=0
+ <title>28.2.4 Controlling Information Written to the Kernel <anchor xml:id="dbdoclet.50438274_marker-1295955" xreflabel=""/>Debug Log</title>
+ <para>Masks are provided in /proc/sys/lnet/subsystem_debug and /proc/sys/lnet/debug to be used with the systctl command to determine what information is to be written to the debug log. The subsystem_debug mask determines the information written to the log based on the subsystem (such as iobdfilter, net, portals, or OSC). The debug mask controls information based on debug type (such as info, error, trace, or alloc).</para>
+ <para>To turn off Lustre debugging completely:</para>
+ <screen>sysctl -w lnet.debug=0
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295959" xreflabel=""/>To turn on full Lustre debugging:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295960" xreflabel=""/>sysctl -w lnet.debug=-1
+ <para>To turn on full Lustre debugging:</para>
+ <screen>sysctl -w lnet.debug=-1
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295961" xreflabel=""/>To turn on logging of messages related to network communications:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295962" xreflabel=""/>sysctl -w lnet.debug=net
+ <para>To turn on logging of messages related to network communications:</para>
+ <screen>sysctl -w lnet.debug=net
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295963" xreflabel=""/>To turn on logging of messages related to network communications and existing debug flags:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295964" xreflabel=""/>sysctl -w lnet.debug=+net
+ <para>To turn on logging of messages related to network communications and existing debug flags:</para>
+ <screen>sysctl -w lnet.debug=+net
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295965" xreflabel=""/>To turn off network logging with changing existing flags:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295966" xreflabel=""/>sysctl -w lnet.debug=-net
+ <para>To turn off network logging with changing existing flags:</para>
+ <screen>sysctl -w lnet.debug=-net
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295967" xreflabel=""/>The various options available to print to kernel debug logs are listed in lnet/include/libcfs/libcfs.h</para>
+ <para>The various options available to print to kernel debug logs are listed in lnet/include/libcfs/libcfs.h</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1295970" xreflabel=""/>28.2.5 <anchor xml:id="dbdoclet.50438274_26909" xreflabel=""/>Troubleshooting with strace<anchor xml:id="dbdoclet.50438274_marker-1295969" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295971" xreflabel=""/>The strace utility provided with the Linux distribution enables system calls to be traced by intercepting all the system calls made by a process and recording the system call name, aruguments, and return values.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295972" xreflabel=""/>To invoke strace on a program, enter:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295973" xreflabel=""/>$ strace <emphasis><program> <args></emphasis>
+ <title>28.2.5 <anchor xml:id="dbdoclet.50438274_26909" xreflabel=""/>Troubleshooting with strace<anchor xml:id="dbdoclet.50438274_marker-1295969" xreflabel=""/></title>
+ <para>The strace utility provided with the Linux distribution enables system calls to be traced by intercepting all the system calls made by a process and recording the system call name, aruguments, and return values.</para>
+ <para>To invoke strace on a program, enter:</para>
+ <screen>$ strace <emphasis><program> <args></emphasis>
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295974" xreflabel=""/>Sometimes, a system call may fork child processes. In this situation, use the -f option of strace to trace the child processes:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295975" xreflabel=""/>$ strace -f <emphasis><program> <args></emphasis>
+ <para>Sometimes, a system call may fork child processes. In this situation, use the -f option of strace to trace the child processes:</para>
+ <screen>$ strace -f <emphasis><program> <args></emphasis>
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295976" xreflabel=""/>To redirect the strace output to a file, enter:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295977" xreflabel=""/>$ strace -o <emphasis><filename> <program> <args></emphasis>
+ <para>To redirect the strace output to a file, enter:</para>
+ <screen>$ strace -o <emphasis><filename> <program> <args></emphasis>
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295978" xreflabel=""/>Use the -ff option, along with -o, to save the trace output in filename.pid, where pid is the process ID of the process being traced. Use the -ttt option to timestamp all lines in the strace output, so they can be correlated to operations in the lustre kernel debug log.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295979" xreflabel=""/>If the debugging is done in UML, save the traces on the host machine. In this example, hostfs is mounted on /r:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295980" xreflabel=""/>$ strace -o /r/tmp/vi.strace
+ <para>Use the -ff option, along with -o, to save the trace output in filename.pid, where pid is the process ID of the process being traced. Use the -ttt option to timestamp all lines in the strace output, so they can be correlated to operations in the lustre kernel debug log.</para>
+ <para>If the debugging is done in UML, save the traces on the host machine. In this example, hostfs is mounted on /r:</para>
+ <screen>$ strace -o /r/tmp/vi.strace
</screen>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1295983" xreflabel=""/>28.2.6 <anchor xml:id="dbdoclet.50438274_54455" xreflabel=""/>Looking at Disk <anchor xml:id="dbdoclet.50438274_marker-1295982" xreflabel=""/>Content</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295984" xreflabel=""/>In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The lfs tool can be used to obtain this information for a given file using the getstripe subcommand. Use a corresponding lfs setstripe command to specify striping attributes for a new file or directory.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295985" xreflabel=""/>The lfsgetstripe utility is written in C; it takes a Lustre filename as input and lists all the objects that form a part of this file. To obtain this information for the file /mnt/lustre/frog in Lustre file system, run:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295986" xreflabel=""/>$ lfs getstripe /mnt/lustre/frog
-<anchor xml:id="dbdoclet.50438274_pgfId-1295987" xreflabel=""/>$
-<anchor xml:id="dbdoclet.50438274_pgfId-1295988" xreflabel=""/> obdix objid
-<anchor xml:id="dbdoclet.50438274_pgfId-1295989" xreflabel=""/> 0 17
-<anchor xml:id="dbdoclet.50438274_pgfId-1295990" xreflabel=""/> 1 4
-</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295991" xreflabel=""/>The debugfs tool is provided in the e2fsprogs package. It can be used for interactive debugging of an ldiskfs file system. The debugfs tool can either be used to check status or modify information in the file system. In Lustre, all objects that belong to a file are stored in an underlying ldiskfs file system on the OSTs. The file system uses the object IDs as the file names. Once the object IDs are known, use the debugfs tool to obtain the attributes of all objects from different OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1295992" xreflabel=""/>A sample run for the /mnt/lustre/frog file used in the above example is shown here:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295993" xreflabel=""/> $ debugfs -c /tmp/ost1
-<anchor xml:id="dbdoclet.50438274_pgfId-1295994" xreflabel=""/> debugfs: cd O
-<anchor xml:id="dbdoclet.50438274_pgfId-1295995" xreflabel=""/> debugfs: cd 0 /* for files in group 0 \
+ <title>28.2.6 <anchor xml:id="dbdoclet.50438274_54455" xreflabel=""/>Looking at Disk <anchor xml:id="dbdoclet.50438274_marker-1295982" xreflabel=""/>Content</title>
+ <para>In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The lfs tool can be used to obtain this information for a given file using the getstripe subcommand. Use a corresponding lfs setstripe command to specify striping attributes for a new file or directory.</para>
+ <para>The lfsgetstripe utility is written in C; it takes a Lustre filename as input and lists all the objects that form a part of this file. To obtain this information for the file /mnt/lustre/frog in Lustre file system, run:</para>
+ <screen>$ lfs getstripe /mnt/lustre/frog
+$
+ obdix objid
+ 0 17
+ 1 4
+</screen>
+ <para>The debugfs tool is provided in the e2fsprogs package. It can be used for interactive debugging of an ldiskfs file system. The debugfs tool can either be used to check status or modify information in the file system. In Lustre, all objects that belong to a file are stored in an underlying ldiskfs file system on the OSTs. The file system uses the object IDs as the file names. Once the object IDs are known, use the debugfs tool to obtain the attributes of all objects from different OSTs.</para>
+ <para>A sample run for the /mnt/lustre/frog file used in the above example is shown here:</para>
+ <screen> $ debugfs -c /tmp/ost1
+ debugfs: cd O
+ debugfs: cd 0 /* for files in group 0 \
*/
-<anchor xml:id="dbdoclet.50438274_pgfId-1295996" xreflabel=""/> debugfs: cd d<objid % 32>
-<anchor xml:id="dbdoclet.50438274_pgfId-1295997" xreflabel=""/> debugfs: stat <objid> /* for getattr on object\
+ debugfs: cd d<objid % 32>
+ debugfs: stat <objid> /* for getattr on object\
*/
-<anchor xml:id="dbdoclet.50438274_pgfId-1295998" xreflabel=""/> debugfs: quit
-<anchor xml:id="dbdoclet.50438274_pgfId-1295999" xreflabel=""/>## Suppose object id is 36, then follow the steps below:
-<anchor xml:id="dbdoclet.50438274_pgfId-1296000" xreflabel=""/> $ debugfs /tmp/ost1
-<anchor xml:id="dbdoclet.50438274_pgfId-1296001" xreflabel=""/> debugfs: cd O
-<anchor xml:id="dbdoclet.50438274_pgfId-1296002" xreflabel=""/> debugfs: cd 0
-<anchor xml:id="dbdoclet.50438274_pgfId-1296003" xreflabel=""/> debugfs: cd d4 /* objid % 32 */
-<anchor xml:id="dbdoclet.50438274_pgfId-1296004" xreflabel=""/> debugfs: stat 36 /* for getattr on obj 4*\
+ debugfs: quit
+## Suppose object id is 36, then follow the steps below:
+ $ debugfs /tmp/ost1
+ debugfs: cd O
+ debugfs: cd 0
+ debugfs: cd d4 /* objid % 32 */
+ debugfs: stat 36 /* for getattr on obj 4*\
/
-<anchor xml:id="dbdoclet.50438274_pgfId-1296005" xreflabel=""/> debugfs: dump 36 /tmp/obj.36 /* dump contents of obj \
+ debugfs: dump 36 /tmp/obj.36 /* dump contents of obj \
4 */
-<anchor xml:id="dbdoclet.50438274_pgfId-1296006" xreflabel=""/> debugfs: quit
+ debugfs: quit
</screen>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1296008" xreflabel=""/>28.2.7 Finding the Lustre <anchor xml:id="dbdoclet.50438274_marker-1296007" xreflabel=""/>UUID of an OST</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296009" xreflabel=""/>To determine the Lustre UUID of an obdfilter disk (for example, if you mix up the cables on your OST devices or the SCSI bus numbering suddenly changes and the SCSI devices get new names), use debugfs to get the last_rcvd file.</para>
+ <title>28.2.7 Finding the Lustre <anchor xml:id="dbdoclet.50438274_marker-1296007" xreflabel=""/>UUID of an OST</title>
+ <para>To determine the Lustre UUID of an obdfilter disk (for example, if you mix up the cables on your OST devices or the SCSI bus numbering suddenly changes and the SCSI devices get new names), use debugfs to get the last_rcvd file.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438274_pgfId-1296010" xreflabel=""/>28.2.8 Printing Debug Messages to the Console</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296011" xreflabel=""/>To dump debug messages to the console (/var/log/messages), set the corresponding debug mask in the printk flag:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296012" xreflabel=""/>sysctl -w lnet.printk=-1
+ <title>28.2.8 Printing Debug Messages to the Console</title>
+ <para>To dump debug messages to the console (/var/log/messages), set the corresponding debug mask in the printk flag:</para>
+ <screen>sysctl -w lnet.printk=-1
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296013" xreflabel=""/>This slows down the system dramatically. It is also possible to selectively enable or disable this capability for particular flags using:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296014" xreflabel=""/>sysctl -w lnet.printk=+vfstrace
-<anchor xml:id="dbdoclet.50438274_pgfId-1296015" xreflabel=""/>sysctl -w lnet.printk=-vfstrace
+ <para>This slows down the system dramatically. It is also possible to selectively enable or disable this capability for particular flags using:</para>
+ <screen>sysctl -w lnet.printk=+vfstrace
+sysctl -w lnet.printk=-vfstrace
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296016" xreflabel=""/>It is possible to disable warning, error , and console messages, though it is strongly recommended to have something like lctldebug_daemon runing to capture this data to a local file system for debugging purposes.</para>
+ <para>It is possible to disable warning, error , and console messages, though it is strongly recommended to have something like lctldebug_daemon runing to capture this data to a local file system for debugging purposes.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1296018" xreflabel=""/>28.2.9 Tracing <anchor xml:id="dbdoclet.50438274_marker-1296017" xreflabel=""/>Lock Traffic</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296019" xreflabel=""/>Lustre has a specific debug type category for tracing lock traffic. Use:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296020" xreflabel=""/>lctl> filter all_types
-<anchor xml:id="dbdoclet.50438274_pgfId-1296021" xreflabel=""/>lctl> show dlmtrace
-<anchor xml:id="dbdoclet.50438274_pgfId-1296022" xreflabel=""/>lctl> debug_kernel [filename]
+ <title>28.2.9 Tracing <anchor xml:id="dbdoclet.50438274_marker-1296017" xreflabel=""/>Lock Traffic</title>
+ <para>Lustre has a specific debug type category for tracing lock traffic. Use:</para>
+ <screen>lctl> filter all_types
+lctl> show dlmtrace
+lctl> debug_kernel [filename]
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438274_80443">
<title>28.3 Lustre Debugging for Developers</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296025" xreflabel=""/>The procedures in this section may be useful to developers debugging Lustre code.</para>
+ <para>The procedures in this section may be useful to developers debugging Lustre code.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1296027" xreflabel=""/>28.3.1 Adding Debugging to the <anchor xml:id="dbdoclet.50438274_marker-1296026" xreflabel=""/>Lustre Source Code</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296028" xreflabel=""/>The debugging infrastructure provides a number of macros that can be used in Lustre source code to aid in debugging or reporting serious errors.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296029" xreflabel=""/>To use these macros, you will need to set the DEBUG_SUBSYSTEM variable at the top of the file as shown below:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296030" xreflabel=""/>#define DEBUG_SUBSYSTEM S_PORTALS
+ <title>28.3.1 Adding Debugging to the <anchor xml:id="dbdoclet.50438274_marker-1296026" xreflabel=""/>Lustre Source Code</title>
+ <para>The debugging infrastructure provides a number of macros that can be used in Lustre source code to aid in debugging or reporting serious errors.</para>
+ <para>To use these macros, you will need to set the DEBUG_SUBSYSTEM variable at the top of the file as shown below:</para>
+ <screen>#define DEBUG_SUBSYSTEM S_PORTALS
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296098" xreflabel=""/>A list of available macros with descritions is provided in the table below.</para>
+ <para>A list of available macros with descritions is provided in the table below.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1296033" xreflabel=""/>Macro</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1296035" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Macro</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296037" xreflabel=""/><emphasis role="bold">LBUG</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296039" xreflabel=""/>A panic-style assertion in the kernel which causes Lustre to dump its circular log to the /tmp/lustre-log file. This file can be retrieved after a reboot. LBUG freezes the thread to allow capture of the panic stack. A system reboot is needed to clear the thread.</para></entry>
+ <entry><para> <emphasis role="bold">LBUG</emphasis></para></entry>
+ <entry><para> A panic-style assertion in the kernel which causes Lustre to dump its circular log to the /tmp/lustre-log file. This file can be retrieved after a reboot. LBUG freezes the thread to allow capture of the panic stack. A system reboot is needed to clear the thread.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296041" xreflabel=""/><emphasis role="bold">LASSERT</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296043" xreflabel=""/>Validates a given expression as true, otherwise calls LBUG. The failed expression is printed on the console, although the values that make up the expression are not printed.</para></entry>
+ <entry><para> <emphasis role="bold">LASSERT</emphasis></para></entry>
+ <entry><para> Validates a given expression as true, otherwise calls LBUG. The failed expression is printed on the console, although the values that make up the expression are not printed.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296045" xreflabel=""/><emphasis role="bold">LASSERTF</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296047" xreflabel=""/>Similar to LASSERT but allows a free-format message to be printed, like printf/printk.</para></entry>
+ <entry><para> <emphasis role="bold">LASSERTF</emphasis></para></entry>
+ <entry><para> Similar to LASSERT but allows a free-format message to be printed, like printf/printk.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296049" xreflabel=""/><emphasis role="bold">CDEBUG</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296051" xreflabel=""/>The basic, most commonly used debug macro that takes just one more argument than standard printf - the debug type. This message adds to the debug log with the debug mask set accordingly. Later, when a user retrieves the log for troubleshooting, they can filter based on this type.</para><para><anchor xml:id="dbdoclet.50438274_pgfId-1296052" xreflabel=""/>CDEBUG(D_INFO, "This is my debug message: the number is %d\n", number).</para></entry>
+ <entry><para> <emphasis role="bold">CDEBUG</emphasis></para></entry>
+ <entry><para> The basic, most commonly used debug macro that takes just one more argument than standard printf - the debug type. This message adds to the debug log with the debug mask set accordingly. Later, when a user retrieves the log for troubleshooting, they can filter based on this type.</para><para>CDEBUG(D_INFO, "This is my debug message: the number is %d\n", number).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296054" xreflabel=""/><emphasis role="bold">CERROR</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296056" xreflabel=""/>Behaves similarly to CDEBUG, but unconditionally prints the message in the debug log and to the console. This is appropriate for serious errors or fatal conditions:</para><para><anchor xml:id="dbdoclet.50438274_pgfId-1296057" xreflabel=""/>CERROR("Something very bad has happened, and the return code is %d.\n", rc);</para></entry>
+ <entry><para> <emphasis role="bold">CERROR</emphasis></para></entry>
+ <entry><para> Behaves similarly to CDEBUG, but unconditionally prints the message in the debug log and to the console. This is appropriate for serious errors or fatal conditions:</para><para>CERROR("Something very bad has happened, and the return code is %d.\n", rc);</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296059" xreflabel=""/><emphasis role="bold">ENTRY and EXIT</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296061" xreflabel=""/>Add messages to aid in call tracing (takes no arguments). When using these macros, cover all exit conditions to avoid confusion when the debug log reports that a function was entered, but never exited.</para></entry>
+ <entry><para> <emphasis role="bold">ENTRY and EXIT</emphasis></para></entry>
+ <entry><para> Add messages to aid in call tracing (takes no arguments). When using these macros, cover all exit conditions to avoid confusion when the debug log reports that a function was entered, but never exited.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296063" xreflabel=""/><emphasis role="bold">LDLM_DEBUG and LDLM_DEBUG_NOLOCK</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296065" xreflabel=""/>Used when tracing MDS and VFS operations for locking. These macros build a thin trace that shows the protocol exchanges between nodes.</para></entry>
+ <entry><para> <emphasis role="bold">LDLM_DEBUG and LDLM_DEBUG_NOLOCK</emphasis></para></entry>
+ <entry><para> Used when tracing MDS and VFS operations for locking. These macros build a thin trace that shows the protocol exchanges between nodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296067" xreflabel=""/><emphasis role="bold">DEBUG_REQ</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296069" xreflabel=""/>Prints information about the given ptlrpc_request structure.</para></entry>
+ <entry><para> <emphasis role="bold">DEBUG_REQ</emphasis></para></entry>
+ <entry><para> Prints information about the given ptlrpc_request structure.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296071" xreflabel=""/><emphasis role="bold">OBD_FAIL_CHECK</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296073" xreflabel=""/>Allows insertion of failure points into the Lustre code. This is useful to generate regression tests that can hit a very specific sequence of events. This works in conjunction with "sysctl -w lustre.fail_loc={fail_loc}" to set a specific failure point for which a given OBD_FAIL_CHECK will test.</para></entry>
+ <entry><para> <emphasis role="bold">OBD_FAIL_CHECK</emphasis></para></entry>
+ <entry><para> Allows insertion of failure points into the Lustre code. This is useful to generate regression tests that can hit a very specific sequence of events. This works in conjunction with "sysctl -w lustre.fail_loc={fail_loc}" to set a specific failure point for which a given OBD_FAIL_CHECK will test.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296075" xreflabel=""/><emphasis role="bold">OBD_FAIL_TIMEOUT</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296077" xreflabel=""/>Similar to OBD_FAIL_CHECK. Useful to simulate hung, blocked or busy processes or network devices. If the given fail_loc is hit, OBD_FAIL_TIMEOUT waits for the specified number of seconds.</para></entry>
+ <entry><para> <emphasis role="bold">OBD_FAIL_TIMEOUT</emphasis></para></entry>
+ <entry><para> Similar to OBD_FAIL_CHECK. Useful to simulate hung, blocked or busy processes or network devices. If the given fail_loc is hit, OBD_FAIL_TIMEOUT waits for the specified number of seconds.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296079" xreflabel=""/><emphasis role="bold">OBD_RACE</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296081" xreflabel=""/>Similar to OBD_FAIL_CHECK. Useful to have multiple processes execute the same code concurrently to provoke locking races. The first process to hit OBD_RACE sleeps until a second process hits OBD_RACE, then both processes continue.</para></entry>
+ <entry><para> <emphasis role="bold">OBD_RACE</emphasis></para></entry>
+ <entry><para> Similar to OBD_FAIL_CHECK. Useful to have multiple processes execute the same code concurrently to provoke locking races. The first process to hit OBD_RACE sleeps until a second process hits OBD_RACE, then both processes continue.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296083" xreflabel=""/><emphasis role="bold">OBD_FAIL_ONCE</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296085" xreflabel=""/>A flag set on a lustre.fail_loc breakpoint to cause the OBD_FAIL_CHECK condition to be hit only one time. Otherwise, a fail_loc is permanent until it is cleared with "sysctl -w lustre.fail_loc=0".</para></entry>
+ <entry><para> <emphasis role="bold">OBD_FAIL_ONCE</emphasis></para></entry>
+ <entry><para> A flag set on a lustre.fail_loc breakpoint to cause the OBD_FAIL_CHECK condition to be hit only one time. Otherwise, a fail_loc is permanent until it is cleared with "sysctl -w lustre.fail_loc=0".</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296087" xreflabel=""/><emphasis role="bold">OBD_FAIL_RAND</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296089" xreflabel=""/>Has OBD_FAIL_CHECK fail randomly; on average every (1 / lustre.fail_val) times.</para></entry>
+ <entry><para> <emphasis role="bold">OBD_FAIL_RAND</emphasis></para></entry>
+ <entry><para> Has OBD_FAIL_CHECK fail randomly; on average every (1 / lustre.fail_val) times.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296091" xreflabel=""/><emphasis role="bold">OBD_FAIL_SKIP</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296093" xreflabel=""/>Has OBD_FAIL_CHECK succeed lustre.fail_val times, and then fail permanently or once with OBD_FAIL_ONCE.</para></entry>
+ <entry><para> <emphasis role="bold">OBD_FAIL_SKIP</emphasis></para></entry>
+ <entry><para> Has OBD_FAIL_CHECK succeed lustre.fail_val times, and then fail permanently or once with OBD_FAIL_ONCE.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296095" xreflabel=""/><emphasis role="bold">OBD_FAIL_SOME</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296097" xreflabel=""/>Has OBD_FAIL_CHECK fail lustre.fail_val times, and then succeed.</para></entry>
+ <entry><para> <emphasis role="bold">OBD_FAIL_SOME</emphasis></para></entry>
+ <entry><para> Has OBD_FAIL_CHECK fail lustre.fail_val times, and then succeed.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1296100" xreflabel=""/>28.3.2 Accessing a Ptlrpc <anchor xml:id="dbdoclet.50438274_marker-1296099" xreflabel=""/>Request History</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296101" xreflabel=""/>Each service maintains a request history, which can be useful for first occurrence troubleshooting.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296102" xreflabel=""/>Ptlrpc is an RPC protocol layered on LNET that deals with stateful servers and has semantics and built-in support for recovery.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296103" xreflabel=""/>A prlrpc request history works as follows:</para>
+ <title>28.3.2 Accessing a Ptlrpc <anchor xml:id="dbdoclet.50438274_marker-1296099" xreflabel=""/>Request History</title>
+ <para>Each service maintains a request history, which can be useful for first occurrence troubleshooting.</para>
+ <para>Ptlrpc is an RPC protocol layered on LNET that deals with stateful servers and has semantics and built-in support for recovery.</para>
+ <para>A prlrpc request history works as follows:</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296104" xreflabel=""/>Request_in_callback() adds the new request to the service's request history.</para>
+ <para>Request_in_callback() adds the new request to the service's request history.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296105" xreflabel=""/>When a request buffer becomes idle, it is added to the service's request buffer history list.</para>
+ <para>When a request buffer becomes idle, it is added to the service's request buffer history list.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296106" xreflabel=""/>Buffers are culled from the service's request buffer history if it has grown above</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296107" xreflabel=""/>req_buffer_history_max and its reqs are removed from the service's request history.</para>
+ <para>Buffers are culled from the service's request buffer history if it has grown above</para>
+ <para>req_buffer_history_max and its reqs are removed from the service's request history.</para>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296108" xreflabel=""/>Request history is accessed and controlled using the following /proc files under the service directory:</para>
+ <para>Request history is accessed and controlled using the following /proc files under the service directory:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296109" xreflabel=""/>req_buffer_history_len</para>
+ <para>req_buffer_history_len</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296110" xreflabel=""/>Number of request buffers currently in the history</para>
+ <para>Number of request buffers currently in the history</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296111" xreflabel=""/>req_buffer_history_max</para>
+ <para>req_buffer_history_max</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296112" xreflabel=""/>Maximum number of request buffers to keep</para>
+ <para>Maximum number of request buffers to keep</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296113" xreflabel=""/>req_history</para>
+ <para>req_history</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296114" xreflabel=""/>The request history</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296115" xreflabel=""/>Requests in the history include "live" requests that are currently being handled. Each line in req_history looks like:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296151" xreflabel=""/><seq>:<target NID>:<client ID>:<xid>:<length>:<phase> <svc specific>
+ <para>The request history</para>
+ <para>Requests in the history include "live" requests that are currently being handled. Each line in req_history looks like:</para>
+ <screen><seq>:<target NID>:<client ID>:<xid>:<length>:<phase> <svc specific>
</screen>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1296118" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1296120" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296122" xreflabel=""/><emphasis role="bold">seq</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296124" xreflabel=""/>Request sequence number</para></entry>
+ <entry><para> <emphasis role="bold">seq</emphasis></para></entry>
+ <entry><para> Request sequence number</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296126" xreflabel=""/><emphasis role="bold">target NID</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296128" xreflabel=""/>Destination NID of the incoming request</para></entry>
+ <entry><para> <emphasis role="bold">target NID</emphasis></para></entry>
+ <entry><para> Destination NID of the incoming request</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296130" xreflabel=""/><emphasis role="bold">client ID</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296132" xreflabel=""/>Client PID and NID</para></entry>
+ <entry><para> <emphasis role="bold">client ID</emphasis></para></entry>
+ <entry><para> Client PID and NID</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296134" xreflabel=""/><emphasis role="bold">xid</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296136" xreflabel=""/>rq_xid</para></entry>
+ <entry><para> <emphasis role="bold">xid</emphasis></para></entry>
+ <entry><para> rq_xid</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296138" xreflabel=""/><emphasis role="bold">length</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296140" xreflabel=""/>Size of the request message</para></entry>
+ <entry><para> <emphasis role="bold">length</emphasis></para></entry>
+ <entry><para> Size of the request message</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296142" xreflabel=""/><emphasis role="bold">phase</emphasis></para></entry>
+ <entry><para> <emphasis role="bold">phase</emphasis></para></entry>
<entry><para><itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296144" xreflabel=""/> New (waiting to be handled or could not be unpacked)</para>
+ <para> New (waiting to be handled or could not be unpacked)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296145" xreflabel=""/> Interpret (unpacked or being handled)</para>
+ <para> Interpret (unpacked or being handled)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296146" xreflabel=""/> Complete (handled)</para>
+ <para> Complete (handled)</para>
</listitem>
</itemizedlist></para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296148" xreflabel=""/><emphasis role="bold">svc specific</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296150" xreflabel=""/>Service-specific request printout. Currently, the only service that does this is the OST (which prints the opcode if the message has been unpacked successfully</para></entry>
+ <entry><para> <emphasis role="bold">svc specific</emphasis></para></entry>
+ <entry><para> Service-specific request printout. Currently, the only service that does this is the OST (which prints the opcode if the message has been unpacked successfully</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438274_pgfId-1296154" xreflabel=""/>28.3.3 Finding Memory <anchor xml:id="dbdoclet.50438274_marker-1296153" xreflabel=""/>Leaks Using leak_finder.pl</title>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296155" xreflabel=""/>Memory leaks can occur in code when memory has been allocated and then not freed once it is no longer required. The leak_finder.pl program provides a way to find memory leaks.</para>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296156" xreflabel=""/>Before running this program, you must turn on debugging to collect all malloc and free entries. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296157" xreflabel=""/>sysctl -w lnet.debug=+malloc
+ <title>28.3.3 Finding Memory <anchor xml:id="dbdoclet.50438274_marker-1296153" xreflabel=""/>Leaks Using leak_finder.pl</title>
+ <para>Memory leaks can occur in code when memory has been allocated and then not freed once it is no longer required. The leak_finder.pl program provides a way to find memory leaks.</para>
+ <para>Before running this program, you must turn on debugging to collect all malloc and free entries. Run:</para>
+ <screen>sysctl -w lnet.debug=+malloc
</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296158" xreflabel=""/>Then complete the following steps:</para>
+ <para>Then complete the following steps:</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296161" xreflabel=""/> 1. Dump the log into a user-specified log file using lctl (see <link xl:href="LustreDebugging.html#50438274_62472">Using the lctl Tool to View Debug Messages</link>).</para>
+ <para> 1. Dump the log into a user-specified log file using lctl (see <link xl:href="LustreDebugging.html#50438274_62472">Using the lctl Tool to View Debug Messages</link>).</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296163" xreflabel=""/> 2. Run the leak finder on the newly-created log dump:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296164" xreflabel=""/>perl leak_finder.pl <ascii-logname>
+ <para> 2. Run the leak finder on the newly-created log dump:</para>
+ <screen>perl leak_finder.pl <ascii-logname>
</screen>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296165" xreflabel=""/>The output is:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296166" xreflabel=""/>malloced 8bytes at a3116744 (called pathcopy)
-<anchor xml:id="dbdoclet.50438274_pgfId-1296167" xreflabel=""/>(lprocfs_status.c:lprocfs_add_vars:80)
-<anchor xml:id="dbdoclet.50438274_pgfId-1296168" xreflabel=""/>freed 8bytes at a3116744 (called pathcopy)
-<anchor xml:id="dbdoclet.50438274_pgfId-1296169" xreflabel=""/>(lprocfs_status.c:lprocfs_add_vars:80)
-</screen>
- <para><anchor xml:id="dbdoclet.50438274_pgfId-1296170" xreflabel=""/>The tool displays the following output to show the leaks found:</para>
- <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296171" xreflabel=""/>Leak:32bytes allocated at a23a8fc(service.c:ptlrpc_init_svc:144,debug file \
+ <para>The output is:</para>
+ <screen>malloced 8bytes at a3116744 (called pathcopy)
+(lprocfs_status.c:lprocfs_add_vars:80)
+freed 8bytes at a3116744 (called pathcopy)
+(lprocfs_status.c:lprocfs_add_vars:80)
+</screen>
+ <para>The tool displays the following output to show the leaks found:</para>
+ <screen>Leak:32bytes allocated at a23a8fc(service.c:ptlrpc_init_svc:144,debug file \
line 241)
</screen>
</section>
<info>
<title xml:id='lustremaintenance.title'>Lustre Maintenance</title>
</info>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1298785" xreflabel=""/>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre maintenance tasks:</para>
+ <para>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre maintenance tasks:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438199_85142"/>Working with Inactive OSTs</para>
</itemizedlist>
<section xml:id="dbdoclet.50438199_42877">
<title>14.1 <anchor xml:id="dbdoclet.50438199_85142" xreflabel=""/>Working with <anchor xml:id="dbdoclet.50438199_marker-1298888" xreflabel=""/>Inactive OSTs</title>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1298890" xreflabel=""/>To mount a client or an MDT with one or more inactive OSTs, run commands similar to this:</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1298891" xreflabel=""/>client> mount -o exclude=testfs-OST0000 -t lustre uml1:/testfs\ /mnt/testfs
- <anchor xml:id="dbdoclet.50438199_pgfId-1298892" xreflabel=""/>client> cat /proc/fs/lustre/lov/testfs-clilov-*/target_obd
+ <para>To mount a client or an MDT with one or more inactive OSTs, run commands similar to this:</para>
+ <screen>client> mount -o exclude=testfs-OST0000 -t lustre uml1:/testfs\ /mnt/testfs
+ client> cat /proc/fs/lustre/lov/testfs-clilov-*/target_obd
</screen>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1298893" xreflabel=""/>To activate an inactive OST on a live client or MDT, use the lctl activate command on the OSC device. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1298894" xreflabel=""/>lctl --device 7 activate
+ <para>To activate an inactive OST on a live client or MDT, use the lctl activate command on the OSC device. For example:</para>
+ <screen>lctl --device 7 activate
</screen>
<section xml:id="dbdoclet.50438199_15240">
<title>14.2 Finding <anchor xml:id="dbdoclet.50438199_marker-1298897" xreflabel=""/>Nodes in the Lustre File System</title>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1298899" xreflabel=""/>There may be situations in which you need to find all nodes in your Lustre file system or get the names of all OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1298900" xreflabel=""/>To get a list of all Lustre nodes, run this command on the MGS:</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1298901" xreflabel=""/># cat /proc/fs/lustre/mgs/MGS/live/*
+ <para>There may be situations in which you need to find all nodes in your Lustre file system or get the names of all OSTs.</para>
+ <para>To get a list of all Lustre nodes, run this command on the MGS:</para>
+ <screen># cat /proc/fs/lustre/mgs/MGS/live/*
</screen>
<note>
<para>
</para>
</note>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1298903" xreflabel=""/>In this example, file system lustre has three nodes, lustre-MDT0000, lustre-OST0000, and lustre-OST0001.</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1298904" xreflabel=""/>cfs21:/tmp# cat /proc/fs/lustre/mgs/MGS/live/*
- <anchor xml:id="dbdoclet.50438199_pgfId-1298905" xreflabel=""/>fsname: lustre
- <anchor xml:id="dbdoclet.50438199_pgfId-1298906" xreflabel=""/>flags: 0x0 gen: 26
- <anchor xml:id="dbdoclet.50438199_pgfId-1298907" xreflabel=""/>lustre-MDT0000
- <anchor xml:id="dbdoclet.50438199_pgfId-1298908" xreflabel=""/>lustre-OST0000
- <anchor xml:id="dbdoclet.50438199_pgfId-1298909" xreflabel=""/>lustre-OST0001
+ <para>In this example, file system lustre has three nodes, lustre-MDT0000, lustre-OST0000, and lustre-OST0001.</para>
+ <screen>cfs21:/tmp# cat /proc/fs/lustre/mgs/MGS/live/*
+ fsname: lustre
+ flags: 0x0 gen: 26
+ lustre-MDT0000
+ lustre-OST0000
+ lustre-OST0001
</screen>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1298910" xreflabel=""/>To get the names of all OSTs, run this command on the MDS:</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1298911" xreflabel=""/># cat /proc/fs/lustre/lov/<fsname>-mdtlov/target_obd
+ <para>To get the names of all OSTs, run this command on the MDS:</para>
+ <screen># cat /proc/fs/lustre/lov/<fsname>-mdtlov/target_obd
</screen>
<note>
<para>
</para>
</note>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1298913" xreflabel=""/>In this example, there are two OSTs, lustre-OST0000 and lustre-OST0001, which are both active.</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1298914" xreflabel=""/>cfs21:/tmp# cat /proc/fs/lustre/lov/lustre-mdtlov/target_obd
- <anchor xml:id="dbdoclet.50438199_pgfId-1298915" xreflabel=""/>0: lustre-OST0000_UUID ACTIVE
- <anchor xml:id="dbdoclet.50438199_pgfId-1298916" xreflabel=""/>1: lustre-OST0001_UUID ACTIVE
+ <para>In this example, there are two OSTs, lustre-OST0000 and lustre-OST0001, which are both active.</para>
+ <screen>cfs21:/tmp# cat /proc/fs/lustre/lov/lustre-mdtlov/target_obd
+ 0: lustre-OST0000_UUID ACTIVE
+ 1: lustre-OST0001_UUID ACTIVE
</screen>
</section>
<section xml:id="dbdoclet.50438199_26070">
<title>14.3 Mounting a Server Without <anchor xml:id="dbdoclet.50438199_marker-1298918" xreflabel=""/>Lustre Service</title>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1298920" xreflabel=""/>If you are using a combined MGS/MDT, but you only want to start the MGS and not the MDT, run this command:</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1298921" xreflabel=""/>mount -t lustre <MDT partition> -o nosvc <mount point>
+ <para>If you are using a combined MGS/MDT, but you only want to start the MGS and not the MDT, run this command:</para>
+ <screen>mount -t lustre <MDT partition> -o nosvc <mount point>
</screen>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1298922" xreflabel=""/>The <MDT partition> variable is the combined MGS/MDT.</para>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1298923" xreflabel=""/>In this example, the combined MGS/MDT is testfs-MDT0000 and the mount point is mnt/test/mdt.</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1298924" xreflabel=""/>$ mount -t lustre -L testfs-MDT0000 -o nosvc /mnt/test/mdt
+ <para>The <MDT partition> variable is the combined MGS/MDT.</para>
+ <para>In this example, the combined MGS/MDT is testfs-MDT0000 and the mount point is mnt/test/mdt.</para>
+ <screen>$ mount -t lustre -L testfs-MDT0000 -o nosvc /mnt/test/mdt
</screen>
</section>
<section xml:id="dbdoclet.50438199_54623">
<title>14.4 Regenerating Lustre <anchor xml:id="dbdoclet.50438199_marker-1305736" xreflabel=""/>Configuration Logs</title>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1304951" xreflabel=""/>If the Lustre system's configuration logs are in a state where the file system cannot be started, use the writeconf command to erase them. After the writeconf command is run and the servers restart, the configuration logs are re-generated and stored on the MGS (as in a new file system).</para>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1304970" xreflabel=""/>You should only use the writeconf command if:</para>
+ <para>If the Lustre system's configuration logs are in a state where the file system cannot be started, use the writeconf command to erase them. After the writeconf command is run and the servers restart, the configuration logs are re-generated and stored on the MGS (as in a new file system).</para>
+ <para>You should only use the writeconf command if:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1304971" xreflabel=""/> The configuration logs are in a state where the file system cannot start</para>
+ <para> The configuration logs are in a state where the file system cannot start</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1304972" xreflabel=""/> A server NID is being changed</para>
+ <para> A server NID is being changed</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1304973" xreflabel=""/>The writeconf command is destructive to some configuration items (i.e., OST pools information and items set via conf_param), and should be used with caution. To avoid problems:</para>
+ <para>The writeconf command is destructive to some configuration items (i.e., OST pools information and items set via conf_param), and should be used with caution. To avoid problems:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1304974" xreflabel=""/> Shut down the file system before running the writeconf command</para>
+ <para> Shut down the file system before running the writeconf command</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1304975" xreflabel=""/> Run the writeconf command on all servers (MDT first, then OSTs)</para>
+ <para> Run the writeconf command on all servers (MDT first, then OSTs)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1306391" xreflabel=""/> Start the file system in this order:</para>
+ <para> Start the file system in this order:</para>
</listitem>
<listitem>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1306392" xreflabel=""/> MGS (or the combined MGS/MDT)</para>
+ <para> MGS (or the combined MGS/MDT)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1306393" xreflabel=""/> MDT</para>
+ <para> MDT</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1306394" xreflabel=""/> OSTs</para>
+ <para> OSTs</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1304977" xreflabel=""/> Lustre clients</para>
+ <para> Lustre clients</para>
</listitem>
</orderedlist>
</listitem>
The OST pools feature enables a group of OSTs to be named for file striping purposes. If you use OST pools, be aware that running the writeconf command erases <emphasis role="bold">all</emphasis> pools information (as well as any other parameters set via lctl conf_param). We recommend that the pools definitions (and conf_param settings) be executed via a script, so they can be reproduced easily after a writeconf is performed.</para>
</caution>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1303394" xreflabel=""/>To regenerate Lustre's system configuration logs:</para>
+ <para>To regenerate Lustre's system configuration logs:</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305772" xreflabel=""/>Shut down the file system in this order.</para>
+ <para>Shut down the file system in this order.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305773" xreflabel=""/>Unmount the clients.</para>
+ <para>Unmount the clients.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305774" xreflabel=""/>Unmount the MDT.</para>
+ <para>Unmount the MDT.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305775" xreflabel=""/>Unmount all OSTs.</para>
+ <para>Unmount all OSTs.</para>
</listitem></orderedlist>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305776" xreflabel=""/>Make sure the the MDT and OST devices are available.</para>
+ <para>Make sure the the MDT and OST devices are available.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305777" xreflabel=""/>Run the writeconf command on all servers.</para>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305778" xreflabel=""/>Run writeconf on the MDT first, and then the OSTs.</para>
+ <para>Run the writeconf command on all servers.</para>
+ <para>Run writeconf on the MDT first, and then the OSTs.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305779" xreflabel=""/>On the MDT, run:</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1305780" xreflabel=""/><mdt node>$ tunefs.lustre --writeconf <device>
+ <para>On the MDT, run:</para>
+ <screen><mdt node>$ tunefs.lustre --writeconf <device>
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305781" xreflabel=""/>On each OST, run:</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1305782" xreflabel=""/><ost node>$ tunefs.lustre --writeconf <device>
+ <para>On each OST, run:</para>
+ <screen><ost node>$ tunefs.lustre --writeconf <device>
</screen>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305783" xreflabel=""/>Restart the file system in this order.</para>
+ <para>Restart the file system in this order.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305784" xreflabel=""/>Mount the MGS (or the combined MGS/MDT).</para>
+ <para>Mount the MGS (or the combined MGS/MDT).</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305785" xreflabel=""/>Mount the MDT.</para>
+ <para>Mount the MDT.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305787" xreflabel=""/>Mount the OSTs.</para>
+ <para>Mount the OSTs.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305811" xreflabel=""/>Mount the clients.</para>
+ <para>Mount the clients.</para>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305788" xreflabel=""/>After the writeconf command is run, the configuration logs are re-generated as servers restart.</para>
+ <para>After the writeconf command is run, the configuration logs are re-generated as servers restart.</para>
</listitem></orderedlist>
</section>
<section xml:id="dbdoclet.50438199_31353">
<title>14.5 Changing a <anchor xml:id="dbdoclet.50438199_marker-1305737" xreflabel=""/>Server NID</title>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1303485" xreflabel=""/>If you need to change the NID on the MDT or an OST, run the writeconf command to erase Lustre configuration information (including server NIDs), and then re-generate the system configuration using updated server NIDs.</para>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1303612" xreflabel=""/>Change a server NID in these situations:</para>
+ <para>If you need to change the NID on the MDT or an OST, run the writeconf command to erase Lustre configuration information (including server NIDs), and then re-generate the system configuration using updated server NIDs.</para>
+ <para>Change a server NID in these situations:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1303524" xreflabel=""/> New server hardware is added to the file system, and the MDS or an OSS is being moved to the new machine</para>
+ <para> New server hardware is added to the file system, and the MDS or an OSS is being moved to the new machine</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1303542" xreflabel=""/> New network card is installed in the server</para>
+ <para> New network card is installed in the server</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1303548" xreflabel=""/> You want to reassign IP addresses</para>
+ <para> You want to reassign IP addresses</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1303474" xreflabel=""/><anchor xml:id="dbdoclet.50438199_DDE_LINK1" xreflabel=""/>To change a server NID:</para>
+ <para><anchor xml:id="dbdoclet.50438199_DDE_LINK1" xreflabel=""/>To change a server NID:</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1303460" xreflabel=""/>Update the LNET configuration in the /etc/modprobe.conf file so the list of server NIDs (lctl list_nids) is correct.</para>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1304468" xreflabel=""/>The lctl list_nids command indicates which network(s) are configured to work with Lustre.</para>
+ <para>Update the LNET configuration in the /etc/modprobe.conf file so the list of server NIDs (lctl list_nids) is correct.</para>
+ <para>The lctl list_nids command indicates which network(s) are configured to work with Lustre.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305816" xreflabel=""/>Shut down the file system in this order.</para>
+ <para>Shut down the file system in this order.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305817" xreflabel=""/>Unmount the clients.</para>
+ <para>Unmount the clients.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305818" xreflabel=""/>Unmount the MDT.</para>
+ <para>Unmount the MDT.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305819" xreflabel=""/>Unmount all OSTs.</para>
+ <para>Unmount all OSTs.</para>
</listitem></orderedlist>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305820" xreflabel=""/>Run the writeconf command on all servers.</para>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305821" xreflabel=""/>Run writeconf on the MDT first, and then the OSTs.</para>
+ <para>Run the writeconf command on all servers.</para>
+ <para>Run writeconf on the MDT first, and then the OSTs.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305822" xreflabel=""/>On the MDT, run:</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1305823" xreflabel=""/><mdt node>$ tunefs.lustre --writeconf <device>
+ <para>On the MDT, run:</para>
+ <screen><mdt node>$ tunefs.lustre --writeconf <device>
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305824" xreflabel=""/>On each OST, run:</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1305825" xreflabel=""/><ost node>$ tunefs.lustre --writeconf <device>
+ <para>On each OST, run:</para>
+ <screen><ost node>$ tunefs.lustre --writeconf <device>
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305826" xreflabel=""/>If the NID on the MGS was changed, communicate the new MGS location to each server. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1305827" xreflabel=""/>tunefs.lustre --erase-param --mgsnode=<new_nid(s)> --writeconf /dev/..
+ <para>If the NID on the MGS was changed, communicate the new MGS location to each server. Run:</para>
+ <screen>tunefs.lustre --erase-param --mgsnode=<new_nid(s)> --writeconf /dev/..
</screen>
</listitem></orderedlist>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305828" xreflabel=""/>Restart the file system in this order.</para>
+ <para>Restart the file system in this order.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305829" xreflabel=""/>Mount the MGS (or the combined MGS/MDT).</para>
+ <para>Mount the MGS (or the combined MGS/MDT).</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305830" xreflabel=""/>Mount the MDT.</para>
+ <para>Mount the MDT.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305831" xreflabel=""/>Mount the OSTs.</para>
+ <para>Mount the OSTs.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305832" xreflabel=""/>Mount the clients.</para>
+ <para>Mount the clients.</para>
</listitem></orderedlist>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1305833" xreflabel=""/>After the writeconf command is run, the configuration logs are re-generated as servers restart, and server NIDs in the updated list_nids file are used.</para>
+ <para>After the writeconf command is run, the configuration logs are re-generated as servers restart, and server NIDs in the updated list_nids file are used.</para>
</section>
<section xml:id="dbdoclet.50438199_22527">
<title>14.6 Adding a New <anchor xml:id="dbdoclet.50438199_marker-1306353" xreflabel=""/>OST to a Lustre File System</title>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1306355" xreflabel=""/>To add an OST to existing Lustre file system:</para>
+ <para>To add an OST to existing Lustre file system:</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1306356" xreflabel=""/> 1. Add a new OST by passing on the following commands, run:</para>
- <screen><anchor xml:id="dbdoclet.50438199_pgfId-1306357" xreflabel=""/>$ mkfs.lustre --fsname=spfs --ost --mgsnode=mds16@tcp0 /dev/sda
- <anchor xml:id="dbdoclet.50438199_pgfId-1306358" xreflabel=""/>$ mkdir -p /mnt/test/ost0
- <anchor xml:id="dbdoclet.50438199_pgfId-1306359" xreflabel=""/>$ mount -t lustre /dev/sda /mnt/test/ost0
+ <para> 1. Add a new OST by passing on the following commands, run:</para>
+ <screen>$ mkfs.lustre --fsname=spfs --ost --mgsnode=mds16@tcp0 /dev/sda
+ $ mkdir -p /mnt/test/ost0
+ $ mount -t lustre /dev/sda /mnt/test/ost0
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1306360" xreflabel=""/> 2. Migrate the data (possibly).</para>
+ <para> 2. Migrate the data (possibly).</para>
</listitem></orderedlist>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1306361" xreflabel=""/>The file system is quite unbalanced when new empty OSTs are added. New file creations are automatically balanced. If this is a scratch file system or files are pruned at a regular interval, then no further work may be needed.</para>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1306362" xreflabel=""/>New files being created will preferentially be placed on the empty OST. As old files are deleted, they will release space on the old OST.</para>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1306363" xreflabel=""/>Files existing prior to the expansion can optionally be rebalanced with an in-place copy, which can be done with a simple script. The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1306364" xreflabel=""/>For example, to rebalance all files within /mnt/lustre/dir, enter:</para>
-<screen><anchor xml:id="dbdoclet.50438199_pgfId-1306365" xreflabel=""/>lfs_migrate /mnt/lustre/file
+<para>The file system is quite unbalanced when new empty OSTs are added. New file creations are automatically balanced. If this is a scratch file system or files are pruned at a regular interval, then no further work may be needed.</para>
+<para>New files being created will preferentially be placed on the empty OST. As old files are deleted, they will release space on the old OST.</para>
+<para>Files existing prior to the expansion can optionally be rebalanced with an in-place copy, which can be done with a simple script. The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
+<para>For example, to rebalance all files within /mnt/lustre/dir, enter:</para>
+<screen>lfs_migrate /mnt/lustre/file
</screen>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1306366" xreflabel=""/>To migrate files within the /test filesystem on OST0004 that are larger than 4GB in size, enter:</para>
-<screen><anchor xml:id="dbdoclet.50438199_pgfId-1306367" xreflabel=""/>lfs find /test -obd test-OST0004 -size +4G | lfs_migrate -y
+<para>To migrate files within the /test filesystem on OST0004 that are larger than 4GB in size, enter:</para>
+<screen>lfs find /test -obd test-OST0004 -size +4G | lfs_migrate -y
</screen>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1306368" xreflabel=""/>See <xref linkend='userutilities'/> (lfs_migrate) for more details.</para>
+<para>See <xref linkend='userutilities'/> (lfs_migrate) for more details.</para>
</section>
<section xml:id="dbdoclet.50438199_14978">
<title>14.7 Removing and Restoring OSTs</title>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1304098" xreflabel=""/>OSTs can be removed from and restored to a Lustre file system. Currently in Lustre, removing an OST really means that the OST is 'deactivated' in the file system, not permanently removed. A removed OST still appears in the file system; do not create a new OST with the same name.</para>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1304099" xreflabel=""/>You may want to remove (deactivate) an OST and prevent new files from being written to it in several situations:</para>
+<para>OSTs can be removed from and restored to a Lustre file system. Currently in Lustre, removing an OST really means that the OST is 'deactivated' in the file system, not permanently removed. A removed OST still appears in the file system; do not create a new OST with the same name.</para>
+<para>You may want to remove (deactivate) an OST and prevent new files from being written to it in several situations:</para>
<itemizedlist><listitem>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1304100" xreflabel=""/> Hard drive has failed and a RAID resync/rebuild is underway</para>
+<para> Hard drive has failed and a RAID resync/rebuild is underway</para>
</listitem>
<listitem>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1304101" xreflabel=""/> OST is nearing its space capacity</para>
+<para> OST is nearing its space capacity</para>
</listitem>
</itemizedlist>
<section remap="h3">
-<title><anchor xml:id="dbdoclet.50438199_pgfId-1298979" xreflabel=""/>14.7.1 Removing an OST from the File System</title>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1306722" xreflabel=""/>OSTs can be removed from a Lustre file system. Currently in Lustre, removing an OST actually means that the OST is 'deactivated' from the file system, not permanently removed. A removed OST still appears in the device listing; you should not normally create a new OST with the same name.</para>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1306724" xreflabel=""/>You may want to deactivate an OST and prevent new files from being written to it in several situations:</para>
+<title>14.7.1 Removing an OST from the File System</title>
+<para>OSTs can be removed from a Lustre file system. Currently in Lustre, removing an OST actually means that the OST is 'deactivated' from the file system, not permanently removed. A removed OST still appears in the device listing; you should not normally create a new OST with the same name.</para>
+<para>You may want to deactivate an OST and prevent new files from being written to it in several situations:</para>
<itemizedlist><listitem>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1306725" xreflabel=""/> OST is nearing its space capacity</para>
+<para> OST is nearing its space capacity</para>
</listitem>
<listitem>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1306726" xreflabel=""/> Hard drive has failed and a RAID resync/rebuild is underway</para>
+<para> Hard drive has failed and a RAID resync/rebuild is underway</para>
</listitem>
<listitem>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1306727" xreflabel=""/> OST storage has failed permanently</para>
+<para> OST storage has failed permanently</para>
</listitem>
</itemizedlist>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1306729" xreflabel=""/>When removing an OST, remember that the MDT does not communicate directly with OSTs. Rather, each OST has a corresponding OSC which communicates with the MDT. It is necessary to determine the device number of the OSC that corresponds to the OST. Then, you use this device number to deactivate the OSC on the MDT.</para>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1298981" xreflabel=""/>To remove an OST from the file system:</para>
+<para>When removing an OST, remember that the MDT does not communicate directly with OSTs. Rather, each OST has a corresponding OSC which communicates with the MDT. It is necessary to determine the device number of the OSC that corresponds to the OST. Then, you use this device number to deactivate the OSC on the MDT.</para>
+<para>To remove an OST from the file system:</para>
<orderedlist><listitem>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1298982" xreflabel=""/>For the OST to be removed, determine the device number of the corresponding OSC on the MDT.</para>
+<para>For the OST to be removed, determine the device number of the corresponding OSC on the MDT.</para>
<orderedlist><listitem>
<para> List all OSCs on the node, along with their device numbers. Run:</para>
-<screen><anchor xml:id="dbdoclet.50438199_pgfId-1298984" xreflabel=""/>lctldl|grep " osc "
+<screen>lctldl|grep " osc "
</screen>
-<para><anchor xml:id="dbdoclet.50438199_pgfId-1298985" xreflabel=""/>This is sample lctldl|grep</para>
+<para>This is sample lctldl|grep</para>
<screen>
11 UP osc lustre-OST-0000-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
12 UP osc lustre-OST-0001-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
<info>
<title xml:id='lustremonitoring.title'>Lustre Monitoring</title>
</info>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1291311" xreflabel=""/>This chapter provides information on monitoring Lustre and includes the following sections:</para>
+ <para>This chapter provides information on monitoring Lustre and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend='dbdoclet.50438273_18711'/>Lustre Changelogs</para>
<section xml:id="dbdoclet.50438273_18711">
<title>12.1 Lustre <anchor xml:id="dbdoclet.50438273_marker-1297383" xreflabel=""/>Changelogs</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296794" xreflabel=""/>The changelogs feature records events that change the file system namespace or file metadata. Changes such as file creation, deletion, renaming, attribute changes, etc. are recorded with the target and parent file identifiers (FIDs), the name of the target, and a timestamp. These records can be used for a variety of purposes:</para>
+ <para>The changelogs feature records events that change the file system namespace or file metadata. Changes such as file creation, deletion, renaming, attribute changes, etc. are recorded with the target and parent file identifiers (FIDs), the name of the target, and a timestamp. These records can be used for a variety of purposes:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296795" xreflabel=""/> Capture recent changes to feed into an archiving system.</para>
+ <para> Capture recent changes to feed into an archiving system.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296796" xreflabel=""/> Use changelog entries to exactly replicate changes in a file system mirror.</para>
+ <para> Use changelog entries to exactly replicate changes in a file system mirror.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296797" xreflabel=""/> Set up "watch scripts" that take action on certain events or directories.</para>
+ <para> Set up "watch scripts" that take action on certain events or directories.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296798" xreflabel=""/> Maintain a rough audit trail (file/directory changes with timestamps, but no user information).</para>
+ <para> Maintain a rough audit trail (file/directory changes with timestamps, but no user information).</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296879" xreflabel=""/>Changelogs record types are:</para>
+ <para>Changelogs record types are:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438273_pgfId-1296808" xreflabel=""/>Value</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438273_pgfId-1296810" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Value</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296812" xreflabel=""/>MARK</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296814" xreflabel=""/>Internal recordkeeping</para></entry>
+ <entry><para> MARK</para></entry>
+ <entry><para> Internal recordkeeping</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296816" xreflabel=""/>CREAT</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296818" xreflabel=""/>Regular file creation</para></entry>
+ <entry><para> CREAT</para></entry>
+ <entry><para> Regular file creation</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296820" xreflabel=""/>MKDIR</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296822" xreflabel=""/>Directory creation</para></entry>
+ <entry><para> MKDIR</para></entry>
+ <entry><para> Directory creation</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296824" xreflabel=""/>HLINK</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296826" xreflabel=""/>Hard link</para></entry>
+ <entry><para> HLINK</para></entry>
+ <entry><para> Hard link</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296828" xreflabel=""/>SLINK</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296830" xreflabel=""/>Soft link</para></entry>
+ <entry><para> SLINK</para></entry>
+ <entry><para> Soft link</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296832" xreflabel=""/>MKNOD</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296834" xreflabel=""/>Other file creation</para></entry>
+ <entry><para> MKNOD</para></entry>
+ <entry><para> Other file creation</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296836" xreflabel=""/>UNLNK</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296838" xreflabel=""/>Regular file removal</para></entry>
+ <entry><para> UNLNK</para></entry>
+ <entry><para> Regular file removal</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296840" xreflabel=""/>RMDIR</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296842" xreflabel=""/>Directory removal</para></entry>
+ <entry><para> RMDIR</para></entry>
+ <entry><para> Directory removal</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296844" xreflabel=""/>RNMFM</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296846" xreflabel=""/>Rename, original</para></entry>
+ <entry><para> RNMFM</para></entry>
+ <entry><para> Rename, original</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296848" xreflabel=""/>RNMTO</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296850" xreflabel=""/>Rename, final</para></entry>
+ <entry><para> RNMTO</para></entry>
+ <entry><para> Rename, final</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296860" xreflabel=""/>IOCTL</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296862" xreflabel=""/>ioctl on file or directory</para></entry>
+ <entry><para> IOCTL</para></entry>
+ <entry><para> ioctl on file or directory</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296864" xreflabel=""/>TRUNC</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296866" xreflabel=""/>Regular file truncated</para></entry>
+ <entry><para> TRUNC</para></entry>
+ <entry><para> Regular file truncated</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296868" xreflabel=""/>SATTR</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296870" xreflabel=""/>Attribute change</para></entry>
+ <entry><para> SATTR</para></entry>
+ <entry><para> Attribute change</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296872" xreflabel=""/>XATTR</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296874" xreflabel=""/>Extended attribute change</para></entry>
+ <entry><para> XATTR</para></entry>
+ <entry><para> Extended attribute change</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296876" xreflabel=""/>UNKNW</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438273_pgfId-1296878" xreflabel=""/>Unknown operation</para></entry>
+ <entry><para> UNKNW</para></entry>
+ <entry><para> Unknown operation</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296880" xreflabel=""/>FID-to-full-pathname and pathname-to-FID functions are also included to map target and parent FIDs into the file system namespace.</para>
+ <para>FID-to-full-pathname and pathname-to-FID functions are also included to map target and parent FIDs into the file system namespace.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1296889" xreflabel=""/>12.1.1 Working with Changelogs</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296890" xreflabel=""/>Several commands are available to work with changelogs.</para>
+ <title>12.1.1 Working with Changelogs</title>
+ <para>Several commands are available to work with changelogs.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1297845" xreflabel=""/>lctl changelog_register</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297846" xreflabel=""/>Because changelog records take up space on the MDT, the system administration must register changelog users. The registrants specify which records they are "done with", and the system purges up to the greatest common record.</para>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297847" xreflabel=""/>To register a new changelog user, run:</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1297848" xreflabel=""/>lctl --device <mdt_device> changelog_register
+ <title>lctl changelog_register</title>
+ <para>Because changelog records take up space on the MDT, the system administration must register changelog users. The registrants specify which records they are "done with", and the system purges up to the greatest common record.</para>
+ <para>To register a new changelog user, run:</para>
+ <screen>lctl --device <mdt_device> changelog_register
</screen>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297849" xreflabel=""/>Changelog entries are not purged beyond a registered user's set point (see lfs changelog_clear).</para>
+ <para>Changelog entries are not purged beyond a registered user's set point (see lfs changelog_clear).</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1297785" xreflabel=""/>lfs changelog</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297786" xreflabel=""/>To display the metadata changes on an MDT (the changelog records), run:</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1297825" xreflabel=""/>lfs changelog <MDT name> [startrec [endrec]]
+ <title>lfs changelog</title>
+ <para>To display the metadata changes on an MDT (the changelog records), run:</para>
+ <screen>lfs changelog <MDT name> [startrec [endrec]]
</screen>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297789" xreflabel=""/>It is optional whether to specify the start and end records.</para>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297836" xreflabel=""/>These are sample changelog records:</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1297790" xreflabel=""/>2 02MKDIR 4298396676 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0]\
+ <para>It is optional whether to specify the start and end records.</para>
+ <para>These are sample changelog records:</para>
+ <screen>2 02MKDIR 4298396676 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0]\
pics
-<anchor xml:id="dbdoclet.50438273_pgfId-1297791" xreflabel=""/>3 01CREAT 4298402264 0x0 t=[0x200000405:0x15fa:0x0] p=[0x200000405:0x15f9:0\
+3 01CREAT 4298402264 0x0 t=[0x200000405:0x15fa:0x0] p=[0x200000405:0x15f9:0\
x0] chloe.jpg
-<anchor xml:id="dbdoclet.50438273_pgfId-1297792" xreflabel=""/>4 06UNLNK 4298404466 0x0 t=[0x200000405:0x15fa:0x0] p=[0x200000405:0x15f9:0\
+4 06UNLNK 4298404466 0x0 t=[0x200000405:0x15fa:0x0] p=[0x200000405:0x15f9:0\
x0] chloe.jpg
-<anchor xml:id="dbdoclet.50438273_pgfId-1297793" xreflabel=""/>5 07RMDIR 4298405394 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0]\
+5 07RMDIR 4298405394 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0]\
pics
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1297809" xreflabel=""/>lfs changelog_clear</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297810" xreflabel=""/>To clear old changelog records for a specific user (records that the user no longer needs), run:</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1297811" xreflabel=""/>lfs changelog_clear <MDT name> <user ID> <endrec>
+ <title>lfs changelog_clear</title>
+ <para>To clear old changelog records for a specific user (records that the user no longer needs), run:</para>
+ <screen>lfs changelog_clear <MDT name> <user ID> <endrec>
</screen>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297812" xreflabel=""/>The changelog_clear command indicates that changelog records previous to <endrec> are no longer of interest to a particular user <user ID>, potentially allowing the MDT to free up disk space. An <endrec> value of 0 indicates the current last record. To run changelog_clear, the changelog user must be registered on the MDT node using lctl.</para>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297813" xreflabel=""/>When all changelog users are done with records < X, the records are deleted.</para>
+ <para>The changelog_clear command indicates that changelog records previous to <endrec> are no longer of interest to a particular user <user ID>, potentially allowing the MDT to free up disk space. An <endrec> value of 0 indicates the current last record. To run changelog_clear, the changelog user must be registered on the MDT node using lctl.</para>
+ <para>When all changelog users are done with records < X, the records are deleted.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1296930" xreflabel=""/>lctl changelog_deregister</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296931" xreflabel=""/>To deregister (unregister) a changelog user, run:</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1296932" xreflabel=""/>lctl --device <mdt_device> changelog_deregister <user ID>
+ <title>lctl changelog_deregister</title>
+ <para>To deregister (unregister) a changelog user, run:</para>
+ <screen>lctl --device <mdt_device> changelog_deregister <user ID>
</screen>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297395" xreflabel=""/> Changelog_deregister cl1 effectively does a changelog_clear cl10 as it deregisters.</para>
+ <para> Changelog_deregister cl1 effectively does a changelog_clear cl10 as it deregisters.</para>
</section>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1296934" xreflabel=""/>12.1.2 Changelog Examples</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297340" xreflabel=""/>This section provides examples of different changelog commands.</para>
+ <title>12.1.2 Changelog Examples</title>
+ <para>This section provides examples of different changelog commands.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1297341" xreflabel=""/>Registering a Changelog User</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296935" xreflabel=""/>To register a new changelog user for a device (lustre-MDT0000):</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1296936" xreflabel=""/># lctl --device lustre-MDT0000 changelog_register
-<anchor xml:id="dbdoclet.50438273_pgfId-1296937" xreflabel=""/>lustre-MDT0000: Registered changelog userid 'cl1'
+ <title>Registering a Changelog User</title>
+ <para>To register a new changelog user for a device (lustre-MDT0000):</para>
+ <screen># lctl --device lustre-MDT0000 changelog_register
+lustre-MDT0000: Registered changelog userid 'cl1'
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1296938" xreflabel=""/>Displaying Changelog Records</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296939" xreflabel=""/>To display changelog records on an MDT (lustre-MDT0000):</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1296940" xreflabel=""/>$ lfs changelog lustre-MDT0000
-<anchor xml:id="dbdoclet.50438273_pgfId-1296941" xreflabel=""/>1 00MARK 19:08:20.890432813 2010.03.24 0x0 t=[0x10001:0x0:0x0] p=[0:0x0:0x\
+ <title>Displaying Changelog Records</title>
+ <para>To display changelog records on an MDT (lustre-MDT0000):</para>
+ <screen>$ lfs changelog lustre-MDT0000
+1 00MARK 19:08:20.890432813 2010.03.24 0x0 t=[0x10001:0x0:0x0] p=[0:0x0:0x\
0] mdd_obd-lustre-MDT0000-0
-<anchor xml:id="dbdoclet.50438273_pgfId-1296942" xreflabel=""/>2 02MKDIR 19:10:21.509659173 2010.03.24 0x0 t=[0x200000420:0x3:0x0] p=[0x61\
+2 02MKDIR 19:10:21.509659173 2010.03.24 0x0 t=[0x200000420:0x3:0x0] p=[0x61\
b4:0xca2c7dde:0x0] mydir
-<anchor xml:id="dbdoclet.50438273_pgfId-1296943" xreflabel=""/>3 14SATTR 19:10:27.329356533 2010.03.24 0x0 t=[0x200000420:0x3:0x0]
-<anchor xml:id="dbdoclet.50438273_pgfId-1297627" xreflabel=""/>4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
+3 14SATTR 19:10:27.329356533 2010.03.24 0x0 t=[0x200000420:0x3:0x0]
+4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
0000420:0x3:0x0] hosts
</screen>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297628" xreflabel=""/>Changelog records include this information:</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1297629" xreflabel=""/>rec#
-<anchor xml:id="dbdoclet.50438273_pgfId-1297630" xreflabel=""/>operation_type(numerical/text)
-<anchor xml:id="dbdoclet.50438273_pgfId-1297631" xreflabel=""/>timestamp
-<anchor xml:id="dbdoclet.50438273_pgfId-1297632" xreflabel=""/>datestamp
-<anchor xml:id="dbdoclet.50438273_pgfId-1297633" xreflabel=""/>flags
-<anchor xml:id="dbdoclet.50438273_pgfId-1297634" xreflabel=""/>t=target_FID
-<anchor xml:id="dbdoclet.50438273_pgfId-1297635" xreflabel=""/>p=parent_FID
-<anchor xml:id="dbdoclet.50438273_pgfId-1297636" xreflabel=""/>target_name
+ <para>Changelog records include this information:</para>
+ <screen>rec#
+operation_type(numerical/text)
+timestamp
+datestamp
+flags
+t=target_FID
+p=parent_FID
+target_name
</screen>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297637" xreflabel=""/>Displayed in this format:</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1297638" xreflabel=""/>rec# operation_type(numerical/text) timestamp datestamp flags t=target_FID \
+ <para>Displayed in this format:</para>
+ <screen>rec# operation_type(numerical/text) timestamp datestamp flags t=target_FID \
p=parent_FID target_name
</screen>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297639" xreflabel=""/>For example:</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1297640" xreflabel=""/>4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
+ <para>For example:</para>
+ <screen>4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
0000420:0x3:0x0] hosts
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1297641" xreflabel=""/>Clearing Changelog Records</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297348" xreflabel=""/>To notify a device that a specific user (cl1) no longer needs records (up to and including 3):</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1296947" xreflabel=""/>$ lfs changelog_clear lustre-MDT0000 cl1 3
+ <title>Clearing Changelog Records</title>
+ <para>To notify a device that a specific user (cl1) no longer needs records (up to and including 3):</para>
+ <screen>$ lfs changelog_clear lustre-MDT0000 cl1 3
</screen>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296958" xreflabel=""/>To confirm that the changelog_clear operation was successful, run lfs changelog; only records after id-3 are listed:</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1297062" xreflabel=""/>$ lfs changelog lustre-MDT0000
-<anchor xml:id="dbdoclet.50438273_pgfId-1297063" xreflabel=""/>4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
+ <para>To confirm that the changelog_clear operation was successful, run lfs changelog; only records after id-3 are listed:</para>
+ <screen>$ lfs changelog lustre-MDT0000
+4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
0000420:0x3:0x0] hosts
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1297051" xreflabel=""/>Deregistering a Changelog User</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297030" xreflabel=""/>To deregister a changelog user (cl1) for a specific device (lustre-MDT0000):</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1296959" xreflabel=""/># lctl --device lustre-MDT0000 changelog_deregister cl1
-<anchor xml:id="dbdoclet.50438273_pgfId-1296960" xreflabel=""/>lustre-MDT0000: Deregistered changelog user 'cl1'
+ <title>Deregistering a Changelog User</title>
+ <para>To deregister a changelog user (cl1) for a specific device (lustre-MDT0000):</para>
+ <screen># lctl --device lustre-MDT0000 changelog_deregister cl1
+lustre-MDT0000: Deregistered changelog user 'cl1'
</screen>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296961" xreflabel=""/>The deregistration operation clears all changelog records for the specified user (cli).</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1296962" xreflabel=""/>$ lfs changelog lustre-MDT0000
-<anchor xml:id="dbdoclet.50438273_pgfId-1296963" xreflabel=""/>5 00MARK 19:13:40.858292517 2010.03.24 0x0 t=[0x40001:0x0:0x0] p=[0:0x0:0x\
+ <para>The deregistration operation clears all changelog records for the specified user (cli).</para>
+ <screen>$ lfs changelog lustre-MDT0000
+5 00MARK 19:13:40.858292517 2010.03.24 0x0 t=[0x40001:0x0:0x0] p=[0:0x0:0x\
0] mdd_obd-lustre-MDT0000-0
</screen>
<informaltable frame="none">
<colspec colname="c1" colwidth="100*"/>
<tbody>
<row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438273_pgfId-1296965" xreflabel=""/>MARK records typically indicate changelog recording status changes.</para></entry>
+ <entry><para><emphasis role="bold">Note -</emphasis>MARK records typically indicate changelog recording status changes.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1296966" xreflabel=""/>Displaying the Changelog Index and Registered Users</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296971" xreflabel=""/>To display the current, maximum changelog index and registered changelog users for a specific device (lustre-MDT0000):</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1296972" xreflabel=""/># lctl get_param mdd.lustre-MDT0000.changelog_users
-<anchor xml:id="dbdoclet.50438273_pgfId-1296973" xreflabel=""/>mdd.lustre-MDT0000.changelog_users=current index: 8
-<anchor xml:id="dbdoclet.50438273_pgfId-1296974" xreflabel=""/>ID index
-<anchor xml:id="dbdoclet.50438273_pgfId-1296975" xreflabel=""/>cl2 8
+ <title>Displaying the Changelog Index and Registered Users</title>
+ <para>To display the current, maximum changelog index and registered changelog users for a specific device (lustre-MDT0000):</para>
+ <screen># lctl get_param mdd.lustre-MDT0000.changelog_users
+mdd.lustre-MDT0000.changelog_users=current index: 8
+ID index
+cl2 8
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1296976" xreflabel=""/>Displaying the Changelog Mask</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296977" xreflabel=""/>To show the current changelog mask on a specific device (lustre-MDT0000):</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1296978" xreflabel=""/># lctl get_param mdd.lustre-MDT0000.changelog_mask
-<anchor xml:id="dbdoclet.50438273_pgfId-1296979" xreflabel=""/>mdd.lustre-MDT0000.changelog_mask=
-<anchor xml:id="dbdoclet.50438273_pgfId-1296980" xreflabel=""/>MARK CREAT MKDIR HLINK SLINK MKNOD UNLNK RMDIR RNMFM RNMTO OPEN CLOSE IOCTL\
+ <title>Displaying the Changelog Mask</title>
+ <para>To show the current changelog mask on a specific device (lustre-MDT0000):</para>
+ <screen># lctl get_param mdd.lustre-MDT0000.changelog_mask
+mdd.lustre-MDT0000.changelog_mask=
+MARK CREAT MKDIR HLINK SLINK MKNOD UNLNK RMDIR RNMFM RNMTO OPEN CLOSE IOCTL\
TRUNC SATTR XATTR HSM
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438273_pgfId-1297457" xreflabel=""/>Setting the Changelog Mask</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297458" xreflabel=""/>To set the current changelog mask on a specific device (lustre-MDT0000):</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1297459" xreflabel=""/># lctl set_param mdd.lustre-MDT0000.changelog_mask=HLINK
-<anchor xml:id="dbdoclet.50438273_pgfId-1297460" xreflabel=""/>mdd.lustre-MDT0000.changelog_mask=HLINK
-<anchor xml:id="dbdoclet.50438273_pgfId-1297288" xreflabel=""/>$ lfs changelog_clear lustre-MDT0000 cl1 0
-<anchor xml:id="dbdoclet.50438273_pgfId-1297293" xreflabel=""/>$ mkdir /mnt/lustre/mydir/foo
-<anchor xml:id="dbdoclet.50438273_pgfId-1297314" xreflabel=""/>$ cp /etc/hosts /mnt/lustre/mydir/foo/file
-<anchor xml:id="dbdoclet.50438273_pgfId-1297316" xreflabel=""/>$ ln /mnt/lustre/mydir/foo/file /mnt/lustre/mydir/myhardlink
+ <title>Setting the Changelog Mask</title>
+ <para>To set the current changelog mask on a specific device (lustre-MDT0000):</para>
+ <screen># lctl set_param mdd.lustre-MDT0000.changelog_mask=HLINK
+mdd.lustre-MDT0000.changelog_mask=HLINK
+$ lfs changelog_clear lustre-MDT0000 cl1 0
+$ mkdir /mnt/lustre/mydir/foo
+$ cp /etc/hosts /mnt/lustre/mydir/foo/file
+$ ln /mnt/lustre/mydir/foo/file /mnt/lustre/mydir/myhardlink
</screen>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297318" xreflabel=""/> Only item types that are in the mask show up in the changelog.</para>
- <screen><anchor xml:id="dbdoclet.50438273_pgfId-1297441" xreflabel=""/>$ lfs changelog lustre-MDT0000
-<anchor xml:id="dbdoclet.50438273_pgfId-1297321" xreflabel=""/>9 03HLINK 19:19:35.171867477 2010.03.24 0x0 t=[0x200000420:0x6:0x0] p=[0x20\
+ <para> Only item types that are in the mask show up in the changelog.</para>
+ <screen>$ lfs changelog lustre-MDT0000
+9 03HLINK 19:19:35.171867477 2010.03.24 0x0 t=[0x200000420:0x6:0x0] p=[0x20\
0000420:0x3:0x0] myhardlink
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438273_81684">
<title>12.2 Lustre <anchor xml:id="dbdoclet.50438273_marker-1297386" xreflabel=""/>Monitoring Tool</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1298196" xreflabel=""/>The Lustre Monitoring Tool (LMT) is a Python-based, distributed system developed and maintained by Lawrence Livermore National Lab (LLNL)). It provides a ''top'' like display of activity on server-side nodes (MDS, OSS and portals routers) on one or more Lustre file systems. It does not provide support for monitoring clients. For more information on LMT, including the setup procedure, see:</para>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296057" xreflabel=""/><link xl:href="http://code.google.com/p/lmt/">http://code.google.com/p/lmt/</link></para>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296102" xreflabel=""/>LMT questions can be directed to:</para>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296120" xreflabel=""/><link xl:href="mailto:lmt-discuss@googlegroups.com">lmt-discuss@googlegroups.com</link></para>
+ <para>The Lustre Monitoring Tool (LMT) is a Python-based, distributed system developed and maintained by Lawrence Livermore National Lab (LLNL)). It provides a ''top'' like display of activity on server-side nodes (MDS, OSS and portals routers) on one or more Lustre file systems. It does not provide support for monitoring clients. For more information on LMT, including the setup procedure, see:</para>
+ <para><link xl:href="http://code.google.com/p/lmt/">http://code.google.com/p/lmt/</link></para>
+ <para>LMT questions can be directed to:</para>
+ <para><link xl:href="mailto:lmt-discuss@googlegroups.com">lmt-discuss@googlegroups.com</link></para>
</section>
<section xml:id="dbdoclet.50438273_80593">
<title>12.3 Collect<anchor xml:id="dbdoclet.50438273_marker-1297391" xreflabel=""/>L</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1293325" xreflabel=""/>CollectL is another tool that can be used to monitor Lustre. You can run CollectL on a Lustre system that has any combination of MDSs, OSTs and clients. The collected data can be written to a file for continuous logging and played back at a later time. It can also be converted to a format suitable for plotting.</para>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1293329" xreflabel=""/>For more information about CollectL, see:</para>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1294650" xreflabel=""/><link xl:href="http://collectl.sourceforge.net">http://collectl.sourceforge.net</link></para>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1294692" xreflabel=""/>Lustre-specific documentation is also available. See:</para>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1294696" xreflabel=""/><link xl:href="http://collectl.sourceforge.net/Tutorial-Lustre.html">http://collectl.sourceforge.net/Tutorial-Lustre.html</link></para>
+ <para>CollectL is another tool that can be used to monitor Lustre. You can run CollectL on a Lustre system that has any combination of MDSs, OSTs and clients. The collected data can be written to a file for continuous logging and played back at a later time. It can also be converted to a format suitable for plotting.</para>
+ <para>For more information about CollectL, see:</para>
+ <para><link xl:href="http://collectl.sourceforge.net">http://collectl.sourceforge.net</link></para>
+ <para>Lustre-specific documentation is also available. See:</para>
+ <para><link xl:href="http://collectl.sourceforge.net/Tutorial-Lustre.html">http://collectl.sourceforge.net/Tutorial-Lustre.html</link></para>
</section>
<section xml:id="dbdoclet.50438273_44185">
<title>12.4 Other Monitoring Options</title>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1295490" xreflabel=""/>A variety of standard tools are available publically.</para>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1298235" xreflabel=""/>Another option is to script a simple monitoring solution that looks at various reports from ipconfig, as well as the procfs files generated by Lustre.</para>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1296750" xreflabel=""/><anchor xml:id="dbdoclet.50438273_67514" xreflabel=""/> </para>
+ <para>A variety of standard tools are available publically.</para>
+ <para>Another option is to script a simple monitoring solution that looks at various reports from ipconfig, as well as the procfs files generated by Lustre.</para>
+ <para><anchor xml:id="dbdoclet.50438273_67514" xreflabel=""/> </para>
</section>
</chapter>
</info>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298785" xreflabel=""/>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre administration tasks:</para>
+ <para>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre administration tasks:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438194_42877"/></para>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438194_42877">
<title>13.1 Mounting by Label</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298853" xreflabel=""/>The file system name is limited to 8 characters. We have encoded the file system and target information in the disk label, so you can mount by label. This allows system administrators to move disks around without worrying about issues such as SCSI disk reordering or getting the /dev/device wrong for a shared target. Soon, file system naming will be made as fail-safe as possible. Currently, Linux disk labels are limited to 16 characters. To identify the target within the file system, 8 characters are reserved, leaving 8 characters for the file system name:</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298854" xreflabel=""/><fsname>-MDT0000 or <fsname>-OST0a19</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298855" xreflabel=""/>To mount by label, use this command:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298856" xreflabel=""/>$ mount -t lustre -L <file system label> <mount point>
+ <para>The file system name is limited to 8 characters. We have encoded the file system and target information in the disk label, so you can mount by label. This allows system administrators to move disks around without worrying about issues such as SCSI disk reordering or getting the /dev/device wrong for a shared target. Soon, file system naming will be made as fail-safe as possible. Currently, Linux disk labels are limited to 16 characters. To identify the target within the file system, 8 characters are reserved, leaving 8 characters for the file system name:</para>
+ <para><fsname>-MDT0000 or <fsname>-OST0a19</para>
+ <para>To mount by label, use this command:</para>
+ <screen>$ mount -t lustre -L <file system label> <mount point>
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298857" xreflabel=""/>This is an example of mount-by-label:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298858" xreflabel=""/>$ mount -t lustre -L testfs-MDT0000 /mnt/mdt
+ <para>This is an example of mount-by-label:</para>
+ <screen>$ mount -t lustre -L testfs-MDT0000 /mnt/mdt
</screen>
<caution><para>Mount-by-label should NOT be used in a multi-path environment.</para></caution>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298860" xreflabel=""/>Although the file system name is internally limited to 8 characters, you can mount the clients at any mount point, so file system users are not subjected to short names. Here is an example:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298861" xreflabel=""/>mount -t lustre uml1@tcp0:/shortfs /mnt/<long-file_system-name>
+ <para>Although the file system name is internally limited to 8 characters, you can mount the clients at any mount point, so file system users are not subjected to short names. Here is an example:</para>
+ <screen>mount -t lustre uml1@tcp0:/shortfs /mnt/<long-file_system-name>
</screen>
</section>
<section xml:id="dbdoclet.50438194_24122">
<title>13.2 Starting <anchor xml:id="dbdoclet.50438194_marker-1305696" xreflabel=""/>Lustre</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1304353" xreflabel=""/>The startup order of Lustre components depends on whether you have a combined MGS/MDT or these components are separate.</para>
+ <para>The startup order of Lustre components depends on whether you have a combined MGS/MDT or these components are separate.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1304354" xreflabel=""/> If you have a combined MGS/MDT, the recommended startup order is OSTs, then the MGS/MDT, and then clients.</para>
+ <para> If you have a combined MGS/MDT, the recommended startup order is OSTs, then the MGS/MDT, and then clients.</para>
</listitem>
<listitem>
<para> </para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1304355" xreflabel=""/> If the MGS and MDT are separate, the recommended startup order is: MGS, then OSTs, then the MDT, and then clients.</para>
+ <para> If the MGS and MDT are separate, the recommended startup order is: MGS, then OSTs, then the MDT, and then clients.</para>
</listitem>
<listitem>
<para> </para>
</section>
<section xml:id="dbdoclet.50438194_84876">
<title>13.3 Mounting a <anchor xml:id="dbdoclet.50438194_marker-1298863" xreflabel=""/>Server</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298865" xreflabel=""/>Starting a Lustre server is straightforward and only involves the mount command. Lustre servers can be added to /etc/fstab:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298866" xreflabel=""/>mount -t lustre
+ <para>Starting a Lustre server is straightforward and only involves the mount command. Lustre servers can be added to /etc/fstab:</para>
+ <screen>mount -t lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298867" xreflabel=""/>The mount command generates output similar to this:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298868" xreflabel=""/>/dev/sda1 on /mnt/test/mdt type lustre (rw)
-<anchor xml:id="dbdoclet.50438194_pgfId-1298869" xreflabel=""/>/dev/sda2 on /mnt/test/ost0 type lustre (rw)
-<anchor xml:id="dbdoclet.50438194_pgfId-1298870" xreflabel=""/>192.168.0.21@tcp:/testfs on /mnt/testfs type lustre (rw)
+ <para>The mount command generates output similar to this:</para>
+ <screen>/dev/sda1 on /mnt/test/mdt type lustre (rw)
+/dev/sda2 on /mnt/test/ost0 type lustre (rw)
+192.168.0.21@tcp:/testfs on /mnt/testfs type lustre (rw)
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298871" xreflabel=""/>In this example, the MDT, an OST (ost0) and file system (testfs) are mounted.</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298872" xreflabel=""/>LABEL=testfs-MDT0000 /mnt/test/mdt lustre defaults,_netdev,noauto 0 0
-<anchor xml:id="dbdoclet.50438194_pgfId-1298873" xreflabel=""/>LABEL=testfs-OST0000 /mnt/test/ost0 lustre defaults,_netdev,noauto 0 0
+ <para>In this example, the MDT, an OST (ost0) and file system (testfs) are mounted.</para>
+ <screen>LABEL=testfs-MDT0000 /mnt/test/mdt lustre defaults,_netdev,noauto 0 0
+LABEL=testfs-OST0000 /mnt/test/ost0 lustre defaults,_netdev,noauto 0 0
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298874" xreflabel=""/>In general, it is wise to specify noauto and let your high-availability (HA) package manage when to mount the device. If you are not using failover, make sure that networking has been started before mounting a Lustre server. RedHat, SuSE, Debian (and perhaps others) use the _netdev flag to ensure that these disks are mounted after the network is up.</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298875" xreflabel=""/>We are mounting by disk label here--the label of a device can be read with e2label. The label of a newly-formatted Lustre server ends in FFFF, meaning that it has yet to be assigned. The assignment takes place when the server is first started, and the disk label is updated.</para>
+ <para>In general, it is wise to specify noauto and let your high-availability (HA) package manage when to mount the device. If you are not using failover, make sure that networking has been started before mounting a Lustre server. RedHat, SuSE, Debian (and perhaps others) use the _netdev flag to ensure that these disks are mounted after the network is up.</para>
+ <para>We are mounting by disk label here--the label of a device can be read with e2label. The label of a newly-formatted Lustre server ends in FFFF, meaning that it has yet to be assigned. The assignment takes place when the server is first started, and the disk label is updated.</para>
<caution><para>Do not do this when the client and OSS are on the same node, as memory pressure between the client and OSS can lead to deadlocks.</para></caution>
</section>
<section xml:id="dbdoclet.50438194_69255">
<title>13.4 Unmounting a<anchor xml:id="dbdoclet.50438194_marker-1298879" xreflabel=""/> Server</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298881" xreflabel=""/>To stop a Lustre server, use the umount <mount point> command.</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298882" xreflabel=""/>For example, to stop ost0 on mount point /mnt/test, run:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298883" xreflabel=""/>$ umount /mnt/test
+ <para>To stop a Lustre server, use the umount <mount point> command.</para>
+ <para>For example, to stop ost0 on mount point /mnt/test, run:</para>
+ <screen>$ umount /mnt/test
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298884" xreflabel=""/>Gracefully stopping a server with the umount command preserves the state of the connected clients. The next time the server is started, it waits for clients to reconnect, and then goes through the recovery procedure.</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298885" xreflabel=""/>If the force (-f) flag is used, then the server evicts all clients and stops WITHOUT recovery. Upon restart, the server does not wait for recovery. Any currently connected clients receive I/O errors until they reconnect.</para>
+ <para>Gracefully stopping a server with the umount command preserves the state of the connected clients. The next time the server is started, it waits for clients to reconnect, and then goes through the recovery procedure.</para>
+ <para>If the force (-f) flag is used, then the server evicts all clients and stops WITHOUT recovery. Upon restart, the server does not wait for recovery. Any currently connected clients receive I/O errors until they reconnect.</para>
<note><para>If you are using loopback devices, use the -d flag. This flag cleans up loop devices and can always be safely specified.</para></note>
</section>
<section xml:id="dbdoclet.50438194_57420">
<title>13.5 Specifying Fail<anchor xml:id="dbdoclet.50438194_marker-1298926" xreflabel=""/>out/Failover Mode for OSTs</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298928" xreflabel=""/>Lustre uses two modes, failout and failover, to handle an OST that has become unreachable because it fails, is taken off the network, is unmounted, etc.</para>
+ <para>Lustre uses two modes, failout and failover, to handle an OST that has become unreachable because it fails, is taken off the network, is unmounted, etc.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298929" xreflabel=""/> In <emphasis>failout</emphasis> mode, Lustre clients immediately receive errors (EIOs) after a timeout, instead of waiting for the OST to recover.</para>
+ <para> In <emphasis>failout</emphasis> mode, Lustre clients immediately receive errors (EIOs) after a timeout, instead of waiting for the OST to recover.</para>
</listitem>
<listitem>
<para> </para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298930" xreflabel=""/> In <emphasis>failover</emphasis> mode, Lustre clients wait for the OST to recover.</para>
+ <para> In <emphasis>failover</emphasis> mode, Lustre clients wait for the OST to recover.</para>
</listitem>
<listitem>
<para> </para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298931" xreflabel=""/>By default, the Lustre file system uses failover mode for OSTs. To specify failout mode instead, run this command:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298932" xreflabel=""/>$ mkfs.lustre --fsname=<fsname> --ost --mgsnode=<MGS node NID> --param="failover\
+ <para>By default, the Lustre file system uses failover mode for OSTs. To specify failout mode instead, run this command:</para>
+ <screen>$ mkfs.lustre --fsname=<fsname> --ost --mgsnode=<MGS node NID> --param="failover\
.mode=failout" <block device name>
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298934" xreflabel=""/>In this example, failout mode is specified for the OSTs on MGS uml1, file system testfs.</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298935" xreflabel=""/>$ mkfs.lustre --fsname=testfs --ost --mgsnode=uml1 --param="failover.mode=fa\
+ <para>In this example, failout mode is specified for the OSTs on MGS uml1, file system testfs.</para>
+ <screen>$ mkfs.lustre --fsname=testfs --ost --mgsnode=uml1 --param="failover.mode=fa\
ilout" /dev/sdb
</screen>
</section>
<section xml:id="dbdoclet.50438194_54138">
<title>13.6 Handling <anchor xml:id="dbdoclet.50438194_marker-1307136" xreflabel=""/>Degraded OST RAID Arrays</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307138" xreflabel=""/>Lustre includes functionality that notifies Lustre if an external RAID array has degraded performance (resulting in reduced overall file system performance), either because a disk has failed and not been replaced, or because a disk was replaced and is undergoing a rebuild. To avoid a global performance slowdown due to a degraded OST, the MDS can avoid the OST for new object allocation if it is notified of the degraded state.</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307139" xreflabel=""/>A parameter for each OST, called degraded, specifies whether the OST is running in degraded mode or not.</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307140" xreflabel=""/>To mark the OST as degraded, use:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307141" xreflabel=""/>lctl set_param obdfilter.{OST_name}.degraded=1</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307142" xreflabel=""/>To mark that the OST is back in normal operation, use:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307143" xreflabel=""/>lctl set_param obdfilter.{OST_name}.degraded=0
-</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307144" xreflabel=""/>To determine if OSTs are currently in degraded mode, use:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307145" xreflabel=""/>lctl get_param obdfilter.*.degraded
-</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307146" xreflabel=""/>If the OST is remounted due to a reboot or other condition, the flag resets to 0.</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307147" xreflabel=""/>It is recommended that this be implemented by an automated script that monitors the status of individual RAID devices.</para>
+ <para>Lustre includes functionality that notifies Lustre if an external RAID array has degraded performance (resulting in reduced overall file system performance), either because a disk has failed and not been replaced, or because a disk was replaced and is undergoing a rebuild. To avoid a global performance slowdown due to a degraded OST, the MDS can avoid the OST for new object allocation if it is notified of the degraded state.</para>
+ <para>A parameter for each OST, called degraded, specifies whether the OST is running in degraded mode or not.</para>
+ <para>To mark the OST as degraded, use:</para>
+ <screen>lctl set_param obdfilter.{OST_name}.degraded=1</screen>
+ <para>To mark that the OST is back in normal operation, use:</para>
+ <screen>lctl set_param obdfilter.{OST_name}.degraded=0
+</screen>
+ <para>To determine if OSTs are currently in degraded mode, use:</para>
+ <screen>lctl get_param obdfilter.*.degraded
+</screen>
+ <para>If the OST is remounted due to a reboot or other condition, the flag resets to 0.</para>
+ <para>It is recommended that this be implemented by an automated script that monitors the status of individual RAID devices.</para>
</section>
<section xml:id="dbdoclet.50438194_88063">
<title>13.7 Running Multiple<anchor xml:id="dbdoclet.50438194_marker-1298939" xreflabel=""/> Lustre File Systems</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298941" xreflabel=""/>There may be situations in which you want to run multiple file systems. This is doable, as long as you follow specific naming conventions.</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298942" xreflabel=""/>By default, the mkfs.lustre command creates a file system named lustre. To specify a different file system name (limited to 8 characters), run this command:</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298943" xreflabel=""/>mkfs.lustre --fsname=<new file system name></para>
+ <para>There may be situations in which you want to run multiple file systems. This is doable, as long as you follow specific naming conventions.</para>
+ <para>By default, the mkfs.lustre command creates a file system named lustre. To specify a different file system name (limited to 8 characters), run this command:</para>
+ <para>mkfs.lustre --fsname=<new file system name></para>
<note><para>The MDT, OSTs and clients in the new file system must share the same name (prepended to the device name). For example, for a new file system named foo, the MDT and two OSTs would be named foo-MDT0000, foo-OST0000, and foo-OST0001.</para></note>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298945" xreflabel=""/>To mount a client on the file system, run:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298946" xreflabel=""/>mount -t lustre mgsnode:/<new fsname> <mountpoint>
+ <para>To mount a client on the file system, run:</para>
+ <screen>mount -t lustre mgsnode:/<new fsname> <mountpoint>
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298947" xreflabel=""/>For example, to mount a client on file system foo at mount point /mnt/lustre1, run:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298948" xreflabel=""/>mount -t lustre mgsnode:/foo /mnt/lustre1
+ <para>For example, to mount a client on file system foo at mount point /mnt/lustre1, run:</para>
+ <screen>mount -t lustre mgsnode:/foo /mnt/lustre1
</screen>
<note><para>If a client(s) will be mounted on several file systems, add the following line to /etc/xattr.conf file to avoid problems when files are moved between the file systems: lustre.* skip</para></note>
<note><para>The MGS is universal; there is only one MGS per Lustre installation, not per file system.</para></note>
<note><para>There is only one file system per MDT. Therefore, specify --mdt --mgs on one file system and --mdt --mgsnode=<MGS node NID> on the other file systems.</para></note>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298951" xreflabel=""/>A Lustre installation with two file systems (foo and bar) could look like this, where the MGS node is mgsnode@tcp0 and the mount points are /mnt/lustre1 and /mnt/lustre2.</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307225" xreflabel=""/>mgsnode# mkfs.lustre --mgs /mnt/lustre1
-<anchor xml:id="dbdoclet.50438194_pgfId-1298953" xreflabel=""/>mdtfoonode# mkfs.lustre --fsname=foo --mdt --mgsnode=mgsnode@tcp0 /mnt/lust\
+ <para>A Lustre installation with two file systems (foo and bar) could look like this, where the MGS node is mgsnode@tcp0 and the mount points are /mnt/lustre1 and /mnt/lustre2.</para>
+ <screen>mgsnode# mkfs.lustre --mgs /mnt/lustre1
+mdtfoonode# mkfs.lustre --fsname=foo --mdt --mgsnode=mgsnode@tcp0 /mnt/lust\
re1
-<anchor xml:id="dbdoclet.50438194_pgfId-1298954" xreflabel=""/>ossfoonode# mkfs.lustre --fsname=foo --ost --mgsnode=mgsnode@tcp0 /mnt/lust\
+ossfoonode# mkfs.lustre --fsname=foo --ost --mgsnode=mgsnode@tcp0 /mnt/lust\
re1
-<anchor xml:id="dbdoclet.50438194_pgfId-1298955" xreflabel=""/>ossfoonode# mkfs.lustre --fsname=foo --ost --mgsnode=mgsnode@tcp0 /mnt/lust\
+ossfoonode# mkfs.lustre --fsname=foo --ost --mgsnode=mgsnode@tcp0 /mnt/lust\
re2
-<anchor xml:id="dbdoclet.50438194_pgfId-1298956" xreflabel=""/>mdtbarnode# mkfs.lustre --fsname=bar --mdt --mgsnode=mgsnode@tcp0 /mnt/lust\
+mdtbarnode# mkfs.lustre --fsname=bar --mdt --mgsnode=mgsnode@tcp0 /mnt/lust\
re1
-<anchor xml:id="dbdoclet.50438194_pgfId-1298957" xreflabel=""/>ossbarnode# mkfs.lustre --fsname=bar --ost --mgsnode=mgsnode@tcp0 /mnt/lust\
+ossbarnode# mkfs.lustre --fsname=bar --ost --mgsnode=mgsnode@tcp0 /mnt/lust\
re1
-<anchor xml:id="dbdoclet.50438194_pgfId-1298958" xreflabel=""/>ossbarnode# mkfs.lustre --fsname=bar --ost --mgsnode=mgsnode@tcp0 /mnt/lust\
+ossbarnode# mkfs.lustre --fsname=bar --ost --mgsnode=mgsnode@tcp0 /mnt/lust\
re2
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298959" xreflabel=""/>To mount a client on file system foo at mount point /mnt/lustre1, run:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298960" xreflabel=""/>mount -t lustre mgsnode@tcp0:/foo /mnt/lustre1
+ <para>To mount a client on file system foo at mount point /mnt/lustre1, run:</para>
+ <screen>mount -t lustre mgsnode@tcp0:/foo /mnt/lustre1
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1298961" xreflabel=""/>To mount a client on file system bar at mount point /mnt/lustre2, run:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1298962" xreflabel=""/>mount -t lustre mgsnode@tcp0:/bar /mnt/lustre2
+ <para>To mount a client on file system bar at mount point /mnt/lustre2, run:</para>
+ <screen>mount -t lustre mgsnode@tcp0:/bar /mnt/lustre2
</screen>
</section>
<section xml:id="dbdoclet.50438194_88980">
<title>13.8 Setting <anchor xml:id="dbdoclet.50438194_marker-1302467" xreflabel=""/>and Retrieving Lustre Parameters</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1301646" xreflabel=""/>Several options are available for setting parameters in Lustre:</para>
+ <para>Several options are available for setting parameters in Lustre:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1301703" xreflabel=""/> When creating a file system, use mkfs.lustre. See <link xl:href="LustreOperations.html#50438194_17237">Setting Parameters with mkfs.lustre</link> below.</para>
+ <para> When creating a file system, use mkfs.lustre. See <link xl:href="LustreOperations.html#50438194_17237">Setting Parameters with mkfs.lustre</link> below.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1301745" xreflabel=""/> When a server is stopped, use tunefs.lustre. See <link xl:href="LustreOperations.html#50438194_55253">Setting Parameters with tunefs.lustre</link> below.</para>
+ <para> When a server is stopped, use tunefs.lustre. See <link xl:href="LustreOperations.html#50438194_55253">Setting Parameters with tunefs.lustre</link> below.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1301707" xreflabel=""/> When the file system is running, use lctl to set or retrieve Lustre parameters. See <link xl:href="LustreOperations.html#50438194_51490">Setting Parameters with lctl</link> and <link xl:href="LustreOperations.html#50438194_63247">Reporting Current Parameter Values</link> below.</para>
+ <para> When the file system is running, use lctl to set or retrieve Lustre parameters. See <link xl:href="LustreOperations.html#50438194_51490">Setting Parameters with lctl</link> and <link xl:href="LustreOperations.html#50438194_63247">Reporting Current Parameter Values</link> below.</para>
</listitem>
</itemizedlist>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438194_pgfId-1301648" xreflabel=""/>13.8.1 <anchor xml:id="dbdoclet.50438194_17237" xreflabel=""/>Setting Parameters with <anchor xml:id="dbdoclet.50438194_marker-1305722" xreflabel=""/>mkfs.lustre</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1301792" xreflabel=""/>When the file system is created, parameters can simply be added as a --param option to the mkfs.lustre command. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1301795" xreflabel=""/>$ mkfs.lustre --mdt --param="sys.timeout=50" /dev/sda
+ <title>13.8.1 <anchor xml:id="dbdoclet.50438194_17237" xreflabel=""/>Setting Parameters with <anchor xml:id="dbdoclet.50438194_marker-1305722" xreflabel=""/>mkfs.lustre</title>
+ <para>When the file system is created, parameters can simply be added as a --param option to the mkfs.lustre command. For example:</para>
+ <screen>$ mkfs.lustre --mdt --param="sys.timeout=50" /dev/sda
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1306965" xreflabel=""/>For more details about creating a file system,see <link xl:href="ConfiguringLustre.html#50438267_88428">Chapter 10</link><emphasis>:</emphasis><link xl:href="ConfiguringLustre.html#50438267_66186">Configuring Lustre</link>. For more details about mkfs.lustre, see <link xl:href="SystemConfigurationUtilities.html#50438219_66186">Chapter 36: System Configuration Utilities</link>.</para>
+ <para>For more details about creating a file system,see <link xl:href="ConfiguringLustre.html#50438267_88428">Chapter 10</link><emphasis>:</emphasis><link xl:href="ConfiguringLustre.html#50438267_66186">Configuring Lustre</link>. For more details about mkfs.lustre, see <link xl:href="SystemConfigurationUtilities.html#50438219_66186">Chapter 36: System Configuration Utilities</link>.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438194_pgfId-1301767" xreflabel=""/>13.8.2 <anchor xml:id="dbdoclet.50438194_55253" xreflabel=""/>Setting Parameters with <anchor xml:id="dbdoclet.50438194_marker-1305720" xreflabel=""/>tunefs.lustre</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1301651" xreflabel=""/>If a server (OSS or MDS) is stopped, parameters can be added using the --param option to the tunefs.lustre command. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1301652" xreflabel=""/>$ tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda
+ <title>13.8.2 <anchor xml:id="dbdoclet.50438194_55253" xreflabel=""/>Setting Parameters with <anchor xml:id="dbdoclet.50438194_marker-1305720" xreflabel=""/>tunefs.lustre</title>
+ <para>If a server (OSS or MDS) is stopped, parameters can be added using the --param option to the tunefs.lustre command. For example:</para>
+ <screen>$ tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1301653" xreflabel=""/>With tunefs.lustre, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old tunefs.lustre parameters and just use newly-specified parameters, run:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1301856" xreflabel=""/>$ tunefs.lustre --erase-params --param=<new parameters> </screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1302108" xreflabel=""/>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <obd|fsname>.<obdtype>.<proc_file_name>=<value>. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1302086" xreflabel=""/>$ tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1
+ <para>With tunefs.lustre, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old tunefs.lustre parameters and just use newly-specified parameters, run:</para>
+ <screen>$ tunefs.lustre --erase-params --param=<new parameters> </screen>
+ <para>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <obd|fsname>.<obdtype>.<proc_file_name>=<value>. For example:</para>
+ <screen>$ tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307012" xreflabel=""/>For more details about tunefs.lustre, see <link xl:href="SystemConfigurationUtilities.html#50438219_66186">Chapter 36: System Configuration Utilities</link>.</para>
+ <para>For more details about tunefs.lustre, see <link xl:href="SystemConfigurationUtilities.html#50438219_66186">Chapter 36: System Configuration Utilities</link>.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438194_pgfId-1301773" xreflabel=""/>13.8.3 <anchor xml:id="dbdoclet.50438194_51490" xreflabel=""/>Setting Parameters <anchor xml:id="dbdoclet.50438194_marker-1305718" xreflabel=""/>with lctl</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1304759" xreflabel=""/>When the file system is running, the lctl command can be used to set parameters (temporary or permanent) and report current parameter values. Temporary parameters are active as long as the server or client is not shut down. Permanent parameters live through server and client reboots.</para>
+ <title>13.8.3 <anchor xml:id="dbdoclet.50438194_51490" xreflabel=""/>Setting Parameters <anchor xml:id="dbdoclet.50438194_marker-1305718" xreflabel=""/>with lctl</title>
+ <para>When the file system is running, the lctl command can be used to set parameters (temporary or permanent) and report current parameter values. Temporary parameters are active as long as the server or client is not shut down. Permanent parameters live through server and client reboots.</para>
<note><para>The lctl list_param command enables users to list all parameters that can be set. See <xref linkend='dbdoclet.50438194_88217'/>.</para></note>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307027" xreflabel=""/>For more details about the lctl command, see the examples in the sections below and <xref linkend='systemconfigurationutilities'/>.</para>
+ <para>For more details about the lctl command, see the examples in the sections below and <xref linkend='systemconfigurationutilities'/>.</para>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438194_pgfId-1307025" xreflabel=""/>13.8.3.1 Setting Temporary Parameters</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1303290" xreflabel=""/>Use lctl set_param to set temporary parameters on the node where it is run. These parameters map to items in /proc/{fs,sys}/{lnet,lustre}. The lctl set_param command uses this syntax:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1302283" xreflabel=""/>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value>
-</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1302281" xreflabel=""/>For example:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1301900" xreflabel=""/># lctl set_param osc.*.max_dirty_mb=1024
-<anchor xml:id="dbdoclet.50438194_pgfId-1304516" xreflabel=""/>osc.myth-OST0000-osc.max_dirty_mb=32
-<anchor xml:id="dbdoclet.50438194_pgfId-1304542" xreflabel=""/>osc.myth-OST0001-osc.max_dirty_mb=32
-<anchor xml:id="dbdoclet.50438194_pgfId-1304545" xreflabel=""/>osc.myth-OST0002-osc.max_dirty_mb=32
-<anchor xml:id="dbdoclet.50438194_pgfId-1304548" xreflabel=""/>osc.myth-OST0003-osc.max_dirty_mb=32
-<anchor xml:id="dbdoclet.50438194_pgfId-1304551" xreflabel=""/>osc.myth-OST0004-osc.max_dirty_mb=32
+ <title>13.8.3.1 Setting Temporary Parameters</title>
+ <para>Use lctl set_param to set temporary parameters on the node where it is run. These parameters map to items in /proc/{fs,sys}/{lnet,lustre}. The lctl set_param command uses this syntax:</para>
+ <screen>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value>
+</screen>
+ <para>For example:</para>
+ <screen># lctl set_param osc.*.max_dirty_mb=1024
+osc.myth-OST0000-osc.max_dirty_mb=32
+osc.myth-OST0001-osc.max_dirty_mb=32
+osc.myth-OST0002-osc.max_dirty_mb=32
+osc.myth-OST0003-osc.max_dirty_mb=32
+osc.myth-OST0004-osc.max_dirty_mb=32
</screen>
</section>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438194_pgfId-1302347" xreflabel=""/>13.8.3.2 <anchor xml:id="dbdoclet.50438194_64195" xreflabel=""/>Setting Permanent Parameters</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1303209" xreflabel=""/>Use the lctl conf_param command to set permanent parameters. In general, the lctl conf_param command can be used to specify any parameter settable in a /proc/fs/lustre file, with its own OBD device. The lctl conf_param command uses this syntax (same as the mkfs.lustre and tunefs.lustre commands):</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1302348" xreflabel=""/><obd|fsname>.<obdtype>.<proc_file_name>=<value>)
-</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1304695" xreflabel=""/>Here are a few examples of lctl conf_param commands:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1302350" xreflabel=""/>$ mgs> lctl conf_param testfs-MDT0000.sys.timeout=40
-<anchor xml:id="dbdoclet.50438194_pgfId-1302351" xreflabel=""/>$ lctl conf_param testfs-MDT0000.mdt.group_upcall=NONE
-<anchor xml:id="dbdoclet.50438194_pgfId-1302352" xreflabel=""/>$ lctl conf_param testfs.llite.max_read_ahead_mb=16
-<anchor xml:id="dbdoclet.50438194_pgfId-1302353" xreflabel=""/>$ lctl conf_param testfs-MDT0000.lov.stripesize=2M
-<anchor xml:id="dbdoclet.50438194_pgfId-1302354" xreflabel=""/>$ lctl conf_param testfs-OST0000.osc.max_dirty_mb=29.15
-<anchor xml:id="dbdoclet.50438194_pgfId-1302355" xreflabel=""/>$ lctl conf_param testfs-OST0000.ost.client_cache_seconds=15
-<anchor xml:id="dbdoclet.50438194_pgfId-1302356" xreflabel=""/>$ lctl conf_param testfs.sys.timeout=40
+ <title>13.8.3.2 <anchor xml:id="dbdoclet.50438194_64195" xreflabel=""/>Setting Permanent Parameters</title>
+ <para>Use the lctl conf_param command to set permanent parameters. In general, the lctl conf_param command can be used to specify any parameter settable in a /proc/fs/lustre file, with its own OBD device. The lctl conf_param command uses this syntax (same as the mkfs.lustre and tunefs.lustre commands):</para>
+ <screen><obd|fsname>.<obdtype>.<proc_file_name>=<value>)
+</screen>
+ <para>Here are a few examples of lctl conf_param commands:</para>
+ <screen>$ mgs> lctl conf_param testfs-MDT0000.sys.timeout=40
+$ lctl conf_param testfs-MDT0000.mdt.group_upcall=NONE
+$ lctl conf_param testfs.llite.max_read_ahead_mb=16
+$ lctl conf_param testfs-MDT0000.lov.stripesize=2M
+$ lctl conf_param testfs-OST0000.osc.max_dirty_mb=29.15
+$ lctl conf_param testfs-OST0000.ost.client_cache_seconds=15
+$ lctl conf_param testfs.sys.timeout=40
</screen>
<caution><para>Parameters specified with the lctlconf_param command are set permanently in the file system's configuration file on the MGS.</para></caution>
</section>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438194_pgfId-1305661" xreflabel=""/>13.8.3.3 <anchor xml:id="dbdoclet.50438194_88217" xreflabel=""/>Listing Parameters</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1305964" xreflabel=""/>To list Lustre or LNET parameters that are available to set, use the lctl list_param command. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1305965" xreflabel=""/>lctl list_param [-FR] <obdtype>.<obdname>
+ <title>13.8.3.3 <anchor xml:id="dbdoclet.50438194_88217" xreflabel=""/>Listing Parameters</title>
+ <para>To list Lustre or LNET parameters that are available to set, use the lctl list_param command. For example:</para>
+ <screen>lctl list_param [-FR] <obdtype>.<obdname>
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1305966" xreflabel=""/>The following arguments are available for the lctl list_param command.</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1305967" xreflabel=""/>-F Add '/', '@' or '=' for directories, symlinks and writeable files, respectively</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1305968" xreflabel=""/>-R Recursively lists all parameters under the specified path</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1305969" xreflabel=""/>For example:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1305970" xreflabel=""/>$ lctl list_param obdfilter.lustre-OST0000
+ <para>The following arguments are available for the lctl list_param command.</para>
+ <para>-F Add '/', '@' or '=' for directories, symlinks and writeable files, respectively</para>
+ <para>-R Recursively lists all parameters under the specified path</para>
+ <para>For example:</para>
+ <screen>$ lctl list_param obdfilter.lustre-OST0000
</screen>
</section>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438194_pgfId-1302343" xreflabel=""/>13.8.3.4 <anchor xml:id="dbdoclet.50438194_63247" xreflabel=""/>Reporting Current <anchor xml:id="dbdoclet.50438194_marker-1302474" xreflabel=""/>Parameter Values</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1302292" xreflabel=""/>To report current Lustre parameter values, use the lctl get_param command with this syntax:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1302296" xreflabel=""/>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name>
-</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1302294" xreflabel=""/>This example reports data on RPC service times.</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1302235" xreflabel=""/>$ lctl get_param -n ost.*.ost_io.timeouts
-<anchor xml:id="dbdoclet.50438194_pgfId-1304578" xreflabel=""/>service : cur 1 worst 30 (at 1257150393, 85d23h58m54s ago) 1 1 1 1
-</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1304580" xreflabel=""/>This example reports the number of inodes available on each OST.</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1304662" xreflabel=""/># lctl get_param osc.*.filesfree
-<anchor xml:id="dbdoclet.50438194_pgfId-1304626" xreflabel=""/>osc.myth-OST0000-osc-ffff88006dd20000.filesfree=217623
-<anchor xml:id="dbdoclet.50438194_pgfId-1304649" xreflabel=""/>osc.myth-OST0001-osc-ffff88006dd20000.filesfree=5075042
-<anchor xml:id="dbdoclet.50438194_pgfId-1304652" xreflabel=""/>osc.myth-OST0002-osc-ffff88006dd20000.filesfree=3762034
-<anchor xml:id="dbdoclet.50438194_pgfId-1304655" xreflabel=""/>osc.myth-OST0003-osc-ffff88006dd20000.filesfree=91052
-
<anchor xml:id="dbdoclet.50438194_pgfId-1304658" xreflabel=""/>osc.myth-OST0004-osc-ffff88006dd20000.filesfree=129651<anchor xml:id="dbdoclet.50438194_88030" xreflabel=""/><anchor xml:id="dbdoclet.50438194_54623" xreflabel=""/></screen>
+ <title>13.8.3.4 <anchor xml:id="dbdoclet.50438194_63247" xreflabel=""/>Reporting Current <anchor xml:id="dbdoclet.50438194_marker-1302474" xreflabel=""/>Parameter Values</title>
+ <para>To report current Lustre parameter values, use the lctl get_param command with this syntax:</para>
+ <screen>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name>
+</screen>
+ <para>This example reports data on RPC service times.</para>
+ <screen>$ lctl get_param -n ost.*.ost_io.timeouts
+service : cur 1 worst 30 (at 1257150393, 85d23h58m54s ago) 1 1 1 1
+</screen>
+ <para>This example reports the number of inodes available on each OST.</para>
+ <screen># lctl get_param osc.*.filesfree
+osc.myth-OST0000-osc-ffff88006dd20000.filesfree=217623
+osc.myth-OST0001-osc-ffff88006dd20000.filesfree=5075042
+osc.myth-OST0002-osc-ffff88006dd20000.filesfree=3762034
+osc.myth-OST0003-osc-ffff88006dd20000.filesfree=91052
+osc.myth-OST0004-osc-ffff88006dd20000.filesfree=129651<anchor xml:id="dbdoclet.50438194_88030" xreflabel=""/><anchor xml:id="dbdoclet.50438194_54623" xreflabel=""/></screen>
</section>
</section>
</section>
<section xml:id="dbdoclet.50438194_41817">
<title>13.9 <anchor xml:id="dbdoclet.50438194_42379" xreflabel=""/><anchor xml:id="dbdoclet.50438194_50129" xreflabel=""/>Specifying NIDs and Fail<anchor xml:id="dbdoclet.50438194_marker-1306313" xreflabel=""/>over</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1306315" xreflabel=""/>If a node has multiple network interfaces, it may have multiple NIDs. When a node is specified, all of its NIDs must be listed, delimited by commas (,) so other nodes can choose the NID that is appropriate for their network interfaces. When failover nodes are specified, they are delimited by a colon (:) or by repeating a keyword (--mgsnode= or --failnode=). To obtain all NIDs from a node (while LNET is running), run:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1306316" xreflabel=""/>lctl list_nids
-</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1306317" xreflabel=""/>This displays the server's NIDs (networks configured to work with Lustre).</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1306319" xreflabel=""/>This example has a combined MGS/MDT failover pair on uml1 and uml2, and a OST failover pair on uml3 and uml4. There are corresponding Elan addresses on uml1 and uml2.</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1306320" xreflabel=""/>uml1> mkfs.lustre --fsname=testfs --mdt --mgs --failnode=uml2,2@elan /dev/sda1
-<anchor xml:id="dbdoclet.50438194_pgfId-1306322" xreflabel=""/>uml1> mount -t lustre /dev/sda1 /mnt/test/mdt
-<anchor xml:id="dbdoclet.50438194_pgfId-1306323" xreflabel=""/>uml3> mkfs.lustre --fsname=testfs --ost --failnode=uml4 --mgsnode=uml1,1@ela\
+ <para>If a node has multiple network interfaces, it may have multiple NIDs. When a node is specified, all of its NIDs must be listed, delimited by commas (,) so other nodes can choose the NID that is appropriate for their network interfaces. When failover nodes are specified, they are delimited by a colon (:) or by repeating a keyword (--mgsnode= or --failnode=). To obtain all NIDs from a node (while LNET is running), run:</para>
+ <screen>lctl list_nids
+</screen>
+ <para>This displays the server's NIDs (networks configured to work with Lustre).</para>
+ <para>This example has a combined MGS/MDT failover pair on uml1 and uml2, and a OST failover pair on uml3 and uml4. There are corresponding Elan addresses on uml1 and uml2.</para>
+ <screen>uml1> mkfs.lustre --fsname=testfs --mdt --mgs --failnode=uml2,2@elan /dev/sda1
+uml1> mount -t lustre /dev/sda1 /mnt/test/mdt
+uml3> mkfs.lustre --fsname=testfs --ost --failnode=uml4 --mgsnode=uml1,1@ela\
n --mgsnode=uml2,2@elan /dev/sdb
-<anchor xml:id="dbdoclet.50438194_pgfId-1306325" xreflabel=""/>uml3> mount -t lustre /dev/sdb /mnt/test/ost0
-<anchor xml:id="dbdoclet.50438194_pgfId-1306326" xreflabel=""/>client> mount -t lustre uml1,1@elan:uml2,2@elan:/testfs /mnt/testfs
-<anchor xml:id="dbdoclet.50438194_pgfId-1306327" xreflabel=""/>uml1> umount /mnt/mdt
-<anchor xml:id="dbdoclet.50438194_pgfId-1306328" xreflabel=""/>uml2> mount -t lustre /dev/sda1 /mnt/test/mdt
-<anchor xml:id="dbdoclet.50438194_pgfId-1306329" xreflabel=""/>uml2> cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status
+uml3> mount -t lustre /dev/sdb /mnt/test/ost0
+client> mount -t lustre uml1,1@elan:uml2,2@elan:/testfs /mnt/testfs
+uml1> umount /mnt/mdt
+uml2> mount -t lustre /dev/sda1 /mnt/test/mdt
+uml2> cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1306330" xreflabel=""/>Where multiple NIDs are specified, comma-separation (for example, uml2,2@elan) means that the two NIDs refer to the same host, and that Lustre needs to choose the "best" one for communication. Colon-separation (for example, uml1:uml2) means that the two NIDs refer to two different hosts, and should be treated as failover locations (Lustre tries the first one, and if that fails, it tries the second one.)</para>
+ <para>Where multiple NIDs are specified, comma-separation (for example, uml2,2@elan) means that the two NIDs refer to the same host, and that Lustre needs to choose the "best" one for communication. Colon-separation (for example, uml1:uml2) means that the two NIDs refer to two different hosts, and should be treated as failover locations (Lustre tries the first one, and if that fails, it tries the second one.)</para>
<note><para>If you have an MGS or MDT configured for failover, perform these steps:</para><para> 1. On the OST, list the NIDs of all MGS nodes at mkfs time.</para><para>OST# mkfs.lustre --fsname sunfs --ost --mgsnode=10.0.0.1</para><para> --mgsnode=10.0.0.2 /dev/{device}</para><para> 2. On the client, mount the file system.</para><para>client# mount -t lustre 10.0.0.1:10.0.0.2:/sunfs /cfs/client/</para></note>
</section>
<section xml:id="dbdoclet.50438194_70905">
<title>13.10 Erasing a <anchor xml:id="dbdoclet.50438194_marker-1307237" xreflabel=""/>File System</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307240" xreflabel=""/>If you want to erase a file system, run this command on your targets:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307241" xreflabel=""/>$ "mkfs.lustre -reformat"
+ <para>If you want to erase a file system, run this command on your targets:</para>
+ <screen>$ "mkfs.lustre -reformat"
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307242" xreflabel=""/>If you are using a separate MGS and want to keep other file systems defined on that MGS, then set the writeconf flag on the MDT for that file system. The writeconf flag causes the configuration logs to be erased; they are regenerated the next time the servers start.</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307243" xreflabel=""/>To set the writeconf flag on the MDT:</para>
+ <para>If you are using a separate MGS and want to keep other file systems defined on that MGS, then set the writeconf flag on the MDT for that file system. The writeconf flag causes the configuration logs to be erased; they are regenerated the next time the servers start.</para>
+ <para>To set the writeconf flag on the MDT:</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307244" xreflabel=""/>Unmount all clients/servers using this file system, run:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307245" xreflabel=""/>$ umount /mnt/lustre
+ <para>Unmount all clients/servers using this file system, run:</para>
+ <screen>$ umount /mnt/lustre
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307246" xreflabel=""/>Erase the file system and, presumably, replace it with another file system, run:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307247" xreflabel=""/>$ mkfs.lustre -reformat --fsname spfs --mdt --mgs /dev/sda
+ <para>Erase the file system and, presumably, replace it with another file system, run:</para>
+ <screen>$ mkfs.lustre -reformat --fsname spfs --mdt --mgs /dev/sda
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307248" xreflabel=""/>If you have a separate MGS (that you do not want to reformat), then add the "writeconf" flag to mkfs.lustre on the MDT, run:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307249" xreflabel=""/>$ mkfs.lustre --reformat --writeconf -fsname spfs --mdt \ --mgs /dev/sda
+ <para>If you have a separate MGS (that you do not want to reformat), then add the "writeconf" flag to mkfs.lustre on the MDT, run:</para>
+ <screen>$ mkfs.lustre --reformat --writeconf -fsname spfs --mdt \ --mgs /dev/sda
</screen>
</listitem></orderedlist>
<note><para>If you have a combined MGS/MDT, reformatting the MDT reformats the MGS as well, causing all configuration information to be lost; you can start building your new file system. Nothing needs to be done with old disks that will not be part of the new file system, just do not mount them.</para></note>
</section>
<section xml:id="dbdoclet.50438194_16954">
<title>13.11 Reclaiming <anchor xml:id="dbdoclet.50438194_marker-1307251" xreflabel=""/>Reserved Disk Space</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307254" xreflabel=""/>All current Lustre installations run the ldiskfs file system internally on service nodes. By default, ldiskfs reserves 5% of the disk space for the root user. In order to reclaim this space, run the following command on your OSSs:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307255" xreflabel=""/>tune2fs [-m reserved_blocks_percent] [device]
+ <para>All current Lustre installations run the ldiskfs file system internally on service nodes. By default, ldiskfs reserves 5% of the disk space for the root user. In order to reclaim this space, run the following command on your OSSs:</para>
+ <screen>tune2fs [-m reserved_blocks_percent] [device]
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307256" xreflabel=""/>You do not need to shut down Lustre before running this command or restart it afterwards.</para>
+ <para>You do not need to shut down Lustre before running this command or restart it afterwards.</para>
</section>
<section xml:id="dbdoclet.50438194_69998">
<title>13.12 Replacing an Existing <anchor xml:id="dbdoclet.50438194_marker-1307278" xreflabel=""/>OST or MDS</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307281" xreflabel=""/>To copy the contents of an existing OST to a new OST (or an old MDS to a new MDS), use one of these methods:</para>
+ <para>To copy the contents of an existing OST to a new OST (or an old MDS to a new MDS), use one of these methods:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307282" xreflabel=""/> Connect the old OST disk and new OST disk to a single machine, mount both, and use rsync to copy all data between the OST file systems.</para>
+ <para> Connect the old OST disk and new OST disk to a single machine, mount both, and use rsync to copy all data between the OST file systems.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307283" xreflabel=""/>For example:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307284" xreflabel=""/>mount -t ldiskfs /dev/old /mnt/ost_old
-<anchor xml:id="dbdoclet.50438194_pgfId-1307285" xreflabel=""/>mount -t ldiskfs /dev/new /mnt/ost_new
-<anchor xml:id="dbdoclet.50438194_pgfId-1307286" xreflabel=""/>rsync -aSv /mnt/ost_old/ /mnt/ost_new
-<anchor xml:id="dbdoclet.50438194_pgfId-1307287" xreflabel=""/># note trailing slash on ost_old/
+ <para>For example:</para>
+ <screen>mount -t ldiskfs /dev/old /mnt/ost_old
+mount -t ldiskfs /dev/new /mnt/ost_new
+rsync -aSv /mnt/ost_old/ /mnt/ost_new
+# note trailing slash on ost_old/
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307288" xreflabel=""/> If you are unable to connect both sets of disk to the same computer, use rsync to copy over the network using rsh (or ssh with -e ssh):</para>
+ <para> If you are unable to connect both sets of disk to the same computer, use rsync to copy over the network using rsh (or ssh with -e ssh):</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307289" xreflabel=""/>rsync -aSvz /mnt/ost_old/ new_ost_node:/mnt/ost_new
+ <screen>rsync -aSvz /mnt/ost_old/ new_ost_node:/mnt/ost_new
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307290" xreflabel=""/> Use the same procedure for the MDS, with one additional step:</para>
+ <para> Use the same procedure for the MDS, with one additional step:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307291" xreflabel=""/>cd /mnt/mds_old; getfattr -R -e base64 -d . > /tmp/mdsea; \<copy all MDS file\
+ <screen>cd /mnt/mds_old; getfattr -R -e base64 -d . > /tmp/mdsea; \<copy all MDS file\
s as above>; cd /mnt/mds_new; setfattr \--restore=/tmp/mdsea
</screen>
</section>
<section xml:id="dbdoclet.50438194_30872">
<title>13.13 Identifying To Which Lustre File an OST Object Belongs</title>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307353" xreflabel=""/>Use this procedure to identify the file containing a given object on a given OST.</para>
+ <para>Use this procedure to identify the file containing a given object on a given OST.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307357" xreflabel=""/>On the OST (as root), run debugfs to display the file identifier (FID) of the file associated with the object.</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307358" xreflabel=""/>For example, if the object is 34976 on /dev/lustre/ost_test2, the debug command is:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307359" xreflabel=""/># debugfs -c -R "stat /O/0/d$((34976 %32))/34976" /dev/lustre/ost_test2
+ <para>On the OST (as root), run debugfs to display the file identifier (FID) of the file associated with the object.</para>
+ <para>For example, if the object is 34976 on /dev/lustre/ost_test2, the debug command is:</para>
+ <screen># debugfs -c -R "stat /O/0/d$((34976 %32))/34976" /dev/lustre/ost_test2
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307360" xreflabel=""/>The command output is:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307361" xreflabel=""/>debugfs 1.41.5.sun2 (23-Apr-2009)
-<anchor xml:id="dbdoclet.50438194_pgfId-1307362" xreflabel=""/>/dev/lustre/ost_test2: catastrophic mode - not reading inode or group bitma\
+ <para>The command output is:</para>
+ <screen>debugfs 1.41.5.sun2 (23-Apr-2009)
+/dev/lustre/ost_test2: catastrophic mode - not reading inode or group bitma\
ps
-<anchor xml:id="dbdoclet.50438194_pgfId-1307363" xreflabel=""/>Inode: 352365 Type: regular Mode: 0666 Flags: 0x80000
-<anchor xml:id="dbdoclet.50438194_pgfId-1307364" xreflabel=""/>Generation: 1574463214 Version: 0xea020000:00000000
-<anchor xml:id="dbdoclet.50438194_pgfId-1307365" xreflabel=""/>User: 500 Group: 500 Size: 260096
-<anchor xml:id="dbdoclet.50438194_pgfId-1307366" xreflabel=""/>File ACL: 0 Directory ACL: 0
-<anchor xml:id="dbdoclet.50438194_pgfId-1307367" xreflabel=""/>Links: 1 Blockcount: 512
-<anchor xml:id="dbdoclet.50438194_pgfId-1307368" xreflabel=""/>Fragment: Address: 0 Number: 0 Size: 0
-<anchor xml:id="dbdoclet.50438194_pgfId-1307369" xreflabel=""/>ctime: 0x4a216b48:00000000 -- Sat May 30 13:22:16 2009
-<anchor xml:id="dbdoclet.50438194_pgfId-1307370" xreflabel=""/>atime: 0x4a216b48:00000000 -- Sat May 30 13:22:16 2009
-<anchor xml:id="dbdoclet.50438194_pgfId-1307371" xreflabel=""/>mtime: 0x4a216b48:00000000 -- Sat May 30 13:22:16 2009
-<anchor xml:id="dbdoclet.50438194_pgfId-1307372" xreflabel=""/>crtime: 0x4a216b3c:975870dc -- Sat May 30 13:22:04 2009
-<anchor xml:id="dbdoclet.50438194_pgfId-1307373" xreflabel=""/>Size of extra inode fields: 24
-<anchor xml:id="dbdoclet.50438194_pgfId-1307374" xreflabel=""/>Extended attributes stored in inode body:
-<anchor xml:id="dbdoclet.50438194_pgfId-1307375" xreflabel=""/>fid = "e2 00 11 00 00 00 00 00 25 43 c1 87 00 00 00 00 a0 88 00 00 00 00 00 \
+Inode: 352365 Type: regular Mode: 0666 Flags: 0x80000
+Generation: 1574463214 Version: 0xea020000:00000000
+User: 500 Group: 500 Size: 260096
+File ACL: 0 Directory ACL: 0
+Links: 1 Blockcount: 512
+Fragment: Address: 0 Number: 0 Size: 0
+ctime: 0x4a216b48:00000000 -- Sat May 30 13:22:16 2009
+atime: 0x4a216b48:00000000 -- Sat May 30 13:22:16 2009
+mtime: 0x4a216b48:00000000 -- Sat May 30 13:22:16 2009
+crtime: 0x4a216b3c:975870dc -- Sat May 30 13:22:04 2009
+Size of extra inode fields: 24
+Extended attributes stored in inode body:
+fid = "e2 00 11 00 00 00 00 00 25 43 c1 87 00 00 00 00 a0 88 00 00 00 00 00 \
00 00 00 00 00 00 00 00 00 " (32)
-<anchor xml:id="dbdoclet.50438194_pgfId-1307376" xreflabel=""/>BLOCKS:
-<anchor xml:id="dbdoclet.50438194_pgfId-1307377" xreflabel=""/>(0-63):47968-48031
-<anchor xml:id="dbdoclet.50438194_pgfId-1307378" xreflabel=""/>TOTAL: 64
+BLOCKS:
+(0-63):47968-48031
+TOTAL: 64
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307379" xreflabel=""/>Note the FID's EA and apply it to the osd_inode_id mapping.</para>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307380" xreflabel=""/>In this example, the FID's EA is:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307381" xreflabel=""/>e2001100000000002543c18700000000a0880000000000000000000000000000
-<anchor xml:id="dbdoclet.50438194_pgfId-1307382" xreflabel=""/>struct osd_inode_id {
-<anchor xml:id="dbdoclet.50438194_pgfId-1307383" xreflabel=""/>__u64 oii_ino; /* inode number */
-<anchor xml:id="dbdoclet.50438194_pgfId-1307384" xreflabel=""/>__u32 oii_gen; /* inode generation */
-<anchor xml:id="dbdoclet.50438194_pgfId-1307385" xreflabel=""/>__u32 oii_pad; /* alignment padding */
-<anchor xml:id="dbdoclet.50438194_pgfId-1307386" xreflabel=""/>};
-</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307387" xreflabel=""/>After swapping, you get an inode number of 0x001100e2 and generation of 0.</para>
+ <para>Note the FID's EA and apply it to the osd_inode_id mapping.</para>
+ <para>In this example, the FID's EA is:</para>
+ <screen>e2001100000000002543c18700000000a0880000000000000000000000000000
+struct osd_inode_id {
+__u64 oii_ino; /* inode number */
+__u32 oii_gen; /* inode generation */
+__u32 oii_pad; /* alignment padding */
+};
+</screen>
+ <para>After swapping, you get an inode number of 0x001100e2 and generation of 0.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307388" xreflabel=""/>On the MDT (as root), use debugfs to find the file associated with the inode.</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307389" xreflabel=""/># debugfs -c -R "ncheck 0x001100e2" /dev/lustre/mdt_test
+ <para>On the MDT (as root), use debugfs to find the file associated with the inode.</para>
+ <screen># debugfs -c -R "ncheck 0x001100e2" /dev/lustre/mdt_test
</screen>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307390" xreflabel=""/>Here is the command output:</para>
- <screen><anchor xml:id="dbdoclet.50438194_pgfId-1307391" xreflabel=""/>debugfs 1.41.5.sun2 (23-Apr-2009)
-<anchor xml:id="dbdoclet.50438194_pgfId-1307392" xreflabel=""/>/dev/lustre/mdt_test: catastrophic mode - not reading inode or group bitmap\
+ <para>Here is the command output:</para>
+ <screen>debugfs 1.41.5.sun2 (23-Apr-2009)
+/dev/lustre/mdt_test: catastrophic mode - not reading inode or group bitmap\
s
-<anchor xml:id="dbdoclet.50438194_pgfId-1307393" xreflabel=""/>Inode Pathname
-<anchor xml:id="dbdoclet.50438194_pgfId-1307394" xreflabel=""/>1114338 /ROOT/brian-laptop-guest/clients/client11/~dmtmp/PWRPNT/ZD16.BMP
+Inode Pathname
+1114338 /ROOT/brian-laptop-guest/clients/client11/~dmtmp/PWRPNT/ZD16.BMP
</screen>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438194_pgfId-1307395" xreflabel=""/>The command lists the inode and pathname associated with the object.</para>
+ <para>The command lists the inode and pathname associated with the object.</para>
<note><para>Debugfs' ''ncheck'' is a brute-force search that may take a long time to complete.</para></note>
<note><para>To find the Lustre file from a disk LBA, follow the steps listed in the document at this URL: <emphasis>http://smartmontools.sourceforge.net/badblockhowto.html. </emphasis> Then, follow the steps above to resolve the Lustre filename.</para></note>
<info>
<title xml:id='lustreproc.title'>LustreProc</title>
</info>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1301083" xreflabel=""/>The /proc file system acts as an interface to internal data structures in the kernel. The /proc variables can be used to control aspects of Lustre performance and provide information.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290340" xreflabel=""/>This chapter describes Lustre /proc entries and includes the following sections:</para>
+ <para>The /proc file system acts as an interface to internal data structures in the kernel. The /proc variables can be used to control aspects of Lustre performance and provide information.</para>
+ <para>This chapter describes Lustre /proc entries and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438271_90999"/></para>
</listitem>
<section xml:id="dbdoclet.50438271_90999">
<title>31.1 Proc Entries for Lustre</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290360" xreflabel=""/>This section describes /proc entries for Lustre.</para>
+ <para>This section describes /proc entries for Lustre.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1290361" xreflabel=""/>31.1.1 Locating Lustre <anchor xml:id="dbdoclet.50438271_marker-1296151" xreflabel=""/>File Systems and Servers</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290362" xreflabel=""/>Use the proc files on the MGS to locate the following:</para>
+ <title>31.1.1 Locating Lustre <anchor xml:id="dbdoclet.50438271_marker-1296151" xreflabel=""/>File Systems and Servers</title>
+ <para>Use the proc files on the MGS to locate the following:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290363" xreflabel=""/> All known file systems</para>
+ <para> All known file systems</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290364" xreflabel=""/># cat /proc/fs/lustre/mgs/MGS/filesystems
-<anchor xml:id="dbdoclet.50438271_pgfId-1291584" xreflabel=""/>spfs
-<anchor xml:id="dbdoclet.50438271_pgfId-1291587" xreflabel=""/>lustre
+ <screen># cat /proc/fs/lustre/mgs/MGS/filesystems
+spfs
+lustre
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290367" xreflabel=""/> The server names participating in a file system (for each file system that has at least one server running)</para>
+ <para> The server names participating in a file system (for each file system that has at least one server running)</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290368" xreflabel=""/># cat /proc/fs/lustre/mgs/MGS/live/spfs
-<anchor xml:id="dbdoclet.50438271_pgfId-1291593" xreflabel=""/>fsname: spfs
-<anchor xml:id="dbdoclet.50438271_pgfId-1291596" xreflabel=""/>flags: 0x0 gen: 7
-<anchor xml:id="dbdoclet.50438271_pgfId-1291599" xreflabel=""/>spfs-MDT0000
-<anchor xml:id="dbdoclet.50438271_pgfId-1291602" xreflabel=""/>spfs-OST0000
+ <screen># cat /proc/fs/lustre/mgs/MGS/live/spfs
+fsname: spfs
+flags: 0x0 gen: 7
+spfs-MDT0000
+spfs-OST0000
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290373" xreflabel=""/>All servers are named according to this convention: <fsname>-<MDT|OST><XXXX> This can be shown for live servers under /proc/fs/lustre/devices:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290374" xreflabel=""/># cat /proc/fs/lustre/devices
-<anchor xml:id="dbdoclet.50438271_pgfId-1290375" xreflabel=""/>0 UP mgs MGS MGS 11
-<anchor xml:id="dbdoclet.50438271_pgfId-1290376" xreflabel=""/>1 UP mgc MGC192.168.10.34@tcp 1f45bb57-d9be-2ddb-c0b0-5431a49226705
-<anchor xml:id="dbdoclet.50438271_pgfId-1290377" xreflabel=""/>2 UP mdt MDS MDS_uuid 3
-<anchor xml:id="dbdoclet.50438271_pgfId-1290378" xreflabel=""/>3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
-<anchor xml:id="dbdoclet.50438271_pgfId-1290379" xreflabel=""/>4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 7
-<anchor xml:id="dbdoclet.50438271_pgfId-1290380" xreflabel=""/>5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438271_pgfId-1290381" xreflabel=""/>6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438271_pgfId-1290382" xreflabel=""/>7 UP lov lustre-clilov-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa04
-<anchor xml:id="dbdoclet.50438271_pgfId-1290383" xreflabel=""/>8 UP mdc lustre-MDT0000-mdc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05
-<anchor xml:id="dbdoclet.50438271_pgfId-1290384" xreflabel=""/>9 UP osc lustre-OST0000-osc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05
-<anchor xml:id="dbdoclet.50438271_pgfId-1290385" xreflabel=""/>10 UP osc lustre-OST0001-osc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05
+ <para>All servers are named according to this convention: <fsname>-<MDT|OST><XXXX> This can be shown for live servers under /proc/fs/lustre/devices:</para>
+ <screen># cat /proc/fs/lustre/devices
+0 UP mgs MGS MGS 11
+1 UP mgc MGC192.168.10.34@tcp 1f45bb57-d9be-2ddb-c0b0-5431a49226705
+2 UP mdt MDS MDS_uuid 3
+3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
+4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 7
+5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
+6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
+7 UP lov lustre-clilov-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa04
+8 UP mdc lustre-MDT0000-mdc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05
+9 UP osc lustre-OST0000-osc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05
+10 UP osc lustre-OST0001-osc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290386" xreflabel=""/>Or from the device label at any time:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290387" xreflabel=""/># e2label /dev/sda
-<anchor xml:id="dbdoclet.50438271_pgfId-1290388" xreflabel=""/>lustre-MDT0000
+ <para>Or from the device label at any time:</para>
+ <screen># e2label /dev/sda
+lustre-MDT0000
</screen>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1290389" xreflabel=""/>31.1.2 Lustre <anchor xml:id="dbdoclet.50438271_marker-1296153" xreflabel=""/>Timeouts</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294163" xreflabel=""/>Lustre uses two types of timeouts.</para>
+ <title>31.1.2 Lustre <anchor xml:id="dbdoclet.50438271_marker-1296153" xreflabel=""/>Timeouts</title>
+ <para>Lustre uses two types of timeouts.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294195" xreflabel=""/>LND timeouts that ensure point-to-point communications complete in finite time in the presence of failures. These timeouts are logged with the S_LND flag set. They may <emphasis>not</emphasis> be printed as console messages, so you should check the Lustre log for D_NETERROR messages, or enable printing of D_NETERROR messages to the console (echo + neterror > /proc/sys/lnet/printk).</para>
+ <para>LND timeouts that ensure point-to-point communications complete in finite time in the presence of failures. These timeouts are logged with the S_LND flag set. They may <emphasis>not</emphasis> be printed as console messages, so you should check the Lustre log for D_NETERROR messages, or enable printing of D_NETERROR messages to the console (echo + neterror > /proc/sys/lnet/printk).</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294211" xreflabel=""/>Congested routers can be a source of spurious LND timeouts. To avoid this, increase the number of LNET router buffers to reduce back-pressure and/or increase LND timeouts on all nodes on all connected networks. You should also consider increasing the total number of LNET router nodes in the system so that the aggregate router bandwidth matches the aggregate server bandwidth.</para>
+ <para>Congested routers can be a source of spurious LND timeouts. To avoid this, increase the number of LNET router buffers to reduce back-pressure and/or increase LND timeouts on all nodes on all connected networks. You should also consider increasing the total number of LNET router nodes in the system so that the aggregate router bandwidth matches the aggregate server bandwidth.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294177" xreflabel=""/>Lustre timeouts that ensure Lustre RPCs complete in finite time in the presence of failures. These timeouts should <emphasis>always</emphasis> be printed as console messages. If Lustre timeouts are not accompanied by LNET timeouts, then you need to increase the lustre timeout on both servers and clients.</para>
+ <para>Lustre timeouts that ensure Lustre RPCs complete in finite time in the presence of failures. These timeouts should <emphasis>always</emphasis> be printed as console messages. If Lustre timeouts are not accompanied by LNET timeouts, then you need to increase the lustre timeout on both servers and clients.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294236" xreflabel=""/>Specific Lustre timeouts are described below.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290390" xreflabel=""/><emphasis role="bold">/proc/sys/lustre/timeout</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290391" xreflabel=""/>This is the time period that a client waits for a server to complete an RPC (default is 100s). Servers wait half of this time for a normal client RPC to complete and a quarter of this time for a single bulk request (read or write of up to 1 MB) to complete. The client pings recoverable targets (MDS and OSTs) at one quarter of the timeout, and the server waits one and a half times the timeout before evicting a client for being "stale."</para>
+ <para>Specific Lustre timeouts are described below.</para>
+ <para><emphasis role="bold">/proc/sys/lustre/timeout</emphasis></para>
+ <para>This is the time period that a client waits for a server to complete an RPC (default is 100s). Servers wait half of this time for a normal client RPC to complete and a quarter of this time for a single bulk request (read or write of up to 1 MB) to complete. The client pings recoverable targets (MDS and OSTs) at one quarter of the timeout, and the server waits one and a half times the timeout before evicting a client for being "stale."</para>
<note><para>Lustre sends periodic 'PING' messages to servers with which it had no communication for a specified period of time. Any network activity on the file system that triggers network traffic toward servers also works as a health check.</para></note>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1292930" xreflabel=""/><emphasis role="bold">/proc/sys/lustre/ldlm_timeout</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290393" xreflabel=""/>This is the time period for which a server will wait for a client to reply to an initial AST (lock cancellation request) where default is 20s for an OST and 6s for an MDS. If the client replies to the AST, the server will give it a normal timeout (half of the client timeout) to flush any dirty data and release the lock.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290394" xreflabel=""/><emphasis role="bold">/proc/sys/lustre/fail_loc</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290395" xreflabel=""/>This is the internal debugging failure hook.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290396" xreflabel=""/>See lustre/include/linux/obd_support.h for the definitions of individual failure locations. The default value is 0 (zero).</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290397" xreflabel=""/>sysctl -w lustre.fail_loc=0x80000122 # drop a single reply
+ <para><emphasis role="bold">/proc/sys/lustre/ldlm_timeout</emphasis></para>
+ <para>This is the time period for which a server will wait for a client to reply to an initial AST (lock cancellation request) where default is 20s for an OST and 6s for an MDS. If the client replies to the AST, the server will give it a normal timeout (half of the client timeout) to flush any dirty data and release the lock.</para>
+ <para><emphasis role="bold">/proc/sys/lustre/fail_loc</emphasis></para>
+ <para>This is the internal debugging failure hook.</para>
+ <para>See lustre/include/linux/obd_support.h for the definitions of individual failure locations. The default value is 0 (zero).</para>
+ <screen>sysctl -w lustre.fail_loc=0x80000122 # drop a single reply
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290398" xreflabel=""/><emphasis role="bold">/proc/sys/lustre/dump_on_timeout</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290399" xreflabel=""/>This triggers dumps of the Lustre debug log when timeouts occur. The default value is 0 (zero).</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294958" xreflabel=""/><emphasis role="bold">/proc/sys/lustre/dump_on_eviction</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294959" xreflabel=""/>This triggers dumps of the Lustre debug log when an eviction occurs. The default value is 0 (zero). By default, debug logs are dumped to the /tmp folder; this location can be changed via /proc.</para>
+ <para><emphasis role="bold">/proc/sys/lustre/dump_on_timeout</emphasis></para>
+ <para>This triggers dumps of the Lustre debug log when timeouts occur. The default value is 0 (zero).</para>
+ <para><emphasis role="bold">/proc/sys/lustre/dump_on_eviction</emphasis></para>
+ <para>This triggers dumps of the Lustre debug log when an eviction occurs. The default value is 0 (zero). By default, debug logs are dumped to the /tmp folder; this location can be changed via /proc.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1292935" xreflabel=""/>31.1.3 Adaptive <anchor xml:id="dbdoclet.50438271_marker-1293380" xreflabel=""/>Timeouts</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1292944" xreflabel=""/>Lustre offers an adaptive mechanism to set RPC timeouts. The adaptive timeouts feature (enabled, by default) causes servers to track actual RPC completion times, and to report estimated completion times for future RPCs back to clients. The clients use these estimates to set their future RPC timeout values. If server request processing slows down for any reason, the RPC completion estimates increase, and the clients allow more time for RPC completion.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1293013" xreflabel=""/>If RPCs queued on the server approach their timeouts, then the server sends an early reply to the client, telling the client to allow more time. In this manner, clients avoid RPC timeouts and disconnect/reconnect cycles. Conversely, as a server speeds up, RPC timeout values decrease, allowing faster detection of non-responsive servers and faster attempts to reconnect to a server's failover partner.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1292945" xreflabel=""/>In previous Lustre versions, the static obd_timeout (/proc/sys/lustre/timeout) value was used as the maximum completion time for all RPCs; this value also affected the client-server ping interval and initial recovery timer. Now, with adaptive timeouts, obd_timeout is only used for the ping interval and initial recovery estimate. When a client reconnects during recovery, the server uses the client's timeout value to reset the recovery wait period; i.e., the server learns how long the client had been willing to wait, and takes this into account when adjusting the recovery period.</para>
+ <title>31.1.3 Adaptive <anchor xml:id="dbdoclet.50438271_marker-1293380" xreflabel=""/>Timeouts</title>
+ <para>Lustre offers an adaptive mechanism to set RPC timeouts. The adaptive timeouts feature (enabled, by default) causes servers to track actual RPC completion times, and to report estimated completion times for future RPCs back to clients. The clients use these estimates to set their future RPC timeout values. If server request processing slows down for any reason, the RPC completion estimates increase, and the clients allow more time for RPC completion.</para>
+ <para>If RPCs queued on the server approach their timeouts, then the server sends an early reply to the client, telling the client to allow more time. In this manner, clients avoid RPC timeouts and disconnect/reconnect cycles. Conversely, as a server speeds up, RPC timeout values decrease, allowing faster detection of non-responsive servers and faster attempts to reconnect to a server's failover partner.</para>
+ <para>In previous Lustre versions, the static obd_timeout (/proc/sys/lustre/timeout) value was used as the maximum completion time for all RPCs; this value also affected the client-server ping interval and initial recovery timer. Now, with adaptive timeouts, obd_timeout is only used for the ping interval and initial recovery estimate. When a client reconnects during recovery, the server uses the client's timeout value to reset the recovery wait period; i.e., the server learns how long the client had been willing to wait, and takes this into account when adjusting the recovery period.</para>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1292947" xreflabel=""/>31.1.3.1 Configuring <anchor xml:id="dbdoclet.50438271_marker-1293381" xreflabel=""/>Adaptive Timeouts</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1292948" xreflabel=""/>One of the goals of adaptive timeouts is to relieve users from having to tune the obd_timeout value. In general, obd_timeout should no longer need to be changed. However, there are several parameters related to adaptive timeouts that users can set. In most situations, the default values should be used.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299111" xreflabel=""/>The following parameters can be set persistently system-wide using lctl conf_param on the MGS. For example, lctl conf_param work1.sys.at_max=1500 sets the at_max value for all servers and clients using the work1 file system.</para>
+ <title>31.1.3.1 Configuring <anchor xml:id="dbdoclet.50438271_marker-1293381" xreflabel=""/>Adaptive Timeouts</title>
+ <para>One of the goals of adaptive timeouts is to relieve users from having to tune the obd_timeout value. In general, obd_timeout should no longer need to be changed. However, there are several parameters related to adaptive timeouts that users can set. In most situations, the default values should be used.</para>
+ <para>The following parameters can be set persistently system-wide using lctl conf_param on the MGS. For example, lctl conf_param work1.sys.at_max=1500 sets the at_max value for all servers and clients using the work1 file system.</para>
<note><para>Nodes using multiple Lustre file systems must use the same at_* values for all file systems.)</para></note>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1294373" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1294375" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294377" xreflabel=""/><emphasis role="bold">at_min</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294379" xreflabel=""/>Sets the minimum adaptive timeout (in seconds). Default value is 0. The at_min parameter is the minimum processing time that a server will report. Clients base their timeouts on this value, but they do not use this value directly. If you experience cases in which, for unknown reasons, the adaptive timeout value is too short and clients time out their RPCs (usually due to temporary network outages), then you can increase the at_min value to compensate for this. Ideally, users should leave at_min set to its default.</para></entry>
+ <entry><para> <emphasis role="bold">at_min</emphasis></para></entry>
+ <entry><para> Sets the minimum adaptive timeout (in seconds). Default value is 0. The at_min parameter is the minimum processing time that a server will report. Clients base their timeouts on this value, but they do not use this value directly. If you experience cases in which, for unknown reasons, the adaptive timeout value is too short and clients time out their RPCs (usually due to temporary network outages), then you can increase the at_min value to compensate for this. Ideally, users should leave at_min set to its default.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294381" xreflabel=""/><emphasis role="bold">at_max</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1299206" xreflabel=""/>Sets the maximum adaptive timeout (in seconds). The at_max parameter is an upper-limit on the service time estimate, and is used as a 'failsafe' in case of rogue/bad/buggy code that would lead to never-ending estimate increases. If at_max is reached, an RPC request is considered 'broken' and should time out.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1294383" xreflabel=""/>Setting at_max to 0 causes adaptive timeouts to be disabled and the old fixed-timeout method (obd_timeout) to be used. This is the default value in Lustre 1.6.5.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1294387" xreflabel=""/> </para>
+ <entry><para> <emphasis role="bold">at_max</emphasis></para></entry>
+ <entry><para> Sets the maximum adaptive timeout (in seconds). The at_max parameter is an upper-limit on the service time estimate, and is used as a 'failsafe' in case of rogue/bad/buggy code that would lead to never-ending estimate increases. If at_max is reached, an RPC request is considered 'broken' and should time out.</para><para>Setting at_max to 0 causes adaptive timeouts to be disabled and the old fixed-timeout method (obd_timeout) to be used. This is the default value in Lustre 1.6.5.</para><para> </para>
<note><para>It is possible that slow hardware might validly cause the service estimate to increase beyond the default value of at_max. In this case, you should increase at_max to the maximum time you are willing to wait for an RPC completion.</para></note></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294390" xreflabel=""/><emphasis role="bold">at_history</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294392" xreflabel=""/>Sets a time period (in seconds) within which adaptive timeouts remember the slowest event that occurred. Default value is 600.</para></entry>
+ <entry><para> <emphasis role="bold">at_history</emphasis></para></entry>
+ <entry><para> Sets a time period (in seconds) within which adaptive timeouts remember the slowest event that occurred. Default value is 600.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294394" xreflabel=""/><emphasis role="bold">at_early_margin</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294396" xreflabel=""/>Sets how far before the deadline Lustre sends an early reply. Default value is 5<footnote><para><anchor xml:id="dbdoclet.50438271_pgfId-1294399" xreflabel=""/>This default was chosen as a reasonable time in which to send a reply from the point at which it was sent.</para></footnote>.</para></entry>
+ <entry><para> <emphasis role="bold">at_early_margin</emphasis></para></entry>
+ <entry><para> Sets how far before the deadline Lustre sends an early reply. Default value is 5<footnote><para>This default was chosen as a reasonable time in which to send a reply from the point at which it was sent.</para></footnote>.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294401" xreflabel=""/><emphasis role="bold">at_extra</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294403" xreflabel=""/>Sets the incremental amount of time that a server asks for, with each early reply. The server does not know how much time the RPC will take, so it asks for a fixed value. Default value is 30<footnote><para><anchor xml:id="dbdoclet.50438271_pgfId-1294406" xreflabel=""/>This default was chosen as a balance between sending too many early replies for the same RPC and overestimating the actual completion time</para></footnote>. When a server finds a queued request about to time out (and needs to send an early reply out), the server adds the at_extra value. If the time expires, the Lustre client enters recovery status and reconnects to restore it to normal status.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1294407" xreflabel=""/>If you see multiple early replies for the same RPC asking for multiple 30-second increases, change the at_extra value to a larger number to cut down on early replies sent and, therefore, network load.</para></entry>
+ <entry><para> <emphasis role="bold">at_extra</emphasis></para></entry>
+ <entry><para> Sets the incremental amount of time that a server asks for, with each early reply. The server does not know how much time the RPC will take, so it asks for a fixed value. Default value is 30<footnote><para>This default was chosen as a balance between sending too many early replies for the same RPC and overestimating the actual completion time</para></footnote>. When a server finds a queued request about to time out (and needs to send an early reply out), the server adds the at_extra value. If the time expires, the Lustre client enters recovery status and reconnects to restore it to normal status.</para><para>If you see multiple early replies for the same RPC asking for multiple 30-second increases, change the at_extra value to a larger number to cut down on early replies sent and, therefore, network load.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294409" xreflabel=""/><emphasis role="bold">ldlm_enqueue_min</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294411" xreflabel=""/>Sets the minimum lock enqueue time. Default value is 100. The ldlm_enqueue time is the maximum of the measured enqueue estimate (influenced by at_min and at_max parameters), multiplied by a weighting factor, and the ldlm_enqueue_min setting. LDLM lock enqueues were based on the obd_timeout value; now they have a dedicated minimum value. Lock enqueues increase as the measured enqueue times increase (similar to adaptive timeouts).</para></entry>
+ <entry><para> <emphasis role="bold">ldlm_enqueue_min</emphasis></para></entry>
+ <entry><para> Sets the minimum lock enqueue time. Default value is 100. The ldlm_enqueue time is the maximum of the measured enqueue estimate (influenced by at_min and at_max parameters), multiplied by a weighting factor, and the ldlm_enqueue_min setting. LDLM lock enqueues were based on the obd_timeout value; now they have a dedicated minimum value. Lock enqueues increase as the measured enqueue times increase (similar to adaptive timeouts).</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1293587" xreflabel=""/>Adaptive timeouts are enabled, by default. To disable adaptive timeouts, at run time, set at_max to 0. On the MGS, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1294471" xreflabel=""/>$ lctl conf_param <fsname>.sys.at_max=0
+ <para>Adaptive timeouts are enabled, by default. To disable adaptive timeouts, at run time, set at_max to 0. On the MGS, run:</para>
+ <screen>$ lctl conf_param <fsname>.sys.at_max=0
</screen>
<note><para>Changing adaptive timeouts status at runtime may cause transient timeout, reconnect, recovery, etc.</para></note>
</section>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1292959" xreflabel=""/>31.1.3.2 Interpreting <anchor xml:id="dbdoclet.50438271_marker-1293383" xreflabel=""/>Adaptive Timeouts Information</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299183" xreflabel=""/>Adaptive timeouts information can be read from /proc/fs/lustre/*/timeouts files (for each service and client) or with the lctl command.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294311" xreflabel=""/>This is an example from the /proc/fs/lustre/*/timeouts files:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1293196" xreflabel=""/>cfs21:~# cat /proc/fs/lustre/ost/OSS/ost_io/timeouts
+ <title>31.1.3.2 Interpreting <anchor xml:id="dbdoclet.50438271_marker-1293383" xreflabel=""/>Adaptive Timeouts Information</title>
+ <para>Adaptive timeouts information can be read from /proc/fs/lustre/*/timeouts files (for each service and client) or with the lctl command.</para>
+ <para>This is an example from the /proc/fs/lustre/*/timeouts files:</para>
+ <screen>cfs21:~# cat /proc/fs/lustre/ost/OSS/ost_io/timeouts
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299178" xreflabel=""/>This is an example using the lctl command:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1294318" xreflabel=""/>$ lctl get_param -n ost.*.ost_io.timeouts
+ <para>This is an example using the lctl command:</para>
+ <screen>$ lctl get_param -n ost.*.ost_io.timeouts
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294307" xreflabel=""/>This is the sample output:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1294322" xreflabel=""/>service : cur 33 worst 34 (at 1193427052, 0d0h26m40s ago) 1 1 33 2
+ <para>This is the sample output:</para>
+ <screen>service : cur 33 worst 34 (at 1193427052, 0d0h26m40s ago) 1 1 33 2
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1292962" xreflabel=""/>The ost_io service on this node is currently reporting an estimate of 33 seconds. The worst RPC service time was 34 seconds, and it happened 26 minutes ago.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1293207" xreflabel=""/>The output also provides a history of service times. In the example, there are 4 "bins" of adaptive_timeout_history, with the maximum RPC time in each bin reported. In 0-150 seconds, the maximum RPC time was 1, with the same result in 150-300 seconds. From 300-450 seconds, the worst (maximum) RPC time was 33 seconds, and from 450-600s the worst time was 2 seconds. The current estimated service time is the maximum value of the 4 bins (33 seconds in this example).</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294437" xreflabel=""/>Service times (as reported by the servers) are also tracked in the client OBDs:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1292965" xreflabel=""/>cfs21:# lctl get_param osc.*.timeouts
-<anchor xml:id="dbdoclet.50438271_pgfId-1292966" xreflabel=""/>last reply : 1193428639, 0d0h00m00s ago
-<anchor xml:id="dbdoclet.50438271_pgfId-1292967" xreflabel=""/>network : cur 1 worst 2 (at 1193427053, 0d0h26m26s ago) 1 1 1\
+ <para>The ost_io service on this node is currently reporting an estimate of 33 seconds. The worst RPC service time was 34 seconds, and it happened 26 minutes ago.</para>
+ <para>The output also provides a history of service times. In the example, there are 4 "bins" of adaptive_timeout_history, with the maximum RPC time in each bin reported. In 0-150 seconds, the maximum RPC time was 1, with the same result in 150-300 seconds. From 300-450 seconds, the worst (maximum) RPC time was 33 seconds, and from 450-600s the worst time was 2 seconds. The current estimated service time is the maximum value of the 4 bins (33 seconds in this example).</para>
+ <para>Service times (as reported by the servers) are also tracked in the client OBDs:</para>
+ <screen>cfs21:# lctl get_param osc.*.timeouts
+last reply : 1193428639, 0d0h00m00s ago
+network : cur 1 worst 2 (at 1193427053, 0d0h26m26s ago) 1 1 1\
1
-<anchor xml:id="dbdoclet.50438271_pgfId-1292968" xreflabel=""/>portal 6 : cur 33 worst 34 (at 1193427052, 0d0h26m27s ago) 33 33 33\
+portal 6 : cur 33 worst 34 (at 1193427052, 0d0h26m27s ago) 33 33 33\
2
-<anchor xml:id="dbdoclet.50438271_pgfId-1292969" xreflabel=""/>portal 28 : cur 1 worst 1 (at 1193426141, 0d0h41m38s ago) 1 1 1\
+portal 28 : cur 1 worst 1 (at 1193426141, 0d0h41m38s ago) 1 1 1\
1
-<anchor xml:id="dbdoclet.50438271_pgfId-1292970" xreflabel=""/>portal 7 : cur 1 worst 1 (at 1193426141, 0d0h41m38s ago) 1 0 1\
+portal 7 : cur 1 worst 1 (at 1193426141, 0d0h41m38s ago) 1 0 1\
1
-<anchor xml:id="dbdoclet.50438271_pgfId-1292971" xreflabel=""/>portal 17 : cur 1 worst 1 (at 1193426177, 0d0h41m02s ago) 1 0 0\
+portal 17 : cur 1 worst 1 (at 1193426177, 0d0h41m02s ago) 1 0 0\
1
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1292972" xreflabel=""/>In this case, RPCs to portal 6, the OST_IO_PORTAL (see lustre/include/lustre/lustre_idl.h), shows the history of what the ost_io portal has reported as the service estimate.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1292974" xreflabel=""/>Server statistic files also show the range of estimates in the normal min/max/sum/sumsq manner.</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1292975" xreflabel=""/>cfs21:~# lctl get_param mdt.*.mdt.stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1292976" xreflabel=""/>...
-<anchor xml:id="dbdoclet.50438271_pgfId-1292977" xreflabel=""/>req_timeout 6 samples [sec] 1 10 15 105
-<anchor xml:id="dbdoclet.50438271_pgfId-1292978" xreflabel=""/>...
+ <para>In this case, RPCs to portal 6, the OST_IO_PORTAL (see lustre/include/lustre/lustre_idl.h), shows the history of what the ost_io portal has reported as the service estimate.</para>
+ <para>Server statistic files also show the range of estimates in the normal min/max/sum/sumsq manner.</para>
+ <screen>cfs21:~# lctl get_param mdt.*.mdt.stats
+...
+req_timeout 6 samples [sec] 1 10 15 105
+...
</screen>
</section>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1290400" xreflabel=""/>31.1.4 LNET <anchor xml:id="dbdoclet.50438271_marker-1296164" xreflabel=""/>Information</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1291795" xreflabel=""/>This section describes /proc entries for LNET information.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290401" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/peers</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290402" xreflabel=""/>Shows all NIDs known to this node and also gives information on the queue state.</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290403" xreflabel=""/># cat /proc/sys/lnet/peers
-<anchor xml:id="dbdoclet.50438271_pgfId-1290404" xreflabel=""/>nid refs state max \
+ <title>31.1.4 LNET <anchor xml:id="dbdoclet.50438271_marker-1296164" xreflabel=""/>Information</title>
+ <para>This section describes /proc entries for LNET information.</para>
+ <para><emphasis role="bold">/proc/sys/lnet/peers</emphasis></para>
+ <para>Shows all NIDs known to this node and also gives information on the queue state.</para>
+ <screen># cat /proc/sys/lnet/peers
+nid refs state max \
rtr min tx min queue
-<anchor xml:id="dbdoclet.50438271_pgfId-1290405" xreflabel=""/>0@lo 1 ~rtr 0 \
+0@lo 1 ~rtr 0 \
0 0 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290406" xreflabel=""/>192.168.10.35@tcp 1 ~rtr 8 8 \
+192.168.10.35@tcp 1 ~rtr 8 8 \
8 8 6 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290407" xreflabel=""/>192.168.10.36@tcp 1 ~rtr 8 8 \
+192.168.10.36@tcp 1 ~rtr 8 8 \
8 8 6 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290408" xreflabel=""/>192.168.10.37@tcp 1 ~rtr 8 8 \
+192.168.10.37@tcp 1 ~rtr 8 8 \
8 8 6 0
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290449" xreflabel=""/>The fields are explained below:</para>
+ <para>The fields are explained below:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1291804" xreflabel=""/>Field</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1291806" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Field</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291808" xreflabel=""/><emphasis role="bold">refs</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291810" xreflabel=""/>A reference count (principally used for debugging)</para></entry>
+ <entry><para> <emphasis role="bold">refs</emphasis></para></entry>
+ <entry><para> A reference count (principally used for debugging)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291812" xreflabel=""/><emphasis role="bold">state</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291863" xreflabel=""/>Only valid to refer to routers. Possible values:</para><itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1291864" xreflabel=""/> ~ rtr (indicates this node is not a router)</para>
+ <entry><para> <emphasis role="bold">state</emphasis></para></entry>
+ <entry><para> Only valid to refer to routers. Possible values:</para><itemizedlist><listitem>
+ <para> ~ rtr (indicates this node is not a router)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1291865" xreflabel=""/> up/down (indicates this node is a router)</para>
+ <para> up/down (indicates this node is a router)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1291871" xreflabel=""/> auto_fail must be enabled</para>
+ <para> auto_fail must be enabled</para>
</listitem>
</itemizedlist></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291816" xreflabel=""/><emphasis role="bold">max</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291818" xreflabel=""/>Maximum number of concurrent sends from this peer</para></entry>
+ <entry><para> <emphasis role="bold">max</emphasis></para></entry>
+ <entry><para> Maximum number of concurrent sends from this peer</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291820" xreflabel=""/><emphasis role="bold">rtr</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291822" xreflabel=""/>Routing buffer credits.</para></entry>
+ <entry><para> <emphasis role="bold">rtr</emphasis></para></entry>
+ <entry><para> Routing buffer credits.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291824" xreflabel=""/><emphasis role="bold">min</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291826" xreflabel=""/>Minimum routing buffer credits seen.</para></entry>
+ <entry><para> <emphasis role="bold">min</emphasis></para></entry>
+ <entry><para> Minimum routing buffer credits seen.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291828" xreflabel=""/><emphasis role="bold">tx</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291830" xreflabel=""/>Send credits.</para></entry>
+ <entry><para> <emphasis role="bold">tx</emphasis></para></entry>
+ <entry><para> Send credits.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291832" xreflabel=""/><emphasis role="bold">min</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291834" xreflabel=""/>Minimum send credits seen.</para></entry>
+ <entry><para> <emphasis role="bold">min</emphasis></para></entry>
+ <entry><para> Minimum send credits seen.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291836" xreflabel=""/><emphasis role="bold">queue</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291838" xreflabel=""/>Total bytes in active/queued sends.</para></entry>
+ <entry><para> <emphasis role="bold">queue</emphasis></para></entry>
+ <entry><para> Total bytes in active/queued sends.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290450" xreflabel=""/>Credits work like a semaphore. At start they are initialized to allow a certain number of operations (8 in this example). LNET keeps a track of the minimum value so that you can see how congested a resource was.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290451" xreflabel=""/>If rtr/tx is less than max, there are operations in progress. The number of operations is equal to rtr or tx subtracted from max.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290452" xreflabel=""/>If rtr/tx is greater that max, there are operations blocking.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290453" xreflabel=""/>LNET also limits concurrent sends and router buffers allocated to a single peer so that no peer can occupy all these resources.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290454" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/nis</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290455" xreflabel=""/># cat /proc/sys/lnet/nis
-<anchor xml:id="dbdoclet.50438271_pgfId-1290456" xreflabel=""/>nid refs peer max \
+ <para>Credits work like a semaphore. At start they are initialized to allow a certain number of operations (8 in this example). LNET keeps a track of the minimum value so that you can see how congested a resource was.</para>
+ <para>If rtr/tx is less than max, there are operations in progress. The number of operations is equal to rtr or tx subtracted from max.</para>
+ <para>If rtr/tx is greater that max, there are operations blocking.</para>
+ <para>LNET also limits concurrent sends and router buffers allocated to a single peer so that no peer can occupy all these resources.</para>
+ <para><emphasis role="bold">/proc/sys/lnet/nis</emphasis></para>
+ <screen># cat /proc/sys/lnet/nis
+nid refs peer max \
tx min
-<anchor xml:id="dbdoclet.50438271_pgfId-1290457" xreflabel=""/>0@lo 3 0 0 \
+0@lo 3 0 0 \
0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290458" xreflabel=""/>192.168.10.34@tcp 4 8 256 \
+192.168.10.34@tcp 4 8 256 \
256 252
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290459" xreflabel=""/>Shows the current queue health on this node. The fields are explained below:</para>
+ <para>Shows the current queue health on this node. The fields are explained below:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1291912" xreflabel=""/>Field</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1291914" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Field</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291963" xreflabel=""/><emphasis role="bold">nid</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291965" xreflabel=""/>Network interface</para></entry>
+ <entry><para> <emphasis role="bold">nid</emphasis></para></entry>
+ <entry><para> Network interface</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291967" xreflabel=""/><emphasis role="bold">refs</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291969" xreflabel=""/>Internal reference counter</para></entry>
+ <entry><para> <emphasis role="bold">refs</emphasis></para></entry>
+ <entry><para> Internal reference counter</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291971" xreflabel=""/><emphasis role="bold">peer</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291973" xreflabel=""/>Number of peer-to-peer send credits on this NID. Credits are used to size buffer pools</para></entry>
+ <entry><para> <emphasis role="bold">peer</emphasis></para></entry>
+ <entry><para> Number of peer-to-peer send credits on this NID. Credits are used to size buffer pools</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291975" xreflabel=""/><emphasis role="bold">max</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291977" xreflabel=""/>Total number of send credits on this NID.</para></entry>
+ <entry><para> <emphasis role="bold">max</emphasis></para></entry>
+ <entry><para> Total number of send credits on this NID.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291979" xreflabel=""/><emphasis role="bold">tx</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291981" xreflabel=""/>Current number of send credits available on this NID.</para></entry>
+ <entry><para> <emphasis role="bold">tx</emphasis></para></entry>
+ <entry><para> Current number of send credits available on this NID.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291983" xreflabel=""/><emphasis role="bold">min</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291985" xreflabel=""/>Lowest number of send credits available on this NID.</para></entry>
+ <entry><para> <emphasis role="bold">min</emphasis></para></entry>
+ <entry><para> Lowest number of send credits available on this NID.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291987" xreflabel=""/><emphasis role="bold">queue</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1291989" xreflabel=""/>Total bytes in active/queued sends.</para></entry>
+ <entry><para> <emphasis role="bold">queue</emphasis></para></entry>
+ <entry><para> Total bytes in active/queued sends.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290494" xreflabel=""/>Subtracting max - tx yields the number of sends currently active. A large or increasing number of active sends may indicate a problem.</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290495" xreflabel=""/># cat /proc/sys/lnet/nis
-<anchor xml:id="dbdoclet.50438271_pgfId-1290496" xreflabel=""/>nid refs peer max \
+ <para>Subtracting max - tx yields the number of sends currently active. A large or increasing number of active sends may indicate a problem.</para>
+ <screen># cat /proc/sys/lnet/nis
+nid refs peer max \
tx min
-<anchor xml:id="dbdoclet.50438271_pgfId-1290497" xreflabel=""/>0@lo 2 0 0 \
+0@lo 2 0 0 \
0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290498" xreflabel=""/>10.67.73.173@tcp 4 8 256 \
+10.67.73.173@tcp 4 8 256 \
256 253
</screen>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1290499" xreflabel=""/>31.1.5 Free Space <anchor xml:id="dbdoclet.50438271_marker-1296165" xreflabel=""/>Distribution</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296514" xreflabel=""/>Free-space stripe weighting, as set, gives a priority of "0" to free space (versus trying to place the stripes "widely" -- nicely distributed across OSSs and OSTs to maximize network balancing). To adjust this priority (as a percentage), use the qos_prio_free proc tunable:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1296515" xreflabel=""/>$ cat /proc/fs/lustre/lov/<fsname>-mdtlov/qos_prio_free
+ <title>31.1.5 Free Space <anchor xml:id="dbdoclet.50438271_marker-1296165" xreflabel=""/>Distribution</title>
+ <para>Free-space stripe weighting, as set, gives a priority of "0" to free space (versus trying to place the stripes "widely" -- nicely distributed across OSSs and OSTs to maximize network balancing). To adjust this priority (as a percentage), use the qos_prio_free proc tunable:</para>
+ <screen>$ cat /proc/fs/lustre/lov/<fsname>-mdtlov/qos_prio_free
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296516" xreflabel=""/>Currently, the default is 90%. You can permanently set this value by running this command on the MGS:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1296517" xreflabel=""/>$ lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90
+ <para>Currently, the default is 90%. You can permanently set this value by running this command on the MGS:</para>
+ <screen>$ lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296518" xreflabel=""/>Setting the priority to 100% means that OSS distribution does not count in the weighting, but the stripe assignment is still done via weighting. If OST 2 has twice as much free space as OST 1, it is twice as likely to be used, but it is NOT guaranteed to be used.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290506" xreflabel=""/>Also note that free-space stripe weighting does not activate until two OSTs are imbalanced by more than 20%. Until then, a faster round-robin stripe allocator is used. (The new round-robin order also maximizes network balancing.)</para>
+ <para>Setting the priority to 100% means that OSS distribution does not count in the weighting, but the stripe assignment is still done via weighting. If OST 2 has twice as much free space as OST 1, it is twice as likely to be used, but it is NOT guaranteed to be used.</para>
+ <para>Also note that free-space stripe weighting does not activate until two OSTs are imbalanced by more than 20%. Until then, a faster round-robin stripe allocator is used. (The new round-robin order also maximizes network balancing.)</para>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438271_pgfId-1296529" xreflabel=""/>31.1.5.1 Managing Stripe Allocation</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296530" xreflabel=""/>The MDS uses two methods to manage stripe allocation and determine which OSTs to use for file object storage:</para>
+ <title>31.1.5.1 Managing Stripe Allocation</title>
+ <para>The MDS uses two methods to manage stripe allocation and determine which OSTs to use for file object storage:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296531" xreflabel=""/><emphasis role="bold">QOS</emphasis></para>
+ <para><emphasis role="bold">QOS</emphasis></para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297308" xreflabel=""/>Quality of Service (QOS) considers an OST's available blocks, speed, and the number of existing objects, etc. Using these criteria, the MDS selects OSTs with more free space more often than OSTs with less free space.</para>
+ <para>Quality of Service (QOS) considers an OST's available blocks, speed, and the number of existing objects, etc. Using these criteria, the MDS selects OSTs with more free space more often than OSTs with less free space.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297309" xreflabel=""/><emphasis role="bold">RR</emphasis></para>
+ <para><emphasis role="bold">RR</emphasis></para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296534" xreflabel=""/>Round-Robin (RR) allocates objects evenly across all OSTs. The RR stripe allocator is faster than QOS, and used often because it distributes space usage/load best in most situations, maximizing network balancing and improving performance.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296535" xreflabel=""/>Whether QOS or RR is used depends on the setting of the qos_threshold_rr proc tunable. The qos_threshold_rr variable specifies a percentage threshold where the use of QOS or RR becomes more/less likely. The qos_threshold_rr tunable can be set as an integer, from 0 to 100, and results in this stripe allocation behavior:</para>
+ <para>Round-Robin (RR) allocates objects evenly across all OSTs. The RR stripe allocator is faster than QOS, and used often because it distributes space usage/load best in most situations, maximizing network balancing and improving performance.</para>
+ <para>Whether QOS or RR is used depends on the setting of the qos_threshold_rr proc tunable. The qos_threshold_rr variable specifies a percentage threshold where the use of QOS or RR becomes more/less likely. The qos_threshold_rr tunable can be set as an integer, from 0 to 100, and results in this stripe allocation behavior:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296536" xreflabel=""/> If qos_threshold_rr is set to 0, then QOS is always used</para>
+ <para> If qos_threshold_rr is set to 0, then QOS is always used</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296537" xreflabel=""/> If qos_threshold_rr is set to 100, then RR is always used</para>
+ <para> If qos_threshold_rr is set to 100, then RR is always used</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296538" xreflabel=""/> The larger the qos_threshold_rr setting, the greater the possibility that RR is used instead of QOS</para>
+ <para> The larger the qos_threshold_rr setting, the greater the possibility that RR is used instead of QOS</para>
</listitem>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438271_78950">
<title>31.2 Lustre I/O <anchor xml:id="dbdoclet.50438271_marker-1290508" xreflabel=""/>Tunables</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290510" xreflabel=""/>The section describes I/O tunables.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290511" xreflabel=""/><emphasis role="bold">/proc/fs/lustre/llite/<fsname>-<uid>/max_cache_mb</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290512" xreflabel=""/># cat /proc/fs/lustre/llite/lustre-ce63ca00/max_cached_mb 128
+ <para>The section describes I/O tunables.</para>
+ <para><emphasis role="bold">/proc/fs/lustre/llite/<fsname>-<uid>/max_cache_mb</emphasis></para>
+ <screen># cat /proc/fs/lustre/llite/lustre-ce63ca00/max_cached_mb 128
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290513" xreflabel=""/>This tunable is the maximum amount of inactive data cached by the client (default is 3/4 of RAM).</para>
+ <para>This tunable is the maximum amount of inactive data cached by the client (default is 3/4 of RAM).</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1290515" xreflabel=""/>31.2.1 Client I/O RPC<anchor xml:id="dbdoclet.50438271_marker-1290514" xreflabel=""/> Stream Tunables</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290516" xreflabel=""/>The Lustre engine always attempts to pack an optimal amount of data into each I/O RPC and attempts to keep a consistent number of issued RPCs in progress at a time. Lustre exposes several tuning variables to adjust behavior according to network conditions and cluster size. Each OSC has its own tree of these tunables. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290517" xreflabel=""/>$ ls -d /proc/fs/lustre/osc/OSC_client_ost1_MNT_client_2 /localhost
-<anchor xml:id="dbdoclet.50438271_pgfId-1290518" xreflabel=""/>/proc/fs/lustre/osc/OSC_uml0_ost1_MNT_localhost
-<anchor xml:id="dbdoclet.50438271_pgfId-1290519" xreflabel=""/>/proc/fs/lustre/osc/OSC_uml0_ost2_MNT_localhost
-<anchor xml:id="dbdoclet.50438271_pgfId-1290520" xreflabel=""/>/proc/fs/lustre/osc/OSC_uml0_ost3_MNT_localhost
-<anchor xml:id="dbdoclet.50438271_pgfId-1290521" xreflabel=""/>$ ls /proc/fs/lustre/osc/OSC_uml0_ost1_MNT_localhost
-<anchor xml:id="dbdoclet.50438271_pgfId-1290522" xreflabel=""/>blocksizefilesfree max_dirty_mb ost_server_uuid stats
+ <title>31.2.1 Client I/O RPC<anchor xml:id="dbdoclet.50438271_marker-1290514" xreflabel=""/> Stream Tunables</title>
+ <para>The Lustre engine always attempts to pack an optimal amount of data into each I/O RPC and attempts to keep a consistent number of issued RPCs in progress at a time. Lustre exposes several tuning variables to adjust behavior according to network conditions and cluster size. Each OSC has its own tree of these tunables. For example:</para>
+ <screen>$ ls -d /proc/fs/lustre/osc/OSC_client_ost1_MNT_client_2 /localhost
+/proc/fs/lustre/osc/OSC_uml0_ost1_MNT_localhost
+/proc/fs/lustre/osc/OSC_uml0_ost2_MNT_localhost
+/proc/fs/lustre/osc/OSC_uml0_ost3_MNT_localhost
+$ ls /proc/fs/lustre/osc/OSC_uml0_ost1_MNT_localhost
+blocksizefilesfree max_dirty_mb ost_server_uuid stats
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290523" xreflabel=""/>... and so on.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290524" xreflabel=""/>RPC stream tunables are described below.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1291134" xreflabel=""/><emphasis role="bold">/proc/fs/lustre/osc/<object name>/max_dirty_mb</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1291135" xreflabel=""/>This tunable controls how many MBs of dirty data can be written and queued up in the OSC. POSIX file writes that are cached contribute to this count. When the limit is reached, additional writes stall until previously-cached writes are written to the server. This may be changed by writing a single ASCII integer to the file. Only values between 0 and 512 are allowable. If 0 is given, no writes are cached. Performance suffers noticeably unless you use large writes (1 MB or more).</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290527" xreflabel=""/><emphasis role="bold">/proc/fs/lustre/osc/<object name>/cur_dirty_bytes</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290528" xreflabel=""/>This tunable is a read-only value that returns the current amount of bytes written and cached on this OSC.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290529" xreflabel=""/><emphasis role="bold">/proc/fs/lustre/osc/<object name>/max_pages_per_rpc</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290530" xreflabel=""/>This tunable is the maximum number of pages that will undergo I/O in a single RPC to the OST. The minimum is a single page and the maximum for this setting is platform dependent (256 for i386/x86_64, possibly less for ia64/PPC with larger PAGE_SIZE), though generally amounts to a total of 1 MB in the RPC.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290531" xreflabel=""/><emphasis role="bold">/proc/fs/lustre/osc/<object name>/max_rpcs_in_flight</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296189" xreflabel=""/>This tunable is the maximum number of concurrent RPCs in flight from an OSC to its OST. If the OSC tries to initiate an RPC but finds that it already has the same number of RPCs outstanding, it will wait to issue further RPCs until some complete. The minimum setting is 1 and maximum setting is 32. If you are looking to improve small file I/O performance, increase the max_rpcs_in_flight value.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290533" xreflabel=""/>To maximize performance, the value for max_dirty_mb is recommended to be 4 * max_pages_per_rpc * max_rpcs_in_flight.</para>
+ <para>... and so on.</para>
+ <para>RPC stream tunables are described below.</para>
+ <para><emphasis role="bold">/proc/fs/lustre/osc/<object name>/max_dirty_mb</emphasis></para>
+ <para>This tunable controls how many MBs of dirty data can be written and queued up in the OSC. POSIX file writes that are cached contribute to this count. When the limit is reached, additional writes stall until previously-cached writes are written to the server. This may be changed by writing a single ASCII integer to the file. Only values between 0 and 512 are allowable. If 0 is given, no writes are cached. Performance suffers noticeably unless you use large writes (1 MB or more).</para>
+ <para><emphasis role="bold">/proc/fs/lustre/osc/<object name>/cur_dirty_bytes</emphasis></para>
+ <para>This tunable is a read-only value that returns the current amount of bytes written and cached on this OSC.</para>
+ <para><emphasis role="bold">/proc/fs/lustre/osc/<object name>/max_pages_per_rpc</emphasis></para>
+ <para>This tunable is the maximum number of pages that will undergo I/O in a single RPC to the OST. The minimum is a single page and the maximum for this setting is platform dependent (256 for i386/x86_64, possibly less for ia64/PPC with larger PAGE_SIZE), though generally amounts to a total of 1 MB in the RPC.</para>
+ <para><emphasis role="bold">/proc/fs/lustre/osc/<object name>/max_rpcs_in_flight</emphasis></para>
+ <para>This tunable is the maximum number of concurrent RPCs in flight from an OSC to its OST. If the OSC tries to initiate an RPC but finds that it already has the same number of RPCs outstanding, it will wait to issue further RPCs until some complete. The minimum setting is 1 and maximum setting is 32. If you are looking to improve small file I/O performance, increase the max_rpcs_in_flight value.</para>
+ <para>To maximize performance, the value for max_dirty_mb is recommended to be 4 * max_pages_per_rpc * max_rpcs_in_flight.</para>
<note><para>The <object name> varies depending on the specific Lustre configuration. For <object name> examples, refer to the sample command output.</para></note>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1290536" xreflabel=""/>31.2.2 Watching the <anchor xml:id="dbdoclet.50438271_marker-1290535" xreflabel=""/>Client RPC Stream</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290537" xreflabel=""/>The same directory contains a rpc_stats file with a histogram showing the composition of previous RPCs. The histogram can be cleared by writing any value into the rpc_stats file.</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290538" xreflabel=""/># cat /proc/fs/lustre/osc/spfs-OST0000-osc-c45f9c00/rpc_stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290539" xreflabel=""/>snapshot_time: 1174867307.156604 (secs.\
+ <title>31.2.2 Watching the <anchor xml:id="dbdoclet.50438271_marker-1290535" xreflabel=""/>Client RPC Stream</title>
+ <para>The same directory contains a rpc_stats file with a histogram showing the composition of previous RPCs. The histogram can be cleared by writing any value into the rpc_stats file.</para>
+ <screen># cat /proc/fs/lustre/osc/spfs-OST0000-osc-c45f9c00/rpc_stats
+snapshot_time: 1174867307.156604 (secs.\
usecs)
-<anchor xml:id="dbdoclet.50438271_pgfId-1290540" xreflabel=""/>read RPCs in flight: 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290541" xreflabel=""/>write RPCs in flight: 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290542" xreflabel=""/>pending write pages: 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290543" xreflabel=""/>pending read pages: 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290544" xreflabel=""/> read write
-<anchor xml:id="dbdoclet.50438271_pgfId-1290545" xreflabel=""/>pages per rpc rpcs % cum % | rpcs \
+read RPCs in flight: 0
+write RPCs in flight: 0
+pending write pages: 0
+pending read pages: 0
+ read write
+pages per rpc rpcs % cum % | rpcs \
% cum %
-<anchor xml:id="dbdoclet.50438271_pgfId-1290546" xreflabel=""/>1: 0 0 0 | 0 \
+1: 0 0 0 | 0 \
0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290547" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1290548" xreflabel=""/> read write
-<anchor xml:id="dbdoclet.50438271_pgfId-1290549" xreflabel=""/>rpcs in flight rpcs % cum % | rpcs \
+
+ read write
+rpcs in flight rpcs % cum % | rpcs \
% cum %
-<anchor xml:id="dbdoclet.50438271_pgfId-1290550" xreflabel=""/>0: 0 0 0 | 0 \
+0: 0 0 0 | 0 \
0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290551" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1290552" xreflabel=""/> read write
-<anchor xml:id="dbdoclet.50438271_pgfId-1290553" xreflabel=""/>offset rpcs % cum % | rpcs \
+
+ read write
+offset rpcs % cum % | rpcs \
% cum %
-<anchor xml:id="dbdoclet.50438271_pgfId-1290554" xreflabel=""/>0: 0 0 0 | 0 \
+0: 0 0 0 | 0 \
0 0
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1298886" xreflabel=""/>Where:</para>
+ <para>Where:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1298854" xreflabel=""/>Field</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1298856" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Field</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298858" xreflabel=""/><emphasis role="bold">{read,write} RPCs in flight</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298860" xreflabel=""/>Number of read/write RPCs issued by the OSC, but not complete at the time of the snapshot. This value should always be less than or equal to max_rpcs_in_flight.</para></entry>
+ <entry><para> <emphasis role="bold">{read,write} RPCs in flight</emphasis></para></entry>
+ <entry><para> Number of read/write RPCs issued by the OSC, but not complete at the time of the snapshot. This value should always be less than or equal to max_rpcs_in_flight.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298862" xreflabel=""/><emphasis role="bold">pending {read,write} pages</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298911" xreflabel=""/>Number of pending read/write pages that have been queued for I/O in the OSC.</para></entry>
+ <entry><para> <emphasis role="bold">pending {read,write} pages</emphasis></para></entry>
+ <entry><para> Number of pending read/write pages that have been queued for I/O in the OSC.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298866" xreflabel=""/><emphasis role="bold">pages per RPC</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298924" xreflabel=""/>When an RPC is sent, the number of pages it consists of is recorded (in order). A single page RPC increments the 0: row.</para></entry>
+ <entry><para> <emphasis role="bold">pages per RPC</emphasis></para></entry>
+ <entry><para> When an RPC is sent, the number of pages it consists of is recorded (in order). A single page RPC increments the 0: row.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298871" xreflabel=""/><emphasis role="bold">RPCs in flight</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298873" xreflabel=""/>When an RPC is sent, the number of other RPCs that are pending is recorded. When the first RPC is sent, the 0: row is incremented. If the first RPC is sent while another is pending, the 1: row is incremented and so on. As each RPC *completes*, the number of pending RPCs is not tabulated.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1298973" xreflabel=""/>This table is a good way to visualize the concurrency of the RPC stream. Ideally, you will see a large clump around the max_rpcs_in_flight value, which shows that the network is being kept busy.</para></entry>
+ <entry><para> <emphasis role="bold">RPCs in flight</emphasis></para></entry>
+ <entry><para> When an RPC is sent, the number of other RPCs that are pending is recorded. When the first RPC is sent, the 0: row is incremented. If the first RPC is sent while another is pending, the 1: row is incremented and so on. As each RPC *completes*, the number of pending RPCs is not tabulated.</para><para>This table is a good way to visualize the concurrency of the RPC stream. Ideally, you will see a large clump around the max_rpcs_in_flight value, which shows that the network is being kept busy.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1299025" xreflabel=""/><emphasis role="bold">offset</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1299027" xreflabel=""/> </para></entry>
+ <entry><para> <emphasis role="bold">offset</emphasis></para></entry>
+ <entry><para> </para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1290565" xreflabel=""/>31.2.3 Client Read-Write <anchor xml:id="dbdoclet.50438271_marker-1290564" xreflabel=""/>Offset Survey</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290566" xreflabel=""/>The offset_stats parameter maintains statistics for occurrences where a series of read or write calls from a process did not access the next sequential location. The offset field is reset to 0 (zero) whenever a different file is read/written.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1293887" xreflabel=""/>Read/write offset statistics are off, by default. The statistics can be activated by writing anything into the offset_stats file.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290567" xreflabel=""/>Example:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290568" xreflabel=""/># cat /proc/fs/lustre/llite/lustre-f57dee00/rw_offset_stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290569" xreflabel=""/>snapshot_time: 1155748884.591028 (secs.usecs)
-<anchor xml:id="dbdoclet.50438271_pgfId-1290570" xreflabel=""/>R/W PID RANGE START RANGE END \
+ <title>31.2.3 Client Read-Write <anchor xml:id="dbdoclet.50438271_marker-1290564" xreflabel=""/>Offset Survey</title>
+ <para>The offset_stats parameter maintains statistics for occurrences where a series of read or write calls from a process did not access the next sequential location. The offset field is reset to 0 (zero) whenever a different file is read/written.</para>
+ <para>Read/write offset statistics are off, by default. The statistics can be activated by writing anything into the offset_stats file.</para>
+ <para>Example:</para>
+ <screen># cat /proc/fs/lustre/llite/lustre-f57dee00/rw_offset_stats
+snapshot_time: 1155748884.591028 (secs.usecs)
+R/W PID RANGE START RANGE END \
SMALLEST EXTENT LARGEST EXTENT OFF\
SET
-<anchor xml:id="dbdoclet.50438271_pgfId-1290571" xreflabel=""/>R 8385 0 128 \
+R 8385 0 128 \
128 128 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290572" xreflabel=""/>R 8385 0 224 \
+R 8385 0 224 \
224 224 -128
-<anchor xml:id="dbdoclet.50438271_pgfId-1290573" xreflabel=""/>W 8385 0 250 \
+W 8385 0 250 \
50 100 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290574" xreflabel=""/>W 8385 100 1110 \
+W 8385 100 1110 \
10 500 -150
-<anchor xml:id="dbdoclet.50438271_pgfId-1290575" xreflabel=""/>W 8384 0 5233 \
+W 8384 0 5233 \
5233 5233 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290576" xreflabel=""/>R 8385 500 600 \
+R 8385 500 600 \
100 100 -610
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290611" xreflabel=""/>Where:</para>
+ <para>Where:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1291997" xreflabel=""/>Field</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1291999" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Field</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292045" xreflabel=""/><emphasis role="bold">R/W</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292047" xreflabel=""/>Whether the non-sequential call was a read or write</para></entry>
+ <entry><para> <emphasis role="bold">R/W</emphasis></para></entry>
+ <entry><para> Whether the non-sequential call was a read or write</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292049" xreflabel=""/><emphasis role="bold">PID</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292051" xreflabel=""/>Process ID which made the read/write call.</para></entry>
+ <entry><para> <emphasis role="bold">PID</emphasis></para></entry>
+ <entry><para> Process ID which made the read/write call.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292053" xreflabel=""/><emphasis role="bold">Range Start/Range End</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292055" xreflabel=""/>Range in which the read/write calls were sequential.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292056" xreflabel=""/> </para></entry>
+ <entry><para> <emphasis role="bold">Range Start/Range End</emphasis></para></entry>
+ <entry><para> Range in which the read/write calls were sequential.</para><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292058" xreflabel=""/><emphasis role="bold">Smallest Extent</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292060" xreflabel=""/>Smallest extent (single read/write) in the corresponding range.</para></entry>
+ <entry><para> <emphasis role="bold">Smallest Extent</emphasis></para></entry>
+ <entry><para> Smallest extent (single read/write) in the corresponding range.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292062" xreflabel=""/><emphasis role="bold">Largest Extent</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292064" xreflabel=""/>Largest extent (single read/write) in the corresponding range.</para></entry>
+ <entry><para> <emphasis role="bold">Largest Extent</emphasis></para></entry>
+ <entry><para> Largest extent (single read/write) in the corresponding range.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292066" xreflabel=""/><emphasis role="bold">Offset</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292068" xreflabel=""/>Difference from the previous range end to the current range start.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292069" xreflabel=""/>For example, Smallest-Extent indicates that the writes in the range 100 to 1110 were sequential, with a minimum write of 10 and a maximum write of 500. This range was started with an offset of -150. That means this is the difference between the last entry's range-end and this entry's range-start for the same file.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292070" xreflabel=""/>The rw_offset_stats file can be cleared by writing to it:</para><screen><anchor xml:id="dbdoclet.50438271_pgfId-1292071" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1292072" xreflabel=""/>echo > /proc/fs/lustre/llite/lustre-f57dee00/rw_offset_stats
+ <entry><para> <emphasis role="bold">Offset</emphasis></para></entry>
+ <entry><para> Difference from the previous range end to the current range start.</para><para>For example, Smallest-Extent indicates that the writes in the range 100 to 1110 were sequential, with a minimum write of 10 and a maximum write of 500. This range was started with an offset of -150. That means this is the difference between the last entry's range-end and this entry's range-start for the same file.</para><para>The rw_offset_stats file can be cleared by writing to it:</para><screen>
+echo > /proc/fs/lustre/llite/lustre-f57dee00/rw_offset_stats
</screen></entry>
</row>
</tbody>
</informaltable>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1290613" xreflabel=""/>31.2.4 Client Read-Write <anchor xml:id="dbdoclet.50438271_marker-1290612" xreflabel=""/>Extents Survey</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290614" xreflabel=""/><emphasis role="bold">Client-Based I/O Extent Size Survey</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290615" xreflabel=""/>The rw_extent_stats histogram in the llite directory shows you the statistics for the sizes of the read-write I/O extents. This file does not maintain the per-process statistics.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290616" xreflabel=""/>Example:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290617" xreflabel=""/>$ cat /proc/fs/lustre/llite/lustre-ee5af200/extents_stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1293904" xreflabel=""/>snapshot_time: 1213828728.348516 (secs.usecs)
-<anchor xml:id="dbdoclet.50438271_pgfId-1294259" xreflabel=""/> read | write
-<anchor xml:id="dbdoclet.50438271_pgfId-1294260" xreflabel=""/>extents calls % cum% | calls % \
+ <title>31.2.4 Client Read-Write <anchor xml:id="dbdoclet.50438271_marker-1290612" xreflabel=""/>Extents Survey</title>
+ <para><emphasis role="bold">Client-Based I/O Extent Size Survey</emphasis></para>
+ <para>The rw_extent_stats histogram in the llite directory shows you the statistics for the sizes of the read-write I/O extents. This file does not maintain the per-process statistics.</para>
+ <para>Example:</para>
+ <screen>$ cat /proc/fs/lustre/llite/lustre-ee5af200/extents_stats
+snapshot_time: 1213828728.348516 (secs.usecs)
+ read | write
+extents calls % cum% | calls % \
cum%
-<anchor xml:id="dbdoclet.50438271_pgfId-1290621" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1294257" xreflabel=""/>0K - 4K : 0 0 0 | 2 2 2
-<anchor xml:id="dbdoclet.50438271_pgfId-1293950" xreflabel=""/>4K - 8K : 0 0 0 | 0 0 2
-<anchor xml:id="dbdoclet.50438271_pgfId-1293918" xreflabel=""/>8K - 16K : 0 0 0 | 0 0 2
-<anchor xml:id="dbdoclet.50438271_pgfId-1293922" xreflabel=""/>16K - 32K : 0 0 0 | 20 23 \
+
+0K - 4K : 0 0 0 | 2 2 2
+4K - 8K : 0 0 0 | 0 0 2
+8K - 16K : 0 0 0 | 0 0 2
+16K - 32K : 0 0 0 | 20 23 \
26
-<anchor xml:id="dbdoclet.50438271_pgfId-1293926" xreflabel=""/>32K - 64K : 0 0 0 | 0 0 \
+32K - 64K : 0 0 0 | 0 0 \
26
-<anchor xml:id="dbdoclet.50438271_pgfId-1293930" xreflabel=""/>64K - 128K : 0 0 0 | 51 60 \
+64K - 128K : 0 0 0 | 51 60 \
86
-<anchor xml:id="dbdoclet.50438271_pgfId-1293934" xreflabel=""/>128K - 256K : 0 0 0 | 0 0 \
+128K - 256K : 0 0 0 | 0 0 \
86
-<anchor xml:id="dbdoclet.50438271_pgfId-1293938" xreflabel=""/>256K - 512K : 0 0 0 | 0 0 \
+256K - 512K : 0 0 0 | 0 0 \
86
-<anchor xml:id="dbdoclet.50438271_pgfId-1293942" xreflabel=""/>512K - 1024K : 0 0 0 | 0 0 \
+512K - 1024K : 0 0 0 | 0 0 \
86
-<anchor xml:id="dbdoclet.50438271_pgfId-1293946" xreflabel=""/>1M - 2M : 0 0 0 | 11 13 100
-<anchor xml:id="dbdoclet.50438271_pgfId-1293908" xreflabel=""/>
+1M - 2M : 0 0 0 | 11 13 100
+
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290622" xreflabel=""/>The file can be cleared by issuing the following command:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290623" xreflabel=""/>$ echo > cat /proc/fs/lustre/llite/lustre-ee5af200/extents_stats
+ <para>The file can be cleared by issuing the following command:</para>
+ <screen>$ echo > cat /proc/fs/lustre/llite/lustre-ee5af200/extents_stats
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290624" xreflabel=""/><emphasis role="bold">Per-Process Client I/O Statistics</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290625" xreflabel=""/>The extents_stats_per_process file maintains the I/O extent size statistics on a per-process basis. So you can track the per-process statistics for the last MAX_PER_PROCESS_HIST processes.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1293971" xreflabel=""/>Example:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1293972" xreflabel=""/>$ cat /proc/fs/lustre/llite/lustre-ee5af200/extents_stats_per_process
-<anchor xml:id="dbdoclet.50438271_pgfId-1293973" xreflabel=""/>snapshot_time: 1213828762.204440 (secs.usecs)
-<anchor xml:id="dbdoclet.50438271_pgfId-1293974" xreflabel=""/> read | write
-<anchor xml:id="dbdoclet.50438271_pgfId-1293975" xreflabel=""/>extents calls % cum% | calls % \
+ <para><emphasis role="bold">Per-Process Client I/O Statistics</emphasis></para>
+ <para>The extents_stats_per_process file maintains the I/O extent size statistics on a per-process basis. So you can track the per-process statistics for the last MAX_PER_PROCESS_HIST processes.</para>
+ <para>Example:</para>
+ <screen>$ cat /proc/fs/lustre/llite/lustre-ee5af200/extents_stats_per_process
+snapshot_time: 1213828762.204440 (secs.usecs)
+ read | write
+extents calls % cum% | calls % \
cum%
-<anchor xml:id="dbdoclet.50438271_pgfId-1293976" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1293998" xreflabel=""/>PID: 11488
-<anchor xml:id="dbdoclet.50438271_pgfId-1293999" xreflabel=""/> 0K - 4K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1293977" xreflabel=""/> 4K - 8K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1293978" xreflabel=""/> 8K - 16K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1293979" xreflabel=""/> 16K - 32K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1293980" xreflabel=""/> 32K - 64K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1293981" xreflabel=""/> 64K - 128K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1293982" xreflabel=""/> 128K - 256K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1293983" xreflabel=""/> 256K - 512K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1294019" xreflabel=""/> 512K - 1024K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1294017" xreflabel=""/> 1M - 2M : 0 0 0 | 10 100 100
-<anchor xml:id="dbdoclet.50438271_pgfId-1294028" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1294031" xreflabel=""/>PID: 11491
-<anchor xml:id="dbdoclet.50438271_pgfId-1294032" xreflabel=""/> 0K - 4K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1294033" xreflabel=""/> 4K - 8K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1294034" xreflabel=""/> 8K - 16K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1294035" xreflabel=""/> 16K - 32K : 0 0 0 | 20 100 100
-<anchor xml:id="dbdoclet.50438271_pgfId-1294036" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1294052" xreflabel=""/>PID: 11424
-<anchor xml:id="dbdoclet.50438271_pgfId-1294053" xreflabel=""/> 0K - 4K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1294054" xreflabel=""/> 4K - 8K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1294055" xreflabel=""/> 8K - 16K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1294056" xreflabel=""/> 16K - 32K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1294070" xreflabel=""/> 32K - 64K : 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1294086" xreflabel=""/> 64K - 128K : 0 0 0 | 16 100 100
-<anchor xml:id="dbdoclet.50438271_pgfId-1294087" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1294060" xreflabel=""/>PID: 11426
-<anchor xml:id="dbdoclet.50438271_pgfId-1294061" xreflabel=""/> 0K - 4K : 0 0 0 | 1 100 100
-<anchor xml:id="dbdoclet.50438271_pgfId-1294096" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1294099" xreflabel=""/>PID: 11429
-<anchor xml:id="dbdoclet.50438271_pgfId-1294100" xreflabel=""/> 0K - 4K : 0 0 0 | 1 100 100
-<anchor xml:id="dbdoclet.50438271_pgfId-1294097" xreflabel=""/>
+
+PID: 11488
+ 0K - 4K : 0 0 0 | 0 0 0
+ 4K - 8K : 0 0 0 | 0 0 0
+ 8K - 16K : 0 0 0 | 0 0 0
+ 16K - 32K : 0 0 0 | 0 0 0
+ 32K - 64K : 0 0 0 | 0 0 0
+ 64K - 128K : 0 0 0 | 0 0 0
+ 128K - 256K : 0 0 0 | 0 0 0
+ 256K - 512K : 0 0 0 | 0 0 0
+ 512K - 1024K : 0 0 0 | 0 0 0
+ 1M - 2M : 0 0 0 | 10 100 100
+
+PID: 11491
+ 0K - 4K : 0 0 0 | 0 0 0
+ 4K - 8K : 0 0 0 | 0 0 0
+ 8K - 16K : 0 0 0 | 0 0 0
+ 16K - 32K : 0 0 0 | 20 100 100
+
+PID: 11424
+ 0K - 4K : 0 0 0 | 0 0 0
+ 4K - 8K : 0 0 0 | 0 0 0
+ 8K - 16K : 0 0 0 | 0 0 0
+ 16K - 32K : 0 0 0 | 0 0 0
+ 32K - 64K : 0 0 0 | 0 0 0
+ 64K - 128K : 0 0 0 | 16 100 100
+
+PID: 11426
+ 0K - 4K : 0 0 0 | 1 100 100
+
+PID: 11429
+ 0K - 4K : 0 0 0 | 1 100 100
+
</screen>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1290632" xreflabel=""/>31.2.5 <anchor xml:id="dbdoclet.50438271_55057" xreflabel=""/> Watching the <anchor xml:id="dbdoclet.50438271_marker-1290631" xreflabel=""/>OST Block I/O Stream</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290633" xreflabel=""/>Similarly, there is a brw_stats histogram in the obdfilter directory which shows you the statistics for number of I/O requests sent to the disk, their size and whether they are contiguous on the disk or not.</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290634" xreflabel=""/>cat /proc/fs/lustre/obdfilter/lustre-OST0000/brw_stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290635" xreflabel=""/>snapshot_time: 1174875636.764630 (secs:usecs)
-<anchor xml:id="dbdoclet.50438271_pgfId-1290636" xreflabel=""/> read write
-<anchor xml:id="dbdoclet.50438271_pgfId-1290637" xreflabel=""/>pages per brw brws % cum % | rpcs % \
+ <title>31.2.5 <anchor xml:id="dbdoclet.50438271_55057" xreflabel=""/> Watching the <anchor xml:id="dbdoclet.50438271_marker-1290631" xreflabel=""/>OST Block I/O Stream</title>
+ <para>Similarly, there is a brw_stats histogram in the obdfilter directory which shows you the statistics for number of I/O requests sent to the disk, their size and whether they are contiguous on the disk or not.</para>
+ <screen>cat /proc/fs/lustre/obdfilter/lustre-OST0000/brw_stats
+snapshot_time: 1174875636.764630 (secs:usecs)
+ read write
+pages per brw brws % cum % | rpcs % \
cum %
-<anchor xml:id="dbdoclet.50438271_pgfId-1290638" xreflabel=""/>1: 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290639" xreflabel=""/> read write
-<anchor xml:id="dbdoclet.50438271_pgfId-1290640" xreflabel=""/>discont pages rpcs % cum % | rpcs % \
+1: 0 0 0 | 0 0 0
+ read write
+discont pages rpcs % cum % | rpcs % \
cum %
-<anchor xml:id="dbdoclet.50438271_pgfId-1290641" xreflabel=""/>1: 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290642" xreflabel=""/> read write
-<anchor xml:id="dbdoclet.50438271_pgfId-1290643" xreflabel=""/>discont blocks rpcs % cum % | rpcs % \
+1: 0 0 0 | 0 0 0
+ read write
+discont blocks rpcs % cum % | rpcs % \
cum %
-<anchor xml:id="dbdoclet.50438271_pgfId-1290644" xreflabel=""/>1: 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290645" xreflabel=""/> read write
-<anchor xml:id="dbdoclet.50438271_pgfId-1290646" xreflabel=""/>dio frags rpcs % cum % | rpcs % cum %
-<anchor xml:id="dbdoclet.50438271_pgfId-1290647" xreflabel=""/>1: 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290648" xreflabel=""/> read write
-<anchor xml:id="dbdoclet.50438271_pgfId-1290649" xreflabel=""/>disk ios in flight rpcs % cum % | rpcs % cum %
-<anchor xml:id="dbdoclet.50438271_pgfId-1290650" xreflabel=""/>1: 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290651" xreflabel=""/> read write
-<anchor xml:id="dbdoclet.50438271_pgfId-1290652" xreflabel=""/>io time (1/1000s) rpcs % cum % | rpcs % cum %
-<anchor xml:id="dbdoclet.50438271_pgfId-1290653" xreflabel=""/>1: 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290654" xreflabel=""/> read write
-<anchor xml:id="dbdoclet.50438271_pgfId-1290655" xreflabel=""/>disk io size rpcs % cum % | rpcs % \
+1: 0 0 0 | 0 0 0
+ read write
+dio frags rpcs % cum % | rpcs % cum %
+1: 0 0 0 | 0 0 0
+ read write
+disk ios in flight rpcs % cum % | rpcs % cum %
+1: 0 0 0 | 0 0 0
+ read write
+io time (1/1000s) rpcs % cum % | rpcs % cum %
+1: 0 0 0 | 0 0 0
+ read write
+disk io size rpcs % cum % | rpcs % \
cum %
-<anchor xml:id="dbdoclet.50438271_pgfId-1290656" xreflabel=""/>1: 0 0 0 | 0 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290657" xreflabel=""/> read write
+1: 0 0 0 | 0 0 0
+ read write
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294291" xreflabel=""/>The fields are explained below:</para>
+ <para>The fields are explained below:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1294276" xreflabel=""/>Field</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1294278" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Field</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294280" xreflabel=""/><emphasis role="bold">pages per brw</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294282" xreflabel=""/>Number of pages per RPC request, which should match aggregate client rpc_stats.</para></entry>
+ <entry><para> <emphasis role="bold">pages per brw</emphasis></para></entry>
+ <entry><para> Number of pages per RPC request, which should match aggregate client rpc_stats.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294284" xreflabel=""/><emphasis role="bold">discont pages</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294286" xreflabel=""/>Number of discontinuities in the logical file offset of each page in a single RPC.</para></entry>
+ <entry><para> <emphasis role="bold">discont pages</emphasis></para></entry>
+ <entry><para> Number of discontinuities in the logical file offset of each page in a single RPC.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294288" xreflabel=""/><emphasis role="bold">discont blocks</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294290" xreflabel=""/>Number of discontinuities in the physical block allocation in the file system for a single RPC.</para></entry>
+ <entry><para> <emphasis role="bold">discont blocks</emphasis></para></entry>
+ <entry><para> Number of discontinuities in the physical block allocation in the file system for a single RPC.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299070" xreflabel=""/>For each Lustre service, the following information is provided:</para>
+ <para>For each Lustre service, the following information is provided:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299055" xreflabel=""/> Number of requests</para>
+ <para> Number of requests</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299083" xreflabel=""/> Request wait time (avg, min, max and std dev)</para>
+ <para> Request wait time (avg, min, max and std dev)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299086" xreflabel=""/> Service idle time (% of elapsed time)</para>
+ <para> Service idle time (% of elapsed time)</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299058" xreflabel=""/>Additionally, data on each Lustre service is provided by service type:</para>
+ <para>Additionally, data on each Lustre service is provided by service type:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299059" xreflabel=""/> Number of requests of this type</para>
+ <para> Number of requests of this type</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299092" xreflabel=""/> Request service time (avg, min, max and std dev)</para>
+ <para> Request service time (avg, min, max and std dev)</para>
</listitem>
</itemizedlist>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1294293" xreflabel=""/>31.2.6 Using File <anchor xml:id="dbdoclet.50438271_marker-1294292" xreflabel=""/>Readahead and Directory Statahead</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295106" xreflabel=""/>Lustre 1.6.5.1 introduced file readahead and directory statahead functionality that read data into memory in anticipation of a process actually requesting the data. File readahead functionality reads file content data into memory. Directory statahead functionality reads metadata into memory. When readahead and/or statahead work well, a data-consuming process finds that the information it needs is available when requested, and it is unnecessary to wait for network I/O.</para>
+ <title>31.2.6 Using File <anchor xml:id="dbdoclet.50438271_marker-1294292" xreflabel=""/>Readahead and Directory Statahead</title>
+ <para>Lustre 1.6.5.1 introduced file readahead and directory statahead functionality that read data into memory in anticipation of a process actually requesting the data. File readahead functionality reads file content data into memory. Directory statahead functionality reads metadata into memory. When readahead and/or statahead work well, a data-consuming process finds that the information it needs is available when requested, and it is unnecessary to wait for network I/O.</para>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1295107" xreflabel=""/>31.2.6.1 Tuning <anchor xml:id="dbdoclet.50438271_marker-1295183" xreflabel=""/>File Readahead</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290680" xreflabel=""/>File readahead is triggered when two or more sequential reads by an application fail to be satisfied by the Linux buffer cache. The size of the initial readahead is 1 MB. Additional readaheads grow linearly, and increment until the readahead cache on the client is full at 40 MB.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290681" xreflabel=""/><emphasis role="bold">/proc/fs/lustre/llite/<fsname>-<uid>/max_read_ahead_mb</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290682" xreflabel=""/>This tunable controls the maximum amount of data readahead on a file. Files are read ahead in RPC-sized chunks (1 MB or the size of read() call, if larger) after the second sequential read on a file descriptor. Random reads are done at the size of the read() call only (no readahead). Reads to non-contiguous regions of the file reset the readahead algorithm, and readahead is not triggered again until there are sequential reads again. To disable readahead, set this tunable to 0. The default value is 40 MB.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290683" xreflabel=""/><emphasis role="bold">/proc/fs/lustre/llite/<fsname>-<uid>/max_read_ahead_whole_mb</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290684" xreflabel=""/>This tunable controls the maximum size of a file that is read in its entirety, regardless of the size of the read().</para>
+ <title>31.2.6.1 Tuning <anchor xml:id="dbdoclet.50438271_marker-1295183" xreflabel=""/>File Readahead</title>
+ <para>File readahead is triggered when two or more sequential reads by an application fail to be satisfied by the Linux buffer cache. The size of the initial readahead is 1 MB. Additional readaheads grow linearly, and increment until the readahead cache on the client is full at 40 MB.</para>
+ <para><emphasis role="bold">/proc/fs/lustre/llite/<fsname>-<uid>/max_read_ahead_mb</emphasis></para>
+ <para>This tunable controls the maximum amount of data readahead on a file. Files are read ahead in RPC-sized chunks (1 MB or the size of read() call, if larger) after the second sequential read on a file descriptor. Random reads are done at the size of the read() call only (no readahead). Reads to non-contiguous regions of the file reset the readahead algorithm, and readahead is not triggered again until there are sequential reads again. To disable readahead, set this tunable to 0. The default value is 40 MB.</para>
+ <para><emphasis role="bold">/proc/fs/lustre/llite/<fsname>-<uid>/max_read_ahead_whole_mb</emphasis></para>
+ <para>This tunable controls the maximum size of a file that is read in its entirety, regardless of the size of the read().</para>
</section>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1295046" xreflabel=""/>31.2.6.2 Tuning Directory <anchor xml:id="dbdoclet.50438271_marker-1295184" xreflabel=""/>Statahead</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295049" xreflabel=""/>When the ls -l process opens a directory, its process ID is recorded. When the first directory entry is ''stated'' with this recorded process ID, a statahead thread is triggered which stats ahead all of the directory entries, in order. The ls -l process can use the stated directory entries directly, improving performance.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295060" xreflabel=""/>/proc/fs/lustre/llite/*/statahead_max</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295152" xreflabel=""/>This tunable controls whether directory statahead is enabled and the maximum statahead count. By default, statahead is active.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295168" xreflabel=""/>To disable statahead, set this tunable to:</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295061" xreflabel=""/>echo 0 > /proc/fs/lustre/llite/*/statahead_max</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295063" xreflabel=""/>To set the maximum statahead count (n), set this tunable to:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1295280" xreflabel=""/>echo n > /proc/fs/lustre/llite/*/statahead_max
+ <title>31.2.6.2 Tuning Directory <anchor xml:id="dbdoclet.50438271_marker-1295184" xreflabel=""/>Statahead</title>
+ <para>When the ls -l process opens a directory, its process ID is recorded. When the first directory entry is ''stated'' with this recorded process ID, a statahead thread is triggered which stats ahead all of the directory entries, in order. The ls -l process can use the stated directory entries directly, improving performance.</para>
+ <para>/proc/fs/lustre/llite/*/statahead_max</para>
+ <para>This tunable controls whether directory statahead is enabled and the maximum statahead count. By default, statahead is active.</para>
+ <para>To disable statahead, set this tunable to:</para>
+ <para>echo 0 > /proc/fs/lustre/llite/*/statahead_max</para>
+ <para>To set the maximum statahead count (n), set this tunable to:</para>
+ <screen>echo n > /proc/fs/lustre/llite/*/statahead_max
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295281" xreflabel=""/>The maximum value of n is 8192.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295282" xreflabel=""/>/proc/fs/lustre/llite/*/statahead_status</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295067" xreflabel=""/>This is a read-only interface that indicates the current statahead status.</para>
+ <para>The maximum value of n is 8192.</para>
+ <para>/proc/fs/lustre/llite/*/statahead_status</para>
+ <para>This is a read-only interface that indicates the current statahead status.</para>
</section>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1290686" xreflabel=""/>31.2.7 OSS <anchor xml:id="dbdoclet.50438271_marker-1296183" xreflabel=""/>Read Cache</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295320" xreflabel=""/>The OSS read cache feature provides read-only caching of data on an OSS. This functionality uses the regular Linux page cache to store the data. Just like caching from a regular filesytem in Linux, OSS read cache uses as much physical memory as is allocated.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295462" xreflabel=""/>OSS read cache improves Lustre performance in these situations:</para>
+ <title>31.2.7 OSS <anchor xml:id="dbdoclet.50438271_marker-1296183" xreflabel=""/>Read Cache</title>
+ <para>The OSS read cache feature provides read-only caching of data on an OSS. This functionality uses the regular Linux page cache to store the data. Just like caching from a regular filesytem in Linux, OSS read cache uses as much physical memory as is allocated.</para>
+ <para>OSS read cache improves Lustre performance in these situations:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295517" xreflabel=""/> Many clients are accessing the same data set (as in HPC applications and when diskless clients boot from Lustre)</para>
+ <para> Many clients are accessing the same data set (as in HPC applications and when diskless clients boot from Lustre)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295521" xreflabel=""/> One client is storing data while another client is reading it (essentially exchanging data via the OST)</para>
+ <para> One client is storing data while another client is reading it (essentially exchanging data via the OST)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295528" xreflabel=""/> A client has very limited caching of its own</para>
+ <para> A client has very limited caching of its own</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295515" xreflabel=""/>OSS read cache offers these benefits:</para>
+ <para>OSS read cache offers these benefits:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295469" xreflabel=""/> Allows OSTs to cache read data more frequently</para>
+ <para> Allows OSTs to cache read data more frequently</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295485" xreflabel=""/> Improves repeated reads to match network speeds instead of disk speeds</para>
+ <para> Improves repeated reads to match network speeds instead of disk speeds</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295488" xreflabel=""/> Provides the building blocks for OST write cache (small-write aggregation)</para>
+ <para> Provides the building blocks for OST write cache (small-write aggregation)</para>
</listitem>
</itemizedlist>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438271_pgfId-1295497" xreflabel=""/>31.2.7.1 Using OSS Read Cache</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295566" xreflabel=""/>OSS read cache is implemented on the OSS, and does not require any special support on the client side. Since OSS read cache uses the memory available in the Linux page cache, you should use I/O patterns to determine the appropriate amount of memory for the cache; if the data is mostly reads, then more cache is required than for writes.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295331" xreflabel=""/>OSS read cache is enabled, by default, and managed by the following tunables:</para>
+ <title>31.2.7.1 Using OSS Read Cache</title>
+ <para>OSS read cache is implemented on the OSS, and does not require any special support on the client side. Since OSS read cache uses the memory available in the Linux page cache, you should use I/O patterns to determine the appropriate amount of memory for the cache; if the data is mostly reads, then more cache is required than for writes.</para>
+ <para>OSS read cache is enabled, by default, and managed by the following tunables:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296609" xreflabel=""/>read_cache_enable controls whether data read from disk during a read request is kept in memory and available for later read requests for the same data, without having to re-read it from disk. By default, read cache is enabled (read_cache_enable = 1).</para>
+ <para>read_cache_enable controls whether data read from disk during a read request is kept in memory and available for later read requests for the same data, without having to re-read it from disk. By default, read cache is enabled (read_cache_enable = 1).</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296624" xreflabel=""/>When the OSS receives a read request from a client, it reads data from disk into its memory and sends the data as a reply to the requests. If read cache is enabled, this data stays in memory after the client's request is finished, and the OSS skips reading data from disk when subsequent read requests for the same are received. The read cache is managed by the Linux kernel globally across all OSTs on that OSS, and the least recently used cache pages will be dropped from memory when the amount of free memory is running low.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295918" xreflabel=""/>If read cache is disabled (read_cache_enable = 0), then the OSS will discard the data after the client's read requests are serviced and, for subsequent read requests, the OSS must read the data from disk.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296677" xreflabel=""/>To disable read cache on all OSTs of an OSS, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1296709" xreflabel=""/>root@oss1# lctl set_param obdfilter.*.read_cache_enable=0
+ <para>When the OSS receives a read request from a client, it reads data from disk into its memory and sends the data as a reply to the requests. If read cache is enabled, this data stays in memory after the client's request is finished, and the OSS skips reading data from disk when subsequent read requests for the same are received. The read cache is managed by the Linux kernel globally across all OSTs on that OSS, and the least recently used cache pages will be dropped from memory when the amount of free memory is running low.</para>
+ <para>If read cache is disabled (read_cache_enable = 0), then the OSS will discard the data after the client's read requests are serviced and, for subsequent read requests, the OSS must read the data from disk.</para>
+ <para>To disable read cache on all OSTs of an OSS, run:</para>
+ <screen>root@oss1# lctl set_param obdfilter.*.read_cache_enable=0
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296681" xreflabel=""/>To re-enable read cache on one OST, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1296722" xreflabel=""/>root@oss1# lctl set_param obdfilter.{OST_name}.read_cache_enable=1
+ <para>To re-enable read cache on one OST, run:</para>
+ <screen>root@oss1# lctl set_param obdfilter.{OST_name}.read_cache_enable=1
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296685" xreflabel=""/>To check if read cache is enabled on all OSTs on an OSS, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1296687" xreflabel=""/>root@oss1# lctl get_param obdfilter.*.read_cache_enable
+ <para>To check if read cache is enabled on all OSTs on an OSS, run:</para>
+ <screen>root@oss1# lctl get_param obdfilter.*.read_cache_enable
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296775" xreflabel=""/>writethrough_cache_enable controls whether data sent to the OSS as a write request is kept in the read cache and available for later reads, or if it is discarded from cache when the write is completed. By default, writethrough cache is enabled (writethrough_cache_enable = 1).</para>
+ <para>writethrough_cache_enable controls whether data sent to the OSS as a write request is kept in the read cache and available for later reads, or if it is discarded from cache when the write is completed. By default, writethrough cache is enabled (writethrough_cache_enable = 1).</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296844" xreflabel=""/>When the OSS receives write requests from a client, it receives data from the client into its memory and writes the data to disk. If writethrough cache is enabled, this data stays in memory after the write request is completed, allowing the OSS to skip reading this data from disk if a later read request, or partial-page write request, for the same data is received.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296871" xreflabel=""/>If writethrough cache is disabled (writethrough_cache_enabled = 0), then the OSS discards the data after the client's write request is completed, and for subsequent read request, or partial-page write request, the OSS must re-read the data from disk.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296892" xreflabel=""/>Enabling writethrough cache is advisable if clients are doing small or unaligned writes that would cause partial-page updates, or if the files written by one node are immediately being accessed by other nodes. Some examples where this might be useful include producer-consumer I/O models or shared-file writes with a different node doing I/O not aligned on 4096-byte boundaries. Disabling writethrough cache is advisable in the case where files are mostly written to the file system but are not re-read within a short time period, or files are only written and re-read by the same node, regardless of whether the I/O is aligned or not.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296922" xreflabel=""/>To disable writethrough cache on all OSTs of an OSS, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1296924" xreflabel=""/>root@oss1# lctl set_param obdfilter.*.writethrough_cache_enable=0
+ <para>When the OSS receives write requests from a client, it receives data from the client into its memory and writes the data to disk. If writethrough cache is enabled, this data stays in memory after the write request is completed, allowing the OSS to skip reading this data from disk if a later read request, or partial-page write request, for the same data is received.</para>
+ <para>If writethrough cache is disabled (writethrough_cache_enabled = 0), then the OSS discards the data after the client's write request is completed, and for subsequent read request, or partial-page write request, the OSS must re-read the data from disk.</para>
+ <para>Enabling writethrough cache is advisable if clients are doing small or unaligned writes that would cause partial-page updates, or if the files written by one node are immediately being accessed by other nodes. Some examples where this might be useful include producer-consumer I/O models or shared-file writes with a different node doing I/O not aligned on 4096-byte boundaries. Disabling writethrough cache is advisable in the case where files are mostly written to the file system but are not re-read within a short time period, or files are only written and re-read by the same node, regardless of whether the I/O is aligned or not.</para>
+ <para>To disable writethrough cache on all OSTs of an OSS, run:</para>
+ <screen>root@oss1# lctl set_param obdfilter.*.writethrough_cache_enable=0
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296926" xreflabel=""/>To re-enable writethrough cache on one OST, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1296928" xreflabel=""/>root@oss1# lctl set_param \obdfilter.{OST_name}.writethrough_cache_enable=1
+ <para>To re-enable writethrough cache on one OST, run:</para>
+ <screen>root@oss1# lctl set_param \obdfilter.{OST_name}.writethrough_cache_enable=1
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296930" xreflabel=""/>To check if writethrough cache is</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1297043" xreflabel=""/>root@oss1# lctl set_param obdfilter.*.writethrough_cache_enable=1
+ <para>To check if writethrough cache is</para>
+ <screen>root@oss1# lctl set_param obdfilter.*.writethrough_cache_enable=1
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297052" xreflabel=""/>readcache_max_filesize controls the maximum size of a file that both the read cache and writethrough cache will try to keep in memory. Files larger than readcache_max_filesize will not be kept in cache for either reads or writes.</para>
+ <para>readcache_max_filesize controls the maximum size of a file that both the read cache and writethrough cache will try to keep in memory. Files larger than readcache_max_filesize will not be kept in cache for either reads or writes.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297105" xreflabel=""/>This can be very useful for workloads where relatively small files are repeatedly accessed by many clients, such as job startup files, executables, log files, etc., but large files are read or written only once. By not putting the larger files into the cache, it is much more likely that more of the smaller files will remain in cache for a longer time.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297126" xreflabel=""/>When setting readcache_max_filesize, the input value can be specified in bytes, or can have a suffix to indicate other binary units such as <emphasis role="bold">K</emphasis>ilobytes, <emphasis role="bold">M</emphasis>egabytes, <emphasis role="bold">G</emphasis>igabytes, <emphasis role="bold">T</emphasis>erabytes, or <emphasis role="bold">P</emphasis>etabytes.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297068" xreflabel=""/>To limit the maximum cached file size to 32MB on all OSTs of an OSS, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1297070" xreflabel=""/>root@oss1# lctl set_param obdfilter.*.readcache_max_filesize=32M
+ <para>This can be very useful for workloads where relatively small files are repeatedly accessed by many clients, such as job startup files, executables, log files, etc., but large files are read or written only once. By not putting the larger files into the cache, it is much more likely that more of the smaller files will remain in cache for a longer time.</para>
+ <para>When setting readcache_max_filesize, the input value can be specified in bytes, or can have a suffix to indicate other binary units such as <emphasis role="bold">K</emphasis>ilobytes, <emphasis role="bold">M</emphasis>egabytes, <emphasis role="bold">G</emphasis>igabytes, <emphasis role="bold">T</emphasis>erabytes, or <emphasis role="bold">P</emphasis>etabytes.</para>
+ <para>To limit the maximum cached file size to 32MB on all OSTs of an OSS, run:</para>
+ <screen>root@oss1# lctl set_param obdfilter.*.readcache_max_filesize=32M
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297072" xreflabel=""/>To disable the maximum cached file size on an OST, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1297074" xreflabel=""/>root@oss1# lctl set_param \obdfilter.{OST_name}.readcache_max_filesize=-1
+ <para>To disable the maximum cached file size on an OST, run:</para>
+ <screen>root@oss1# lctl set_param \obdfilter.{OST_name}.readcache_max_filesize=-1
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297076" xreflabel=""/>To check the current maximum cached file size on all OSTs of an OSS, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1297049" xreflabel=""/>root@oss1# lctl get_param obdfilter.*.readcache_max_filesize
+ <para>To check the current maximum cached file size on all OSTs of an OSS, run:</para>
+ <screen>root@oss1# lctl get_param obdfilter.*.readcache_max_filesize
</screen>
</section>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438271_pgfId-1300258" xreflabel=""/>31.2.8 OSS Asynchronous Journal Commit</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300265" xreflabel=""/>The OSS asynchronous journal commit feature synchronously writes data to disk without forcing a journal flush. This reduces the number of seeks and significantly improves performance on some hardware.</para>
+ <title>31.2.8 OSS Asynchronous Journal Commit</title>
+ <para>The OSS asynchronous journal commit feature synchronously writes data to disk without forcing a journal flush. This reduces the number of seeks and significantly improves performance on some hardware.</para>
<note><para>Asynchronous journal commit cannot work with O_DIRECT writes, a journal flush is still forced.</para></note>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300718" xreflabel=""/>When asynchronous journal commit is enabled, client nodes keep data in the page cache (a page reference). Lustre clients monitor the last committed transaction number (transno) in messages sent from the OSS to the clients. When a client sees that the last committed transno reported by the OSS is >=bulk write transno, it releases the reference on the corresponding pages. To avoid page references being held for too long on clients after a bulk write, a 7 second ping request is scheduled (jbd commit time is 5 seconds) after the bulk write reply is received, so the OSS has an opportunity to report the last committed transno.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300661" xreflabel=""/>If the OSS crashes before the journal commit occurs, then the intermediate data is lost. However, new OSS recovery functionality (introduced in the asynchronous journal commit feature), causes clients to replay their write requests and compensate for the missing disk updates by restoring the state of the file system.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300287" xreflabel=""/>To enable asynchronous journal commit, set the sync_journal parameter to zero (sync_journal=0):</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1300289" xreflabel=""/>$ lctl set_param obdfilter.*.sync_journal=0
-<anchor xml:id="dbdoclet.50438271_pgfId-1300410" xreflabel=""/>obdfilter.lol-OST0001.sync_journal=0
+ <para>When asynchronous journal commit is enabled, client nodes keep data in the page cache (a page reference). Lustre clients monitor the last committed transaction number (transno) in messages sent from the OSS to the clients. When a client sees that the last committed transno reported by the OSS is >=bulk write transno, it releases the reference on the corresponding pages. To avoid page references being held for too long on clients after a bulk write, a 7 second ping request is scheduled (jbd commit time is 5 seconds) after the bulk write reply is received, so the OSS has an opportunity to report the last committed transno.</para>
+ <para>If the OSS crashes before the journal commit occurs, then the intermediate data is lost. However, new OSS recovery functionality (introduced in the asynchronous journal commit feature), causes clients to replay their write requests and compensate for the missing disk updates by restoring the state of the file system.</para>
+ <para>To enable asynchronous journal commit, set the sync_journal parameter to zero (sync_journal=0):</para>
+ <screen>$ lctl set_param obdfilter.*.sync_journal=0
+obdfilter.lol-OST0001.sync_journal=0
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300292" xreflabel=""/>By default, sync_journal is disabled (sync_journal=1), which forces a journal flush after every bulk write.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300727" xreflabel=""/>When asynchronous journal commit is used, clients keep a page reference until the journal transaction commits. This can cause problems when a client receives a blocking callback, because pages need to be removed from the page cache, but they cannot be removed because of the extra page reference.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300728" xreflabel=""/>This problem is solved by forcing a journal flush on lock cancellation. When this happens, the client is granted the metadata blocks that have hit the disk, and it can safely release the page reference before processing the blocking callback. The parameter which controls this action is sync_on_lock_cancel, which can be set to the following values:</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300441" xreflabel=""/>always: Always force a journal flush on lock cancellation</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300530" xreflabel=""/>blocking: Force a journal flush only when the local cancellation is due to a blocking callback</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300533" xreflabel=""/>never: Do not force any journal flush</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300297" xreflabel=""/>Here is an example of sync_on_lock_cancel being set not to force a journal flush:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1300298" xreflabel=""/>$ lctl get_param obdfilter.*.sync_on_lock_cancel
-<anchor xml:id="dbdoclet.50438271_pgfId-1300299" xreflabel=""/>obdfilter.lol-OST0001.sync_on_lock_cancel=never
+ <para>By default, sync_journal is disabled (sync_journal=1), which forces a journal flush after every bulk write.</para>
+ <para>When asynchronous journal commit is used, clients keep a page reference until the journal transaction commits. This can cause problems when a client receives a blocking callback, because pages need to be removed from the page cache, but they cannot be removed because of the extra page reference.</para>
+ <para>This problem is solved by forcing a journal flush on lock cancellation. When this happens, the client is granted the metadata blocks that have hit the disk, and it can safely release the page reference before processing the blocking callback. The parameter which controls this action is sync_on_lock_cancel, which can be set to the following values:</para>
+ <para>always: Always force a journal flush on lock cancellation</para>
+ <para>blocking: Force a journal flush only when the local cancellation is due to a blocking callback</para>
+ <para>never: Do not force any journal flush</para>
+ <para>Here is an example of sync_on_lock_cancel being set not to force a journal flush:</para>
+ <screen>$ lctl get_param obdfilter.*.sync_on_lock_cancel
+obdfilter.lol-OST0001.sync_on_lock_cancel=never
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300317" xreflabel=""/>By default, sync_on_lock_cancel is set to never, because asynchronous journal commit is disabled by default.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300592" xreflabel=""/>When asynchronous journal commit is enabled (sync_journal=0), sync_on_lock_cancel is automatically set to always, if it was previously set to never.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1300596" xreflabel=""/>Similarly, when asynchronous journal commit is disabled, (sync_journal=1), sync_on_lock_cancel is enforced to never.</para>
+ <para>By default, sync_on_lock_cancel is set to never, because asynchronous journal commit is disabled by default.</para>
+ <para>When asynchronous journal commit is enabled (sync_journal=0), sync_on_lock_cancel is automatically set to always, if it was previously set to never.</para>
+ <para>Similarly, when asynchronous journal commit is disabled, (sync_journal=1), sync_on_lock_cancel is enforced to never.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1297046" xreflabel=""/>31.2.9 mballoc <anchor xml:id="dbdoclet.50438271_marker-1297045" xreflabel=""/>History</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290687" xreflabel=""/><emphasis role="bold">/proc/fs/ldiskfs/sda/mb_history</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290688" xreflabel=""/>Multi-Block-Allocate (mballoc), enables Lustre to ask ldiskfs to allocate multiple blocks with a single request to the block allocator. Typically, an ldiskfs file system allocates only one block per time. Each mballoc-enabled partition has this file. This is sample output:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290689" xreflabel=""/>pid inode goal result found grps cr \ me\
+ <title>31.2.9 mballoc <anchor xml:id="dbdoclet.50438271_marker-1297045" xreflabel=""/>History</title>
+ <para><emphasis role="bold">/proc/fs/ldiskfs/sda/mb_history</emphasis></para>
+ <para>Multi-Block-Allocate (mballoc), enables Lustre to ask ldiskfs to allocate multiple blocks with a single request to the block allocator. Typically, an ldiskfs file system allocates only one block per time. Each mballoc-enabled partition has this file. This is sample output:</para>
+ <screen>pid inode goal result found grps cr \ me\
rge tail broken
-<anchor xml:id="dbdoclet.50438271_pgfId-1290690" xreflabel=""/>2838 139267 17/12288/1 17/12288/1 1 0 0 \
+2838 139267 17/12288/1 17/12288/1 1 0 0 \
\ M 1 8192
-<anchor xml:id="dbdoclet.50438271_pgfId-1290691" xreflabel=""/>2838 139267 17/12289/1 17/12289/1 1 0 0 \
+2838 139267 17/12289/1 17/12289/1 1 0 0 \
\ M 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290692" xreflabel=""/>2838 139267 17/12290/1 17/12290/1 1 0 0 \
+2838 139267 17/12290/1 17/12290/1 1 0 0 \
\ M 1 2
-<anchor xml:id="dbdoclet.50438271_pgfId-1290693" xreflabel=""/>2838 24577 3/12288/1 3/12288/1 1 0 0 \
+2838 24577 3/12288/1 3/12288/1 1 0 0 \
\ M 1 8192
-<anchor xml:id="dbdoclet.50438271_pgfId-1290694" xreflabel=""/>2838 24578 3/12288/1 3/771/1 1 1 1 \
+2838 24578 3/12288/1 3/771/1 1 1 1 \
\ 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290695" xreflabel=""/>2838 32769 4/12288/1 4/12288/1 1 0 0 \
+2838 32769 4/12288/1 4/12288/1 1 0 0 \
\ M 1 8192
-<anchor xml:id="dbdoclet.50438271_pgfId-1290696" xreflabel=""/>2838 32770 4/12288/1 4/12289/1 13 1 1 \
+2838 32770 4/12288/1 4/12289/1 13 1 1 \
\ 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290697" xreflabel=""/>2838 32771 4/12288/1 5/771/1 26 2 1 \
+2838 32771 4/12288/1 5/771/1 26 2 1 \
\ 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290698" xreflabel=""/>2838 32772 4/12288/1 5/896/1 31 2 1 \
+2838 32772 4/12288/1 5/896/1 31 2 1 \
\ 1 128
-<anchor xml:id="dbdoclet.50438271_pgfId-1290699" xreflabel=""/>2838 32773 4/12288/1 5/897/1 31 2 1 \
+2838 32773 4/12288/1 5/897/1 31 2 1 \
\ 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290700" xreflabel=""/>2828 32774 4/12288/1 5/898/1 31 2 1 \
+2828 32774 4/12288/1 5/898/1 31 2 1 \
\ 1 2
-<anchor xml:id="dbdoclet.50438271_pgfId-1290701" xreflabel=""/>2838 32775 4/12288/1 5/899/1 31 2 1 \
+2838 32775 4/12288/1 5/899/1 31 2 1 \
\ 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290702" xreflabel=""/>2838 32776 4/12288/1 5/900/1 31 2 1 \
+2838 32776 4/12288/1 5/900/1 31 2 1 \
\ 1 4
-<anchor xml:id="dbdoclet.50438271_pgfId-1290703" xreflabel=""/>2838 32777 4/12288/1 5/901/1 31 2 1 \
+2838 32777 4/12288/1 5/901/1 31 2 1 \
\ 0 0
-<anchor xml:id="dbdoclet.50438271_pgfId-1290704" xreflabel=""/>2838 32778 4/12288/1 5/902/1 31 2 1 \
+2838 32778 4/12288/1 5/902/1 31 2 1 \
\ 1 2
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290791" xreflabel=""/>The parameters are described below:</para>
+ <para>The parameters are described below:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1292192" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1292194" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292196" xreflabel=""/><emphasis role="bold">pid</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292198" xreflabel=""/>Process that made the allocation.</para></entry>
+ <entry><para> <emphasis role="bold">pid</emphasis></para></entry>
+ <entry><para> Process that made the allocation.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292200" xreflabel=""/><emphasis role="bold">inode</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292202" xreflabel=""/>inode number allocated blocks</para></entry>
+ <entry><para> <emphasis role="bold">inode</emphasis></para></entry>
+ <entry><para> inode number allocated blocks</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292204" xreflabel=""/><emphasis role="bold">goal</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292206" xreflabel=""/>Initial request that came to mballoc (group/block-in-group/number-of-blocks)</para></entry>
+ <entry><para> <emphasis role="bold">goal</emphasis></para></entry>
+ <entry><para> Initial request that came to mballoc (group/block-in-group/number-of-blocks)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292208" xreflabel=""/><emphasis role="bold">result</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292210" xreflabel=""/>What mballoc actually found for this request.</para></entry>
+ <entry><para> <emphasis role="bold">result</emphasis></para></entry>
+ <entry><para> What mballoc actually found for this request.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292212" xreflabel=""/><emphasis role="bold">found</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292214" xreflabel=""/>Number of free chunks mballoc found and measured before the final decision.</para></entry>
+ <entry><para> <emphasis role="bold">found</emphasis></para></entry>
+ <entry><para> Number of free chunks mballoc found and measured before the final decision.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292216" xreflabel=""/><emphasis role="bold">grps</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292218" xreflabel=""/>Number of groups mballoc scanned to satisfy the request.</para></entry>
+ <entry><para> <emphasis role="bold">grps</emphasis></para></entry>
+ <entry><para> Number of groups mballoc scanned to satisfy the request.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292220" xreflabel=""/><emphasis role="bold">cr</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292222" xreflabel=""/>Stage at which mballoc found the result:</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292223" xreflabel=""/><emphasis role="bold">0</emphasis> - best in terms of resource allocation. The request was 1MB or larger and was satisfied directly via the kernel buddy allocator.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292224" xreflabel=""/><emphasis role="bold">1</emphasis> - regular stage (good at resource consumption)</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292225" xreflabel=""/><emphasis role="bold">2</emphasis> - fs is quite fragmented (not that bad at resource consumption)</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292226" xreflabel=""/><emphasis role="bold">3</emphasis> - fs is very fragmented (worst at resource consumption)</para></entry>
+ <entry><para> <emphasis role="bold">cr</emphasis></para></entry>
+ <entry><para> Stage at which mballoc found the result:</para><para><emphasis role="bold">0</emphasis> - best in terms of resource allocation. The request was 1MB or larger and was satisfied directly via the kernel buddy allocator.</para><para><emphasis role="bold">1</emphasis> - regular stage (good at resource consumption)</para><para><emphasis role="bold">2</emphasis> - fs is quite fragmented (not that bad at resource consumption)</para><para><emphasis role="bold">3</emphasis> - fs is very fragmented (worst at resource consumption)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292228" xreflabel=""/><emphasis role="bold">queue</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292230" xreflabel=""/>Total bytes in active/queued sends.</para></entry>
+ <entry><para> <emphasis role="bold">queue</emphasis></para></entry>
+ <entry><para> Total bytes in active/queued sends.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292232" xreflabel=""/><emphasis role="bold">merge</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292234" xreflabel=""/>Whether the request hit the goal. This is good as extents code can now merge new blocks to existing extent, eliminating the need for extents tree growth.</para></entry>
+ <entry><para> <emphasis role="bold">merge</emphasis></para></entry>
+ <entry><para> Whether the request hit the goal. This is good as extents code can now merge new blocks to existing extent, eliminating the need for extents tree growth.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292236" xreflabel=""/><emphasis role="bold">tail</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292238" xreflabel=""/>Number of blocks left free after the allocation breaks large free chunks.</para></entry>
+ <entry><para> <emphasis role="bold">tail</emphasis></para></entry>
+ <entry><para> Number of blocks left free after the allocation breaks large free chunks.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292240" xreflabel=""/><emphasis role="bold">broken</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292242" xreflabel=""/>How large the broken chunk was.</para></entry>
+ <entry><para> <emphasis role="bold">broken</emphasis></para></entry>
+ <entry><para> How large the broken chunk was.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290792" xreflabel=""/>Most customers are probably interested in found/cr. If cr is 0 1 and found is less than 100, then mballoc is doing quite well.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290793" xreflabel=""/>Also, number-of-blocks-in-request (third number in the goal triple) can tell the number of blocks requested by the obdfilter. If the obdfilter is doing a lot of small requests (just few blocks), then either the client is processing input/output to a lot of small files, or something may be wrong with the client (because it is better if client sends large input/output requests). This can be investigated with the OSC rpc_stats or OST brw_stats mentioned above.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290794" xreflabel=""/>Number of groups scanned (grps column) should be small. If it reaches a few dozen often, then either your disk file system is pretty fragmented or mballoc is doing something wrong in the group selection part.</para>
+ <para>Most customers are probably interested in found/cr. If cr is 0 1 and found is less than 100, then mballoc is doing quite well.</para>
+ <para>Also, number-of-blocks-in-request (third number in the goal triple) can tell the number of blocks requested by the obdfilter. If the obdfilter is doing a lot of small requests (just few blocks), then either the client is processing input/output to a lot of small files, or something may be wrong with the client (because it is better if client sends large input/output requests). This can be investigated with the OSC rpc_stats or OST brw_stats mentioned above.</para>
+ <para>Number of groups scanned (grps column) should be small. If it reaches a few dozen often, then either your disk file system is pretty fragmented or mballoc is doing something wrong in the group selection part.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1290796" xreflabel=""/>31.2.10 mballoc3<anchor xml:id="dbdoclet.50438271_marker-1290795" xreflabel=""/> Tunables</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290800" xreflabel=""/>Lustre version 1.6.1 and later includes mballoc3, which was built on top of mballoc2. By default, mballoc3 is enabled, and adds these features:</para>
+ <title>31.2.10 mballoc3<anchor xml:id="dbdoclet.50438271_marker-1290795" xreflabel=""/> Tunables</title>
+ <para>Lustre version 1.6.1 and later includes mballoc3, which was built on top of mballoc2. By default, mballoc3 is enabled, and adds these features:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290801" xreflabel=""/> Pre-allocation for single files (helps to resist fragmentation)</para>
+ <para> Pre-allocation for single files (helps to resist fragmentation)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290802" xreflabel=""/> Pre-allocation for a group of files (helps to pack small files into large, contiguous chunks)</para>
+ <para> Pre-allocation for a group of files (helps to pack small files into large, contiguous chunks)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290803" xreflabel=""/> Stream allocation (helps to decrease the seek rate)</para>
+ <para> Stream allocation (helps to decrease the seek rate)</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290830" xreflabel=""/>The following mballoc3 tunables are available:</para>
+ <para>The following mballoc3 tunables are available:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1292250" xreflabel=""/>Field</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1292252" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Field</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292283" xreflabel=""/><emphasis role="bold">stats</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292285" xreflabel=""/>Enables/disables the collection of statistics. Collected statistics can be found</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292286" xreflabel=""/>in /proc/fs/ldiskfs2/<dev>/mb_history.</para></entry>
+ <entry><para> <emphasis role="bold">stats</emphasis></para></entry>
+ <entry><para> Enables/disables the collection of statistics. Collected statistics can be found</para><para>in /proc/fs/ldiskfs2/<dev>/mb_history.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292288" xreflabel=""/><emphasis role="bold">max_to_scan</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292290" xreflabel=""/>Maximum number of free chunks that mballoc finds before a final decision to avoid livelock.</para></entry>
+ <entry><para> <emphasis role="bold">max_to_scan</emphasis></para></entry>
+ <entry><para> Maximum number of free chunks that mballoc finds before a final decision to avoid livelock.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292292" xreflabel=""/><emphasis role="bold">min_to_scan</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292294" xreflabel=""/>Minimum number of free chunks that mballoc finds before a final decision. This is useful for a very small request, to resist fragmentation of big free chunks.</para></entry>
+ <entry><para> <emphasis role="bold">min_to_scan</emphasis></para></entry>
+ <entry><para> Minimum number of free chunks that mballoc finds before a final decision. This is useful for a very small request, to resist fragmentation of big free chunks.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292296" xreflabel=""/><emphasis role="bold">order2_req</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292298" xreflabel=""/>For requests equal to 2^N (where N >= order2_req), a very fast search via buddy structures is used.</para></entry>
+ <entry><para> <emphasis role="bold">order2_req</emphasis></para></entry>
+ <entry><para> For requests equal to 2^N (where N >= order2_req), a very fast search via buddy structures is used.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292300" xreflabel=""/><emphasis role="bold">stream_req</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292302" xreflabel=""/>Requests smaller or equal to this value are packed together to form large write I/Os.</para></entry>
+ <entry><para> <emphasis role="bold">stream_req</emphasis></para></entry>
+ <entry><para> Requests smaller or equal to this value are packed together to form large write I/Os.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290831" xreflabel=""/>The following tunables, providing more control over allocation policy, will be available in the next version:</para>
+ <para>The following tunables, providing more control over allocation policy, will be available in the next version:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1292371" xreflabel=""/>Field</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1292373" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Field</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292405" xreflabel=""/><emphasis role="bold">stats</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292407" xreflabel=""/>Enables/disables the collection of statistics. Collected statistics can be found in /proc/fs/ldiskfs2/<dev>/mb_history.</para></entry>
+ <entry><para> <emphasis role="bold">stats</emphasis></para></entry>
+ <entry><para> Enables/disables the collection of statistics. Collected statistics can be found in /proc/fs/ldiskfs2/<dev>/mb_history.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292409" xreflabel=""/><emphasis role="bold">max_to_scan</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292411" xreflabel=""/>Maximum number of free chunks that mballoc finds before a final decision to avoid livelock.</para></entry>
+ <entry><para> <emphasis role="bold">max_to_scan</emphasis></para></entry>
+ <entry><para> Maximum number of free chunks that mballoc finds before a final decision to avoid livelock.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292413" xreflabel=""/><emphasis role="bold">min_to_scan</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292415" xreflabel=""/>Minimum number of free chunks that mballoc finds before a final decision. This is useful for a very small request, to resist fragmentation of big free chunks.</para></entry>
+ <entry><para> <emphasis role="bold">min_to_scan</emphasis></para></entry>
+ <entry><para> Minimum number of free chunks that mballoc finds before a final decision. This is useful for a very small request, to resist fragmentation of big free chunks.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292417" xreflabel=""/><emphasis role="bold">order2_req</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292419" xreflabel=""/>For requests equal to 2^N (where N >= order2_req), a very fast search via buddy structures is used.</para></entry>
+ <entry><para> <emphasis role="bold">order2_req</emphasis></para></entry>
+ <entry><para> For requests equal to 2^N (where N >= order2_req), a very fast search via buddy structures is used.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292392" xreflabel=""/><emphasis role="bold">small_req</emphasis></para></entry>
- <entry morerows="1"><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292427" xreflabel=""/>All requests are divided into 3 categories:</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292428" xreflabel=""/>< small_req (packed together to form large, aggregated requests)</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292429" xreflabel=""/>< large_req (allocated mostly in linearly)</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292430" xreflabel=""/>> large_req (very large requests so the arm seek does not matter)</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292431" xreflabel=""/>The idea is that we try to pack small requests to form large requests, and then place all large requests (including compound from the small ones) close to one another, causing as few arm seeks as possible.</para></entry>
+ <entry><para> <emphasis role="bold">small_req</emphasis></para></entry>
+ <entry morerows="1"><para> All requests are divided into 3 categories:</para><para>< small_req (packed together to form large, aggregated requests)</para><para>< large_req (allocated mostly in linearly)</para><para>> large_req (very large requests so the arm seek does not matter)</para><para>The idea is that we try to pack small requests to form large requests, and then place all large requests (including compound from the small ones) close to one another, causing as few arm seeks as possible.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292437" xreflabel=""/><emphasis role="bold">large_req</emphasis></para></entry>
+ <entry><para> <emphasis role="bold">large_req</emphasis></para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292449" xreflabel=""/><emphasis role="bold">prealloc_table</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292451" xreflabel=""/>The amount of space to preallocate depends on the current file size. The idea is that for small files we do not need 1 MB preallocations and for large files, 1 MB preallocations are not large enough; it is better to preallocate 4 MB.</para></entry>
+ <entry><para> <emphasis role="bold">prealloc_table</emphasis></para></entry>
+ <entry><para> The amount of space to preallocate depends on the current file size. The idea is that for small files we do not need 1 MB preallocations and for large files, 1 MB preallocations are not large enough; it is better to preallocate 4 MB.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292453" xreflabel=""/><emphasis role="bold">group_prealloc</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292455" xreflabel=""/>The amount of space preallocated for small requests to be grouped.</para></entry>
+ <entry><para> <emphasis role="bold">group_prealloc</emphasis></para></entry>
+ <entry><para> The amount of space preallocated for small requests to be grouped.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1290875" xreflabel=""/>31.2.11 <anchor xml:id="dbdoclet.50438271_13474" xreflabel=""/>Lo<anchor xml:id="dbdoclet.50438271_marker-1290874" xreflabel=""/>cking</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290876" xreflabel=""/><emphasis role="bold">/proc/fs/lustre/ldlm/ldlm/namespaces/<OSC name|MDC name>/lru_size</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290877" xreflabel=""/>The lru_size parameter is used to control the number of client-side locks in an LRU queue. LRU size is dynamic, based on load. This optimizes the number of locks available to nodes that have different workloads (e.g., login/build nodes vs. compute nodes vs. backup nodes).</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297289" xreflabel=""/>The total number of locks available is a function of the server's RAM. The default limit is 50 locks/1 MB of RAM. If there is too much memory pressure, then the LRU size is shrunk. The number of locks on the server is limited to {number of OST/MDT on node} * {number of clients} * {client lru_size}.</para>
+ <title>31.2.11 <anchor xml:id="dbdoclet.50438271_13474" xreflabel=""/>Lo<anchor xml:id="dbdoclet.50438271_marker-1290874" xreflabel=""/>cking</title>
+ <para><emphasis role="bold">/proc/fs/lustre/ldlm/ldlm/namespaces/<OSC name|MDC name>/lru_size</emphasis></para>
+ <para>The lru_size parameter is used to control the number of client-side locks in an LRU queue. LRU size is dynamic, based on load. This optimizes the number of locks available to nodes that have different workloads (e.g., login/build nodes vs. compute nodes vs. backup nodes).</para>
+ <para>The total number of locks available is a function of the server's RAM. The default limit is 50 locks/1 MB of RAM. If there is too much memory pressure, then the LRU size is shrunk. The number of locks on the server is limited to {number of OST/MDT on node} * {number of clients} * {client lru_size}.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294671" xreflabel=""/> To enable automatic LRU sizing, set the lru_size parameter to 0. In this case, the lru_size parameter shows the current number of locks being used on the export. (In Lustre 1.6.5.1 and later, LRU sizing is enabled, by default.)</para>
+ <para> To enable automatic LRU sizing, set the lru_size parameter to 0. In this case, the lru_size parameter shows the current number of locks being used on the export. (In Lustre 1.6.5.1 and later, LRU sizing is enabled, by default.)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294717" xreflabel=""/> To specify a maximum number of locks, set the lru_size parameter to a value > 0 (former numbers are okay, 100 * CPU_NR). We recommend that you only increase the LRU size on a few login nodes where users access the file system interactively.</para>
+ <para> To specify a maximum number of locks, set the lru_size parameter to a value > 0 (former numbers are okay, 100 * CPU_NR). We recommend that you only increase the LRU size on a few login nodes where users access the file system interactively.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290878" xreflabel=""/>To clear the LRU on a single client, and as a result flush client cache, without changing the lru_size value:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1294737" xreflabel=""/>$ lctl set_param ldlm.namespaces.<osc_name|mdc_name>.lru_size=clear
+ <para>To clear the LRU on a single client, and as a result flush client cache, without changing the lru_size value:</para>
+ <screen>$ lctl set_param ldlm.namespaces.<osc_name|mdc_name>.lru_size=clear
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290880" xreflabel=""/>If you shrink the LRU size below the number of existing unused locks, then the unused locks are canceled immediately. Use echo clear to cancel all locks without changing the value.</para>
+ <para>If you shrink the LRU size below the number of existing unused locks, then the unused locks are canceled immediately. Use echo clear to cancel all locks without changing the value.</para>
<note><para>Currently, the lru_size parameter can only be set temporarily with lctl set_param; it cannot be set permanently.</para></note>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294998" xreflabel=""/>To disable LRU sizing, run this command on the Lustre clients:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1295001" xreflabel=""/>$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
+ <para>To disable LRU sizing, run this command on the Lustre clients:</para>
+ <screen>$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295002" xreflabel=""/>Replace NR_CPU value with the number of CPUs on the node.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297273" xreflabel=""/>To determine the number of locks being granted:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1297276" xreflabel=""/>$ lctl get_param ldlm.namespaces.*.pool.limit
+ <para>Replace NR_CPU value with the number of CPUs on the node.</para>
+ <para>To determine the number of locks being granted:</para>
+ <screen>$ lctl get_param ldlm.namespaces.*.pool.limit
</screen>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438271_pgfId-1296246" xreflabel=""/>31.2.12 <anchor xml:id="dbdoclet.50438271_87260" xreflabel=""/>Setting MDS and OSS Thread Counts</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1298326" xreflabel=""/>MDS and OSS thread counts (minimum and maximum) can be set via the {min,max}_thread_count tunable. For each service, a new /proc/fs/lustre/{service}/*/thread_{min,max,started} entry is created. The tunable, {service}.thread_{min,max,started}, can be used to set the minimum and maximum thread counts or get the current number of running threads for the following services.</para>
+ <title>31.2.12 <anchor xml:id="dbdoclet.50438271_87260" xreflabel=""/>Setting MDS and OSS Thread Counts</title>
+ <para>MDS and OSS thread counts (minimum and maximum) can be set via the {min,max}_thread_count tunable. For each service, a new /proc/fs/lustre/{service}/*/thread_{min,max,started} entry is created. The tunable, {service}.thread_{min,max,started}, can be used to set the minimum and maximum thread counts or get the current number of running threads for the following services.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298639" xreflabel=""/><emphasis role="bold">Service</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298641" xreflabel=""/><emphasis role="bold">Description</emphasis></para></entry>
+ <entry><para> <emphasis role="bold">Service</emphasis></para></entry>
+ <entry><para> <emphasis role="bold">Description</emphasis></para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298556" xreflabel=""/>mdt.MDS.mds</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298558" xreflabel=""/>normal metadata ops</para></entry>
+ <entry><para> mdt.MDS.mds</para></entry>
+ <entry><para> normal metadata ops</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298560" xreflabel=""/>mdt.MDS.mds_readpage</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298562" xreflabel=""/>metadata readdir</para></entry>
+ <entry><para> mdt.MDS.mds_readpage</para></entry>
+ <entry><para> metadata readdir</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298564" xreflabel=""/>mdt.MDS.mds_setattr</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298566" xreflabel=""/>metadata setattr</para></entry>
+ <entry><para> mdt.MDS.mds_setattr</para></entry>
+ <entry><para> metadata setattr</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298581" xreflabel=""/>ost.OSS.ost</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298583" xreflabel=""/>normal data</para></entry>
+ <entry><para> ost.OSS.ost</para></entry>
+ <entry><para> normal data</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298585" xreflabel=""/>ost.OSS.ost_io</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298587" xreflabel=""/>bulk data IO</para></entry>
+ <entry><para> ost.OSS.ost_io</para></entry>
+ <entry><para> bulk data IO</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298589" xreflabel=""/>ost.OSS.ost_create</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298591" xreflabel=""/>OST object pre-creation service</para></entry>
+ <entry><para> ost.OSS.ost_create</para></entry>
+ <entry><para> OST object pre-creation service</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298593" xreflabel=""/>ldlm.services.ldlm_canceld</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298595" xreflabel=""/>DLM lock cancel</para></entry>
+ <entry><para> ldlm.services.ldlm_canceld</para></entry>
+ <entry><para> DLM lock cancel</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298568" xreflabel=""/>ldlm.services.ldlm_cbd</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1298570" xreflabel=""/>DLM lock grant</para></entry>
+ <entry><para> ldlm.services.ldlm_cbd</para></entry>
+ <entry><para> DLM lock grant</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299411" xreflabel=""/> To temporarily set this tunable, run:</para>
+ <para> To temporarily set this tunable, run:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1299412" xreflabel=""/># lctl {get,set}_param {service}.thread_{min,max,started}
+ <screen># lctl {get,set}_param {service}.thread_{min,max,started}
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299413" xreflabel=""/> To permanently set this tunable, run:</para>
+ <para> To permanently set this tunable, run:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1299414" xreflabel=""/># lctl conf_param {service}.thread_{min,max,started} </screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299409" xreflabel=""/>The following examples show how to set thread counts and get the number of running threads for the ost_io service.</para>
+ <screen># lctl conf_param {service}.thread_{min,max,started} </screen>
+ <para>The following examples show how to set thread counts and get the number of running threads for the ost_io service.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299429" xreflabel=""/> To get the number of running threads, run:</para>
+ <para> To get the number of running threads, run:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1299270" xreflabel=""/># lctl get_param ost.OSS.ost_io.threads_started</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299276" xreflabel=""/>The command output will be similar to this:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1299282" xreflabel=""/>ost.OSS.ost_io.threads_started=128
+ <screen># lctl get_param ost.OSS.ost_io.threads_started</screen>
+ <para>The command output will be similar to this:</para>
+ <screen>ost.OSS.ost_io.threads_started=128
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299257" xreflabel=""/> To set the maximum number of threads (512), run:</para>
+ <para> To set the maximum number of threads (512), run:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1299294" xreflabel=""/># lctl get_param ost.OSS.ost_io.threads_max
+ <screen># lctl get_param ost.OSS.ost_io.threads_max
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299330" xreflabel=""/>The command output will be:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1299331" xreflabel=""/>ost.OSS.ost_io.threads_max=512
+ <para>The command output will be:</para>
+ <screen>ost.OSS.ost_io.threads_max=512
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299306" xreflabel=""/> To set the maximum thread count to 256 instead of 512 (to avoid overloading the storage or for an array with requests), run:</para>
+ <para> To set the maximum thread count to 256 instead of 512 (to avoid overloading the storage or for an array with requests), run:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1299316" xreflabel=""/># lctl set_param ost.OSS.ost_io.threads_max=256
+ <screen># lctl set_param ost.OSS.ost_io.threads_max=256
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299352" xreflabel=""/>The command output will be:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1299353" xreflabel=""/>ost.OSS.ost_io.threads_max=256
+ <para>The command output will be:</para>
+ <screen>ost.OSS.ost_io.threads_max=256
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299368" xreflabel=""/> To check if the new threads_max setting is active, run:</para>
+ <para> To check if the new threads_max setting is active, run:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1299375" xreflabel=""/># lctl get_param ost.OSS.ost_io.threads_max
+ <screen># lctl get_param ost.OSS.ost_io.threads_max
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1299381" xreflabel=""/>The command output will be similar to this:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1299382" xreflabel=""/>ost.OSS.ost_io.threads_max=256
+ <para>The command output will be similar to this:</para>
+ <screen>ost.OSS.ost_io.threads_max=256
</screen>
<note><para>Currently, the maximum thread count setting is advisory because Lustre does not reduce the number of service threads in use, even if that number exceeds the threads_max value. Lustre does not stop service threads once they are started.</para></note>
</section>
</section>
<section xml:id="dbdoclet.50438271_83523">
<title>31.3 Debug</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290884" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/debug</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1298213" xreflabel=""/>By default, Lustre generates a detailed log of all operations to aid in debugging. The level of debugging can affect the performance or speed you achieve with Lustre. Therefore, it is useful to reduce this overhead by turning down the debug level<footnote><para><anchor xml:id="dbdoclet.50438271_pgfId-1298216" xreflabel=""/>This controls the level of Lustre debugging kept in the internal log buffer. It does not alter the level of debugging that goes to syslog.</para></footnote> to improve performance. Raise the debug level when you need to collect the logs for debugging problems. The debugging mask can be set with "symbolic names" instead of the numerical values that were used in prior releases. The new symbolic format is shown in the examples below.</para>
+ <para><emphasis role="bold">/proc/sys/lnet/debug</emphasis></para>
+ <para>By default, Lustre generates a detailed log of all operations to aid in debugging. The level of debugging can affect the performance or speed you achieve with Lustre. Therefore, it is useful to reduce this overhead by turning down the debug level<footnote><para>This controls the level of Lustre debugging kept in the internal log buffer. It does not alter the level of debugging that goes to syslog.</para></footnote> to improve performance. Raise the debug level when you need to collect the logs for debugging problems. The debugging mask can be set with "symbolic names" instead of the numerical values that were used in prior releases. The new symbolic format is shown in the examples below.</para>
<note><para>All of the commands below must be run as root; note the # nomenclature.</para></note>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1298218" xreflabel=""/>To verify the debug level used by examining the sysctl that controls debugging, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1297891" xreflabel=""/># sysctl lnet.debug
-<anchor xml:id="dbdoclet.50438271_pgfId-1297995" xreflabel=""/>lnet.debug = ioctl neterror warning error emerg ha config console
+ <para>To verify the debug level used by examining the sysctl that controls debugging, run:</para>
+ <screen># sysctl lnet.debug
+lnet.debug = ioctl neterror warning error emerg ha config console
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297894" xreflabel=""/>To turn off debugging (except for network error debugging), run this command on all concerned nodes:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1297895" xreflabel=""/># sysctl -w lnet.debug="neterror"
-<anchor xml:id="dbdoclet.50438271_pgfId-1297938" xreflabel=""/>lnet.debug = neterror
+ <para>To turn off debugging (except for network error debugging), run this command on all concerned nodes:</para>
+ <screen># sysctl -w lnet.debug="neterror"
+lnet.debug = neterror
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1298239" xreflabel=""/>To turn off debugging completely, run this command on all concerned nodes:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1298240" xreflabel=""/># sysctl -w lnet.debug=0
-<anchor xml:id="dbdoclet.50438271_pgfId-1298657" xreflabel=""/>lnet.debug = 0
+ <para>To turn off debugging completely, run this command on all concerned nodes:</para>
+ <screen># sysctl -w lnet.debug=0
+lnet.debug = 0
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1298237" xreflabel=""/>To set an appropriate debug level for a production environment, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1298260" xreflabel=""/># sysctl -w lnet.debug="warning dlmtrace error emerg ha rpctrace vfstrace"
-<anchor xml:id="dbdoclet.50438271_pgfId-1298658" xreflabel=""/>lnet.debug = warning dlmtrace error emerg ha rpctrace vfstrace
+ <para>To set an appropriate debug level for a production environment, run:</para>
+ <screen># sysctl -w lnet.debug="warning dlmtrace error emerg ha rpctrace vfstrace"
+lnet.debug = warning dlmtrace error emerg ha rpctrace vfstrace
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1298258" xreflabel=""/>The flags above collect enough high-level information to aid debugging, but they do not cause any serious performance impact.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1298069" xreflabel=""/>To clear all flags and set new ones, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1297899" xreflabel=""/># sysctl -w lnet.debug="warning"
-<anchor xml:id="dbdoclet.50438271_pgfId-1297944" xreflabel=""/>lnet.debug = warning
+ <para>The flags above collect enough high-level information to aid debugging, but they do not cause any serious performance impact.</para>
+ <para>To clear all flags and set new ones, run:</para>
+ <screen># sysctl -w lnet.debug="warning"
+lnet.debug = warning
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297902" xreflabel=""/>To add new flags to existing ones, prefix them with a "+":</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1297903" xreflabel=""/># sysctl -w lnet.debug="+neterror +ha"
-<anchor xml:id="dbdoclet.50438271_pgfId-1297950" xreflabel=""/>lnet.debug = +neterror +ha
-<anchor xml:id="dbdoclet.50438271_pgfId-1297905" xreflabel=""/># sysctl lnet.debug
-<anchor xml:id="dbdoclet.50438271_pgfId-1297953" xreflabel=""/>lnet.debug = neterror warning ha
+ <para>To add new flags to existing ones, prefix them with a "+":</para>
+ <screen># sysctl -w lnet.debug="+neterror +ha"
+lnet.debug = +neterror +ha
+# sysctl lnet.debug
+lnet.debug = neterror warning ha
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297908" xreflabel=""/>To remove flags, prefix them with a "-":</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1297909" xreflabel=""/># sysctl -w lnet.debug="-ha"
-<anchor xml:id="dbdoclet.50438271_pgfId-1297962" xreflabel=""/>lnet.debug = -ha
-<anchor xml:id="dbdoclet.50438271_pgfId-1297911" xreflabel=""/># sysctl lnet.debug
-<anchor xml:id="dbdoclet.50438271_pgfId-1297974" xreflabel=""/>lnet.debug = neterror warning
+ <para>To remove flags, prefix them with a "-":</para>
+ <screen># sysctl -w lnet.debug="-ha"
+lnet.debug = -ha
+# sysctl lnet.debug
+lnet.debug = neterror warning
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297914" xreflabel=""/>You can verify and change the debug level using the /proc interface in Lustre. To use the flags with /proc, run:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1297916" xreflabel=""/># cat /proc/sys/lnet/debug
-<anchor xml:id="dbdoclet.50438271_pgfId-1298027" xreflabel=""/>neterror warning
-<anchor xml:id="dbdoclet.50438271_pgfId-1298104" xreflabel=""/># echo "+ha" > /proc/sys/lnet/debug
-<anchor xml:id="dbdoclet.50438271_pgfId-1298108" xreflabel=""/># cat /proc/sys/lnet/debug
-<anchor xml:id="dbdoclet.50438271_pgfId-1298037" xreflabel=""/>neterror warning ha
-<anchor xml:id="dbdoclet.50438271_pgfId-1297921" xreflabel=""/># echo "-warning" > /proc/sys/lnet/debug
-<anchor xml:id="dbdoclet.50438271_pgfId-1297922" xreflabel=""/># cat /proc/sys/lnet/debug
-<anchor xml:id="dbdoclet.50438271_pgfId-1298040" xreflabel=""/>neterror ha
+ <para>You can verify and change the debug level using the /proc interface in Lustre. To use the flags with /proc, run:</para>
+ <screen># cat /proc/sys/lnet/debug
+neterror warning
+# echo "+ha" > /proc/sys/lnet/debug
+# cat /proc/sys/lnet/debug
+neterror warning ha
+# echo "-warning" > /proc/sys/lnet/debug
+# cat /proc/sys/lnet/debug
+neterror ha
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290897" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/subsystem_debug</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290898" xreflabel=""/>This controls the debug logs3 for subsystems (see S_* definitions).</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290899" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/debug_path</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290900" xreflabel=""/>This indicates the location where debugging symbols should be stored for gdb. The default is set to /r/tmp/lustre-log-localhost.localdomain.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290901" xreflabel=""/>These values can also be set via sysctl -w lnet.debug={value}</para>
+ <para><emphasis role="bold">/proc/sys/lnet/subsystem_debug</emphasis></para>
+ <para>This controls the debug logs3 for subsystems (see S_* definitions).</para>
+ <para><emphasis role="bold">/proc/sys/lnet/debug_path</emphasis></para>
+ <para>This indicates the location where debugging symbols should be stored for gdb. The default is set to /r/tmp/lustre-log-localhost.localdomain.</para>
+ <para>These values can also be set via sysctl -w lnet.debug={value}</para>
<note><para>The above entries only exist when Lustre has already been loaded.</para></note>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294968" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/panic_on_lbug</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294969" xreflabel=""/>This causes Lustre to call ''panic'' when it detects an internal problem (an LBUG); panic crashes the node. This is particularly useful when a kernel crash dump utility is configured. The crash dump is triggered when the internal inconsistency is detected by Lustre.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294977" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/upcall</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294978" xreflabel=""/>This allows you to specify the path to the binary which will be invoked when an LBUG is encountered. This binary is called with four parameters. The first one is the string ''LBUG''. The second one is the file where the LBUG occurred. The third one is the function name. The fourth one is the line number in the file.</para>
+ <para><emphasis role="bold">/proc/sys/lnet/panic_on_lbug</emphasis></para>
+ <para>This causes Lustre to call ''panic'' when it detects an internal problem (an LBUG); panic crashes the node. This is particularly useful when a kernel crash dump utility is configured. The crash dump is triggered when the internal inconsistency is detected by Lustre.</para>
+ <para><emphasis role="bold">/proc/sys/lnet/upcall</emphasis></para>
+ <para>This allows you to specify the path to the binary which will be invoked when an LBUG is encountered. This binary is called with four parameters. The first one is the string ''LBUG''. The second one is the file where the LBUG occurred. The third one is the function name. The fourth one is the line number in the file.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438271_pgfId-1290905" xreflabel=""/>31.3.1 RPC Information for Other OBD Devices</title>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290906" xreflabel=""/>Some OBD devices maintain a count of the number of RPC events that they process. Sometimes these events are more specific to operations of the device, like llite, than actual raw RPC counts.</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290907" xreflabel=""/>$ find /proc/fs/lustre/ -name stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290908" xreflabel=""/>/proc/fs/lustre/osc/lustre-OST0001-osc-ce63ca00/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290909" xreflabel=""/>/proc/fs/lustre/osc/lustre-OST0000-osc-ce63ca00/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290910" xreflabel=""/>/proc/fs/lustre/osc/lustre-OST0001-osc/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290911" xreflabel=""/>/proc/fs/lustre/osc/lustre-OST0000-osc/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290912" xreflabel=""/>/proc/fs/lustre/mdt/MDS/mds_readpage/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290913" xreflabel=""/>/proc/fs/lustre/mdt/MDS/mds_setattr/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290914" xreflabel=""/>/proc/fs/lustre/mdt/MDS/mds/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290915" xreflabel=""/>/proc/fs/lustre/mds/lustre-MDT0000/exports/ab206805-0630-6647-8543-d24265c9\
+ <title>31.3.1 RPC Information for Other OBD Devices</title>
+ <para>Some OBD devices maintain a count of the number of RPC events that they process. Sometimes these events are more specific to operations of the device, like llite, than actual raw RPC counts.</para>
+ <screen>$ find /proc/fs/lustre/ -name stats
+/proc/fs/lustre/osc/lustre-OST0001-osc-ce63ca00/stats
+/proc/fs/lustre/osc/lustre-OST0000-osc-ce63ca00/stats
+/proc/fs/lustre/osc/lustre-OST0001-osc/stats
+/proc/fs/lustre/osc/lustre-OST0000-osc/stats
+/proc/fs/lustre/mdt/MDS/mds_readpage/stats
+/proc/fs/lustre/mdt/MDS/mds_setattr/stats
+/proc/fs/lustre/mdt/MDS/mds/stats
+/proc/fs/lustre/mds/lustre-MDT0000/exports/ab206805-0630-6647-8543-d24265c9\
1a3d/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290916" xreflabel=""/>/proc/fs/lustre/mds/lustre-MDT0000/exports/08ac6584-6c4a-3536-2c6d-b36cf9cb\
+/proc/fs/lustre/mds/lustre-MDT0000/exports/08ac6584-6c4a-3536-2c6d-b36cf9cb\
daa0/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290917" xreflabel=""/>/proc/fs/lustre/mds/lustre-MDT0000/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290918" xreflabel=""/>/proc/fs/lustre/ldlm/services/ldlm_canceld/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290919" xreflabel=""/>/proc/fs/lustre/ldlm/services/ldlm_cbd/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290920" xreflabel=""/>/proc/fs/lustre/llite/lustre-ce63ca00/stats
+/proc/fs/lustre/mds/lustre-MDT0000/stats
+/proc/fs/lustre/ldlm/services/ldlm_canceld/stats
+/proc/fs/lustre/ldlm/services/ldlm_cbd/stats
+/proc/fs/lustre/llite/lustre-ce63ca00/stats
</screen>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438271_pgfId-1290921" xreflabel=""/>31.3.1.1 Interpreting OST Statistics</title>
+ <title>31.3.1.1 Interpreting OST Statistics</title>
<note><para>See also <xref linkend="dbdoclet.50438219_84890"/> (llobdstat) and <xref linkend="dbdoclet.50438273_80593"/> (CollectL).</para></note>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1301139" xreflabel=""/>The OST .../stats files can be used to track client statistics (client activity) for each OST. It is possible to get a periodic dump of values from these file (for example, every 10 seconds), that show the RPC rates (similar to iostat) by using the llstat.pl tool:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290922" xreflabel=""/># llstat /proc/fs/lustre/osc/lustre-OST0000-osc/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290923" xreflabel=""/>/usr/bin/llstat: STATS on 09/14/07 /proc/fs/lustre/osc/lustre-OST0000-osc/s\
+ <para>The OST .../stats files can be used to track client statistics (client activity) for each OST. It is possible to get a periodic dump of values from these file (for example, every 10 seconds), that show the RPC rates (similar to iostat) by using the llstat.pl tool:</para>
+ <screen># llstat /proc/fs/lustre/osc/lustre-OST0000-osc/stats
+/usr/bin/llstat: STATS on 09/14/07 /proc/fs/lustre/osc/lustre-OST0000-osc/s\
tats on 192.168.10.34@tcp
-<anchor xml:id="dbdoclet.50438271_pgfId-1290924" xreflabel=""/>snapshot_time 1189732762.835363
-<anchor xml:id="dbdoclet.50438271_pgfId-1290925" xreflabel=""/>ost_create 1
-<anchor xml:id="dbdoclet.50438271_pgfId-1290926" xreflabel=""/>ost_get_info 1
-<anchor xml:id="dbdoclet.50438271_pgfId-1290927" xreflabel=""/>ost_connect 1
-<anchor xml:id="dbdoclet.50438271_pgfId-1290928" xreflabel=""/>ost_set_info 1
-<anchor xml:id="dbdoclet.50438271_pgfId-1290929" xreflabel=""/>obd_ping 212
+snapshot_time 1189732762.835363
+ost_create 1
+ost_get_info 1
+ost_connect 1
+ost_set_info 1
+obd_ping 212
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290930" xreflabel=""/>To clear the statistics, give the -c option to llstat.pl. To specify how frequently the statistics should be cleared (in seconds), use an integer for the -i option. This is sample output with -c and -i10 options used, providing statistics every 10s):</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1290931" xreflabel=""/>$ llstat -c -i10 /proc/fs/lustre/ost/OSS/ost_io/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1290932" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1290933" xreflabel=""/>/usr/bin/llstat: STATS on 06/06/07 /proc/fs/lustre/ost/OSS/ost_io/ stats on\
+ <para>To clear the statistics, give the -c option to llstat.pl. To specify how frequently the statistics should be cleared (in seconds), use an integer for the -i option. This is sample output with -c and -i10 options used, providing statistics every 10s):</para>
+ <screen>$ llstat -c -i10 /proc/fs/lustre/ost/OSS/ost_io/stats
+
+/usr/bin/llstat: STATS on 06/06/07 /proc/fs/lustre/ost/OSS/ost_io/ stats on\
192.168.16.35@tcp
-<anchor xml:id="dbdoclet.50438271_pgfId-1290934" xreflabel=""/>snapshot_time 1181074093.276072
-<anchor xml:id="dbdoclet.50438271_pgfId-1290935" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1290936" xreflabel=""/>/proc/fs/lustre/ost/OSS/ost_io/stats @ 1181074103.284895
-<anchor xml:id="dbdoclet.50438271_pgfId-1290937" xreflabel=""/>Name Cur.Count Cur.Rate #Events Unit \
+snapshot_time 1181074093.276072
+
+/proc/fs/lustre/ost/OSS/ost_io/stats @ 1181074103.284895
+Name Cur.Count Cur.Rate #Events Unit \
\last min avg max stddev
-<anchor xml:id="dbdoclet.50438271_pgfId-1290938" xreflabel=""/>req_waittime 8 0 8 [usec] \
+req_waittime 8 0 8 [usec] \
2078\ 34 259.75 868 317.49
-<anchor xml:id="dbdoclet.50438271_pgfId-1290939" xreflabel=""/>req_qdepth 8 0 8 [reqs] 1\ \
+req_qdepth 8 0 8 [reqs] 1\ \
0 0.12 1 0.35
-<anchor xml:id="dbdoclet.50438271_pgfId-1290940" xreflabel=""/>req_active 8 0 8 [reqs] 11\ \
+req_active 8 0 8 [reqs] 11\ \
1 1.38 2 0.52
-<anchor xml:id="dbdoclet.50438271_pgfId-1290941" xreflabel=""/>reqbuf_avail 8 0 8 [bufs] \
+reqbuf_avail 8 0 8 [bufs] \
511\ 63 63.88 64 0.35
-<anchor xml:id="dbdoclet.50438271_pgfId-1290942" xreflabel=""/>ost_write 8 0 8 [bytes] 1697677\\
+ost_write 8 0 8 [bytes] 1697677\\
72914 212209.62 387579 91874.29
-<anchor xml:id="dbdoclet.50438271_pgfId-1290943" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1290944" xreflabel=""/>/proc/fs/lustre/ost/OSS/ost_io/stats @ 1181074113.290180
-<anchor xml:id="dbdoclet.50438271_pgfId-1290945" xreflabel=""/>Name Cur.Count Cur.Rate #Events Unit \
+
+/proc/fs/lustre/ost/OSS/ost_io/stats @ 1181074113.290180
+Name Cur.Count Cur.Rate #Events Unit \
\last min avg max stddev
-<anchor xml:id="dbdoclet.50438271_pgfId-1290946" xreflabel=""/>req_waittime 31 3 39 [usec] \
+req_waittime 31 3 39 [usec] \
30011\ 34 822.79 12245 2047.71
-<anchor xml:id="dbdoclet.50438271_pgfId-1290947" xreflabel=""/>req_qdepth 31 3 39 [reqs] 0\ \
+req_qdepth 31 3 39 [reqs] 0\ \
0 0.03 1 0.16
-<anchor xml:id="dbdoclet.50438271_pgfId-1290948" xreflabel=""/>req_active 31 3 39 [reqs] 58\ \
+req_active 31 3 39 [reqs] 58\ \
1 1.77 3 0.74
-<anchor xml:id="dbdoclet.50438271_pgfId-1290949" xreflabel=""/>reqbuf_avail 31 3 39 [bufs] \
+reqbuf_avail 31 3 39 [bufs] \
1977\ 63 63.79 64 0.41
-<anchor xml:id="dbdoclet.50438271_pgfId-1290950" xreflabel=""/>ost_write 30 3 38 [bytes] 10284679\
+ost_write 30 3 38 [bytes] 10284679\
\ 15019 315325.16 910694 197776.51
-<anchor xml:id="dbdoclet.50438271_pgfId-1290951" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438271_pgfId-1290952" xreflabel=""/>/proc/fs/lustre/ost/OSS/ost_io/stats @ 1181074123.325560
-<anchor xml:id="dbdoclet.50438271_pgfId-1290953" xreflabel=""/>Name Cur.Count Cur.Rate #Events Unit \
+
+/proc/fs/lustre/ost/OSS/ost_io/stats @ 1181074123.325560
+Name Cur.Count Cur.Rate #Events Unit \
\last min avg max stddev
-<anchor xml:id="dbdoclet.50438271_pgfId-1290954" xreflabel=""/>req_waittime 21 2 60 [usec] \
+req_waittime 21 2 60 [usec] \
14970\ 34 784.32 12245 1878.66
-<anchor xml:id="dbdoclet.50438271_pgfId-1290955" xreflabel=""/>req_qdepth 21 2 60 [reqs] 0\ \
+req_qdepth 21 2 60 [reqs] 0\ \
0 0.02 1 0.13
-<anchor xml:id="dbdoclet.50438271_pgfId-1290956" xreflabel=""/>req_active 21 2 60 [reqs] 33\ \
+req_active 21 2 60 [reqs] 33\ \
1 1.70 3 0.70
-<anchor xml:id="dbdoclet.50438271_pgfId-1290957" xreflabel=""/>reqbuf_avail 21 2 60 [bufs] \
+reqbuf_avail 21 2 60 [bufs] \
1341\ 63 63.82 64 0.39
-<anchor xml:id="dbdoclet.50438271_pgfId-1290958" xreflabel=""/>ost_write 21 2 59 [bytes] 7648424\\
+ost_write 21 2 59 [bytes] 7648424\\
15019 332725.08 910694 180397.87
</screen>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1292572" xreflabel=""/>Where:</para>
+ <para>Where:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1292580" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1292582" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292631" xreflabel=""/><emphasis role="bold">Cur. Count</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292633" xreflabel=""/>Number of events of each type sent in the last interval (in this example, 10s)</para></entry>
+ <entry><para> <emphasis role="bold">Cur. Count</emphasis></para></entry>
+ <entry><para> Number of events of each type sent in the last interval (in this example, 10s)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292635" xreflabel=""/><emphasis role="bold">Cur. Rate</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292637" xreflabel=""/>Number of events per second in the last interval</para></entry>
+ <entry><para> <emphasis role="bold">Cur. Rate</emphasis></para></entry>
+ <entry><para> Number of events per second in the last interval</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292639" xreflabel=""/><emphasis role="bold">#Events</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292641" xreflabel=""/>Total number of such events since the system started</para></entry>
+ <entry><para> <emphasis role="bold">#Events</emphasis></para></entry>
+ <entry><para> Total number of such events since the system started</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292643" xreflabel=""/><emphasis role="bold">Unit</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292645" xreflabel=""/>Unit of measurement for that statistic (microseconds, requests, buffers)</para></entry>
+ <entry><para> <emphasis role="bold">Unit</emphasis></para></entry>
+ <entry><para> Unit of measurement for that statistic (microseconds, requests, buffers)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292647" xreflabel=""/><emphasis role="bold">last</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292649" xreflabel=""/>Average rate of these events (in units/event) for the last interval during which they arrived. For instance, in the above mentioned case of ost_destroy it took an average of 736 microseconds per destroy for the 400 object destroys in the previous 10 seconds.</para></entry>
+ <entry><para> <emphasis role="bold">last</emphasis></para></entry>
+ <entry><para> Average rate of these events (in units/event) for the last interval during which they arrived. For instance, in the above mentioned case of ost_destroy it took an average of 736 microseconds per destroy for the 400 object destroys in the previous 10 seconds.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292651" xreflabel=""/><emphasis role="bold">min</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292653" xreflabel=""/>Minimum rate (in units/events) since the service started</para></entry>
+ <entry><para> <emphasis role="bold">min</emphasis></para></entry>
+ <entry><para> Minimum rate (in units/events) since the service started</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292655" xreflabel=""/><emphasis role="bold">avg</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292657" xreflabel=""/>Average rate</para></entry>
+ <entry><para> <emphasis role="bold">avg</emphasis></para></entry>
+ <entry><para> Average rate</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292659" xreflabel=""/><emphasis role="bold">max</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292661" xreflabel=""/>Maximum rate</para></entry>
+ <entry><para> <emphasis role="bold">max</emphasis></para></entry>
+ <entry><para> Maximum rate</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292663" xreflabel=""/><emphasis role="bold">stddev</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292665" xreflabel=""/>Standard deviation (not measured in all cases)</para></entry>
+ <entry><para> <emphasis role="bold">stddev</emphasis></para></entry>
+ <entry><para> Standard deviation (not measured in all cases)</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1292739" xreflabel=""/>The events common to all services are:</para>
+ <para>The events common to all services are:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1292673" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1292675" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292720" xreflabel=""/><emphasis role="bold">req_waittime</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292722" xreflabel=""/>Amount of time a request waited in the queue before being handled by an available server thread.</para></entry>
+ <entry><para> <emphasis role="bold">req_waittime</emphasis></para></entry>
+ <entry><para> Amount of time a request waited in the queue before being handled by an available server thread.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292724" xreflabel=""/><emphasis role="bold">req_qdepth</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292726" xreflabel=""/>Number of requests waiting to be handled in the queue for this service.</para></entry>
+ <entry><para> <emphasis role="bold">req_qdepth</emphasis></para></entry>
+ <entry><para> Number of requests waiting to be handled in the queue for this service.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292728" xreflabel=""/><emphasis role="bold">req_active</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292730" xreflabel=""/>Number of requests currently being handled.</para></entry>
+ <entry><para> <emphasis role="bold">req_active</emphasis></para></entry>
+ <entry><para> Number of requests currently being handled.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292732" xreflabel=""/><emphasis role="bold">reqbuf_avail</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292734" xreflabel=""/>Number of unsolicited lnet request buffers for this service.</para></entry>
+ <entry><para> <emphasis role="bold">reqbuf_avail</emphasis></para></entry>
+ <entry><para> Number of unsolicited lnet request buffers for this service.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1291033" xreflabel=""/>Some service-specific events of interest are:</para>
+ <para>Some service-specific events of interest are:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1292473" xreflabel=""/>Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438271_pgfId-1292475" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292507" xreflabel=""/><emphasis role="bold">ldlm_enqueue</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292509" xreflabel=""/>Time it takes to enqueue a lock (this includes file open on the MDS)</para></entry>
+ <entry><para> <emphasis role="bold">ldlm_enqueue</emphasis></para></entry>
+ <entry><para> Time it takes to enqueue a lock (this includes file open on the MDS)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292511" xreflabel=""/><emphasis role="bold">mds_reint</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292513" xreflabel=""/>Time it takes to process an MDS modification record (includes create, mkdir, unlink, rename and setattr)</para></entry>
+ <entry><para> <emphasis role="bold">mds_reint</emphasis></para></entry>
+ <entry><para> Time it takes to process an MDS modification record (includes create, mkdir, unlink, rename and setattr)</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438271_pgfId-1291461" xreflabel=""/>31.3.1.2 Interpreting MDT Statistics</title>
+ <title>31.3.1.2 Interpreting MDT Statistics</title>
<note><para>See also <xref linkend="dbdoclet.50438219_84890"/> (llobdstat) and <xref linkend="dbdoclet.50438273_80593"/> (CollectL).</para></note>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297382" xreflabel=""/>The MDT .../stats files can be used to track MDT statistics for the MDS. Here is sample output for an MDT stats file:</para>
- <screen><anchor xml:id="dbdoclet.50438271_pgfId-1297438" xreflabel=""/># cat /proc/fs/lustre/mds/*-MDT0000/stats
-<anchor xml:id="dbdoclet.50438271_pgfId-1297466" xreflabel=""/>snapshot_time 1244832003.676892 secs.usecs
-<anchor xml:id="dbdoclet.50438271_pgfId-1297469" xreflabel=""/>open 2 samples [reqs]
-<anchor xml:id="dbdoclet.50438271_pgfId-1297472" xreflabel=""/>close 1 samples [reqs]
-<anchor xml:id="dbdoclet.50438271_pgfId-1297475" xreflabel=""/>getxattr 3 samples [reqs]
-<anchor xml:id="dbdoclet.50438271_pgfId-1297478" xreflabel=""/>process_config 1 samples [reqs]
-<anchor xml:id="dbdoclet.50438271_pgfId-1297481" xreflabel=""/>connect 2 samples [reqs]
-<anchor xml:id="dbdoclet.50438271_pgfId-1297484" xreflabel=""/>disconnect 2 samples [reqs]
-<anchor xml:id="dbdoclet.50438271_pgfId-1297487" xreflabel=""/>statfs 3 samples [reqs]
-<anchor xml:id="dbdoclet.50438271_pgfId-1297490" xreflabel=""/>setattr 1 samples [reqs]
-<anchor xml:id="dbdoclet.50438271_pgfId-1297493" xreflabel=""/>getattr 3 samples [reqs]
-<anchor xml:id="dbdoclet.50438271_pgfId-1297496" xreflabel=""/>llog_init 6 samples [reqs]
-<anchor xml:id="dbdoclet.50438271_pgfId-1297499" xreflabel=""/>notify 16 samples [reqs]
+ <para>The MDT .../stats files can be used to track MDT statistics for the MDS. Here is sample output for an MDT stats file:</para>
+ <screen># cat /proc/fs/lustre/mds/*-MDT0000/stats
+snapshot_time 1244832003.676892 secs.usecs
+open 2 samples [reqs]
+close 1 samples [reqs]
+getxattr 3 samples [reqs]
+process_config 1 samples [reqs]
+connect 2 samples [reqs]
+disconnect 2 samples [reqs]
+statfs 3 samples [reqs]
+setattr 1 samples [reqs]
+getattr 3 samples [reqs]
+llog_init 6 samples [reqs]
+notify 16 samples [reqs]
</screen>
</section>
</section>
<info>
<title xml:id='lustreprogramminginterfaces.title'>Lustre Programming Interfaces</title>
</info>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1293209" xreflabel=""/>This chapter describes public programming interfaces to control various aspects of Lustre from userspace. These interfaces are generally not guaranteed to remain unchanged over time, although we will make an effort to notify the user community well in advance of major changes. This chapter includes the following section:</para>
+ <para>This chapter describes public programming interfaces to control various aspects of Lustre from userspace. These interfaces are generally not guaranteed to remain unchanged over time, although we will make an effort to notify the user community well in advance of major changes. This chapter includes the following section:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438291_32926"/></para>
<section xml:id="dbdoclet.50438291_32926">
<title>33.1 User/Group <anchor xml:id="dbdoclet.50438291_marker-1293215" xreflabel=""/>Cache Upcall</title>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1293217" xreflabel=""/>This section describes user and group upcall.</para>
+ <para>This section describes user and group upcall.</para>
<note><para>For information on a universal UID/GID, see <link xl:href="InstallingLustre.html#50438261_19503">Environmental Requirements</link>.</para></note>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438291_pgfId-1293218" xreflabel=""/>33.1.1 Name</title>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1293219" xreflabel=""/>Use /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall to look up a given user's group membership.</para>
+ <title>33.1.1 Name</title>
+ <para>Use /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall to look up a given user's group membership.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438291_pgfId-1293220" xreflabel=""/>33.1.2 Description</title>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1293221" xreflabel=""/>The group upcall file contains the path to an executable that, when installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the mds_grp_downcall_data data structure (see <link xl:href="LustreProgrammingInterfaces.html#50438291_33759">Data Structures</link>) and write it to the /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info pseudo-file.</para>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1293225" xreflabel=""/>For a sample upcall program, see lustre/utils/l_getgroups.c in the Lustre source distribution.</para>
+ <title>33.1.2 Description</title>
+ <para>The group upcall file contains the path to an executable that, when installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the mds_grp_downcall_data data structure (see <link xl:href="LustreProgrammingInterfaces.html#50438291_33759">Data Structures</link>) and write it to the /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info pseudo-file.</para>
+ <para>For a sample upcall program, see lustre/utils/l_getgroups.c in the Lustre source distribution.</para>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438291_pgfId-1293226" xreflabel=""/>33.1.2.1 Primary and Secondary Groups</title>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1293227" xreflabel=""/>The mechanism for the primary/secondary group is as follows:</para>
+ <title>33.1.2.1 Primary and Secondary Groups</title>
+ <para>The mechanism for the primary/secondary group is as follows:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1293228" xreflabel=""/> The MDS issues an upcall (set per MDS) to map the numeric UID to the supplementary group(s).</para>
+ <para> The MDS issues an upcall (set per MDS) to map the numeric UID to the supplementary group(s).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1293229" xreflabel=""/> If there is no upcall or if there is an upcall and it fails, supplementary groups will be added as supplied by the client (as they are now).</para>
+ <para> If there is no upcall or if there is an upcall and it fails, supplementary groups will be added as supplied by the client (as they are now).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1293230" xreflabel=""/> The default upcall is /usr/sbin/l_getidentity, which can interact with the user/group database to obtain UID/GID/suppgid. The user/group database depends on authentication configuration, and can be local /etc/passwd, NIS, LDAP, etc. If necessary, the administrator can use a parse utility to set /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall. If the upcall interface is set to NONE, then upcall is disabled. The MDS uses the UID/GID/suppgid supplied by the client.</para>
+ <para> The default upcall is /usr/sbin/l_getidentity, which can interact with the user/group database to obtain UID/GID/suppgid. The user/group database depends on authentication configuration, and can be local /etc/passwd, NIS, LDAP, etc. If necessary, the administrator can use a parse utility to set /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall. If the upcall interface is set to NONE, then upcall is disabled. The MDS uses the UID/GID/suppgid supplied by the client.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1293231" xreflabel=""/> The default group upcall is set by mkfs.lustre. Use tunefs.lustre --param or echo{path}>/proc/fs/lustre/mds/{mdsname}/group_upcall</para>
+ <para> The default group upcall is set by mkfs.lustre. Use tunefs.lustre --param or echo{path}>/proc/fs/lustre/mds/{mdsname}/group_upcall</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1294341" xreflabel=""/> The Lustre administrator can specify permissions for a specific UID by configuring /etc/lustre/perm.conf on the MDS. As commented in lustre/utils/l_getidentity.c</para>
+ <para> The Lustre administrator can specify permissions for a specific UID by configuring /etc/lustre/perm.conf on the MDS. As commented in lustre/utils/l_getidentity.c</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438291_pgfId-1294527" xreflabel=""/>/** permission file format is like this: * {nid} {uid} {perms} * * '*' nid \
+ <screen>/** permission file format is like this: * {nid} {uid} {perms} * * '*' nid \
means any nid* '*' uid means any uid* the valid values for perms are:* setu\
id/setgid/setgrp/rmtacl -- enable corresponding perm* nosetuid/nosetgid/nos\
etgrp/normtacl -- disable corresponding perm* they can be listed together, \
is preferential,* when they are in different lines (items), the latter is \
preferential,* '*' nid is as default perm, and is not preferential.*/
</screen>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1294343" xreflabel=""/>Currently, rmtacl/normtacl can be ignored (part of security functionality), and used for remote clients. The /usr/sbin/l_getidentity utility can parse /etc/lustre/perm.conf to obtain permission mask for specified UID.</para>
+ <para>Currently, rmtacl/normtacl can be ignored (part of security functionality), and used for remote clients. The /usr/sbin/l_getidentity utility can parse /etc/lustre/perm.conf to obtain permission mask for specified UID.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1294268" xreflabel=""/> To avoid repeated upcalls, the MDS caches supplemental group information. Use /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_expire to set the cache time (default is 600 seconds). The kernel waits for the upcall to complete (at most, 5 seconds) and takes the "failure" behavior as described. Set the wait time in /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_acquire_expire (default is 15 seconds). Cached entries are flushed by writing to /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_flush.</para>
+ <para> To avoid repeated upcalls, the MDS caches supplemental group information. Use /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_expire to set the cache time (default is 600 seconds). The kernel waits for the upcall to complete (at most, 5 seconds) and takes the "failure" behavior as described. Set the wait time in /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_acquire_expire (default is 15 seconds). Cached entries are flushed by writing to /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_flush.</para>
</listitem>
</itemizedlist>
</section>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438291_pgfId-1293233" xreflabel=""/>33.1.3 Parameters</title>
+ <title>33.1.3 Parameters</title>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1293234" xreflabel=""/> Name of the MDS service</para>
+ <para> Name of the MDS service</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1293235" xreflabel=""/> Numeric UID</para>
+ <para> Numeric UID</para>
</listitem>
</itemizedlist>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438291_pgfId-1293237" xreflabel=""/>33.1.4 <anchor xml:id="dbdoclet.50438291_33759" xreflabel=""/>Data Structures</title>
- <screen><anchor xml:id="dbdoclet.50438291_pgfId-1293240" xreflabel=""/>struct identity_downcall_data {
-<anchor xml:id="dbdoclet.50438291_pgfId-1293241" xreflabel=""/> __u32 idd_magic;
-<anchor xml:id="dbdoclet.50438291_pgfId-1293242" xreflabel=""/> __u32 idd_err;
-<anchor xml:id="dbdoclet.50438291_pgfId-1293243" xreflabel=""/> __u32 idd_uid;
-<anchor xml:id="dbdoclet.50438291_pgfId-1293244" xreflabel=""/> __u32 idd_gid;
-<anchor xml:id="dbdoclet.50438291_pgfId-1293245" xreflabel=""/> __u32 idd_nperms;
-<anchor xml:id="dbdoclet.50438291_pgfId-1294332" xreflabel=""/> struct perm_downcall_data idd_perms[N_PERMS_MAX];
-<anchor xml:id="dbdoclet.50438291_pgfId-1294322" xreflabel=""/> __u32 idd_ngroups;
-<anchor xml:id="dbdoclet.50438291_pgfId-1293246" xreflabel=""/> __u32 idd_groups[0];
-<anchor xml:id="dbdoclet.50438291_pgfId-1293247" xreflabel=""/>};
+ <title>33.1.4 <anchor xml:id="dbdoclet.50438291_33759" xreflabel=""/>Data Structures</title>
+ <screen>struct identity_downcall_data {
+ __u32 idd_magic;
+ __u32 idd_err;
+ __u32 idd_uid;
+ __u32 idd_gid;
+ __u32 idd_nperms;
+ struct perm_downcall_data idd_perms[N_PERMS_MAX];
+ __u32 idd_ngroups;
+ __u32 idd_groups[0];
+};
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438291_73963">
<title>33.2 l_getgroups<anchor xml:id="dbdoclet.50438291_marker-1294565" xreflabel=""/> Utility</title>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1294567" xreflabel=""/>The l_getgroups utility handles Lustre user/group cache upcall.</para>
+ <para>The l_getgroups utility handles Lustre user/group cache upcall.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438291_pgfId-1294568" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438291_pgfId-1294569" xreflabel=""/>l_getgroups [-v] [-d|mdsname] uid]
-<anchor xml:id="dbdoclet.50438291_pgfId-1294570" xreflabel=""/>l_getgroups [-v] -s
+ <title>Synopsis</title>
+ <screen>l_getgroups [-v] [-d|mdsname] uid]
+l_getgroups [-v] -s
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438291_pgfId-1294571" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1294572" xreflabel=""/>The group upcall file contains the path to an executable that, when properly installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the mds_grp_downcall_data data structure (see Data structures) and write it to the /proc/fs/lustre/mds/mds-service/group_info pseudo-file.</para>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1294573" xreflabel=""/>l_getgroups is the reference implementation of the user/group cache upcall.</para>
+ <title>Description</title>
+ <para>The group upcall file contains the path to an executable that, when properly installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the mds_grp_downcall_data data structure (see Data structures) and write it to the /proc/fs/lustre/mds/mds-service/group_info pseudo-file.</para>
+ <para>l_getgroups is the reference implementation of the user/group cache upcall.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438291_pgfId-1294574" xreflabel=""/>Files</title>
- <para><anchor xml:id="dbdoclet.50438291_pgfId-1294575" xreflabel=""/>/proc/fs/lustre/mds/mds-service/group_upcall</para>
+ <title>Files</title>
+ <para>/proc/fs/lustre/mds/mds-service/group_upcall</para>
</section>
</section>
</chapter>
<info>
<title xml:id='lustrerecovery.title'>Lustre Recovery</title>
</info>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1292351" xreflabel=""/>This chapter describes how recovery is implemented in Lustre and includes the following sections:</para>
+ <para>This chapter describes how recovery is implemented in Lustre and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438268_58047"/></para>
</listitem>
<section xml:id="dbdoclet.50438268_58047">
<title>30.1 Recovery Overview</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291584" xreflabel=""/>Lustre's recovery feature is responsible for dealing with node or network failure and returning the cluster to a consistent, performant state. Because Lustre allows servers to perform asynchronous update operations to the on-disk file system (i.e., the server can reply without waiting for the update to synchronously commit to disk), the clients may have state in memory that is newer than what the server can recover from disk after a crash.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291585" xreflabel=""/>A handful of different types of failures can cause recovery to occur:</para>
+ <para>Lustre's recovery feature is responsible for dealing with node or network failure and returning the cluster to a consistent, performant state. Because Lustre allows servers to perform asynchronous update operations to the on-disk file system (i.e., the server can reply without waiting for the update to synchronously commit to disk), the clients may have state in memory that is newer than what the server can recover from disk after a crash.</para>
+ <para>A handful of different types of failures can cause recovery to occur:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291586" xreflabel=""/> Client (compute node) failure</para>
+ <para> Client (compute node) failure</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291587" xreflabel=""/> MDS failure (and failover)</para>
+ <para> MDS failure (and failover)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291588" xreflabel=""/> OST failure (and failover)</para>
+ <para> OST failure (and failover)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291589" xreflabel=""/> Transient network partition</para>
+ <para> Transient network partition</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291597" xreflabel=""/>Currently, all Lustre failure and recovery operations are based on the concept of connection failure; all imports or exports associated with a given connection are considered to fail if any of them fail.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1290652" xreflabel=""/>For information on Lustre recovery, see <xref linkend="dbdoclet.50438268_65824"/>. For information on recovering from a corrupt file system, see <xref linkend="dbdoclet.50438268_83826"/>. For information on resolving orphaned objects, a common issue after recovery, see <xref linkend='troubleshootingrecovery'/> (Working with Orphaned Objects).</para>
+ <para>Currently, all Lustre failure and recovery operations are based on the concept of connection failure; all imports or exports associated with a given connection are considered to fail if any of them fail.</para>
+ <para>For information on Lustre recovery, see <xref linkend="dbdoclet.50438268_65824"/>. For information on recovering from a corrupt file system, see <xref linkend="dbdoclet.50438268_83826"/>. For information on resolving orphaned objects, a common issue after recovery, see <xref linkend='troubleshootingrecovery'/> (Working with Orphaned Objects).</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438268_pgfId-1287395" xreflabel=""/>30.1.1 <anchor xml:id="dbdoclet.50438268_96839" xreflabel=""/>Client <anchor xml:id="dbdoclet.50438268_marker-1287394" xreflabel=""/>Failure</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287396" xreflabel=""/>Recovery from client failure in Lustre is based on lock revocation and other resources, so surviving clients can continue their work uninterrupted. If a client fails to timely respond to a blocking lock callback from the Distributed Lock Manager (DLM) or fails to communicate with the server in a long period of time (i.e., no pings), the client is forcibly removed from the cluster (evicted). This enables other clients to acquire locks blocked by the dead client's locks, and also frees resources (file handles, export data) associated with that client. Note that this scenario can be caused by a network partition, as well as an actual client node system failure. <xref linkend="dbdoclet.50438268_96876"/> describes this case in more detail.</para>
+ <title>30.1.1 <anchor xml:id="dbdoclet.50438268_96839" xreflabel=""/>Client <anchor xml:id="dbdoclet.50438268_marker-1287394" xreflabel=""/>Failure</title>
+ <para>Recovery from client failure in Lustre is based on lock revocation and other resources, so surviving clients can continue their work uninterrupted. If a client fails to timely respond to a blocking lock callback from the Distributed Lock Manager (DLM) or fails to communicate with the server in a long period of time (i.e., no pings), the client is forcibly removed from the cluster (evicted). This enables other clients to acquire locks blocked by the dead client's locks, and also frees resources (file handles, export data) associated with that client. Note that this scenario can be caused by a network partition, as well as an actual client node system failure. <xref linkend="dbdoclet.50438268_96876"/> describes this case in more detail.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438268_pgfId-1290714" xreflabel=""/>30.1.2 <anchor xml:id="dbdoclet.50438268_43796" xreflabel=""/>Client <anchor xml:id="dbdoclet.50438268_marker-1292164" xreflabel=""/>Eviction</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291609" xreflabel=""/>If a client is not behaving properly from the server's point of view, it will be evicted. This ensures that the whole file system can continue to function in the presence of failed or misbehaving clients. An evicted client must invalidate all locks, which in turn, results in all cached inodes becoming invalidated and all cached data being flushed.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291610" xreflabel=""/>Reasons why a client might be evicted:</para>
+ <title>30.1.2 <anchor xml:id="dbdoclet.50438268_43796" xreflabel=""/>Client <anchor xml:id="dbdoclet.50438268_marker-1292164" xreflabel=""/>Eviction</title>
+ <para>If a client is not behaving properly from the server's point of view, it will be evicted. This ensures that the whole file system can continue to function in the presence of failed or misbehaving clients. An evicted client must invalidate all locks, which in turn, results in all cached inodes becoming invalidated and all cached data being flushed.</para>
+ <para>Reasons why a client might be evicted:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291611" xreflabel=""/> Failure to respond to a server request in a timely manner</para>
+ <para> Failure to respond to a server request in a timely manner</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291612" xreflabel=""/> Blocking lock callback (i.e., client holds lock that another client/server wants)</para>
+ <para> Blocking lock callback (i.e., client holds lock that another client/server wants)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291613" xreflabel=""/> Lock completion callback (i.e., client is granted lock previously held by another client)</para>
+ <para> Lock completion callback (i.e., client is granted lock previously held by another client)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291614" xreflabel=""/> Lock glimpse callback (i.e., client is asked for size of object by another client)</para>
+ <para> Lock glimpse callback (i.e., client is asked for size of object by another client)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291615" xreflabel=""/> Server shutdown notification (with simplified interoperability)</para>
+ <para> Server shutdown notification (with simplified interoperability)</para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291616" xreflabel=""/> Failure to ping the server in a timely manner, unless the server is receiving no RPC traffic at all (which may indicate a network partition).</para>
+ <para> Failure to ping the server in a timely manner, unless the server is receiving no RPC traffic at all (which may indicate a network partition).</para>
</listitem>
</itemizedlist>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438268_pgfId-1287398" xreflabel=""/>30.1.3 <anchor xml:id="dbdoclet.50438268_37508" xreflabel=""/>MDS Failure <anchor xml:id="dbdoclet.50438268_marker-1287397" xreflabel=""/>(Failover)</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291624" xreflabel=""/>Highly-available (HA) Lustre operation requires that the metadata server have a peer configured for failover, including the use of a shared storage device for the MDT backing file system. The actual mechanism for detecting peer failure, power off (STONITH) of the failed peer (to prevent it from continuing to modify the shared disk), and takeover of the Lustre MDS service on the backup node depends on external HA software such as Heartbeat. It is also possible to have MDS recovery with a single MDS node. In this case, recovery will take as long as is needed for the single MDS to be restarted.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291625" xreflabel=""/>When clients detect an MDS failure (either by timeouts of in-flight requests or idle-time ping messages), they connect to the new backup MDS and use the Metadata Replay protocol. Metadata Replay is responsible for ensuring that the backup MDS re-acquires state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1290890" xreflabel=""/>The reconnection to a new (or restarted) MDS is managed by the file system configuration loaded by the client when the file system is first mounted. If a failover MDS has been configured (using the --failnode= option to mkfs.lustre or tunefs.lustre), the client tries to reconnect to both the primary and backup MDS until one of them responds that the failed MDT is again available. At that point, the client begins recovery. For more information, see <xref linkend="dbdoclet.50438268_65824"/>.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1290891" xreflabel=""/>Transaction numbers are used to ensure that operations are replayed in the order they were originally performed, so that they are guaranteed to succeed and present the same filesystem state as before the failure. In addition, clients inform the new server of their existing lock state (including locks that have not yet been granted). All metadata and lock replay must complete before new, non-recovery operations are permitted. In addition, only clients that were connected at the time of MDS failure are permitted to reconnect during the recovery window, to avoid the introduction of state changes that might conflict with what is being replayed by previously-connected clients.</para>
+ <title>30.1.3 <anchor xml:id="dbdoclet.50438268_37508" xreflabel=""/>MDS Failure <anchor xml:id="dbdoclet.50438268_marker-1287397" xreflabel=""/>(Failover)</title>
+ <para>Highly-available (HA) Lustre operation requires that the metadata server have a peer configured for failover, including the use of a shared storage device for the MDT backing file system. The actual mechanism for detecting peer failure, power off (STONITH) of the failed peer (to prevent it from continuing to modify the shared disk), and takeover of the Lustre MDS service on the backup node depends on external HA software such as Heartbeat. It is also possible to have MDS recovery with a single MDS node. In this case, recovery will take as long as is needed for the single MDS to be restarted.</para>
+ <para>When clients detect an MDS failure (either by timeouts of in-flight requests or idle-time ping messages), they connect to the new backup MDS and use the Metadata Replay protocol. Metadata Replay is responsible for ensuring that the backup MDS re-acquires state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk.</para>
+ <para>The reconnection to a new (or restarted) MDS is managed by the file system configuration loaded by the client when the file system is first mounted. If a failover MDS has been configured (using the --failnode= option to mkfs.lustre or tunefs.lustre), the client tries to reconnect to both the primary and backup MDS until one of them responds that the failed MDT is again available. At that point, the client begins recovery. For more information, see <xref linkend="dbdoclet.50438268_65824"/>.</para>
+ <para>Transaction numbers are used to ensure that operations are replayed in the order they were originally performed, so that they are guaranteed to succeed and present the same filesystem state as before the failure. In addition, clients inform the new server of their existing lock state (including locks that have not yet been granted). All metadata and lock replay must complete before new, non-recovery operations are permitted. In addition, only clients that were connected at the time of MDS failure are permitted to reconnect during the recovery window, to avoid the introduction of state changes that might conflict with what is being replayed by previously-connected clients.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438268_pgfId-1289241" xreflabel=""/>30.1.4 <anchor xml:id="dbdoclet.50438268_28881" xreflabel=""/>OST <anchor xml:id="dbdoclet.50438268_marker-1289240" xreflabel=""/>Failure (Failover)</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291633" xreflabel=""/>When an OST fails or has communication problems with the client, the default action is that the corresponding OSC enters recovery, and I/O requests going to that OST are blocked waiting for OST recovery or failover. It is possible to administratively mark the OSC as <emphasis>inactive</emphasis> on the client, in which case file operations that involve the failed OST will return an IO error (-EIO). Otherwise, the application waits until the OST has recovered or the client process is interrupted (e.g. ,with <emphasis>CTRL-C</emphasis>).</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1290917" xreflabel=""/>The MDS (via the LOV) detects that an OST is unavailable and skips it when assigning objects to new files. When the OST is restarted or re-establishes communication with the MDS, the MDS and OST automatically perform orphan recovery to destroy any objects that belong to files that were deleted while the OST was unavailable. For more information, see <xref linkend='troubleshootingrecovery'/> (Working with Orphaned Objects).</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1290921" xreflabel=""/>While the OSC to OST operation recovery protocol is the same as that between the MDC and MDT using the Metadata Replay protocol, typically the OST commits bulk write operations to disk synchronously and each reply indicates that the request is already committed and the data does not need to be saved for recovery. In some cases, the OST replies to the client before the operation is committed to disk (e.g. truncate, destroy, setattr, and I/O operations in very new versions of Lustre), and normal replay and resend handling is done, including resending of the bulk writes. In this case, the client keeps a copy of the data available in memory until the server indicates that the write has committed to disk.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1290922" xreflabel=""/>To force an OST recovery, unmount the OST and then mount it again. If the OST was connected to clients before it failed, then a recovery process starts after the remount, enabling clients to reconnect to the OST and replay transactions in their queue. When the OST is in recovery mode, all new client connections are refused until the recovery finishes. The recovery is complete when either all previously-connected clients reconnect and their transactions are replayed or a client connection attempt times out. If a connection attempt times out, then all clients waiting to reconnect (and their transactions) are lost.</para>
+ <title>30.1.4 <anchor xml:id="dbdoclet.50438268_28881" xreflabel=""/>OST <anchor xml:id="dbdoclet.50438268_marker-1289240" xreflabel=""/>Failure (Failover)</title>
+ <para>When an OST fails or has communication problems with the client, the default action is that the corresponding OSC enters recovery, and I/O requests going to that OST are blocked waiting for OST recovery or failover. It is possible to administratively mark the OSC as <emphasis>inactive</emphasis> on the client, in which case file operations that involve the failed OST will return an IO error (-EIO). Otherwise, the application waits until the OST has recovered or the client process is interrupted (e.g. ,with <emphasis>CTRL-C</emphasis>).</para>
+ <para>The MDS (via the LOV) detects that an OST is unavailable and skips it when assigning objects to new files. When the OST is restarted or re-establishes communication with the MDS, the MDS and OST automatically perform orphan recovery to destroy any objects that belong to files that were deleted while the OST was unavailable. For more information, see <xref linkend='troubleshootingrecovery'/> (Working with Orphaned Objects).</para>
+ <para>While the OSC to OST operation recovery protocol is the same as that between the MDC and MDT using the Metadata Replay protocol, typically the OST commits bulk write operations to disk synchronously and each reply indicates that the request is already committed and the data does not need to be saved for recovery. In some cases, the OST replies to the client before the operation is committed to disk (e.g. truncate, destroy, setattr, and I/O operations in very new versions of Lustre), and normal replay and resend handling is done, including resending of the bulk writes. In this case, the client keeps a copy of the data available in memory until the server indicates that the write has committed to disk.</para>
+ <para>To force an OST recovery, unmount the OST and then mount it again. If the OST was connected to clients before it failed, then a recovery process starts after the remount, enabling clients to reconnect to the OST and replay transactions in their queue. When the OST is in recovery mode, all new client connections are refused until the recovery finishes. The recovery is complete when either all previously-connected clients reconnect and their transactions are replayed or a client connection attempt times out. If a connection attempt times out, then all clients waiting to reconnect (and their transactions) are lost.</para>
<note><para>If you know an OST will not recover a previously-connected client (if, for example, the client has crashed), you can manually abort the recovery using this command:</para><para>lctl --device <OST device number> abort_recovery To determine an OST's device number and device name, run the lctl dl command. Sample lctl dl command output is shown below:</para><para>7 UP obdfilter ddn_data-OST0009 ddn_data-OST0009_UUID 1159 In this example, 7 is the OST device number. The device name is ddn_data-OST0009. In most instances, the device name can be used in place of the device number.</para></note>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438268_pgfId-1289389" xreflabel=""/>30.1.5 <anchor xml:id="dbdoclet.50438268_96876" xreflabel=""/>Network <anchor xml:id="dbdoclet.50438268_marker-1289388" xreflabel=""/>Partition</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291641" xreflabel=""/>Network failures may be transient. To avoid invoking recovery, the client tries, initially, to re-send any timed out request to the server. If the resend also fails, the client tries to re-establish a connection to the server. Clients can detect harmless partition upon reconnect if the server has not had any reason to evict the client.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291642" xreflabel=""/>If a request was processed by the server, but the reply was dropped (i.e., did not arrive back at the client), the server must reconstruct the reply when the client resends the request, rather than performing the same request twice.</para>
+ <title>30.1.5 <anchor xml:id="dbdoclet.50438268_96876" xreflabel=""/>Network <anchor xml:id="dbdoclet.50438268_marker-1289388" xreflabel=""/>Partition</title>
+ <para>Network failures may be transient. To avoid invoking recovery, the client tries, initially, to re-send any timed out request to the server. If the resend also fails, the client tries to re-establish a connection to the server. Clients can detect harmless partition upon reconnect if the server has not had any reason to evict the client.</para>
+ <para>If a request was processed by the server, but the reply was dropped (i.e., did not arrive back at the client), the server must reconstruct the reply when the client resends the request, rather than performing the same request twice.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1289291" xreflabel=""/>30.1.6 Failed Recovery</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1290945" xreflabel=""/>In the case of failed recovery, a client is evicted by the server and must reconnect after having flushed its saved state related to that server, as described in <link xl:href="LustreRecovery.html#50438268_43796">Client Eviction</link>, above. Failed recovery might occur for a number of reasons, including:</para>
+ <title>30.1.6 Failed Recovery</title>
+ <para>In the case of failed recovery, a client is evicted by the server and must reconnect after having flushed its saved state related to that server, as described in <link xl:href="LustreRecovery.html#50438268_43796">Client Eviction</link>, above. Failed recovery might occur for a number of reasons, including:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1290949" xreflabel=""/> Failure of recovery</para>
+ <para> Failure of recovery</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1290951" xreflabel=""/> Recovery fails if the operations of one client directly depend on the operations of another client that failed to participate in recovery. Otherwise, Version Based Recovery (VBR) allows recovery to proceed for all of the connected clients, and only missing clients are evicted.</para>
+ <para> Recovery fails if the operations of one client directly depend on the operations of another client that failed to participate in recovery. Otherwise, Version Based Recovery (VBR) allows recovery to proceed for all of the connected clients, and only missing clients are evicted.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1290965" xreflabel=""/> Manual abort of recovery</para>
+ <para> Manual abort of recovery</para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1290953" xreflabel=""/> Manual eviction by the administrator</para>
+ <para> Manual eviction by the administrator</para>
</listitem>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438268_65824">
<title>30.2 Metadata <anchor xml:id="dbdoclet.50438268_marker-1292175" xreflabel=""/>Replay</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287399" xreflabel=""/>Highly available Lustre operation requires that the MDS have a peer configured for failover, including the use of a shared storage device for the MDS backing file system. When a client detects an MDS failure, it connects to the new MDS and uses the metadata replay protocol to replay its requests.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1288905" xreflabel=""/>Metadata replay ensures that the failover MDS re-accumulates state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk.</para>
+ <para>Highly available Lustre operation requires that the MDS have a peer configured for failover, including the use of a shared storage device for the MDS backing file system. When a client detects an MDS failure, it connects to the new MDS and uses the metadata replay protocol to replay its requests.</para>
+ <para>Metadata replay ensures that the failover MDS re-accumulates state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1290989" xreflabel=""/>30.2.1 XID Numbers</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1290993" xreflabel=""/>Each request sent by the client contains an XID number, which is a client-unique, monotonically increasing 64-bit integer. The initial value of the XID is chosen so that it is highly unlikely that the same client node reconnecting to the same server after a reboot would have the same XID sequence. The XID is used by the client to order all of the requests that it sends, until such a time that the request is assigned a transaction number. The XID is also used in Reply Reconstruction to uniquely identify per-client requests at the server.</para>
+ <title>30.2.1 XID Numbers</title>
+ <para>Each request sent by the client contains an XID number, which is a client-unique, monotonically increasing 64-bit integer. The initial value of the XID is chosen so that it is highly unlikely that the same client node reconnecting to the same server after a reboot would have the same XID sequence. The XID is used by the client to order all of the requests that it sends, until such a time that the request is assigned a transaction number. The XID is also used in Reply Reconstruction to uniquely identify per-client requests at the server.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1288625" xreflabel=""/>30.2.2 Transaction Numbers</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291008" xreflabel=""/>Each client request processed by the server that involves any state change (metadata update, file open, write, etc., depending on server type) is assigned a transaction number by the server that is a target-unique, monontonically increasing, server-wide 64-bit integer. The transaction number for each file system-modifying request is sent back to the client along with the reply to that client request. The transaction numbers allow the client and server to unambiguously order every modification to the file system in case recovery is needed.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291005" xreflabel=""/>Each reply sent to a client (regardless of request type) also contains the last committed transaction number that indicates the highest transaction number committed to the file system. The ldiskfs backing file system that Lustre uses enforces the requirement that any earlier disk operation will always be committed to disk before a later disk operation, so the last committed transaction number also reports that any requests with a lower transaction number have been committed to disk.</para>
+ <title>30.2.2 Transaction Numbers</title>
+ <para>Each client request processed by the server that involves any state change (metadata update, file open, write, etc., depending on server type) is assigned a transaction number by the server that is a target-unique, monontonically increasing, server-wide 64-bit integer. The transaction number for each file system-modifying request is sent back to the client along with the reply to that client request. The transaction numbers allow the client and server to unambiguously order every modification to the file system in case recovery is needed.</para>
+ <para>Each reply sent to a client (regardless of request type) also contains the last committed transaction number that indicates the highest transaction number committed to the file system. The ldiskfs backing file system that Lustre uses enforces the requirement that any earlier disk operation will always be committed to disk before a later disk operation, so the last committed transaction number also reports that any requests with a lower transaction number have been committed to disk.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1288774" xreflabel=""/>30.2.3 Replay and Resend</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1288780" xreflabel=""/>Lustre recovery can be separated into two distinct types of operations: <emphasis>replay</emphasis> and <emphasis>resend</emphasis>.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291023" xreflabel=""/><emphasis>Replay</emphasis> operations are those for which the client received a reply from the server that the operation had been successfully completed. These operations need to be redone in exactly the same manner after a server restart as had been reported before the server failed. Replay can only happen if the server failed; otherwise it will not have lost any state in memory.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291021" xreflabel=""/><emphasis>Resend</emphasis> operations are those for which the client never received a reply, so their final state is unknown to the client. The client sends unanswered requests to the server again in XID order, and again awaits a reply for each one. In some cases, resent requests have been handled and committed to disk by the server (possibly also having dependent operations committed), in which case, the server performs reply reconstruction for the lost reply. In other cases, the server did not receive the lost request at all and processing proceeds as with any normal request. These are what happen in the case of a network interruption. It is also possible that the server received the request, but was unable to reply or commit it to disk before failure.</para>
+ <title>30.2.3 Replay and Resend</title>
+ <para>Lustre recovery can be separated into two distinct types of operations: <emphasis>replay</emphasis> and <emphasis>resend</emphasis>.</para>
+ <para><emphasis>Replay</emphasis> operations are those for which the client received a reply from the server that the operation had been successfully completed. These operations need to be redone in exactly the same manner after a server restart as had been reported before the server failed. Replay can only happen if the server failed; otherwise it will not have lost any state in memory.</para>
+ <para><emphasis>Resend</emphasis> operations are those for which the client never received a reply, so their final state is unknown to the client. The client sends unanswered requests to the server again in XID order, and again awaits a reply for each one. In some cases, resent requests have been handled and committed to disk by the server (possibly also having dependent operations committed), in which case, the server performs reply reconstruction for the lost reply. In other cases, the server did not receive the lost request at all and processing proceeds as with any normal request. These are what happen in the case of a network interruption. It is also possible that the server received the request, but was unable to reply or commit it to disk before failure.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1288655" xreflabel=""/>30.2.4 Client Replay List</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291046" xreflabel=""/>All file system-modifying requests have the potential to be required for server state recovery (replay) in case of a server failure. Replies that have an assigned transaction number that is higher than the last committed transaction number received in any reply from each server are preserved for later replay in a per-server replay list. As each reply is received from the server, it is checked to see if it has a higher last committed transaction number than the previous highest last committed number. Most requests that now have a lower transaction number can safely be removed from the replay list. One exception to this rule is for open requests, which need to be saved for replay until the file is closed so that the MDS can properly reference count open-unlinked files.</para>
+ <title>30.2.4 Client Replay List</title>
+ <para>All file system-modifying requests have the potential to be required for server state recovery (replay) in case of a server failure. Replies that have an assigned transaction number that is higher than the last committed transaction number received in any reply from each server are preserved for later replay in a per-server replay list. As each reply is received from the server, it is checked to see if it has a higher last committed transaction number than the previous highest last committed number. Most requests that now have a lower transaction number can safely be removed from the replay list. One exception to this rule is for open requests, which need to be saved for replay until the file is closed so that the MDS can properly reference count open-unlinked files.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1291084" xreflabel=""/>30.2.5 Server Recovery</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291095" xreflabel=""/>A server enters recovery if it was not shut down cleanly. If, upon startup, if any client entries are in the last_rcvd file for any previously connected clients, the server enters recovery mode and waits for these previously-connected clients to reconnect and begin replaying or resending their requests. This allows the server to recreate state that was exposed to clients (a request that completed successfully) but was not committed to disk before failure.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291096" xreflabel=""/>In the absence of any client connection attempts, the server waits indefinitely for the clients to reconnect. This is intended to handle the case where the server has a network problem and clients are unable to reconnect and/or if the server needs to be restarted repeatedly to resolve some problem with hardware or software. Once the server detects client connection attempts - either new clients or previously-connected clients - a recovery timer starts and forces recovery to finish in a finite time regardless of whether the previously-connected clients are available or not.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291097" xreflabel=""/>If no client entries are present in the last_rcvd file, or if the administrator manually aborts recovery, the server does not wait for client reconnection and proceeds to allow all clients to connect.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291098" xreflabel=""/>As clients connect, the server gathers information from each one to determine how long the recovery needs to take. Each client reports its connection UUID, and the server does a lookup for this UUID in the last_rcvd file to determine if this client was previously connected. If not, the client is refused connection and it will retry until recovery is completed. Each client reports its last seen transaction, so the server knows when all transactions have been replayed. The client also reports the amount of time that it was previously waiting for request completion so that the server can estimate how long some clients might need to detect the server failure and reconnect.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291093" xreflabel=""/>If the client times out during replay, it attempts to reconnect. If the client is unable to reconnect, REPLAY fails and it returns to DISCON state. It is possible that clients will timeout frequently during REPLAY, so reconnection should not delay an already slow process more than necessary. We can mitigate this by increasing the timeout during replay.</para>
+ <title>30.2.5 Server Recovery</title>
+ <para>A server enters recovery if it was not shut down cleanly. If, upon startup, if any client entries are in the last_rcvd file for any previously connected clients, the server enters recovery mode and waits for these previously-connected clients to reconnect and begin replaying or resending their requests. This allows the server to recreate state that was exposed to clients (a request that completed successfully) but was not committed to disk before failure.</para>
+ <para>In the absence of any client connection attempts, the server waits indefinitely for the clients to reconnect. This is intended to handle the case where the server has a network problem and clients are unable to reconnect and/or if the server needs to be restarted repeatedly to resolve some problem with hardware or software. Once the server detects client connection attempts - either new clients or previously-connected clients - a recovery timer starts and forces recovery to finish in a finite time regardless of whether the previously-connected clients are available or not.</para>
+ <para>If no client entries are present in the last_rcvd file, or if the administrator manually aborts recovery, the server does not wait for client reconnection and proceeds to allow all clients to connect.</para>
+ <para>As clients connect, the server gathers information from each one to determine how long the recovery needs to take. Each client reports its connection UUID, and the server does a lookup for this UUID in the last_rcvd file to determine if this client was previously connected. If not, the client is refused connection and it will retry until recovery is completed. Each client reports its last seen transaction, so the server knows when all transactions have been replayed. The client also reports the amount of time that it was previously waiting for request completion so that the server can estimate how long some clients might need to detect the server failure and reconnect.</para>
+ <para>If the client times out during replay, it attempts to reconnect. If the client is unable to reconnect, REPLAY fails and it returns to DISCON state. It is possible that clients will timeout frequently during REPLAY, so reconnection should not delay an already slow process more than necessary. We can mitigate this by increasing the timeout during replay.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1291090" xreflabel=""/>30.2.6 Request Replay</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291116" xreflabel=""/>If a client was previously connected, it gets a response from the server telling it that the server is in recovery and what the last committed transaction number on disk is. The client can then iterate through its replay list and use this last committed transaction number to prune any previously-committed requests. It replays any newer requests to the server in transaction number order, one at a time, waiting for a reply from the server before replaying the next request.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291114" xreflabel=""/>Open requests that are on the replay list may have a transaction number lower than the server's last committed transaction number. The server processes those open requests immediately. The server then processes replayed requests from all of the clients in transaction number order, starting at the last committed transaction number to ensure that the state is updated on disk in exactly the same manner as it was before the crash. As each replayed request is processed, the last committed transaction is incremented. If the server receives a replay request from a client that is higher than the current last committed transaction, that request is put aside until other clients provide the intervening transactions. In this manner, the server replays requests in the same sequence as they were previously executed on the server until either all clients are out of requests to replay or there is a gap in a sequence.</para>
+ <title>30.2.6 Request Replay</title>
+ <para>If a client was previously connected, it gets a response from the server telling it that the server is in recovery and what the last committed transaction number on disk is. The client can then iterate through its replay list and use this last committed transaction number to prune any previously-committed requests. It replays any newer requests to the server in transaction number order, one at a time, waiting for a reply from the server before replaying the next request.</para>
+ <para>Open requests that are on the replay list may have a transaction number lower than the server's last committed transaction number. The server processes those open requests immediately. The server then processes replayed requests from all of the clients in transaction number order, starting at the last committed transaction number to ensure that the state is updated on disk in exactly the same manner as it was before the crash. As each replayed request is processed, the last committed transaction is incremented. If the server receives a replay request from a client that is higher than the current last committed transaction, that request is put aside until other clients provide the intervening transactions. In this manner, the server replays requests in the same sequence as they were previously executed on the server until either all clients are out of requests to replay or there is a gap in a sequence.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1288925" xreflabel=""/>30.2.7 Gaps in the Replay Sequence</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291145" xreflabel=""/>In some cases, a gap may occur in the reply sequence. This might be caused by lost replies, where the request was processed and committed to disk but the reply was not received by the client. It can also be caused by clients missing from recovery due to partial network failure or client death.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291146" xreflabel=""/>In the case where all clients have reconnected, but there is a gap in the replay sequence the only possibility is that some requests were processed by the server but the reply was lost. Since the client must still have these requests in its resend list, they are processed after recovery is finished.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1288931" xreflabel=""/>In the case where all clients have not reconnected, it is likely that the failed clients had requests that will no longer be replayed. The VBR feature is used to determine if a request following a transaction gap is safe to be replayed. Each item in the file system (MDS inode or OST object) stores on disk the number of the last transaction in which it was modified. Each reply from the server contains the previous version number of the objects that it affects. During VBR replay, the server matches the previous version numbers in the resend request against the current version number. If the versions match, the request is the next one that affects the object and can be safely replayed. For more information, see <link xl:href="LustreRecovery.html#50438268_80068">Version-based Recovery</link>.</para>
+ <title>30.2.7 Gaps in the Replay Sequence</title>
+ <para>In some cases, a gap may occur in the reply sequence. This might be caused by lost replies, where the request was processed and committed to disk but the reply was not received by the client. It can also be caused by clients missing from recovery due to partial network failure or client death.</para>
+ <para>In the case where all clients have reconnected, but there is a gap in the replay sequence the only possibility is that some requests were processed by the server but the reply was lost. Since the client must still have these requests in its resend list, they are processed after recovery is finished.</para>
+ <para>In the case where all clients have not reconnected, it is likely that the failed clients had requests that will no longer be replayed. The VBR feature is used to determine if a request following a transaction gap is safe to be replayed. Each item in the file system (MDS inode or OST object) stores on disk the number of the last transaction in which it was modified. Each reply from the server contains the previous version number of the objects that it affects. During VBR replay, the server matches the previous version numbers in the resend request against the current version number. If the versions match, the request is the next one that affects the object and can be safely replayed. For more information, see <link xl:href="LustreRecovery.html#50438268_80068">Version-based Recovery</link>.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1288935" xreflabel=""/>30.2.8 Lock Recovery</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291167" xreflabel=""/>If all requests were replayed successfully and all clients reconnected, clients then do lock replay locks -- that is, every client sends information about every lock it holds from this server and its state (whenever it was granted or not, what mode, what properties and so on), and then recovery completes successfully. Currently, Lustre does not do lock verification and just trusts clients to present an accurate lock state. This does not impart any security concerns since Lustre 1.x clients are trusted for other information (e.g. user ID) during normal operation also.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291165" xreflabel=""/>After all of the saved requests and locks have been replayed, the client sends an MDS_GETSTATUS request with last-replay flag set. The reply to that request is held back until all clients have completed replay (sent the same flagged getstatus request), so that clients don't send non-recovery requests before recovery is complete.</para>
+ <title>30.2.8 Lock Recovery</title>
+ <para>If all requests were replayed successfully and all clients reconnected, clients then do lock replay locks -- that is, every client sends information about every lock it holds from this server and its state (whenever it was granted or not, what mode, what properties and so on), and then recovery completes successfully. Currently, Lustre does not do lock verification and just trusts clients to present an accurate lock state. This does not impart any security concerns since Lustre 1.x clients are trusted for other information (e.g. user ID) during normal operation also.</para>
+ <para>After all of the saved requests and locks have been replayed, the client sends an MDS_GETSTATUS request with last-replay flag set. The reply to that request is held back until all clients have completed replay (sent the same flagged getstatus request), so that clients don't send non-recovery requests before recovery is complete.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1288940" xreflabel=""/>30.2.9 Request Resend</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1288653" xreflabel=""/>Once all of the previously-shared state has been recovered on the server (the target file system is up-to-date with client cache and the server has recreated locks representing the locks held by the client), the client can resend any requests that did not receive an earlier reply. This processing is done like normal request processing, and, in some cases, the server may do reply reconstruction.</para>
+ <title>30.2.9 Request Resend</title>
+ <para>Once all of the previously-shared state has been recovered on the server (the target file system is up-to-date with client cache and the server has recreated locks representing the locks held by the client), the client can resend any requests that did not receive an earlier reply. This processing is done like normal request processing, and, in some cases, the server may do reply reconstruction.</para>
</section>
</section>
<section xml:id="dbdoclet.50438268_23736">
<title>30.3 Reply <anchor xml:id="dbdoclet.50438268_marker-1292176" xreflabel=""/>Reconstruction</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289740" xreflabel=""/>When a reply is dropped, the MDS needs to be able to reconstruct the reply when the original request is re-sent. This must be done without repeating any non-idempotent operations, while preserving the integrity of the locking system. In the event of MDS failover, the information used to reconstruct the reply must be serialized on the disk in transactions that are joined or nested with those operating on the disk.</para>
+ <para>When a reply is dropped, the MDS needs to be able to reconstruct the reply when the original request is re-sent. This must be done without repeating any non-idempotent operations, while preserving the integrity of the locking system. In the event of MDS failover, the information used to reconstruct the reply must be serialized on the disk in transactions that are joined or nested with those operating on the disk.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1289741" xreflabel=""/>30.3.1 Required State</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289743" xreflabel=""/>For the majority of requests, it is sufficient for the server to store three pieces of data in the last_rcvd file:</para>
+ <title>30.3.1 Required State</title>
+ <para>For the majority of requests, it is sufficient for the server to store three pieces of data in the last_rcvd file:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289808" xreflabel=""/> XID of the request</para>
+ <para> XID of the request</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289814" xreflabel=""/> Resulting transno (if any)</para>
+ <para> Resulting transno (if any)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289821" xreflabel=""/> Result code (req->rq_status)</para>
+ <para> Result code (req->rq_status)</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289749" xreflabel=""/>For open requests, the "disposition" of the open must also be stored.</para>
+ <para>For open requests, the "disposition" of the open must also be stored.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1289826" xreflabel=""/>30.3.2 Reconstruction of Open Replies</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289752" xreflabel=""/>An open reply consists of up to three pieces of information (in addition to the contents of the "request log"):</para>
+ <title>30.3.2 Reconstruction of Open Replies</title>
+ <para>An open reply consists of up to three pieces of information (in addition to the contents of the "request log"):</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289840" xreflabel=""/> File handle</para>
+ <para> File handle</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289874" xreflabel=""/> Lock handle</para>
+ <para> Lock handle</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289850" xreflabel=""/> mds_body with information about the file created (for O_CREAT)</para>
+ <para> mds_body with information about the file created (for O_CREAT)</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289738" xreflabel=""/>The disposition, status and request data (re-sent intact by the client) are sufficient to determine which type of lock handle was granted, whether an open file handle was created, and which resource should be described in the mds_body.</para>
+ <para>The disposition, status and request data (re-sent intact by the client) are sufficient to determine which type of lock handle was granted, whether an open file handle was created, and which resource should be described in the mds_body.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1289767" xreflabel=""/>Finding the File Handle</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289769" xreflabel=""/>The file handle can be found in the XID of the request and the list of per-export open file handles. The file handle contains the resource/FID.</para>
+ <title>Finding the File Handle</title>
+ <para>The file handle can be found in the XID of the request and the list of per-export open file handles. The file handle contains the resource/FID.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1291192" xreflabel=""/>Finding the Resource/fid</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1291193" xreflabel=""/>The file handle contains the resource/fid.</para>
+ <title>Finding the Resource/fid</title>
+ <para>The file handle contains the resource/fid.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1289773" xreflabel=""/>Finding the Lock Handle</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1289765" xreflabel=""/>The lock handle can be found by walking the list of granted locks for the resource looking for one with the appropriate remote file handle (present in the re-sent request). Verify that the lock has the right mode (determined by performing the disposition/request/status analysis above) and is granted to the proper client.</para>
+ <title>Finding the Lock Handle</title>
+ <para>The lock handle can be found by walking the list of granted locks for the resource looking for one with the appropriate remote file handle (present in the re-sent request). Verify that the lock has the right mode (determined by performing the disposition/request/status analysis above) and is granted to the proper client.</para>
</section>
</section>
</section>
<section xml:id="dbdoclet.50438268_80068">
<title>30.4 Version-based <anchor xml:id="dbdoclet.50438268_marker-1288580" xreflabel=""/>Recovery</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287888" xreflabel=""/>The Version-based Recovery (VBR) feature improves Lustre reliability in cases where client requests (RPCs) fail to replay during recovery
- <footnote><para><anchor xml:id="dbdoclet.50438268_pgfId-1288438" xreflabel=""/>There are two scenarios under which client RPCs are not replayed: (1) Non-functioning or isolated clients do not reconnect, and they cannot replay their RPCs, causing a gap in the replay sequence. These clients get errors and are evicted. (2) Functioning clients connect, but they cannot replay some or all of their RPCs that occurred after the gap caused by the non-functioning/isolated clients. These clients get errors (caused by the failed clients). With VBR, these requests have a better chance to replay because the "gaps" are only related to specific files that the missing client(s) changed.</para></footnote>.
+ <para>The Version-based Recovery (VBR) feature improves Lustre reliability in cases where client requests (RPCs) fail to replay during recovery
+ <footnote><para>There are two scenarios under which client RPCs are not replayed: (1) Non-functioning or isolated clients do not reconnect, and they cannot replay their RPCs, causing a gap in the replay sequence. These clients get errors and are evicted. (2) Functioning clients connect, but they cannot replay some or all of their RPCs that occurred after the gap caused by the non-functioning/isolated clients. These clients get errors (caused by the failed clients). With VBR, these requests have a better chance to replay because the "gaps" are only related to specific files that the missing client(s) changed.</para></footnote>.
.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287894" xreflabel=""/>In pre-VBR versions of Lustre, if the MGS or an OST went down and then recovered, a recovery process was triggered in which clients attempted to replay their requests. Clients were only allowed to replay RPCs in serial order. If a particular client could not replay its requests, then those requests were lost as well as the requests of clients later in the sequence. The ''downstream'' clients never got to replay their requests because of the wait on the earlier client'™s RPCs. Eventually, the recovery period would time out (so the component could accept new requests), leaving some number of clients evicted and their requests and data lost.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287896" xreflabel=""/>With VBR, the recovery mechanism does not result in the loss of clients or their data, because changes in inode versions are tracked, and more clients are able to reintegrate into the cluster. With VBR, inode tracking looks like this:</para>
+ <para>In pre-VBR versions of Lustre, if the MGS or an OST went down and then recovered, a recovery process was triggered in which clients attempted to replay their requests. Clients were only allowed to replay RPCs in serial order. If a particular client could not replay its requests, then those requests were lost as well as the requests of clients later in the sequence. The ''downstream'' clients never got to replay their requests because of the wait on the earlier client'™s RPCs. Eventually, the recovery period would time out (so the component could accept new requests), leaving some number of clients evicted and their requests and data lost.</para>
+ <para>With VBR, the recovery mechanism does not result in the loss of clients or their data, because changes in inode versions are tracked, and more clients are able to reintegrate into the cluster. With VBR, inode tracking looks like this:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1288169" xreflabel=""/> Each inode<footnote><para><anchor xml:id="dbdoclet.50438268_pgfId-1288489" xreflabel=""/>Usually, there are two inodes, a parent and a child.</para></footnote> stores a version, that is, the number of the last transaction (transno) in which the inode was changed.</para>
+ <para> Each inode<footnote><para>Usually, there are two inodes, a parent and a child.</para></footnote> stores a version, that is, the number of the last transaction (transno) in which the inode was changed.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1288212" xreflabel=""/> When an inode is about to be changed, a pre-operation version of the inode is saved in the client'™s data.</para>
+ <para> When an inode is about to be changed, a pre-operation version of the inode is saved in the client'™s data.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1288241" xreflabel=""/> The client keeps the pre-operation inode version and the post-operation version (transaction number) for replay, and sends them in the event of a server failure.</para>
+ <para> The client keeps the pre-operation inode version and the post-operation version (transaction number) for replay, and sends them in the event of a server failure.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1288505" xreflabel=""/> If the pre-operation version matches, then the request is replayed. The post-operation version is assigned on all inodes modified in the request.</para>
+ <para> If the pre-operation version matches, then the request is replayed. The post-operation version is assigned on all inodes modified in the request.</para>
</listitem>
</itemizedlist>
<note><para>An RPC can contain up to four pre-operation versions, because several inodes can be involved in an operation. In the case of a ''rename'' operation, four different inodes can be modified.</para></note>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287777" xreflabel=""/>During normal operation, the server:</para>
+ <para>During normal operation, the server:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287779" xreflabel=""/> Updates the versions of all inodes involved in a given operation</para>
+ <para> Updates the versions of all inodes involved in a given operation</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287952" xreflabel=""/> Returns the old and new inode versions to the client with the reply</para>
+ <para> Returns the old and new inode versions to the client with the reply</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287979" xreflabel=""/>When the recovery mechanism is underway, VBR follows these steps:</para>
+ <para>When the recovery mechanism is underway, VBR follows these steps:</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287980" xreflabel=""/>VBR only allows clients to replay transactions if the affected inodes have the same version as during the original execution of the transactions, even if there is gap in transactions due to a missed client.</para>
+ <para>VBR only allows clients to replay transactions if the affected inodes have the same version as during the original execution of the transactions, even if there is gap in transactions due to a missed client.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287992" xreflabel=""/>The server attempts to execute every transaction that the client offers, even if it encounters a re-integration failure.</para>
+ <para>The server attempts to execute every transaction that the client offers, even if it encounters a re-integration failure.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1288004" xreflabel=""/>When the replay is complete, the client and server check if a replay failed on any transaction because of inode version mismatch. If the versions match, the client gets a successful re-integration message. If the versions do not match, then the client is evicted.</para>
+ <para>When the replay is complete, the client and server check if a replay failed on any transaction because of inode version mismatch. If the versions match, the client gets a successful re-integration message. If the versions do not match, then the client is evicted.</para>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1288023" xreflabel=""/>VBR recovery is fully transparent to users. It may lead to slightly longer recovery times if the cluster loses several clients during server recovery.</para>
+ <para>VBR recovery is fully transparent to users. It may lead to slightly longer recovery times if the cluster loses several clients during server recovery.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438268_pgfId-1287803" xreflabel=""/>30.4.1 <anchor xml:id="dbdoclet.50438268_marker-1288583" xreflabel=""/>VBR Messages</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287805" xreflabel=""/>The VBR feature is built into the Lustre recovery functionality. It cannot be disabled. These are some VBR messages that may be displayed:</para>
- <screen><anchor xml:id="dbdoclet.50438268_pgfId-1287821" xreflabel=""/>DEBUG_REQ(D_WARNING, req, "Version mismatch during replay\n");
+ <title>30.4.1 <anchor xml:id="dbdoclet.50438268_marker-1288583" xreflabel=""/>VBR Messages</title>
+ <para>The VBR feature is built into the Lustre recovery functionality. It cannot be disabled. These are some VBR messages that may be displayed:</para>
+ <screen>DEBUG_REQ(D_WARNING, req, "Version mismatch during replay\n");
</screen>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287823" xreflabel=""/>This message indicates why the client was evicted. No action is needed.</para>
- <screen><anchor xml:id="dbdoclet.50438268_pgfId-1287825" xreflabel=""/>CWARN("%s: version recovery fails, reconnecting\n");
+ <para>This message indicates why the client was evicted. No action is needed.</para>
+ <screen>CWARN("%s: version recovery fails, reconnecting\n");
</screen>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287827" xreflabel=""/>This message indicates why the recovery failed. No action is needed.</para>
+ <para>This message indicates why the recovery failed. No action is needed.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438268_pgfId-1287839" xreflabel=""/>30.4.2 Tips for <anchor xml:id="dbdoclet.50438268_marker-1288584" xreflabel=""/>Using VBR</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1287767" xreflabel=""/>VBR will be successful for clients which do not share data with other client. Therefore, the strategy for reliable use of VBR is to store a client'™s data in its own directory, where possible. VBR can recover these clients, even if other clients are lost.</para>
+ <title>30.4.2 Tips for <anchor xml:id="dbdoclet.50438268_marker-1288584" xreflabel=""/>Using VBR</title>
+ <para>VBR will be successful for clients which do not share data with other client. Therefore, the strategy for reliable use of VBR is to store a client'™s data in its own directory, where possible. VBR can recover these clients, even if other clients are lost.</para>
</section>
</section>
<section xml:id="dbdoclet.50438268_83826">
<title>30.5 Commit on <anchor xml:id="dbdoclet.50438268_marker-1292182" xreflabel=""/>Share</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1292074" xreflabel=""/>The commit-on-share (COS) feature makes Lustre recovery more reliable by preventing missing clients from causing cascading evictions of other clients. With COS enabled, if some Lustre clients miss the recovery window after a reboot or a server failure, the remaining clients are not evicted.</para>
+ <para>The commit-on-share (COS) feature makes Lustre recovery more reliable by preventing missing clients from causing cascading evictions of other clients. With COS enabled, if some Lustre clients miss the recovery window after a reboot or a server failure, the remaining clients are not evicted.</para>
<note><para>The commit-on-share feature is enabled, by default.</para></note>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1292075" xreflabel=""/>30.5.1 Working with Commit on Share</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1292076" xreflabel=""/>To illustrate how COS works, let's first look at the old recovery scenario. After a service restart, the MDS would boot and enter recovery mode. Clients began reconnecting and replaying their uncommitted transactions. Clients could replay transactions independently as long as their transactions did not depend on each other (one client's transactions did not depend on a different client's transactions). The MDS is able to determine whether one transaction is dependent on another transaction via the <link xl:href="LustreRecovery.html#50438268_80068">Version-based Recovery</link> feature.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1292080" xreflabel=""/>If there was a dependency between client transactions (for example, creating and deleting the same file), and one or more clients did not reconnect in time, then some clients may have been evicted because their transactions depended on transactions from the missing clients. Evictions of those clients caused more clients to be evicted and so on, resulting in "cascading" client evictions.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1292081" xreflabel=""/>COS addresses the problem of cascading evictions by eliminating dependent transactions between clients. It ensures that one transaction is committed to disk if another client performs a transaction dependent on the first one. With no dependent, uncommitted transactions to apply, the clients replay their requests independently without the risk of being evicted.</para>
+ <title>30.5.1 Working with Commit on Share</title>
+ <para>To illustrate how COS works, let's first look at the old recovery scenario. After a service restart, the MDS would boot and enter recovery mode. Clients began reconnecting and replaying their uncommitted transactions. Clients could replay transactions independently as long as their transactions did not depend on each other (one client's transactions did not depend on a different client's transactions). The MDS is able to determine whether one transaction is dependent on another transaction via the <link xl:href="LustreRecovery.html#50438268_80068">Version-based Recovery</link> feature.</para>
+ <para>If there was a dependency between client transactions (for example, creating and deleting the same file), and one or more clients did not reconnect in time, then some clients may have been evicted because their transactions depended on transactions from the missing clients. Evictions of those clients caused more clients to be evicted and so on, resulting in "cascading" client evictions.</para>
+ <para>COS addresses the problem of cascading evictions by eliminating dependent transactions between clients. It ensures that one transaction is committed to disk if another client performs a transaction dependent on the first one. With no dependent, uncommitted transactions to apply, the clients replay their requests independently without the risk of being evicted.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438268_pgfId-1292099" xreflabel=""/>30.5.2 Tuning Commit On Share</title>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1292100" xreflabel=""/>Commit on Share can be enabled or disabled using the mdt.commit_on_sharing tunable (0/1). This tunable can be set when the MDS is created (mkfs.lustre) or when the Lustre file system is active, using the lctl set/get_param or lctl conf_param commands.</para>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1292101" xreflabel=""/>To set a default value for COS (disable/enable) when the file system is created, use:</para>
- <screen><anchor xml:id="dbdoclet.50438268_pgfId-1292102" xreflabel=""/>--param mdt.commit_on_sharing=0/1
+ <title>30.5.2 Tuning Commit On Share</title>
+ <para>Commit on Share can be enabled or disabled using the mdt.commit_on_sharing tunable (0/1). This tunable can be set when the MDS is created (mkfs.lustre) or when the Lustre file system is active, using the lctl set/get_param or lctl conf_param commands.</para>
+ <para>To set a default value for COS (disable/enable) when the file system is created, use:</para>
+ <screen>--param mdt.commit_on_sharing=0/1
</screen>
- <para><anchor xml:id="dbdoclet.50438268_pgfId-1292103" xreflabel=""/>To disable or enable COS when the file system is running, use:</para>
- <screen><anchor xml:id="dbdoclet.50438268_pgfId-1292104" xreflabel=""/>lctl set_param mdt.*.commit_on_sharing=0/1
+ <para>To disable or enable COS when the file system is running, use:</para>
+ <screen>lctl set_param mdt.*.commit_on_sharing=0/1
</screen>
<note><para>Enabling COS may cause the MDS to do a large number of synchronous disk operations, hurting performance. Placing the ldiskfs journal on a low-latency external device may improve file system performance.</para></note>
</section>
<info>
<title xml:id='lustretroubleshooting.title'>Lustre Troubleshooting</title>
</info>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291311" xreflabel=""/>This chapter provides information to troubleshoot Lustre, submit a Lustre bug, and Lustre performance tips. It includes the following sections:</para>
+ <para>This chapter provides information to troubleshoot Lustre, submit a Lustre bug, and Lustre performance tips. It includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438198_11171"/></para>
</listitem>
<section xml:id="dbdoclet.50438198_11171">
<title>26.1 Lustre Error Messages</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291322" xreflabel=""/>Several resources are available to help troubleshoot Lustre. This section describes error numbers, error messages and logs.</para>
+ <para>Several resources are available to help troubleshoot Lustre. This section describes error numbers, error messages and logs.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1292773" xreflabel=""/>26.1.1 Error <anchor xml:id="dbdoclet.50438198_marker-1296744" xreflabel=""/>Numbers</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1292777" xreflabel=""/>Error numbers for Lustre come from the Linux errno.h, and are located in /usr/include/asm/errno.h. Lustre does not use all of the available Linux error numbers. The exact meaning of an error number depends on where it is used. Here is a summary of the basic errors that Lustre users may encounter.</para>
+ <title>26.1.1 Error <anchor xml:id="dbdoclet.50438198_marker-1296744" xreflabel=""/>Numbers</title>
+ <para>Error numbers for Lustre come from the Linux errno.h, and are located in /usr/include/asm/errno.h. Lustre does not use all of the available Linux error numbers. The exact meaning of an error number depends on where it is used. Here is a summary of the basic errors that Lustre users may encounter.</para>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
<colspec colname="c3" colwidth="33*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438198_pgfId-1292782" xreflabel=""/>Error Number</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438198_pgfId-1292850" xreflabel=""/>Error Name</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438198_pgfId-1292784" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Error Number</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Error Name</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292786" xreflabel=""/>-1</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292852" xreflabel=""/>-EPERM</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292788" xreflabel=""/>Permission is denied.</para></entry>
+ <entry><para> -1</para></entry>
+ <entry><para> -EPERM</para></entry>
+ <entry><para> Permission is denied.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292790" xreflabel=""/>-2</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292854" xreflabel=""/>-ENOENT</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292797" xreflabel=""/>The requested file or directory does not exist.</para></entry>
+ <entry><para> -2</para></entry>
+ <entry><para> -ENOENT</para></entry>
+ <entry><para> The requested file or directory does not exist.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292799" xreflabel=""/>-4</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292856" xreflabel=""/>-EINTR</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292801" xreflabel=""/>The operation was interrupted (usually CTRL-C or a killing process).</para></entry>
+ <entry><para> -4</para></entry>
+ <entry><para> -EINTR</para></entry>
+ <entry><para> The operation was interrupted (usually CTRL-C or a killing process).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292803" xreflabel=""/>-5</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292858" xreflabel=""/>-EIO</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292805" xreflabel=""/>The operation failed with a read or write error.</para></entry>
+ <entry><para> -5</para></entry>
+ <entry><para> -EIO</para></entry>
+ <entry><para> The operation failed with a read or write error.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292810" xreflabel=""/>-19</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292860" xreflabel=""/>-ENODEV</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292816" xreflabel=""/>No such device is available. The server stopped or failed over.</para></entry>
+ <entry><para> -19</para></entry>
+ <entry><para> -ENODEV</para></entry>
+ <entry><para> No such device is available. The server stopped or failed over.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292818" xreflabel=""/>-22</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292862" xreflabel=""/>-EINVAL</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292820" xreflabel=""/>The parameter contains an invalid value.</para></entry>
+ <entry><para> -22</para></entry>
+ <entry><para> -EINVAL</para></entry>
+ <entry><para> The parameter contains an invalid value.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292906" xreflabel=""/>-28</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292908" xreflabel=""/>-ENOSPC</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292910" xreflabel=""/>The file system is out-of-space or out of inodes. Use lfs df (query the amount of file system space) or lfs df -i (query the number of inodes).</para></entry>
+ <entry><para> -28</para></entry>
+ <entry><para> -ENOSPC</para></entry>
+ <entry><para> The file system is out-of-space or out of inodes. Use lfs df (query the amount of file system space) or lfs df -i (query the number of inodes).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292900" xreflabel=""/>-30</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292902" xreflabel=""/>-EROFS</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292904" xreflabel=""/>The file system is read-only, likely due to a detected error.</para></entry>
+ <entry><para> -30</para></entry>
+ <entry><para> -EROFS</para></entry>
+ <entry><para> The file system is read-only, likely due to a detected error.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292894" xreflabel=""/>-43</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292896" xreflabel=""/>-EIDRM</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292898" xreflabel=""/>The UID/GID does not match any known UID/GID on the MDS. Update etc/hosts and etc/group on the MDS to add the missing user or group.</para></entry>
+ <entry><para> -43</para></entry>
+ <entry><para> -EIDRM</para></entry>
+ <entry><para> The UID/GID does not match any known UID/GID on the MDS. Update etc/hosts and etc/group on the MDS to add the missing user or group.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292888" xreflabel=""/>-107</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292890" xreflabel=""/>-ENOTCONN</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292892" xreflabel=""/>The client is not connected to this server.</para></entry>
+ <entry><para> -107</para></entry>
+ <entry><para> -ENOTCONN</para></entry>
+ <entry><para> The client is not connected to this server.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292882" xreflabel=""/>-110</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292884" xreflabel=""/>-ETIMEDOUT</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438198_pgfId-1292886" xreflabel=""/>The operation took too long and timed out.</para></entry>
+ <entry><para> -110</para></entry>
+ <entry><para> -ETIMEDOUT</para></entry>
+ <entry><para> The operation took too long and timed out.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291324" xreflabel=""/>26.1.2 <anchor xml:id="dbdoclet.50438198_40669" xreflabel=""/>Viewing Error <anchor xml:id="dbdoclet.50438198_marker-1291323" xreflabel=""/>Messages</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291325" xreflabel=""/>As Lustre code runs on the kernel, single-digit error codes display to the application; these error codes are an indication of the problem. Refer to the kernel console log (dmesg) for all recent kernel messages from that node. On the node, /var/log/messages holds a log of all messages for at least the past day.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291328" xreflabel=""/>The error message initiates with "LustreError" in the console log and provides a short description of:</para>
+ <title>26.1.2 <anchor xml:id="dbdoclet.50438198_40669" xreflabel=""/>Viewing Error <anchor xml:id="dbdoclet.50438198_marker-1291323" xreflabel=""/>Messages</title>
+ <para>As Lustre code runs on the kernel, single-digit error codes display to the application; these error codes are an indication of the problem. Refer to the kernel console log (dmesg) for all recent kernel messages from that node. On the node, /var/log/messages holds a log of all messages for at least the past day.</para>
+ <para>The error message initiates with "LustreError" in the console log and provides a short description of:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291329" xreflabel=""/> What the problem is</para>
+ <para> What the problem is</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291330" xreflabel=""/> Which process ID had trouble</para>
+ <para> Which process ID had trouble</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291331" xreflabel=""/> Which server node it was communicating with, and so on.</para>
+ <para> Which server node it was communicating with, and so on.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291332" xreflabel=""/>Lustre logs are dumped to /proc/sys/lnet/debug_path.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296082" xreflabel=""/>Collect the first group of messages related to a problem, and any messages that precede "LBUG" or "assertion failure" errors. Messages that mention server nodes (OST or MDS) are specific to that server; you must collect similar messages from the relevant server console logs.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291333" xreflabel=""/>Another Lustre debug log holds information for Lustre action for a short period of time which, in turn, depends on the processes on the node to use Lustre. Use the following command to extract debug logs on each of the nodes, run</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291334" xreflabel=""/>$ lctl dk <filename>
+ <para>Lustre logs are dumped to /proc/sys/lnet/debug_path.</para>
+ <para>Collect the first group of messages related to a problem, and any messages that precede "LBUG" or "assertion failure" errors. Messages that mention server nodes (OST or MDS) are specific to that server; you must collect similar messages from the relevant server console logs.</para>
+ <para>Another Lustre debug log holds information for Lustre action for a short period of time which, in turn, depends on the processes on the node to use Lustre. Use the following command to extract debug logs on each of the nodes, run</para>
+ <screen>$ lctl dk <filename>
</screen>
<note><para>LBUG freezes the thread to allow capture of the panic stack. A system reboot is needed to clear the thread.</para></note>
</section>
</section>
<section xml:id="dbdoclet.50438198_30989">
<title>26.2 Reporting a Lustre <anchor xml:id="dbdoclet.50438198_marker-1296753" xreflabel=""/>Bug</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1292557" xreflabel=""/>If, after troubleshooting your Lustre system, you cannot resolve the problem, consider reporting a Lustre bug. The process for reporting a bug is described in the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</link>.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1297414" xreflabel=""/>You can also post a question to the <link xl:href="http://wiki.lustre.org/index.php/Lustre_Mailing_Lists">lustre-discuss mailing list</link> or search the <link xl:href="http://groups.google.com/group/lustre-discuss-list">lustre-discuss Archives</link> for information about your issue.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1297376" xreflabel=""/>A Lustre diagnostics tool is available for downloading at: <link xl:href="http://downloads.lustre.org/public/tools/lustre-diagnostics/">http://downloads.lustre.org/public/tools/lustre-diagnostics/</link></para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1298089" xreflabel=""/>You can run this tool to capture diagnostics output to include in the reported bug. To run this tool, enter one of these commands:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1292528" xreflabel=""/># lustre-diagnostics -t <bugzilla bug #>
-<anchor xml:id="dbdoclet.50438198_pgfId-1292529" xreflabel=""/># lustre-diagnostics.
-</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1292530" xreflabel=""/>Output is sent directly to the terminal. Use normal file redirection to send the output to a file, and then manually attach the file to the bug you are submitting.</para>
+ <para>If, after troubleshooting your Lustre system, you cannot resolve the problem, consider reporting a Lustre bug. The process for reporting a bug is described in the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</link>.</para>
+ <para>You can also post a question to the <link xl:href="http://wiki.lustre.org/index.php/Lustre_Mailing_Lists">lustre-discuss mailing list</link> or search the <link xl:href="http://groups.google.com/group/lustre-discuss-list">lustre-discuss Archives</link> for information about your issue.</para>
+ <para>A Lustre diagnostics tool is available for downloading at: <link xl:href="http://downloads.lustre.org/public/tools/lustre-diagnostics/">http://downloads.lustre.org/public/tools/lustre-diagnostics/</link></para>
+ <para>You can run this tool to capture diagnostics output to include in the reported bug. To run this tool, enter one of these commands:</para>
+ <screen># lustre-diagnostics -t <bugzilla bug #>
+# lustre-diagnostics.
+</screen>
+ <para>Output is sent directly to the terminal. Use normal file redirection to send the output to a file, and then manually attach the file to the bug you are submitting.</para>
</section>
<section xml:id="dbdoclet.50438198_93109">
<title>26.3 Common Lustre Problems</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291338" xreflabel=""/>This section describes how to address common issues encountered with Lustre.</para>
+ <para>This section describes how to address common issues encountered with Lustre.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291350" xreflabel=""/>26.3.1 OST Object is <anchor xml:id="dbdoclet.50438198_marker-1291349" xreflabel=""/>Missing or Damaged</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291351" xreflabel=""/>If the OSS fails to find an object or finds a damaged object, this message appears:</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291717" xreflabel=""/>OST object missing or damaged (OST "ost1", object 98148, error -2)</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291352" xreflabel=""/>If the reported error is -2 (-ENOENT, or "No such file or directory"), then the object is missing. This can occur either because the MDS and OST are out of sync, or because an OST object was corrupted and deleted.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291353" xreflabel=""/>If you have recovered the file system from a disk failure by using e2fsck, then unrecoverable objects may have been deleted or moved to /lost+found on the raw OST partition. Because files on the MDS still reference these objects, attempts to access them produce this error.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291354" xreflabel=""/>If you have recovered a backup of the raw MDS or OST partition, then the restored partition is very likely to be out of sync with the rest of your cluster. No matter which server partition you restored from backup, files on the MDS may reference objects which no longer exist (or did not exist when the backup was taken); accessing those files produces this error.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291355" xreflabel=""/>If neither of those descriptions is applicable to your situation, then it is possible that you have discovered a programming error that allowed the servers to get out of sync. Please report this condition to the Lustre group, and we will investigate.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291356" xreflabel=""/>If the reported error is anything else (such as -5, "I/O error"), it likely indicates a storage failure. The low-level file system returns this error if it is unable to read from the storage device.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291358" xreflabel=""/><emphasis role="bold">Suggested Action</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291359" xreflabel=""/>If the reported error is -2, you can consider checking in /lost+found on your raw OST device, to see if the missing object is there. However, it is likely that this object is lost forever, and that the file that references the object is now partially or completely lost. Restore this file from backup, or salvage what you can and delete it.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291360" xreflabel=""/>If the reported error is anything else, then you should immediately inspect this server for storage problems.</para>
+ <title>26.3.1 OST Object is <anchor xml:id="dbdoclet.50438198_marker-1291349" xreflabel=""/>Missing or Damaged</title>
+ <para>If the OSS fails to find an object or finds a damaged object, this message appears:</para>
+ <para>OST object missing or damaged (OST "ost1", object 98148, error -2)</para>
+ <para>If the reported error is -2 (-ENOENT, or "No such file or directory"), then the object is missing. This can occur either because the MDS and OST are out of sync, or because an OST object was corrupted and deleted.</para>
+ <para>If you have recovered the file system from a disk failure by using e2fsck, then unrecoverable objects may have been deleted or moved to /lost+found on the raw OST partition. Because files on the MDS still reference these objects, attempts to access them produce this error.</para>
+ <para>If you have recovered a backup of the raw MDS or OST partition, then the restored partition is very likely to be out of sync with the rest of your cluster. No matter which server partition you restored from backup, files on the MDS may reference objects which no longer exist (or did not exist when the backup was taken); accessing those files produces this error.</para>
+ <para>If neither of those descriptions is applicable to your situation, then it is possible that you have discovered a programming error that allowed the servers to get out of sync. Please report this condition to the Lustre group, and we will investigate.</para>
+ <para>If the reported error is anything else (such as -5, "I/O error"), it likely indicates a storage failure. The low-level file system returns this error if it is unable to read from the storage device.</para>
+ <para><emphasis role="bold">Suggested Action</emphasis></para>
+ <para>If the reported error is -2, you can consider checking in /lost+found on your raw OST device, to see if the missing object is there. However, it is likely that this object is lost forever, and that the file that references the object is now partially or completely lost. Restore this file from backup, or salvage what you can and delete it.</para>
+ <para>If the reported error is anything else, then you should immediately inspect this server for storage problems.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291362" xreflabel=""/>26.3.2 OSTs <anchor xml:id="dbdoclet.50438198_marker-1291361" xreflabel=""/>Become Read-Only</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291363" xreflabel=""/>If the SCSI devices are inaccessible to Lustre at the block device level, then ldiskfs remounts the device read-only to prevent file system corruption. This is a normal behavior. The status in /proc/fs/lustre/health_check also shows "not healthy" on the affected nodes.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1293032" xreflabel=""/>To determine what caused the "not healthy" condition:</para>
+ <title>26.3.2 OSTs <anchor xml:id="dbdoclet.50438198_marker-1291361" xreflabel=""/>Become Read-Only</title>
+ <para>If the SCSI devices are inaccessible to Lustre at the block device level, then ldiskfs remounts the device read-only to prevent file system corruption. This is a normal behavior. The status in /proc/fs/lustre/health_check also shows "not healthy" on the affected nodes.</para>
+ <para>To determine what caused the "not healthy" condition:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1293041" xreflabel=""/> Examine the consoles of all servers for any error indications</para>
+ <para> Examine the consoles of all servers for any error indications</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1293045" xreflabel=""/> Examine the syslogs of all servers for any LustreErrors or LBUG</para>
+ <para> Examine the syslogs of all servers for any LustreErrors or LBUG</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1293046" xreflabel=""/> Check the health of your system hardware and network. (Are the disks working as expected, is the network dropping packets?)</para>
+ <para> Check the health of your system hardware and network. (Are the disks working as expected, is the network dropping packets?)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1293055" xreflabel=""/> Consider what was happening on the cluster at the time. Does this relate to a specific user workload or a system load condition? Is the condition reproducible? Does it happen at a specific time (day, week or month)?</para>
+ <para> Consider what was happening on the cluster at the time. Does this relate to a specific user workload or a system load condition? Is the condition reproducible? Does it happen at a specific time (day, week or month)?</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291365" xreflabel=""/>To recover from this problem, you must restart Lustre services using these file systems. There is no other way to know that the I/O made it to disk, and the state of the cache may be inconsistent with what is on disk.</para>
+ <para>To recover from this problem, you must restart Lustre services using these file systems. There is no other way to know that the I/O made it to disk, and the state of the cache may be inconsistent with what is on disk.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291367" xreflabel=""/>26.3.3 Identifying a <anchor xml:id="dbdoclet.50438198_marker-1291366" xreflabel=""/>Missing OST</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291368" xreflabel=""/>If an OST is missing for any reason, you may need to know what files are affected. Although an OST is missing, the files system should be operational. From any mounted client node, generate a list of files that reside on the affected OST. It is advisable to mark the missing OST as 'unavailable' so clients and the MDS do not time out trying to contact it.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291369" xreflabel=""/> 1. Generate a list of devices and determine the OST's device number. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291370" xreflabel=""/>$ lctl dl
+ <title>26.3.3 Identifying a <anchor xml:id="dbdoclet.50438198_marker-1291366" xreflabel=""/>Missing OST</title>
+ <para>If an OST is missing for any reason, you may need to know what files are affected. Although an OST is missing, the files system should be operational. From any mounted client node, generate a list of files that reside on the affected OST. It is advisable to mark the missing OST as 'unavailable' so clients and the MDS do not time out trying to contact it.</para>
+ <para> 1. Generate a list of devices and determine the OST's device number. Run:</para>
+ <screen>$ lctl dl
</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1293115" xreflabel=""/>The lctl dl command output lists the device name and number, along with the device UUID and the number of references on the device.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291371" xreflabel=""/> 2. Deactivate the OST (on the OSS at the MDS). Run:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291372" xreflabel=""/>$ lctl --device <OST device name or number> deactivate
+ <para>The lctl dl command output lists the device name and number, along with the device UUID and the number of references on the device.</para>
+ <para> 2. Deactivate the OST (on the OSS at the MDS). Run:</para>
+ <screen>$ lctl --device <OST device name or number> deactivate
</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291373" xreflabel=""/>The OST device number or device name is generated by the lctl dl command.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1293067" xreflabel=""/>The deactivate command prevents clients from creating new objects on the specified OST, although you can still access the OST for reading.</para>
+ <para>The OST device number or device name is generated by the lctl dl command.</para>
+ <para>The deactivate command prevents clients from creating new objects on the specified OST, although you can still access the OST for reading.</para>
<note><para>If the OST later becomes available it needs to be reactivated, run:</para><para># lctl --device <OST device name or number> activate</para></note>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291376" xreflabel=""/> 3. Determine all files that are striped over the missing OST, run:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291377" xreflabel=""/># lfs getstripe -r -O {OST_UUID} /mountpoint
+ <para> 3. Determine all files that are striped over the missing OST, run:</para>
+ <screen># lfs getstripe -r -O {OST_UUID} /mountpoint
</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291378" xreflabel=""/>This returns a simple list of filenames from the affected file system.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291379" xreflabel=""/> 4. If necessary, you can read the valid parts of a striped file, run:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291380" xreflabel=""/># dd if=filename of=new_filename bs=4k conv=sync,noerror
+ <para>This returns a simple list of filenames from the affected file system.</para>
+ <para> 4. If necessary, you can read the valid parts of a striped file, run:</para>
+ <screen># dd if=filename of=new_filename bs=4k conv=sync,noerror
</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291381" xreflabel=""/> 5. You can delete these files with the unlink or munlink command.</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291382" xreflabel=""/># unlink|munlink filename {filename ...}
+ <para> 5. You can delete these files with the unlink or munlink command.</para>
+ <screen># unlink|munlink filename {filename ...}
</screen>
<note><para>There is no functional difference between the unlink and munlink commands. The unlink command is for newer Linux distributions. You can run munlink if unlink is not available.</para><para> When you run the unlink or munlink command, the file on the MDS is permanently removed.</para></note>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291384" xreflabel=""/> 6. If you need to know, specifically, which parts of the file are missing data, then you first need to determine the file layout (striping pattern), which includes the index of the missing OST). Run:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291385" xreflabel=""/># lfs getstripe -v {filename}
-</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291386" xreflabel=""/> 7. Use this computation is to determine which offsets in the file are affected: [(C*N + X)*S, (C*N + X)*S + S - 1], N = { 0, 1, 2, ...}</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291388" xreflabel=""/>where:</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291389" xreflabel=""/>C = stripe count</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291390" xreflabel=""/>S = stripe size</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291391" xreflabel=""/>X = index of bad OST for this file</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291392" xreflabel=""/>For example, for a 2 stripe file, stripe size = 1M, the bad OST is at index 0, and you have holes in the file at: [(2*N + 0)*1M, (2*N + 0)*1M + 1M - 1], N = { 0, 1, 2, ...}</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291394" xreflabel=""/>If the file system cannot be mounted, currently there is no way that parses metadata directly from an MDS. If the bad OST does not start, options to mount the file system are to provide a loop device OST in its place or replace it with a newly-formatted OST. In that case, the missing objects are created and are read as zero-filled.</para>
+ <para> 6. If you need to know, specifically, which parts of the file are missing data, then you first need to determine the file layout (striping pattern), which includes the index of the missing OST). Run:</para>
+ <screen># lfs getstripe -v {filename}
+</screen>
+ <para> 7. Use this computation is to determine which offsets in the file are affected: [(C*N + X)*S, (C*N + X)*S + S - 1], N = { 0, 1, 2, ...}</para>
+ <para>where:</para>
+ <para>C = stripe count</para>
+ <para>S = stripe size</para>
+ <para>X = index of bad OST for this file</para>
+ <para>For example, for a 2 stripe file, stripe size = 1M, the bad OST is at index 0, and you have holes in the file at: [(2*N + 0)*1M, (2*N + 0)*1M + 1M - 1], N = { 0, 1, 2, ...}</para>
+ <para>If the file system cannot be mounted, currently there is no way that parses metadata directly from an MDS. If the bad OST does not start, options to mount the file system are to provide a loop device OST in its place or replace it with a newly-formatted OST. In that case, the missing objects are created and are read as zero-filled.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291436" xreflabel=""/>26.3.4 <anchor xml:id="dbdoclet.50438198_69657" xreflabel=""/>Fixing a Bad LAST_ID on an OST</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296775" xreflabel=""/>Each OST contains a LAST_ID file, which holds the last object (pre-)created by the MDS <footnote><para><anchor xml:id="dbdoclet.50438198_pgfId-1296778" xreflabel=""/>The contents of the LAST_ID file must be accurate regarding the actual objects that exist on the OST.</para></footnote>. The MDT contains a lov_objid file, with values that represent the last object the MDS has allocated to a file.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296779" xreflabel=""/>During normal operation, the MDT keeps some pre-created (but unallocated) objects on the OST, and the relationship between LAST_ID and lov_objid should be LAST_ID <= lov_objid. Any difference in the file values results in objects being created on the OST when it next connects to the MDS. These objects are never actually allocated to a file, since they are of 0 length (empty), but they do no harm. Creating empty objects enables the OST to catch up to the MDS, so normal operations resume.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296780" xreflabel=""/>However, in the case where lov_objid < LAST_ID, bad things can happen as the MDS is not aware of objects that have already been allocated on the OST, and it reallocates them to new files, overwriting their existing contents.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296781" xreflabel=""/>Here is the rule to avoid this scenario:</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296782" xreflabel=""/>LAST_ID >= lov_objid and LAST_ID == last_physical_object and lov_objid >= last_used_object</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296783" xreflabel=""/>Although the lov_objid value should be equal to the last_used_object value, the above rule suffices to keep Lustre happy at the expense of a few leaked objects.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296784" xreflabel=""/>In situations where there is on-disk corruption of the OST, for example caused by running with write cache enabled on the disks, the LAST_ID value may become inconsistent and result in a message similar to:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296785" xreflabel=""/>"filter_precreate()) HOME-OST0003: Serious error:
-<anchor xml:id="dbdoclet.50438198_pgfId-1296786" xreflabel=""/>objid 3478673 already exists; is this filesystem corrupt?"
-</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296787" xreflabel=""/>A related situation may happen if there is a significant discrepancy between the record of previously-created objects on the OST and the previously-allocated objects on the MDS, for example if the MDS has been corrupted, or restored from backup, which may cause significant data loss if left unchecked. This produces a message like:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296788" xreflabel=""/>"HOME-OST0003: ignoring bogus orphan destroy request:
-<anchor xml:id="dbdoclet.50438198_pgfId-1296789" xreflabel=""/>obdid 3438673 last_id 3478673"
-</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296797" xreflabel=""/>To recover from this situation, determine and set a reasonable LAST_ID value.</para>
+ <title>26.3.4 <anchor xml:id="dbdoclet.50438198_69657" xreflabel=""/>Fixing a Bad LAST_ID on an OST</title>
+ <para>Each OST contains a LAST_ID file, which holds the last object (pre-)created by the MDS <footnote><para>The contents of the LAST_ID file must be accurate regarding the actual objects that exist on the OST.</para></footnote>. The MDT contains a lov_objid file, with values that represent the last object the MDS has allocated to a file.</para>
+ <para>During normal operation, the MDT keeps some pre-created (but unallocated) objects on the OST, and the relationship between LAST_ID and lov_objid should be LAST_ID <= lov_objid. Any difference in the file values results in objects being created on the OST when it next connects to the MDS. These objects are never actually allocated to a file, since they are of 0 length (empty), but they do no harm. Creating empty objects enables the OST to catch up to the MDS, so normal operations resume.</para>
+ <para>However, in the case where lov_objid < LAST_ID, bad things can happen as the MDS is not aware of objects that have already been allocated on the OST, and it reallocates them to new files, overwriting their existing contents.</para>
+ <para>Here is the rule to avoid this scenario:</para>
+ <para>LAST_ID >= lov_objid and LAST_ID == last_physical_object and lov_objid >= last_used_object</para>
+ <para>Although the lov_objid value should be equal to the last_used_object value, the above rule suffices to keep Lustre happy at the expense of a few leaked objects.</para>
+ <para>In situations where there is on-disk corruption of the OST, for example caused by running with write cache enabled on the disks, the LAST_ID value may become inconsistent and result in a message similar to:</para>
+ <screen>"filter_precreate()) HOME-OST0003: Serious error:
+objid 3478673 already exists; is this filesystem corrupt?"
+</screen>
+ <para>A related situation may happen if there is a significant discrepancy between the record of previously-created objects on the OST and the previously-allocated objects on the MDS, for example if the MDS has been corrupted, or restored from backup, which may cause significant data loss if left unchecked. This produces a message like:</para>
+ <screen>"HOME-OST0003: ignoring bogus orphan destroy request:
+obdid 3438673 last_id 3478673"
+</screen>
+ <para>To recover from this situation, determine and set a reasonable LAST_ID value.</para>
<note><para>The file system must be stopped on all servers before performing this procedure.</para></note>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296799" xreflabel=""/>For hex < -> decimal translations:</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296800" xreflabel=""/>Use GDB:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296801" xreflabel=""/>(gdb) p /x 15028
-<anchor xml:id="dbdoclet.50438198_pgfId-1296802" xreflabel=""/>$2 = 0x3ab4
+ <para>For hex < -> decimal translations:</para>
+ <para>Use GDB:</para>
+ <screen>(gdb) p /x 15028
+$2 = 0x3ab4
</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296803" xreflabel=""/>Or bc:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296804" xreflabel=""/>echo "obase=16; 15028" | bc
+ <para>Or bc:</para>
+ <screen>echo "obase=16; 15028" | bc
</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296805" xreflabel=""/> 1. Determine a reasonable value for the LAST_ID file. Check on the MDS:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296806" xreflabel=""/># mount -t ldiskfs /dev/<mdsdev> /mnt/mds
-<anchor xml:id="dbdoclet.50438198_pgfId-1296807" xreflabel=""/># od -Ax -td8 /mnt/mds/lov_objid
+ <para> 1. Determine a reasonable value for the LAST_ID file. Check on the MDS:</para>
+ <screen># mount -t ldiskfs /dev/<mdsdev> /mnt/mds
+# od -Ax -td8 /mnt/mds/lov_objid
</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296808" xreflabel=""/>There is one entry for each OST, in OST index order. This is what the MDS thinks is the last in-use object.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296809" xreflabel=""/> 2. Determine the OST index for this OST.</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296810" xreflabel=""/># od -Ax -td4 /mnt/ost/last_rcvd
+ <para>There is one entry for each OST, in OST index order. This is what the MDS thinks is the last in-use object.</para>
+ <para> 2. Determine the OST index for this OST.</para>
+ <screen># od -Ax -td4 /mnt/ost/last_rcvd
</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296811" xreflabel=""/>It will have it at offset 0x8c.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296812" xreflabel=""/> 3. Check on the OST. Use debugfs to check the LAST_ID value:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296813" xreflabel=""/>debugfs -c -R 'dump /O/0/LAST_ID /tmp/LAST_ID' /dev/XXX ; od -Ax -td8 /tmp/\
+ <para>It will have it at offset 0x8c.</para>
+ <para> 3. Check on the OST. Use debugfs to check the LAST_ID value:</para>
+ <screen>debugfs -c -R 'dump /O/0/LAST_ID /tmp/LAST_ID' /dev/XXX ; od -Ax -td8 /tmp/\
LAST_ID"
</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296814" xreflabel=""/> 4. Check the objects on the OST:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296815" xreflabel=""/>mount -rt ldiskfs /dev/{ostdev} /mnt/ost
-<anchor xml:id="dbdoclet.50438198_pgfId-1296816" xreflabel=""/># note the ls below is a number one and not a letter L
-<anchor xml:id="dbdoclet.50438198_pgfId-1296817" xreflabel=""/>ls -1s /mnt/ost/O/0/d* | grep -v [a-z] |
-<anchor xml:id="dbdoclet.50438198_pgfId-1296818" xreflabel=""/>sort -k2 -n > /tmp/objects.{diskname}
-<anchor xml:id="dbdoclet.50438198_pgfId-1296819" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438198_pgfId-1296820" xreflabel=""/>tail -30 /tmp/objects.{diskname}
-</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296821" xreflabel=""/>This shows you the OST state. There may be some pre-created orphans. Check for zero-length objects. Any zero-length objects with IDs higher than LAST_ID should be deleted. New objects will be pre-created.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296832" xreflabel=""/>If the OST LAST_ID value matches that for the objects existing on the OST, then it is possible the lov_objid file on the MDS is incorrect. Delete the lov_objid file on the MDS and it will be re-created from the LAST_ID on the OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296833" xreflabel=""/>If you determine the LAST_ID file on the OST is incorrect (that is, it does not match what objects exist, does not match the MDS lov_objid value), then you have decided on a proper value for LAST_ID.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296834" xreflabel=""/>Once you have decided on a proper value for LAST_ID, use this repair procedure.</para>
+ <para> 4. Check the objects on the OST:</para>
+ <screen>mount -rt ldiskfs /dev/{ostdev} /mnt/ost
+# note the ls below is a number one and not a letter L
+ls -1s /mnt/ost/O/0/d* | grep -v [a-z] |
+sort -k2 -n > /tmp/objects.{diskname}
+
+tail -30 /tmp/objects.{diskname}
+</screen>
+ <para>This shows you the OST state. There may be some pre-created orphans. Check for zero-length objects. Any zero-length objects with IDs higher than LAST_ID should be deleted. New objects will be pre-created.</para>
+ <para>If the OST LAST_ID value matches that for the objects existing on the OST, then it is possible the lov_objid file on the MDS is incorrect. Delete the lov_objid file on the MDS and it will be re-created from the LAST_ID on the OSTs.</para>
+ <para>If you determine the LAST_ID file on the OST is incorrect (that is, it does not match what objects exist, does not match the MDS lov_objid value), then you have decided on a proper value for LAST_ID.</para>
+ <para>Once you have decided on a proper value for LAST_ID, use this repair procedure.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296835" xreflabel=""/>Access:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296836" xreflabel=""/>mount -t ldiskfs /dev/{ostdev} /mnt/ost
+ <para>Access:</para>
+ <screen>mount -t ldiskfs /dev/{ostdev} /mnt/ost
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296837" xreflabel=""/>Check the current:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296838" xreflabel=""/>od -Ax -td8 /mnt/ost/O/0/LAST_ID
+ <para>Check the current:</para>
+ <screen>od -Ax -td8 /mnt/ost/O/0/LAST_ID
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296839" xreflabel=""/>Be very safe, only work on backups:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296840" xreflabel=""/>cp /mnt/ost/O/0/LAST_ID /tmp/LAST_ID
+ <para>Be very safe, only work on backups:</para>
+ <screen>cp /mnt/ost/O/0/LAST_ID /tmp/LAST_ID
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296841" xreflabel=""/>Convert binary to text:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296842" xreflabel=""/>xxd /tmp/LAST_ID /tmp/LAST_ID.asc
+ <para>Convert binary to text:</para>
+ <screen>xxd /tmp/LAST_ID /tmp/LAST_ID.asc
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296843" xreflabel=""/>Fix:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296844" xreflabel=""/>vi /tmp/LAST_ID.asc
+ <para>Fix:</para>
+ <screen>vi /tmp/LAST_ID.asc
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296845" xreflabel=""/>Convert to binary:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296846" xreflabel=""/>xxd -r /tmp/LAST_ID.asc /tmp/LAST_ID.new
+ <para>Convert to binary:</para>
+ <screen>xxd -r /tmp/LAST_ID.asc /tmp/LAST_ID.new
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296847" xreflabel=""/>Verify:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296848" xreflabel=""/>od -Ax -td8 /tmp/LAST_ID.new
+ <para>Verify:</para>
+ <screen>od -Ax -td8 /tmp/LAST_ID.new
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296849" xreflabel=""/>Replace:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296850" xreflabel=""/>cp /tmp/LAST_ID.new /mnt/ost/O/0/LAST_ID
+ <para>Replace:</para>
+ <screen>cp /tmp/LAST_ID.new /mnt/ost/O/0/LAST_ID
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1296851" xreflabel=""/>Clean up:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1296852" xreflabel=""/>umount /mnt/ost
+ <para>Clean up:</para>
+ <screen>umount /mnt/ost
</screen>
</listitem></orderedlist>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291447" xreflabel=""/>26.3.5 Handling/Debugging <anchor xml:id="dbdoclet.50438198_marker-1291446" xreflabel=""/>"Bind: Address already in use" Error</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291448" xreflabel=""/>During startup, Lustre may report a bind: Address already in use error and reject to start the operation. This is caused by a portmap service (often NFS locking) which starts before Lustre and binds to the default port 988. You must have port 988 open from firewall or IP tables for incoming connections on the client, OSS, and MDS nodes. LNET will create three outgoing connections on available, reserved ports to each client-server pair, starting with 1023, 1022 and 1021.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291449" xreflabel=""/>Unfortunately, you cannot set sunprc to avoid port 988. If you receive this error, do the following:</para>
+ <title>26.3.5 Handling/Debugging <anchor xml:id="dbdoclet.50438198_marker-1291446" xreflabel=""/>"Bind: Address already in use" Error</title>
+ <para>During startup, Lustre may report a bind: Address already in use error and reject to start the operation. This is caused by a portmap service (often NFS locking) which starts before Lustre and binds to the default port 988. You must have port 988 open from firewall or IP tables for incoming connections on the client, OSS, and MDS nodes. LNET will create three outgoing connections on available, reserved ports to each client-server pair, starting with 1023, 1022 and 1021.</para>
+ <para>Unfortunately, you cannot set sunprc to avoid port 988. If you receive this error, do the following:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291450" xreflabel=""/> Start Lustre before starting any service that uses sunrpc.</para>
+ <para> Start Lustre before starting any service that uses sunrpc.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291451" xreflabel=""/> Use a port other than 988 for Lustre. This is configured in /etc/modprobe.conf as an option to the LNET module. For example:</para>
+ <para> Use a port other than 988 for Lustre. This is configured in /etc/modprobe.conf as an option to the LNET module. For example:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291452" xreflabel=""/>options lnet accept_port=988
+ <screen>options lnet accept_port=988
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291453" xreflabel=""/> Add modprobe ptlrpc to your system startup scripts before the service that uses sunrpc. This causes Lustre to bind to port 988 and sunrpc to select a different port.</para>
+ <para> Add modprobe ptlrpc to your system startup scripts before the service that uses sunrpc. This causes Lustre to bind to port 988 and sunrpc to select a different port.</para>
</listitem>
</itemizedlist>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291471" xreflabel=""/>26.3.6 Handling/Debugging <anchor xml:id="dbdoclet.50438198_marker-1291470" xreflabel=""/>Error "- 28"</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1297002" xreflabel=""/>A Linux error -28 (ENOSPC) that occurs during a write or sync operation indicates that an existing file residing on an OST could not be rewritten or updated because the OST was full, or nearly full. To verify if this is the case, on a client on which the OST is mounted, enter :</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1297980" xreflabel=""/>lfs df -h
+ <title>26.3.6 Handling/Debugging <anchor xml:id="dbdoclet.50438198_marker-1291470" xreflabel=""/>Error "- 28"</title>
+ <para>A Linux error -28 (ENOSPC) that occurs during a write or sync operation indicates that an existing file residing on an OST could not be rewritten or updated because the OST was full, or nearly full. To verify if this is the case, on a client on which the OST is mounted, enter :</para>
+ <screen>lfs df -h
</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1297979" xreflabel=""/>To address this issue, you can do one of the following:</para>
+ <para>To address this issue, you can do one of the following:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1297887" xreflabel=""/> Expand the disk space on the OST.</para>
+ <para> Expand the disk space on the OST.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1297888" xreflabel=""/> Copy or stripe the file to a less full OST.</para>
+ <para> Copy or stripe the file to a less full OST.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1297889" xreflabel=""/>A Linux error -28 (ENOSPC) that occurs when a new file is being created may indicate that the MDS has run out of inodes and needs to be made larger. Newly created files do not written to full OSTs, while existing files continue to reside on the OST where they were initially created. To view inode information on the MDS, enter:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1297985" xreflabel=""/>lfs df -i
+ <para>A Linux error -28 (ENOSPC) that occurs when a new file is being created may indicate that the MDS has run out of inodes and needs to be made larger. Newly created files do not written to full OSTs, while existing files continue to reside on the OST where they were initially created. To view inode information on the MDS, enter:</para>
+ <screen>lfs df -i
</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1297871" xreflabel=""/>Typically, Lustre reports this error to your application. If the application is checking the return code from its function calls, then it decodes it into a textual error message such as No space left on device. Both versions of the error message also appear in the system log.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1298031" xreflabel=""/>For more information about the lfs df command, see <link xl:href="ManagingStripingFreeSpace.html#50438209_35838">Checking File System Free Space</link>.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1297882" xreflabel=""/>Although it is less efficient, you can also use the grep command to determine which OST or MDS is running out of space. To check the free space and inodes on a client, enter:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291475" xreflabel=""/>grep '[0-9]' /proc/fs/lustre/osc/*/kbytes{free,avail,total}
-<anchor xml:id="dbdoclet.50438198_pgfId-1291476" xreflabel=""/>grep '[0-9]' /proc/fs/lustre/osc/*/files{free,total}
-<anchor xml:id="dbdoclet.50438198_pgfId-1291477" xreflabel=""/>grep '[0-9]' /proc/fs/lustre/mdc/*/kbytes{free,avail,total}
-<anchor xml:id="dbdoclet.50438198_pgfId-1291478" xreflabel=""/>grep '[0-9]' /proc/fs/lustre/mdc/*/files{free,total}
+ <para>Typically, Lustre reports this error to your application. If the application is checking the return code from its function calls, then it decodes it into a textual error message such as No space left on device. Both versions of the error message also appear in the system log.</para>
+ <para>For more information about the lfs df command, see <link xl:href="ManagingStripingFreeSpace.html#50438209_35838">Checking File System Free Space</link>.</para>
+ <para>Although it is less efficient, you can also use the grep command to determine which OST or MDS is running out of space. To check the free space and inodes on a client, enter:</para>
+ <screen>grep '[0-9]' /proc/fs/lustre/osc/*/kbytes{free,avail,total}
+grep '[0-9]' /proc/fs/lustre/osc/*/files{free,total}
+grep '[0-9]' /proc/fs/lustre/mdc/*/kbytes{free,avail,total}
+grep '[0-9]' /proc/fs/lustre/mdc/*/files{free,total}
</screen>
<note><para>You can find other numeric error codes along with a short name and text description in /usr/include/asm/errno.h.</para></note>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291481" xreflabel=""/>26.3.7 Triggering <anchor xml:id="dbdoclet.50438198_marker-1291480" xreflabel=""/>Watchdog for PID NNN</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291482" xreflabel=""/>In some cases, a server node triggers a watchdog timer and this causes a process stack to be dumped to the console along with a Lustre kernel debug log being dumped into /tmp (by default). The presence of a watchdog timer does NOT mean that the thread OOPSed, but rather that it is taking longer time than expected to complete a given operation. In some cases, this situation is expected.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291483" xreflabel=""/>For example, if a RAID rebuild is really slowing down I/O on an OST, it might trigger watchdog timers to trip. But another message follows shortly thereafter, indicating that the thread in question has completed processing (after some number of seconds). Generally, this indicates a transient problem. In other cases, it may legitimately signal that a thread is stuck because of a software error (lock inversion, for example).</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291876" xreflabel=""/>Lustre: 0:0:(watchdog.c:122:lcw_cb())
-</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291877" xreflabel=""/>The above message indicates that the watchdog is active for pid 933:</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291878" xreflabel=""/>It was inactive for 100000ms:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291487" xreflabel=""/>Lustre: 0:0:(linux-debug.c:132:portals_debug_dumpstack())
-</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291488" xreflabel=""/>Showing stack for process:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291489" xreflabel=""/>933 ll_ost_25 D F896071A 0 933 1 934 932 (L-TLB)
-<anchor xml:id="dbdoclet.50438198_pgfId-1291490" xreflabel=""/>f6d87c60 00000046 00000000 f896071a f8def7cc 00002710 00001822 2da48cae
-<anchor xml:id="dbdoclet.50438198_pgfId-1291491" xreflabel=""/>0008cf1a f6d7c220 f6d7c3d0 f6d86000 f3529648 f6d87cc4 f3529640 f8961d3d
-<anchor xml:id="dbdoclet.50438198_pgfId-1291492" xreflabel=""/>00000010 f6d87c9c ca65a13c 00001fff 00000001 00000001 00000000 00000001
-</screen>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291493" xreflabel=""/>Call trace:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291494" xreflabel=""/>filter_do_bio+0x3dd/0xb90 [obdfilter]
-<anchor xml:id="dbdoclet.50438198_pgfId-1291495" xreflabel=""/>default_wake_function+0x0/0x20
-<anchor xml:id="dbdoclet.50438198_pgfId-1291496" xreflabel=""/>filter_direct_io+0x2fb/0x990 [obdfilter]
-<anchor xml:id="dbdoclet.50438198_pgfId-1291497" xreflabel=""/>filter_preprw_read+0x5c5/0xe00 [obdfilter]
-<anchor xml:id="dbdoclet.50438198_pgfId-1291498" xreflabel=""/>lustre_swab_niobuf_remote+0x0/0x30 [ptlrpc]
-<anchor xml:id="dbdoclet.50438198_pgfId-1291499" xreflabel=""/>ost_brw_read+0x18df/0x2400 [ost]
-<anchor xml:id="dbdoclet.50438198_pgfId-1291500" xreflabel=""/>ost_handle+0x14c2/0x42d0 [ost]
-<anchor xml:id="dbdoclet.50438198_pgfId-1291501" xreflabel=""/>ptlrpc_server_handle_request+0x870/0x10b0 [ptlrpc]
-<anchor xml:id="dbdoclet.50438198_pgfId-1291502" xreflabel=""/>ptlrpc_main+0x42e/0x7c0 [ptlrpc]
+ <title>26.3.7 Triggering <anchor xml:id="dbdoclet.50438198_marker-1291480" xreflabel=""/>Watchdog for PID NNN</title>
+ <para>In some cases, a server node triggers a watchdog timer and this causes a process stack to be dumped to the console along with a Lustre kernel debug log being dumped into /tmp (by default). The presence of a watchdog timer does NOT mean that the thread OOPSed, but rather that it is taking longer time than expected to complete a given operation. In some cases, this situation is expected.</para>
+ <para>For example, if a RAID rebuild is really slowing down I/O on an OST, it might trigger watchdog timers to trip. But another message follows shortly thereafter, indicating that the thread in question has completed processing (after some number of seconds). Generally, this indicates a transient problem. In other cases, it may legitimately signal that a thread is stuck because of a software error (lock inversion, for example).</para>
+ <screen>Lustre: 0:0:(watchdog.c:122:lcw_cb())
+</screen>
+ <para>The above message indicates that the watchdog is active for pid 933:</para>
+ <para>It was inactive for 100000ms:</para>
+ <screen>Lustre: 0:0:(linux-debug.c:132:portals_debug_dumpstack())
+</screen>
+ <para>Showing stack for process:</para>
+ <screen>933 ll_ost_25 D F896071A 0 933 1 934 932 (L-TLB)
+f6d87c60 00000046 00000000 f896071a f8def7cc 00002710 00001822 2da48cae
+0008cf1a f6d7c220 f6d7c3d0 f6d86000 f3529648 f6d87cc4 f3529640 f8961d3d
+00000010 f6d87c9c ca65a13c 00001fff 00000001 00000001 00000000 00000001
+</screen>
+ <para>Call trace:</para>
+ <screen>filter_do_bio+0x3dd/0xb90 [obdfilter]
+default_wake_function+0x0/0x20
+filter_direct_io+0x2fb/0x990 [obdfilter]
+filter_preprw_read+0x5c5/0xe00 [obdfilter]
+lustre_swab_niobuf_remote+0x0/0x30 [ptlrpc]
+ost_brw_read+0x18df/0x2400 [ost]
+ost_handle+0x14c2/0x42d0 [ost]
+ptlrpc_server_handle_request+0x870/0x10b0 [ptlrpc]
+ptlrpc_main+0x42e/0x7c0 [ptlrpc]
</screen>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291504" xreflabel=""/>26.3.8 Handling <anchor xml:id="dbdoclet.50438198_marker-1291503" xreflabel=""/>Timeouts on Initial Lustre Setup</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291505" xreflabel=""/>If you come across timeouts or hangs on the initial setup of your Lustre system, verify that name resolution for servers and clients is working correctly. Some distributions configure /etc/hosts sts so the name of the local machine (as reported by the 'hostname' command) is mapped to local host (127.0.0.1) instead of a proper IP address.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291506" xreflabel=""/>This might produce this error:</para>
- <screen><anchor xml:id="dbdoclet.50438198_pgfId-1291507" xreflabel=""/>LustreError:(ldlm_handle_cancel()) received cancel for unknown lock cookie
-<anchor xml:id="dbdoclet.50438198_pgfId-1291508" xreflabel=""/>0xe74021a4b41b954e from nid 0x7f000001 (0:127.0.0.1)
+ <title>26.3.8 Handling <anchor xml:id="dbdoclet.50438198_marker-1291503" xreflabel=""/>Timeouts on Initial Lustre Setup</title>
+ <para>If you come across timeouts or hangs on the initial setup of your Lustre system, verify that name resolution for servers and clients is working correctly. Some distributions configure /etc/hosts sts so the name of the local machine (as reported by the 'hostname' command) is mapped to local host (127.0.0.1) instead of a proper IP address.</para>
+ <para>This might produce this error:</para>
+ <screen>LustreError:(ldlm_handle_cancel()) received cancel for unknown lock cookie
+0xe74021a4b41b954e from nid 0x7f000001 (0:127.0.0.1)
</screen>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291510" xreflabel=""/>26.3.9 Handling/Debugging <anchor xml:id="dbdoclet.50438198_marker-1291509" xreflabel=""/>"LustreError: xxx went back in time"</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291511" xreflabel=""/>Each time Lustre changes the state of the disk file system, it records a unique transaction number. Occasionally, when committing these transactions to the disk, the last committed transaction number displays to other nodes in the cluster to assist the recovery. Therefore, the promised transactions remain absolutely safe on the disappeared disk.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291512" xreflabel=""/>This situation arises when:</para>
+ <title>26.3.9 Handling/Debugging <anchor xml:id="dbdoclet.50438198_marker-1291509" xreflabel=""/>"LustreError: xxx went back in time"</title>
+ <para>Each time Lustre changes the state of the disk file system, it records a unique transaction number. Occasionally, when committing these transactions to the disk, the last committed transaction number displays to other nodes in the cluster to assist the recovery. Therefore, the promised transactions remain absolutely safe on the disappeared disk.</para>
+ <para>This situation arises when:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291513" xreflabel=""/> You are using a disk device that claims to have data written to disk before it actually does, as in case of a device with a large cache. If that disk device crashes or loses power in a way that causes the loss of the cache, there can be a loss of transactions that you believe are committed. This is a very serious event, and you should run e2fsck against that storage before restarting Lustre.</para>
+ <para> You are using a disk device that claims to have data written to disk before it actually does, as in case of a device with a large cache. If that disk device crashes or loses power in a way that causes the loss of the cache, there can be a loss of transactions that you believe are committed. This is a very serious event, and you should run e2fsck against that storage before restarting Lustre.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291514" xreflabel=""/> As per the Lustre requirement, the shared storage used for failover is completely cache-coherent. This ensures that if one server takes over for another, it sees the most up-to-date and accurate copy of the data. In case of the failover of the server, if the shared storage does not provide cache coherency between all of its ports, then Lustre can produce an error.</para>
+ <para> As per the Lustre requirement, the shared storage used for failover is completely cache-coherent. This ensures that if one server takes over for another, it sees the most up-to-date and accurate copy of the data. In case of the failover of the server, if the shared storage does not provide cache coherency between all of its ports, then Lustre can produce an error.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291515" xreflabel=""/>If you know the exact reason for the error, then it is safe to proceed with no further action. If you do not know the reason, then this is a serious issue and you should explore it with your disk vendor.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291516" xreflabel=""/>If the error occurs during failover, examine your disk cache settings. If it occurs after a restart without failover, try to determine how the disk can report that a write succeeded, then lose the Data Device corruption or Disk Errors.</para>
+ <para>If you know the exact reason for the error, then it is safe to proceed with no further action. If you do not know the reason, then this is a serious issue and you should explore it with your disk vendor.</para>
+ <para>If the error occurs during failover, examine your disk cache settings. If it occurs after a restart without failover, try to determine how the disk can report that a write succeeded, then lose the Data Device corruption or Disk Errors.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291518" xreflabel=""/>26.3.10 Lustre Error: <anchor xml:id="dbdoclet.50438198_marker-1291517" xreflabel=""/>"Slow Start_Page_Write"</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291519" xreflabel=""/>The slow start_page_write message appears when the operation takes an extremely long time to allocate a batch of memory pages. Use these pages to receive network traffic first, and then write to disk.</para>
+ <title>26.3.10 Lustre Error: <anchor xml:id="dbdoclet.50438198_marker-1291517" xreflabel=""/>"Slow Start_Page_Write"</title>
+ <para>The slow start_page_write message appears when the operation takes an extremely long time to allocate a batch of memory pages. Use these pages to receive network traffic first, and then write to disk.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291521" xreflabel=""/>26.3.11 Drawbacks in Doing <anchor xml:id="dbdoclet.50438198_marker-1291520" xreflabel=""/>Multi-client O_APPEND Writes</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291522" xreflabel=""/>It is possible to do multi-client O_APPEND writes to a single file, but there are few drawbacks that may make this a sub-optimal solution. These drawbacks are:</para>
+ <title>26.3.11 Drawbacks in Doing <anchor xml:id="dbdoclet.50438198_marker-1291520" xreflabel=""/>Multi-client O_APPEND Writes</title>
+ <para>It is possible to do multi-client O_APPEND writes to a single file, but there are few drawbacks that may make this a sub-optimal solution. These drawbacks are:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291523" xreflabel=""/> Each client needs to take an EOF lock on all the OSTs, as it is difficult to know which OST holds the end of the file until you check all the OSTs. As all the clients are using the same O_APPEND, there is significant locking overhead.</para>
+ <para> Each client needs to take an EOF lock on all the OSTs, as it is difficult to know which OST holds the end of the file until you check all the OSTs. As all the clients are using the same O_APPEND, there is significant locking overhead.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291524" xreflabel=""/> The second client cannot get all locks until the end of the writing of the first client, as the taking serializes all writes from the clients.</para>
+ <para> The second client cannot get all locks until the end of the writing of the first client, as the taking serializes all writes from the clients.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291525" xreflabel=""/> To avoid deadlocks, the taking of these locks occurs in a known, consistent order. As a client cannot know which OST holds the next piece of the file until the client has locks on all OSTS, there is a need of these locks in case of a striped file.</para>
+ <para> To avoid deadlocks, the taking of these locks occurs in a known, consistent order. As a client cannot know which OST holds the next piece of the file until the client has locks on all OSTS, there is a need of these locks in case of a striped file.</para>
</listitem>
</itemizedlist>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291882" xreflabel=""/>26.3.12 Slowdown Occurs <anchor xml:id="dbdoclet.50438198_marker-1291921" xreflabel=""/>During Lustre Startup</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291894" xreflabel=""/>When Lustre starts, the Lustre file system needs to read in data from the disk. For the very first mdsrate run after the reboot, the MDS needs to wait on all the OSTs for object pre-creation. This causes a slowdown to occur when Lustre starts up.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291896" xreflabel=""/>After the file system has been running for some time, it contains more data in cache and hence, the variability caused by reading critical metadata from disk is mostly eliminated. The file system now reads data from the cache.</para>
+ <title>26.3.12 Slowdown Occurs <anchor xml:id="dbdoclet.50438198_marker-1291921" xreflabel=""/>During Lustre Startup</title>
+ <para>When Lustre starts, the Lustre file system needs to read in data from the disk. For the very first mdsrate run after the reboot, the MDS needs to wait on all the OSTs for object pre-creation. This causes a slowdown to occur when Lustre starts up.</para>
+ <para>After the file system has been running for some time, it contains more data in cache and hence, the variability caused by reading critical metadata from disk is mostly eliminated. The file system now reads data from the cache.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1291922" xreflabel=""/>26.3.13 Log Message 'Out of <anchor xml:id="dbdoclet.50438198_marker-1292113" xreflabel=""/>Memory' on OST</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291934" xreflabel=""/>When planning the hardware for an OSS node, consider the memory usage of several components in the Lustre system. If insufficient memory is available, an 'out of memory' message can be logged.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1292123" xreflabel=""/>During normal operation, several conditions indicate insufficient RAM on a server node:</para>
+ <title>26.3.13 Log Message 'Out of <anchor xml:id="dbdoclet.50438198_marker-1292113" xreflabel=""/>Memory' on OST</title>
+ <para>When planning the hardware for an OSS node, consider the memory usage of several components in the Lustre system. If insufficient memory is available, an 'out of memory' message can be logged.</para>
+ <para>During normal operation, several conditions indicate insufficient RAM on a server node:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291969" xreflabel=""/> kernel "Out of memory" and/or "oom-killer" messages</para>
+ <para> kernel "Out of memory" and/or "oom-killer" messages</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1292105" xreflabel=""/> Lustre "kmalloc of 'mmm' (NNNN bytes) failed..." messages</para>
+ <para> Lustre "kmalloc of 'mmm' (NNNN bytes) failed..." messages</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1292053" xreflabel=""/> Lustre or kernel stack traces showing processes stuck in "try_to_free_pages"</para>
+ <para> Lustre or kernel stack traces showing processes stuck in "try_to_free_pages"</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1292421" xreflabel=""/>For information on determining the MDS memory and OSS memory requirements, see <link xl:href="SettingUpLustreSystem.html#50438256_26456">Determining Memory Requirements</link>.</para>
+ <para>For information on determining the MDS memory and OSS memory requirements, see <link xl:href="SettingUpLustreSystem.html#50438256_26456">Determining Memory Requirements</link>.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438198_pgfId-1294801" xreflabel=""/>26.3.14 Setting SCSI <anchor xml:id="dbdoclet.50438198_marker-1294800" xreflabel=""/>I/O Sizes</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1294802" xreflabel=""/>Some SCSI drivers default to a maximum I/O size that is too small for good Lustre performance. we have fixed quite a few drivers, but you may still find that some drivers give unsatisfactory performance with Lustre. As the default value is hard-coded, you need to recompile the drivers to change their default. On the other hand, some drivers may have a wrong default set.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1294803" xreflabel=""/>If you suspect bad I/O performance and an analysis of Lustre statistics indicates that I/O is not 1 MB, check /sys/block/<device>/queue/max_sectors_kb. If the max_sectors_kb value is less than 1024, set it to at least 1024 to improve performance. If changing max_sectors_kb does not change the I/O size as reported by Lustre, you may want to examine the SCSI driver code.</para>
+ <title>26.3.14 Setting SCSI <anchor xml:id="dbdoclet.50438198_marker-1294800" xreflabel=""/>I/O Sizes</title>
+ <para>Some SCSI drivers default to a maximum I/O size that is too small for good Lustre performance. we have fixed quite a few drivers, but you may still find that some drivers give unsatisfactory performance with Lustre. As the default value is hard-coded, you need to recompile the drivers to change their default. On the other hand, some drivers may have a wrong default set.</para>
+ <para>If you suspect bad I/O performance and an analysis of Lustre statistics indicates that I/O is not 1 MB, check /sys/block/<device>/queue/max_sectors_kb. If the max_sectors_kb value is less than 1024, set it to at least 1024 to improve performance. If changing max_sectors_kb does not change the I/O size as reported by Lustre, you may want to examine the SCSI driver code.</para>
</section>
</section>
</chapter>
<info>
<title xml:id='lustretuning.title'>Lustre Tuning</title>
</info>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1291087" xreflabel=""/>This chapter contains information about tuning Lustre for better performance and includes the following sections:</para>
+ <para>This chapter contains information about tuning Lustre for better performance and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438272_55226"/></para>
<section xml:id="dbdoclet.50438272_55226">
<title>25.1 Optimizing the Number of Service Threads</title>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295659" xreflabel=""/>An OSS can have a minimum of 2 service threads and a maximum of 512 service threads. The number of service threads is a function of how much RAM and how many CPUs are on each OSS node (1 thread / 128MB * num_cpus). If the load on the OSS node is high, new service threads will be started in order to process more requests concurrently, up to 4x the initial number of threads (subject to the maximum of 512). For a 2GB 2-CPU system, the default thread count is 32 and the maximum thread count is 128.</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295660" xreflabel=""/>Increasing the size of the thread pool may help when:</para>
+ <para>An OSS can have a minimum of 2 service threads and a maximum of 512 service threads. The number of service threads is a function of how much RAM and how many CPUs are on each OSS node (1 thread / 128MB * num_cpus). If the load on the OSS node is high, new service threads will be started in order to process more requests concurrently, up to 4x the initial number of threads (subject to the maximum of 512). For a 2GB 2-CPU system, the default thread count is 32 and the maximum thread count is 128.</para>
+ <para>Increasing the size of the thread pool may help when:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295661" xreflabel=""/> Several OSTs are exported from a single OSS</para>
+ <para> Several OSTs are exported from a single OSS</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295662" xreflabel=""/> Back-end storage is running synchronously</para>
+ <para> Back-end storage is running synchronously</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295663" xreflabel=""/> I/O completions take excessive time due to slow storage</para>
+ <para> I/O completions take excessive time due to slow storage</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295664" xreflabel=""/>Decreasing the size of the thread pool may help if:</para>
+ <para>Decreasing the size of the thread pool may help if:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295665" xreflabel=""/> Clients are overwhelming the storage capacity</para>
+ <para> Clients are overwhelming the storage capacity</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295666" xreflabel=""/> There are lots of "slow I/O" or similar messages</para>
+ <para> There are lots of "slow I/O" or similar messages</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295667" xreflabel=""/>Increasing the number of I/O threads allows the kernel and storage to aggregate many writes together for more efficient disk I/O. The OSS thread pool is shared--each thread allocates approximately 1.5 MB (maximum RPC size + 0.5 MB) for internal I/O buffers.</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295668" xreflabel=""/>It is very important to consider memory consumption when increasing the thread pool size. Drives are only able to sustain a certain amount of parallel I/O activity before performance is degraded, due to the high number of seeks and the OST threads just waiting for I/O. In this situation, it may be advisable to decrease the load by decreasing the number of OST threads.</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295669" xreflabel=""/>Determining the optimum number of OST threads is a process of trial and error, and varies for each particular configuration. Variables include the number of OSTs on each OSS, number and speed of disks, RAID configuration, and available RAM. You may want to start with a number of OST threads equal to the number of actual disk spindles on the node. If you use RAID, subtract any dead spindles not used for actual data (e.g., 1 of N of spindles for RAID5, 2 of N spindles for RAID6), and monitor the performance of clients during usual workloads. If performance is degraded, increase the thread count and see how that works until performance is degraded again or you reach satisfactory performance.</para>
+ <para>Increasing the number of I/O threads allows the kernel and storage to aggregate many writes together for more efficient disk I/O. The OSS thread pool is shared--each thread allocates approximately 1.5 MB (maximum RPC size + 0.5 MB) for internal I/O buffers.</para>
+ <para>It is very important to consider memory consumption when increasing the thread pool size. Drives are only able to sustain a certain amount of parallel I/O activity before performance is degraded, due to the high number of seeks and the OST threads just waiting for I/O. In this situation, it may be advisable to decrease the load by decreasing the number of OST threads.</para>
+ <para>Determining the optimum number of OST threads is a process of trial and error, and varies for each particular configuration. Variables include the number of OSTs on each OSS, number and speed of disks, RAID configuration, and available RAM. You may want to start with a number of OST threads equal to the number of actual disk spindles on the node. If you use RAID, subtract any dead spindles not used for actual data (e.g., 1 of N of spindles for RAID5, 2 of N spindles for RAID6), and monitor the performance of clients during usual workloads. If performance is degraded, increase the thread count and see how that works until performance is degraded again or you reach satisfactory performance.</para>
<note><para>If there are too many threads, the latency for individual I/O requests can become very high and should be avoided. Set the desired maximum thread count permanently using the method described above.</para></note>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438272_pgfId-1295614" xreflabel=""/>25.1.1 <anchor xml:id="dbdoclet.50438272_60005" xreflabel=""/>Specifying the OSS Service <anchor xml:id="dbdoclet.50438272_marker-1294858" xreflabel=""/>Thread Count</title>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1291118" xreflabel=""/>The oss_num_threads parameter enables the number of OST service threads to be specified at module load time on the OSS nodes:</para>
- <screen><anchor xml:id="dbdoclet.50438272_pgfId-1291119" xreflabel=""/>options ost oss_num_threads={N}</screen>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1292633" xreflabel=""/>After startup, the minimum and maximum number of OSS thread counts can be set via the {service}.thread_{min,max,started} tunable. To change the tunable at runtime, run:</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1293688" xreflabel=""/>lctl {get,set}_param {service}.thread_{min,max,started}</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1293692" xreflabel=""/>For details, see <link xl:href="LustreProc.html#50438271_87260">Setting MDS and OSS Thread Counts</link>.</para>
+ <title>25.1.1 <anchor xml:id="dbdoclet.50438272_60005" xreflabel=""/>Specifying the OSS Service <anchor xml:id="dbdoclet.50438272_marker-1294858" xreflabel=""/>Thread Count</title>
+ <para>The oss_num_threads parameter enables the number of OST service threads to be specified at module load time on the OSS nodes:</para>
+ <screen>options ost oss_num_threads={N}</screen>
+ <para>After startup, the minimum and maximum number of OSS thread counts can be set via the {service}.thread_{min,max,started} tunable. To change the tunable at runtime, run:</para>
+ <para>lctl {get,set}_param {service}.thread_{min,max,started}</para>
+ <para>For details, see <link xl:href="LustreProc.html#50438271_87260">Setting MDS and OSS Thread Counts</link>.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438272_pgfId-1293756" xreflabel=""/>25.1.2 Specifying the MDS Service <anchor xml:id="dbdoclet.50438272_marker-1293755" xreflabel=""/>Thread Count</title>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1293828" xreflabel=""/>The mds_num_threads parameter enables the number of MDS service threads to be specified at module load time on the MDS node:</para>
- <screen><anchor xml:id="dbdoclet.50438272_pgfId-1291131" xreflabel=""/>options mds mds_num_threads={N}</screen>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1293699" xreflabel=""/>After startup, the minimum and maximum number of MDS thread counts can be set via the {service}.thread_{min,max,started} tunable. To change the tunable at runtime, run:</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1293700" xreflabel=""/>lctl {get,set}_param {service}.thread_{min,max,started}</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1293704" xreflabel=""/>For details, see <link xl:href="LustreProc.html#50438271_87260">Setting MDS and OSS Thread Counts</link>.</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1294918" xreflabel=""/>At this time, no testing has been done to determine the optimal number of MDS threads. The default value varies, based on server size, up to a maximum of 32. The maximum number of threads (MDS_MAX_THREADS) is 512.</para>
+ <title>25.1.2 Specifying the MDS Service <anchor xml:id="dbdoclet.50438272_marker-1293755" xreflabel=""/>Thread Count</title>
+ <para>The mds_num_threads parameter enables the number of MDS service threads to be specified at module load time on the MDS node:</para>
+ <screen>options mds mds_num_threads={N}</screen>
+ <para>After startup, the minimum and maximum number of MDS thread counts can be set via the {service}.thread_{min,max,started} tunable. To change the tunable at runtime, run:</para>
+ <para>lctl {get,set}_param {service}.thread_{min,max,started}</para>
+ <para>For details, see <link xl:href="LustreProc.html#50438271_87260">Setting MDS and OSS Thread Counts</link>.</para>
+ <para>At this time, no testing has been done to determine the optimal number of MDS threads. The default value varies, based on server size, up to a maximum of 32. The maximum number of threads (MDS_MAX_THREADS) is 512.</para>
<note><para>The OSS and MDS automatically start new service threads dynamically, in response to server load within a factor of 4. The default value is calculated the same way as before. Setting the _mu_threads module parameter disables automatic thread creation behavior.</para></note>
</section>
</section>
<section xml:id="dbdoclet.50438272_73839">
<title>25.2 Tuning LNET Parameters</title>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295625" xreflabel=""/>This section describes LNET tunables. that may be necessary on some systems to improve performance. To test the performance of your Lustre network, see <link xl:href="LNETSelfTest.html#50438223_71556">Chapter 23</link>: <link xl:href="LNETSelfTest.html#50438223_21832">Testing Lustre Network Performance (LNET Self-Test)</link>.</para>
+ <para>This section describes LNET tunables. that may be necessary on some systems to improve performance. To test the performance of your Lustre network, see <link xl:href="LNETSelfTest.html#50438223_71556">Chapter 23</link>: <link xl:href="LNETSelfTest.html#50438223_21832">Testing Lustre Network Performance (LNET Self-Test)</link>.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438272_pgfId-1291141" xreflabel=""/>25.2.1 Transmit and Receive Buffer Size</title>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1291142" xreflabel=""/>The kernel allocates buffers for sending and receiving messages on a network.</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295680" xreflabel=""/>ksocklnd has separate parameters for the transmit and receive buffers.</para>
- <screen><anchor xml:id="dbdoclet.50438272_pgfId-1291143" xreflabel=""/>options ksocklnd tx_buffer_size=0 rx_buffer_size=0
+ <title>25.2.1 Transmit and Receive Buffer Size</title>
+ <para>The kernel allocates buffers for sending and receiving messages on a network.</para>
+ <para>ksocklnd has separate parameters for the transmit and receive buffers.</para>
+ <screen>options ksocklnd tx_buffer_size=0 rx_buffer_size=0
</screen>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1291144" xreflabel=""/>If these parameters are left at the default value (0), the system automatically tunes the transmit and receive buffer size. In almost every case, this default produces the best performance. Do not attempt to tune these parameters unless you are a network expert.</para>
+ <para>If these parameters are left at the default value (0), the system automatically tunes the transmit and receive buffer size. In almost every case, this default produces the best performance. Do not attempt to tune these parameters unless you are a network expert.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438272_pgfId-1291145" xreflabel=""/>25.2.2 Hardware Interrupts (enable_irq_affinity)</title>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295715" xreflabel=""/>The hardware interrupts that are generated by network adapters may be handled by any CPU in the system. In some cases, we would like network traffic to remain local to a single CPU to help keep the processor cache warm and minimize the impact of context switches. This is helpful when an SMP system has more than one network interface and ideal when the number of interfaces equals the number of CPUs. To enable the enable_irq_affinity parameter, enter:</para>
- <screen><anchor xml:id="dbdoclet.50438272_pgfId-1295742" xreflabel=""/>options ksocklnd enable_irq_affinity=1
+ <title>25.2.2 Hardware Interrupts (enable_irq_affinity)</title>
+ <para>The hardware interrupts that are generated by network adapters may be handled by any CPU in the system. In some cases, we would like network traffic to remain local to a single CPU to help keep the processor cache warm and minimize the impact of context switches. This is helpful when an SMP system has more than one network interface and ideal when the number of interfaces equals the number of CPUs. To enable the enable_irq_affinity parameter, enter:</para>
+ <screen>options ksocklnd enable_irq_affinity=1
</screen>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295711" xreflabel=""/>In other cases, if you have an SMP platform with a single fast interface such as 10Gb Ethernet and more than two CPUs, you may see performance improve by turning this parameter off.</para>
- <screen><anchor xml:id="dbdoclet.50438272_pgfId-1295736" xreflabel=""/>options ksocklnd enable_irq_affinity=0
+ <para>In other cases, if you have an SMP platform with a single fast interface such as 10Gb Ethernet and more than two CPUs, you may see performance improve by turning this parameter off.</para>
+ <screen>options ksocklnd enable_irq_affinity=0
</screen>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295685" xreflabel=""/>By default, this parameter is off. As always, you should test the performance to compare the impact of changing this parameter.</para>
+ <para>By default, this parameter is off. As always, you should test the performance to compare the impact of changing this parameter.</para>
</section>
</section>
<section xml:id="dbdoclet.50438272_25884">
<title>25.3 Lockless <anchor xml:id="dbdoclet.50438272_marker-1291703" xreflabel=""/>I/O Tunables</title>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1291569" xreflabel=""/>The lockless I/O tunable feature allows servers to ask clients to do lockless I/O (liblustre-style where the server does the locking) on contended files.</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1291570" xreflabel=""/>The lockless I/O patch introduces these tunables:</para>
+ <para>The lockless I/O tunable feature allows servers to ask clients to do lockless I/O (liblustre-style where the server does the locking) on contended files.</para>
+ <para>The lockless I/O patch introduces these tunables:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1291571" xreflabel=""/> OST-side:</para>
+ <para> OST-side:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438272_pgfId-1292156" xreflabel=""/>/proc/fs/lustre/ldlm/namespaces/filter-lustre-*
+ <screen>/proc/fs/lustre/ldlm/namespaces/filter-lustre-*
</screen>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1292119" xreflabel=""/>contended_locks - If the number of lock conflicts in the scan of granted and waiting queues at contended_locks is exceeded, the resource is considered to be contended.</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1292125" xreflabel=""/>contention_seconds - The resource keeps itself in a contended state as set in the parameter.</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1292131" xreflabel=""/>max_nolock_bytes - Server-side locking set only for requests less than the blocks set in the max_nolock_bytes parameter. If this tunable is set to zero (0), it disables server-side locking for read/write requests.</para>
+ <para>contended_locks - If the number of lock conflicts in the scan of granted and waiting queues at contended_locks is exceeded, the resource is considered to be contended.</para>
+ <para>contention_seconds - The resource keeps itself in a contended state as set in the parameter.</para>
+ <para>max_nolock_bytes - Server-side locking set only for requests less than the blocks set in the max_nolock_bytes parameter. If this tunable is set to zero (0), it disables server-side locking for read/write requests.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1291576" xreflabel=""/> Client-side:</para>
+ <para> Client-side:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438272_pgfId-1291577" xreflabel=""/>/proc/fs/lustre/llite/lustre-*
+ <screen>/proc/fs/lustre/llite/lustre-*
</screen>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1292113" xreflabel=""/>contention_seconds - llite inode remembers its contended state for the time specified in this parameter.</para>
+ <para>contention_seconds - llite inode remembers its contended state for the time specified in this parameter.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1291579" xreflabel=""/> Client-side statistics:</para>
+ <para> Client-side statistics:</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1291580" xreflabel=""/>The /proc/fs/lustre/llite/lustre-*/stats file has new rows for lockless I/O statistics.</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1291566" xreflabel=""/>lockless_read_bytes and lockless_write_bytes - To count the total bytes read or written, the client makes its own decisions based on the request size. The client does not communicate with the server if the request size is smaller than the min_nolock_size, without acquiring locks by the client.</para>
+ <para>The /proc/fs/lustre/llite/lustre-*/stats file has new rows for lockless I/O statistics.</para>
+ <para>lockless_read_bytes and lockless_write_bytes - To count the total bytes read or written, the client makes its own decisions based on the request size. The client does not communicate with the server if the request size is smaller than the min_nolock_size, without acquiring locks by the client.</para>
</section>
<section xml:id="dbdoclet.50438272_80545">
<title>25.4 Improving Lustre <anchor xml:id="dbdoclet.50438272_marker-1295851" xreflabel=""/>Performance When Working with Small Files</title>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295854" xreflabel=""/>A Lustre environment where an application writes small file chunks from many clients to a single file will result in bad I/O performance. To improve Lustre'™s performance with small files:</para>
+ <para>A Lustre environment where an application writes small file chunks from many clients to a single file will result in bad I/O performance. To improve Lustre'™s performance with small files:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295855" xreflabel=""/> Have the application aggregate writes some amount before submitting them to Lustre. By default, Lustre enforces POSIX coherency semantics, so it results in lock ping-pong between client nodes if they are all writing to the same file at one time.</para>
+ <para> Have the application aggregate writes some amount before submitting them to Lustre. By default, Lustre enforces POSIX coherency semantics, so it results in lock ping-pong between client nodes if they are all writing to the same file at one time.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295856" xreflabel=""/> Have the application do 4kB O_DIRECT sized I/O to the file and disable locking on the output file. This avoids partial-page IO submissions and, by disabling locking, you avoid contention between clients.</para>
+ <para> Have the application do 4kB O_DIRECT sized I/O to the file and disable locking on the output file. This avoids partial-page IO submissions and, by disabling locking, you avoid contention between clients.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295857" xreflabel=""/> Have the application write contiguous data.</para>
+ <para> Have the application write contiguous data.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295858" xreflabel=""/> Add more disks or use SSD disks for the OSTs. This dramatically improves the IOPS rate. Consider creating larger OSTs rather than many smaller OSTs due to less overhead (journal, connections, etc).</para>
+ <para> Add more disks or use SSD disks for the OSTs. This dramatically improves the IOPS rate. Consider creating larger OSTs rather than many smaller OSTs due to less overhead (journal, connections, etc).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295859" xreflabel=""/> Use RAID-1+0 OSTs instead of RAID-5/6. There is RAID parity overhead for writing small chunks of data to disk.</para>
+ <para> Use RAID-1+0 OSTs instead of RAID-5/6. There is RAID parity overhead for writing small chunks of data to disk.</para>
</listitem>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438272_45406">
<title>25.5 Understanding Why Write Performance Is Better Than Read Performance</title>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295894" xreflabel=""/>Typically, the performance of write operations on a Lustre cluster is better than read operations. When doing writes, all clients are sending write RPCs asynchronously. The RPCs are allocated, and written to disk in the order they arrive. In many cases, this allows the back-end storage to aggregate writes efficiently.</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295895" xreflabel=""/>In the case of read operations, the reads from clients may come in a different order and need a lot of seeking to get read from the disk. This noticeably hampers the read throughput.</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295896" xreflabel=""/>Currently, there is no readahead on the OSTs themselves, though the clients do readahead. If there are lots of clients doing reads it would not be possible to do any readahead in any case because of memory consumption (consider that even a single RPC (1 MB) readahead for 1000 clients would consume 1 GB of RAM).</para>
- <para><anchor xml:id="dbdoclet.50438272_pgfId-1295897" xreflabel=""/>For file systems that use socklnd (TCP, Ethernet) as interconnect, there is also additional CPU overhead because the client cannot receive data without copying it from the network buffers. In the write case, the client CAN send data without the additional data copy. This means that the client is more likely to become CPU-bound during reads than writes.</para>
+ <para>Typically, the performance of write operations on a Lustre cluster is better than read operations. When doing writes, all clients are sending write RPCs asynchronously. The RPCs are allocated, and written to disk in the order they arrive. In many cases, this allows the back-end storage to aggregate writes efficiently.</para>
+ <para>In the case of read operations, the reads from clients may come in a different order and need a lot of seeking to get read from the disk. This noticeably hampers the read throughput.</para>
+ <para>Currently, there is no readahead on the OSTs themselves, though the clients do readahead. If there are lots of clients doing reads it would not be possible to do any readahead in any case because of memory consumption (consider that even a single RPC (1 MB) readahead for 1000 clients would consume 1 GB of RAM).</para>
+ <para>For file systems that use socklnd (TCP, Ethernet) as interconnect, there is also additional CPU overhead because the client cannot receive data without copying it from the network buffers. In the write case, the client CAN send data without the additional data copy. This means that the client is more likely to become CPU-bound during reads than writes.</para>
</section>
</chapter>
<section xml:id="dbdoclet.50438213_13563">
<title>20.1 Lustre Failover and <anchor xml:id="dbdoclet.50438213_marker-1301522" xreflabel=""/>Multiple-Mount Protection</title>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1292341" xreflabel=""/>The failover functionality in Lustre is implemented with the multiple-mount protection (MMP) feature, which protects the file system from being mounted simultaneously to more than one node. This feature is important in a shared storage environment (for example, when a failover pair of OSTs share a partition).</para>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1292342" xreflabel=""/>Lustre's backend file system, ldiskfs, supports the MMP mechanism. A block in the file system is updated by a kmmpd daemon at one second intervals, and a sequence number is written in this block. If the file system is cleanly unmounted, then a special "clean" sequence is written to this block. When mounting the file system, ldiskfs checks if the MMP block has a clean sequence or not.</para>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1292344" xreflabel=""/>Even if the MMP block has a clean sequence, ldiskfs waits for some interval to guard against the following situations:</para>
+ <para>The failover functionality in Lustre is implemented with the multiple-mount protection (MMP) feature, which protects the file system from being mounted simultaneously to more than one node. This feature is important in a shared storage environment (for example, when a failover pair of OSTs share a partition).</para>
+ <para>Lustre's backend file system, ldiskfs, supports the MMP mechanism. A block in the file system is updated by a kmmpd daemon at one second intervals, and a sequence number is written in this block. If the file system is cleanly unmounted, then a special "clean" sequence is written to this block. When mounting the file system, ldiskfs checks if the MMP block has a clean sequence or not.</para>
+ <para>Even if the MMP block has a clean sequence, ldiskfs waits for some interval to guard against the following situations:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1292346" xreflabel=""/> If I/O traffic is heavy, it may take longer for the MMP block to be updated.</para>
+ <para> If I/O traffic is heavy, it may take longer for the MMP block to be updated.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1292404" xreflabel=""/> If another node is trying to mount the same file system, a "race" condition may occur.</para>
+ <para> If another node is trying to mount the same file system, a "race" condition may occur.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1292349" xreflabel=""/>With MMP enabled, mounting a clean file system takes at least 10 seconds. If the file system was not cleanly unmounted, then the file system mount may require additional time.</para>
+ <para>With MMP enabled, mounting a clean file system takes at least 10 seconds. If the file system was not cleanly unmounted, then the file system mount may require additional time.</para>
<note><para>The MMP feature is only supported on Linux kernel versions >= 2.6.9.</para></note>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438213_pgfId-1292355" xreflabel=""/>20.1.1 Working with Multiple-Mount Protection</title>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1300904" xreflabel=""/>On a new Lustre file system, MMP is automatically enabled by mkfs.lustre at format time if failover is being used and the kernel and e2fsprogs version support it. On an existing file system, a Lustre administrator can manually enable MMP when the file system is unmounted.</para>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1294096" xreflabel=""/>Use the following commands to determine whether MMP is running in Lustre and to enable or disable the MMP feature.</para>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1294086" xreflabel=""/>To determine if MMP is enabled, run:</para>
- <screen><anchor xml:id="dbdoclet.50438213_pgfId-1294100" xreflabel=""/>dumpe2fs -h <device>|grep mmp
+ <title>20.1.1 Working with Multiple-Mount Protection</title>
+ <para>On a new Lustre file system, MMP is automatically enabled by mkfs.lustre at format time if failover is being used and the kernel and e2fsprogs version support it. On an existing file system, a Lustre administrator can manually enable MMP when the file system is unmounted.</para>
+ <para>Use the following commands to determine whether MMP is running in Lustre and to enable or disable the MMP feature.</para>
+ <para>To determine if MMP is enabled, run:</para>
+ <screen>dumpe2fs -h <device>|grep mmp
</screen>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1292415" xreflabel=""/>Here is a sample command:</para>
- <screen><anchor xml:id="dbdoclet.50438213_pgfId-1292417" xreflabel=""/>dumpe2fs -h /dev/sdc | grep mmp
-<anchor xml:id="dbdoclet.50438213_pgfId-1294106" xreflabel=""/>Filesystem features: has_journal ext_attr resize_inode dir_index
-<anchor xml:id="dbdoclet.50438213_pgfId-1294109" xreflabel=""/>filetype extent mmp sparse_super large_file uninit_bg
+ <para>Here is a sample command:</para>
+ <screen>dumpe2fs -h /dev/sdc | grep mmp
+Filesystem features: has_journal ext_attr resize_inode dir_index
+filetype extent mmp sparse_super large_file uninit_bg
</screen>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1292423" xreflabel=""/>To manually disable MMP, run:</para>
- <screen><anchor xml:id="dbdoclet.50438213_pgfId-1294115" xreflabel=""/>tune2fs -O ^mmp <device>
+ <para>To manually disable MMP, run:</para>
+ <screen>tune2fs -O ^mmp <device>
</screen>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1292487" xreflabel=""/>To manually enable MMP, run:</para>
- <screen><anchor xml:id="dbdoclet.50438213_pgfId-1294119" xreflabel=""/>tune2fs -O mmp <device>
+ <para>To manually enable MMP, run:</para>
+ <screen>tune2fs -O mmp <device>
</screen>
- <para><anchor xml:id="dbdoclet.50438213_pgfId-1292426" xreflabel=""/>When MMP is enabled, if ldiskfs detects multiple mount attempts after the file system is mounted, it blocks these later mount attempts and reports the time when the MMP block was last updated, the node name, and the device name of the node where the file system is currently mounted.</para>
+ <para>When MMP is enabled, if ldiskfs detects multiple mount attempts after the file system is mounted, it blocks these later mount attempts and reports the time when the MMP block was last updated, the node name, and the device name of the node where the file system is currently mounted.</para>
</section>
</section>
</chapter>
<info>
<title xml:id='managingfilesystemio.title'>Managing the File System and I/O</title>
</info>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1291802" xreflabel=""/>This chapter describes file striping and I/O options, and includes the following sections:</para>
+ <para>This chapter describes file striping and I/O options, and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438211_17536"/></para>
<section xml:id="dbdoclet.50438211_17536">
<title>19.1 Handling <anchor xml:id="dbdoclet.50438211_marker-1295529" xreflabel=""/>Full OSTs</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296599" xreflabel=""/>Sometimes a Lustre file system becomes unbalanced, often due to incorrectly-specified stripe settings, or when very large files are created that are not striped over all of the OSTs. If an OST is full and an attempt is made to write more information to the file system, an error occurs. The procedures below describe how to handle a full OST.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296601" xreflabel=""/>The MDS will normally handle space balancing automatically at file creation time, and this procedure is normally not needed, but may be desirable in certain circumstances (e.g. when creating very large files that would consume more than the total free space of the full OSTs).</para>
+ <para>Sometimes a Lustre file system becomes unbalanced, often due to incorrectly-specified stripe settings, or when very large files are created that are not striped over all of the OSTs. If an OST is full and an attempt is made to write more information to the file system, an error occurs. The procedures below describe how to handle a full OST.</para>
+ <para>The MDS will normally handle space balancing automatically at file creation time, and this procedure is normally not needed, but may be desirable in certain circumstances (e.g. when creating very large files that would consume more than the total free space of the full OSTs).</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438211_pgfId-1294599" xreflabel=""/>19.1.1 <anchor xml:id="dbdoclet.50438211_54434" xreflabel=""/>Checking OST Space Usage</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1294600" xreflabel=""/>The example below shows an unbalanced file system:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1294601" xreflabel=""/>root@LustreClient01 ~]# lfs df -h
-<anchor xml:id="dbdoclet.50438211_pgfId-1294602" xreflabel=""/>UUID bytes Used Available \
+ <title>19.1.1 <anchor xml:id="dbdoclet.50438211_54434" xreflabel=""/>Checking OST Space Usage</title>
+ <para>The example below shows an unbalanced file system:</para>
+ <screen>root@LustreClient01 ~]# lfs df -h
+UUID bytes Used Available \
Use% Mounted on
-<anchor xml:id="dbdoclet.50438211_pgfId-1294603" xreflabel=""/>lustre-MDT0000_UUID 4.4G 214.5M 3.9G \
+lustre-MDT0000_UUID 4.4G 214.5M 3.9G \
4% /mnt/lustre[MDT:0]
-<anchor xml:id="dbdoclet.50438211_pgfId-1294604" xreflabel=""/>lustre-OST0000_UUID 2.0G 751.3M 1.1G \
+lustre-OST0000_UUID 2.0G 751.3M 1.1G \
37% /mnt/lustre[OST:0]
-<anchor xml:id="dbdoclet.50438211_pgfId-1294605" xreflabel=""/>lustre-OST0001_UUID 2.0G 755.3M 1.1G \
+lustre-OST0001_UUID 2.0G 755.3M 1.1G \
37% /mnt/lustre[OST:1]
-<anchor xml:id="dbdoclet.50438211_pgfId-1294606" xreflabel=""/>lustre-OST0002_UUID 2.0G 1.7G 155.1M \
+lustre-OST0002_UUID 2.0G 1.7G 155.1M \
86% /mnt/lustre[OST:2] <-
-<anchor xml:id="dbdoclet.50438211_pgfId-1294607" xreflabel=""/>lustre-OST0003_UUID 2.0G 751.3M 1.1G \
+lustre-OST0003_UUID 2.0G 751.3M 1.1G \
37% /mnt/lustre[OST:3]
-<anchor xml:id="dbdoclet.50438211_pgfId-1294608" xreflabel=""/>lustre-OST0004_UUID 2.0G 747.3M 1.1G \
+lustre-OST0004_UUID 2.0G 747.3M 1.1G \
37% /mnt/lustre[OST:4]
-<anchor xml:id="dbdoclet.50438211_pgfId-1294609" xreflabel=""/>lustre-OST0005_UUID 2.0G 743.3M 1.1G \
+lustre-OST0005_UUID 2.0G 743.3M 1.1G \
36% /mnt/lustre[OST:5]
-<anchor xml:id="dbdoclet.50438211_pgfId-1294610" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438211_pgfId-1294611" xreflabel=""/>filesystem summary: 11.8G 5.4G 5.8G \
+
+filesystem summary: 11.8G 5.4G 5.8G \
45% /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1294626" xreflabel=""/>In this case, OST:2 is almost full and when an attempt is made to write additional information to the file system (even with uniform striping over all the OSTs), the write command fails as follows:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1294627" xreflabel=""/>[root@LustreClient01 ~]# lfs setstripe /mnt/lustre 4M 0 -1
-<anchor xml:id="dbdoclet.50438211_pgfId-1294628" xreflabel=""/>[root@LustreClient01 ~]# dd if=/dev/zero of=/mnt/lustre/test_3 \ bs=10M cou\
+ <para>In this case, OST:2 is almost full and when an attempt is made to write additional information to the file system (even with uniform striping over all the OSTs), the write command fails as follows:</para>
+ <screen>[root@LustreClient01 ~]# lfs setstripe /mnt/lustre 4M 0 -1
+[root@LustreClient01 ~]# dd if=/dev/zero of=/mnt/lustre/test_3 \ bs=10M cou\
nt=100
-<anchor xml:id="dbdoclet.50438211_pgfId-1294629" xreflabel=""/>dd: writing `/mnt/lustre/test_3': No space left on device
-<anchor xml:id="dbdoclet.50438211_pgfId-1294630" xreflabel=""/>98+0 records in
-<anchor xml:id="dbdoclet.50438211_pgfId-1294631" xreflabel=""/>97+0 records out
-<anchor xml:id="dbdoclet.50438211_pgfId-1294632" xreflabel=""/>1017192448 bytes (1.0 GB) copied, 23.2411 seconds, 43.8 MB/s
+dd: writing `/mnt/lustre/test_3': No space left on device
+98+0 records in
+97+0 records out
+1017192448 bytes (1.0 GB) copied, 23.2411 seconds, 43.8 MB/s
</screen>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438211_pgfId-1294633" xreflabel=""/>19.1.2 Taking a Full OST Offline</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296618" xreflabel=""/>To avoid running out of space in the file system, if the OST usage is imbalanced and one or more OSTs are close to being full while there are others that have a lot of space, the full OSTs may optionally be deactivated at the MDS to prevent the MDS from allocating new objects there.</para>
+ <title>19.1.2 Taking a Full OST Offline</title>
+ <para>To avoid running out of space in the file system, if the OST usage is imbalanced and one or more OSTs are close to being full while there are others that have a lot of space, the full OSTs may optionally be deactivated at the MDS to prevent the MDS from allocating new objects there.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1294635" xreflabel=""/>Log into the MDS server:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1294636" xreflabel=""/>[root@LustreClient01 ~]# ssh root@192.168.0.10
-<anchor xml:id="dbdoclet.50438211_pgfId-1294637" xreflabel=""/>root@192.168.0.10's password:
-<anchor xml:id="dbdoclet.50438211_pgfId-1294638" xreflabel=""/>Last login: Wed Nov 26 13:35:12 2008 from 192.168.0.6
+ <para>Log into the MDS server:</para>
+ <screen>[root@LustreClient01 ~]# ssh root@192.168.0.10
+root@192.168.0.10's password:
+Last login: Wed Nov 26 13:35:12 2008 from 192.168.0.6
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1294639" xreflabel=""/>Use the lctl dl command to show the status of all file system components:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1294640" xreflabel=""/>[root@mds ~]# lctl dl
-<anchor xml:id="dbdoclet.50438211_pgfId-1294641" xreflabel=""/>0 UP mgs MGS MGS 9
-<anchor xml:id="dbdoclet.50438211_pgfId-1294642" xreflabel=""/>1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294643" xreflabel=""/>2 UP mdt MDS MDS_uuid 3
-<anchor xml:id="dbdoclet.50438211_pgfId-1294644" xreflabel=""/>3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
-<anchor xml:id="dbdoclet.50438211_pgfId-1294645" xreflabel=""/>4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294646" xreflabel=""/>5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294647" xreflabel=""/>6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294648" xreflabel=""/>7 UP osc lustre-OST0002-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294649" xreflabel=""/>8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294650" xreflabel=""/>9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294651" xreflabel=""/>10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5
+ <para>Use the lctl dl command to show the status of all file system components:</para>
+ <screen>[root@mds ~]# lctl dl
+0 UP mgs MGS MGS 9
+1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
+2 UP mdt MDS MDS_uuid 3
+3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
+4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
+5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
+6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
+7 UP osc lustre-OST0002-osc lustre-mdtlov_UUID 5
+8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
+9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
+10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1294652" xreflabel=""/>Use lctl deactivate to take the full OST offline:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1294653" xreflabel=""/>[root@mds ~]# lctl --device 7 deactivate
+ <para>Use lctl deactivate to take the full OST offline:</para>
+ <screen>[root@mds ~]# lctl --device 7 deactivate
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1294664" xreflabel=""/>Display the status of the file system components:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1294665" xreflabel=""/>[root@mds ~]# lctl dl
-<anchor xml:id="dbdoclet.50438211_pgfId-1294666" xreflabel=""/>0 UP mgs MGS MGS 9
-<anchor xml:id="dbdoclet.50438211_pgfId-1294667" xreflabel=""/>1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294668" xreflabel=""/>2 UP mdt MDS MDS_uuid 3
-<anchor xml:id="dbdoclet.50438211_pgfId-1294669" xreflabel=""/>3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
-<anchor xml:id="dbdoclet.50438211_pgfId-1294670" xreflabel=""/>4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294671" xreflabel=""/>5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294672" xreflabel=""/>6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294673" xreflabel=""/>7 IN osc lustre-OST0002-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294674" xreflabel=""/>8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294675" xreflabel=""/>9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294676" xreflabel=""/>10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5
+ <para>Display the status of the file system components:</para>
+ <screen>[root@mds ~]# lctl dl
+0 UP mgs MGS MGS 9
+1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
+2 UP mdt MDS MDS_uuid 3
+3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
+4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
+5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
+6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
+7 IN osc lustre-OST0002-osc lustre-mdtlov_UUID 5
+8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
+9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
+10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5
</screen>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296630" xreflabel=""/>The device list shows that OST0002 is now inactive. When new files are created in the file system, they will only use the remaining active OSTs. Either manual space rebalancing can be done by migrating data to other OSTs, as shown in the next section, or normal file deletion and creation can be allowed to passively rebalance the space usage.</para>
+ <para>The device list shows that OST0002 is now inactive. When new files are created in the file system, they will only use the remaining active OSTs. Either manual space rebalancing can be done by migrating data to other OSTs, as shown in the next section, or normal file deletion and creation can be allowed to passively rebalance the space usage.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438211_pgfId-1294681" xreflabel=""/>19.1.3 Migrating Data within a File System</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296644" xreflabel=""/>As stripes cannot be moved within the file system, data must be migrated manually by copying and renaming the file, removing the original file, and renaming the new file with the original file name. The simplest way to do this is to use the lfs_migrate command (see <link xl:href="UserUtilities.html#50438206_42260">lfs_migrate</link>). However, the steps for migrating a file by hand are also shown here for reference.</para>
+ <title>19.1.3 Migrating Data within a File System</title>
+ <para>As stripes cannot be moved within the file system, data must be migrated manually by copying and renaming the file, removing the original file, and renaming the new file with the original file name. The simplest way to do this is to use the lfs_migrate command (see <link xl:href="UserUtilities.html#50438206_42260">lfs_migrate</link>). However, the steps for migrating a file by hand are also shown here for reference.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1294683" xreflabel=""/>Identify the file(s) to be moved.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296660" xreflabel=""/>In the example below, output from the getstripe command indicates that the file test_2 is located entirely on OST2:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1296668" xreflabel=""/>[client]# lfs getstripe /mnt/lustre/test_2
-<anchor xml:id="dbdoclet.50438211_pgfId-1296669" xreflabel=""/>/mnt/lustre/test_2
-<anchor xml:id="dbdoclet.50438211_pgfId-1296670" xreflabel=""/>obdidx objid objid group
-<anchor xml:id="dbdoclet.50438211_pgfId-1296671" xreflabel=""/> 2 8 0x8 0
+ <para>Identify the file(s) to be moved.</para>
+ <para>In the example below, output from the getstripe command indicates that the file test_2 is located entirely on OST2:</para>
+ <screen>[client]# lfs getstripe /mnt/lustre/test_2
+/mnt/lustre/test_2
+obdidx objid objid group
+ 2 8 0x8 0
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296685" xreflabel=""/>To move single object(s), create a new copy and remove the original. Enter:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1296698" xreflabel=""/>[client]# cp -a /mnt/lustre/test_2 /mnt/lustre/test_2.tmp
-<anchor xml:id="dbdoclet.50438211_pgfId-1296699" xreflabel=""/>[client]# mv /mnt/lustre/test_2.tmp /mnt/lustre/test_2
+ <para>To move single object(s), create a new copy and remove the original. Enter:</para>
+ <screen>[client]# cp -a /mnt/lustre/test_2 /mnt/lustre/test_2.tmp
+[client]# mv /mnt/lustre/test_2.tmp /mnt/lustre/test_2
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296711" xreflabel=""/>To migrate large files from one or more OSTs, enter:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1296718" xreflabel=""/>[client]# lfs find --ost {OST_UUID} -size +1G | lfs_migrate -y
+ <para>To migrate large files from one or more OSTs, enter:</para>
+ <screen>[client]# lfs find --ost {OST_UUID} -size +1G | lfs_migrate -y
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296709" xreflabel=""/>Check the file system balance.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296435" xreflabel=""/>The df output in the example below shows a more balanced system compared to the df output in the example in <link xl:href="ManagingFileSystemIO.html#50438211_54434">Checking OST Space Usage</link>.</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1296729" xreflabel=""/>[client]# lfs df -h
-<anchor xml:id="dbdoclet.50438211_pgfId-1296730" xreflabel=""/>UUID bytes Used Available Use% \
+ <para>Check the file system balance.</para>
+ <para>The df output in the example below shows a more balanced system compared to the df output in the example in <link xl:href="ManagingFileSystemIO.html#50438211_54434">Checking OST Space Usage</link>.</para>
+ <screen>[client]# lfs df -h
+UUID bytes Used Available Use% \
Mounted on
-<anchor xml:id="dbdoclet.50438211_pgfId-1296731" xreflabel=""/>lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% \
+lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% \
/mnt/lustre[MDT:0]
-<anchor xml:id="dbdoclet.50438211_pgfId-1296732" xreflabel=""/>lustre-OST0000_UUID 2.0G 1.3G 598.1M 65% \
+lustre-OST0000_UUID 2.0G 1.3G 598.1M 65% \
/mnt/lustre[OST:0]
-<anchor xml:id="dbdoclet.50438211_pgfId-1296733" xreflabel=""/>lustre-OST0001_UUID 2.0G 1.3G 594.1M 65% \
+lustre-OST0001_UUID 2.0G 1.3G 594.1M 65% \
/mnt/lustre[OST:1]
-<anchor xml:id="dbdoclet.50438211_pgfId-1296734" xreflabel=""/>lustre-OST0002_UUID 2.0G 913.4M 1000.0M 45% \
+lustre-OST0002_UUID 2.0G 913.4M 1000.0M 45% \
/mnt/lustre[OST:2]
-<anchor xml:id="dbdoclet.50438211_pgfId-1296735" xreflabel=""/>lustre-OST0003_UUID 2.0G 1.3G 602.1M 65% \
+lustre-OST0003_UUID 2.0G 1.3G 602.1M 65% \
/mnt/lustre[OST:3]
-<anchor xml:id="dbdoclet.50438211_pgfId-1296736" xreflabel=""/>lustre-OST0004_UUID 2.0G 1.3G 606.1M 64% \
+lustre-OST0004_UUID 2.0G 1.3G 606.1M 64% \
/mnt/lustre[OST:4]
-<anchor xml:id="dbdoclet.50438211_pgfId-1296737" xreflabel=""/>lustre-OST0005_UUID 2.0G 1.3G 610.1M 64% \
+lustre-OST0005_UUID 2.0G 1.3G 610.1M 64% \
/mnt/lustre[OST:5]
-<anchor xml:id="dbdoclet.50438211_pgfId-1296738" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438211_pgfId-1296739" xreflabel=""/>filesystem summary: 11.8G 7.3G 3.9G 61% \
+
+filesystem summary: 11.8G 7.3G 3.9G 61% \
/mnt/lustre
</screen>
</listitem></orderedlist>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438211_pgfId-1296756" xreflabel=""/>19.1.4 Returning an Inactive OST Back Online</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296764" xreflabel=""/>Once the deactivated OST(s) no longer are severely imbalanced, due to either active or passive data redistribution, they should be reactivated so they will again have new files allocated on them.</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1294729" xreflabel=""/>[mds]# lctl --device 7 activate
-<anchor xml:id="dbdoclet.50438211_pgfId-1294730" xreflabel=""/>[mds]# lctl dl
-<anchor xml:id="dbdoclet.50438211_pgfId-1294731" xreflabel=""/> 0 UP mgs MGS MGS 9
-<anchor xml:id="dbdoclet.50438211_pgfId-1294732" xreflabel=""/> 1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-816dd1e813 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294733" xreflabel=""/> 2 UP mdt MDS MDS_uuid 3
-<anchor xml:id="dbdoclet.50438211_pgfId-1294734" xreflabel=""/> 3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
-<anchor xml:id="dbdoclet.50438211_pgfId-1294735" xreflabel=""/> 4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294736" xreflabel=""/> 5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294737" xreflabel=""/> 6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294738" xreflabel=""/> 7 UP osc lustre-OST0002-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294739" xreflabel=""/> 8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294740" xreflabel=""/> 9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
-<anchor xml:id="dbdoclet.50438211_pgfId-1294741" xreflabel=""/> 10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID
+ <title>19.1.4 Returning an Inactive OST Back Online</title>
+ <para>Once the deactivated OST(s) no longer are severely imbalanced, due to either active or passive data redistribution, they should be reactivated so they will again have new files allocated on them.</para>
+ <screen>[mds]# lctl --device 7 activate
+[mds]# lctl dl
+ 0 UP mgs MGS MGS 9
+ 1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-816dd1e813 5
+ 2 UP mdt MDS MDS_uuid 3
+ 3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
+ 4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
+ 5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
+ 6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
+ 7 UP osc lustre-OST0002-osc lustre-mdtlov_UUID 5
+ 8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
+ 9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
+ 10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438211_75549">
<title>19.2 Creating and Managing <anchor xml:id="dbdoclet.50438211_marker-1295531" xreflabel=""/>OST Pools</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293507" xreflabel=""/><anchor xml:id="dbdoclet.50438211_81568" xreflabel=""/>The OST pools feature enables users to group OSTs together to make object placement more flexible. A 'pool' is the name associated with an arbitrary subset of OSTs in a Lustre cluster.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293508" xreflabel=""/>OST pools follow these rules:</para>
+ <para><anchor xml:id="dbdoclet.50438211_81568" xreflabel=""/>The OST pools feature enables users to group OSTs together to make object placement more flexible. A 'pool' is the name associated with an arbitrary subset of OSTs in a Lustre cluster.</para>
+ <para>OST pools follow these rules:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293509" xreflabel=""/> An OST can be a member of multiple pools.</para>
+ <para> An OST can be a member of multiple pools.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293510" xreflabel=""/> No ordering of OSTs in a pool is defined or implied.</para>
+ <para> No ordering of OSTs in a pool is defined or implied.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1294572" xreflabel=""/> Stripe allocation within a pool follows the same rules as the normal stripe allocator.</para>
+ <para> Stripe allocation within a pool follows the same rules as the normal stripe allocator.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293511" xreflabel=""/> OST membership in a pool is flexible, and can change over time.</para>
+ <para> OST membership in a pool is flexible, and can change over time.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293512" xreflabel=""/>When an OST pool is defined, it can be used to allocate files. When file or directory striping is set to a pool, only OSTs in the pool are candidates for striping. If a stripe_index is specified which refers to an OST that is not a member of the pool, an error is returned.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293513" xreflabel=""/>OST pools are used only at file creation. If the definition of a pool changes (an OST is added or removed or the pool is destroyed), already-created files are not affected.</para>
+ <para>When an OST pool is defined, it can be used to allocate files. When file or directory striping is set to a pool, only OSTs in the pool are candidates for striping. If a stripe_index is specified which refers to an OST that is not a member of the pool, an error is returned.</para>
+ <para>OST pools are used only at file creation. If the definition of a pool changes (an OST is added or removed or the pool is destroyed), already-created files are not affected.</para>
<note><para>An error (EINVAL) results if you create a file using an empty pool.</para></note>
<note><para>If a directory has pool striping set and the pool is subsequently removed, the new files created in this directory have the (non-pool) default striping pattern for that directory applied and no error is returned.</para></note>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438211_pgfId-1293517" xreflabel=""/>19.2.1 <anchor xml:id="dbdoclet.50438211_71392" xreflabel=""/>Working with OST Pools</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293518" xreflabel=""/>OST pools are defined in the configuration log on the MGS. Use the lctl command to:</para>
+ <title>19.2.1 <anchor xml:id="dbdoclet.50438211_71392" xreflabel=""/>Working with OST Pools</title>
+ <para>OST pools are defined in the configuration log on the MGS. Use the lctl command to:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293519" xreflabel=""/> Create/destroy a pool</para>
+ <para> Create/destroy a pool</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293520" xreflabel=""/> Add/remove OSTs in a pool</para>
+ <para> Add/remove OSTs in a pool</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1295826" xreflabel=""/> List pools and OSTs in a specific pool</para>
+ <para> List pools and OSTs in a specific pool</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1295827" xreflabel=""/>The lctl command MUST be run on the MGS. Another requirement for managing OST pools is to either have the MDT and MGS on the same node or have a Lustre client mounted on the MGS node, if it is separate from the MDS. This is needed to validate the pool commands being run are correct.</para>
+ <para>The lctl command MUST be run on the MGS. Another requirement for managing OST pools is to either have the MDT and MGS on the same node or have a Lustre client mounted on the MGS node, if it is separate from the MDS. This is needed to validate the pool commands being run are correct.</para>
<caution><para>Running the writeconf command on the MDS erases all pools information (as well as any other parameters set using lctl conf_param). We recommend that the pools definitions (and conf_param settings) be executed using a script, so they can be reproduced easily after a writeconf is performed.</para></caution>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293524" xreflabel=""/>To create a new pool, run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293525" xreflabel=""/>lctl pool_new <fsname>.<poolname>
+ <para>To create a new pool, run:</para>
+ <screen>lctl pool_new <fsname>.<poolname>
</screen>
<note><para>The pool name is an ASCII string up to 16 characters.</para></note>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293527" xreflabel=""/>To add the named OST to a pool, run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293528" xreflabel=""/>lctl pool_add <fsname>.<poolname> <ost_list>
+ <para>To add the named OST to a pool, run:</para>
+ <screen>lctl pool_add <fsname>.<poolname> <ost_list>
</screen>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293529" xreflabel=""/>Where:</para>
+ <para>Where:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1295844" xreflabel=""/><ost_list> is <fsname->OST<index_range>[_UUID]</para>
+ <para><ost_list> is <fsname->OST<index_range>[_UUID]</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1295845" xreflabel=""/><index_range> is <ost_index_start>-<ost_index_end>[,<index_range>] or <ost_index_start>-<ost_index_end>/<step></para>
+ <para><index_range> is <ost_index_start>-<ost_index_end>[,<index_range>] or <ost_index_start>-<ost_index_end>/<step></para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1294344" xreflabel=""/>If the leading <fsname> and/or ending _UUID are missing, they are automatically added.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293531" xreflabel=""/> For example, to add even-numbered OSTs to pool1 on file system lustre, run a single command (add) to add many OSTs to the pool at one time:</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293532" xreflabel=""/>lctl pool_add lustre.pool1 OST[0-10/2]</para>
+ <para>If the leading <fsname> and/or ending _UUID are missing, they are automatically added.</para>
+ <para> For example, to add even-numbered OSTs to pool1 on file system lustre, run a single command (add) to add many OSTs to the pool at one time:</para>
+ <para>lctl pool_add lustre.pool1 OST[0-10/2]</para>
<note><para>Each time an OST is added to a pool, a new llog configuration record is created. For convenience, you can run a single command.</para></note>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293534" xreflabel=""/>To remove a named OST from a pool, run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293535" xreflabel=""/>lctl pool_remove <fsname>.<poolname> <ost_list>
+ <para>To remove a named OST from a pool, run:</para>
+ <screen>lctl pool_remove <fsname>.<poolname> <ost_list>
</screen>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293536" xreflabel=""/>To destroy a pool, run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293537" xreflabel=""/>lctl pool_destroy <fsname>.<poolname>
+ <para>To destroy a pool, run:</para>
+ <screen>lctl pool_destroy <fsname>.<poolname>
</screen>
<note><para>All OSTs must be removed from a pool before it can be destroyed.</para></note>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293539" xreflabel=""/>To list pools in the named file system, run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293540" xreflabel=""/>lctl pool_list <fsname> | <pathname>
+ <para>To list pools in the named file system, run:</para>
+ <screen>lctl pool_list <fsname> | <pathname>
</screen>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293541" xreflabel=""/>To list OSTs in a named pool, run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293542" xreflabel=""/>lctl pool_list <fsname>.<poolname>
+ <para>To list OSTs in a named pool, run:</para>
+ <screen>lctl pool_list <fsname>.<poolname>
</screen>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438211_pgfId-1293543" xreflabel=""/>19.2.1.1 Using the lfs Command with OST Pools</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293544" xreflabel=""/>Several lfs commands can be run with OST pools. Use the lfs setstripe command to associate a directory with an OST pool. This causes all new regular files and directories in the directory to be created in the pool. The lfs command can be used to list pools in a file system and OSTs in a named pool.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293545" xreflabel=""/>To associate a directory with a pool, so all new files and directories will be created in the pool, run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293546" xreflabel=""/>lfs setstripe --pool|-p pool_name <filename|dirname>
+ <title>19.2.1.1 Using the lfs Command with OST Pools</title>
+ <para>Several lfs commands can be run with OST pools. Use the lfs setstripe command to associate a directory with an OST pool. This causes all new regular files and directories in the directory to be created in the pool. The lfs command can be used to list pools in a file system and OSTs in a named pool.</para>
+ <para>To associate a directory with a pool, so all new files and directories will be created in the pool, run:</para>
+ <screen>lfs setstripe --pool|-p pool_name <filename|dirname>
</screen>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293547" xreflabel=""/>To set striping patterns, run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293548" xreflabel=""/>lfs setstripe [--size|-s stripe_size] [--offset|-o start_ost]
-<anchor xml:id="dbdoclet.50438211_pgfId-1293549" xreflabel=""/> [--count|-c stripe_count] [--pool|-p pool_name]
-<anchor xml:id="dbdoclet.50438211_pgfId-1293550" xreflabel=""/> <dir|filename>
+ <para>To set striping patterns, run:</para>
+ <screen>lfs setstripe [--size|-s stripe_size] [--offset|-o start_ost]
+ [--count|-c stripe_count] [--pool|-p pool_name]
+ <dir|filename>
</screen>
<note><para>If you specify striping with an invalid pool name, because the pool does not exist or the pool name was mistyped, lfs setstripe returns an error. Run lfs pool_list to make sure the pool exists and the pool name is entered correctly.</para></note>
<note><para>The --pool option for lfs setstripe is compatible with other modifiers. For example, you can set striping on a directory to use an explicit starting index.</para></note>
</section>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438211_pgfId-1293557" xreflabel=""/>19.2.2 <anchor xml:id="dbdoclet.50438211_62780" xreflabel=""/>Tips for Using OST Pools</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293558" xreflabel=""/>Here are several suggestions for using OST pools.</para>
+ <title>19.2.2 <anchor xml:id="dbdoclet.50438211_62780" xreflabel=""/>Tips for Using OST Pools</title>
+ <para>Here are several suggestions for using OST pools.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293559" xreflabel=""/> A directory or file can be given an extended attribute (EA), that restricts striping to a pool.</para>
+ <para> A directory or file can be given an extended attribute (EA), that restricts striping to a pool.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293560" xreflabel=""/> Pools can be used to group OSTs with the same technology or performance (slower or faster), or that are preferred for certain jobs. Examples are SATA OSTs versus SAS OSTs or remote OSTs versus local OSTs.</para>
+ <para> Pools can be used to group OSTs with the same technology or performance (slower or faster), or that are preferred for certain jobs. Examples are SATA OSTs versus SAS OSTs or remote OSTs versus local OSTs.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293561" xreflabel=""/> A file created in an OST pool tracks the pool by keeping the pool name in the file LOV EA.</para>
+ <para> A file created in an OST pool tracks the pool by keeping the pool name in the file LOV EA.</para>
</listitem>
</itemizedlist>
</section>
</section>
<section xml:id="dbdoclet.50438211_11204">
<title>19.3 Adding an OST to a Lustre File System</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296450" xreflabel=""/>To add an OST to existing Lustre file system:</para>
+ <para>To add an OST to existing Lustre file system:</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296451" xreflabel=""/>Add a new OST by passing on the following commands, run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1296452" xreflabel=""/>$ mkfs.lustre --fsname=spfs --ost --mgsnode=mds16@tcp0 /dev/sda
-<anchor xml:id="dbdoclet.50438211_pgfId-1296453" xreflabel=""/>$ mkdir -p /mnt/test/ost0
-<anchor xml:id="dbdoclet.50438211_pgfId-1296454" xreflabel=""/>$ mount -t lustre /dev/sda /mnt/test/ost0
+ <para>Add a new OST by passing on the following commands, run:</para>
+ <screen>$ mkfs.lustre --fsname=spfs --ost --mgsnode=mds16@tcp0 /dev/sda
+$ mkdir -p /mnt/test/ost0
+$ mount -t lustre /dev/sda /mnt/test/ost0
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296455" xreflabel=""/>Migrate the data (possibly).</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296456" xreflabel=""/>The file system is quite unbalanced when new empty OSTs are added. New file creations are automatically balanced. If this is a scratch file system or files are pruned at a regular interval, then no further work may be needed. Files existing prior to the expansion can be rebalanced with an in-place copy, which can be done with a simple script.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296457" xreflabel=""/>The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296458" xreflabel=""/>A very clever migration script would do the following:</para>
+ <para>Migrate the data (possibly).</para>
+ <para>The file system is quite unbalanced when new empty OSTs are added. New file creations are automatically balanced. If this is a scratch file system or files are pruned at a regular interval, then no further work may be needed. Files existing prior to the expansion can be rebalanced with an in-place copy, which can be done with a simple script.</para>
+ <para>The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
+ <para>A very clever migration script would do the following:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296459" xreflabel=""/> Examine the current distribution of data.</para>
+ <para> Examine the current distribution of data.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296460" xreflabel=""/> Calculate how much data should move from each full OST to the empty ones.</para>
+ <para> Calculate how much data should move from each full OST to the empty ones.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296461" xreflabel=""/> Search for files on a given full OST (using lfs getstripe).</para>
+ <para> Search for files on a given full OST (using lfs getstripe).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296462" xreflabel=""/> Force the new destination OST (using lfs setstripe).</para>
+ <para> Force the new destination OST (using lfs setstripe).</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296463" xreflabel=""/> Copy only enough files to address the imbalance.</para>
+ <para> Copy only enough files to address the imbalance.</para>
</listitem>
</itemizedlist>
</listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1296464" xreflabel=""/>If a Lustre administrator wants to explore this approach further, per-OST disk-usage statistics can be found under /proc/fs/lustre/osc/*/rpc_stats</para>
+ <para>If a Lustre administrator wants to explore this approach further, per-OST disk-usage statistics can be found under /proc/fs/lustre/osc/*/rpc_stats</para>
</section>
<section xml:id="dbdoclet.50438211_80295">
<title>19.4 Performing <anchor xml:id="dbdoclet.50438211_marker-1291962" xreflabel=""/>Direct I/O</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1291964" xreflabel=""/>Lustre supports the O_DIRECT flag to open.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1291965" xreflabel=""/>Applications using the read() and write() calls must supply buffers aligned on a page boundary (usually 4 K). If the alignment is not correct, the call returns -EINVAL. Direct I/O may help performance in cases where the client is doing a large amount of I/O and is CPU-bound (CPU utilization 100%).</para>
+ <para>Lustre supports the O_DIRECT flag to open.</para>
+ <para>Applications using the read() and write() calls must supply buffers aligned on a page boundary (usually 4 K). If the alignment is not correct, the call returns -EINVAL. Direct I/O may help performance in cases where the client is doing a large amount of I/O and is CPU-bound (CPU utilization 100%).</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438211_pgfId-1291967" xreflabel=""/>19.4.1 Making File System Objects Immutable</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1291968" xreflabel=""/>An immutable file or directory is one that cannot be modified, renamed or removed. To do this:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1291969" xreflabel=""/>chattr +i <file>
+ <title>19.4.1 Making File System Objects Immutable</title>
+ <para>An immutable file or directory is one that cannot be modified, renamed or removed. To do this:</para>
+ <screen>chattr +i <file>
</screen>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1291970" xreflabel=""/>To remove this flag, use chattr -i</para>
+ <para>To remove this flag, use chattr -i</para>
</section>
</section>
<section xml:id="dbdoclet.50438211_61024">
<title>19.5 Other I/O Options</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1291973" xreflabel=""/>This section describes other I/O options, including checksums.</para>
+ <para>This section describes other I/O options, including checksums.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438211_pgfId-1291975" xreflabel=""/>19.5.1 Lustre <anchor xml:id="dbdoclet.50438211_marker-1291974" xreflabel=""/>Checksums</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293844" xreflabel=""/>To guard against network data corruption, a Lustre client can perform two types of data checksums: in-memory (for data in client memory) and wire (for data sent over the network). For each checksum type, a 32-bit checksum of the data read or written on both the client and server is computed, to ensure that the data has not been corrupted in transit over the network. The ldiskfs backing file system does NOT do any persistent checksumming, so it does not detect corruption of data in the OST file system.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1292598" xreflabel=""/>The checksumming feature is enabled, by default, on individual client nodes. If the client or OST detects a checksum mismatch, then an error is logged in the syslog of the form:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1292603" xreflabel=""/>LustreError: BAD WRITE CHECKSUM: changed in transit before arrival at OST: \
+ <title>19.5.1 Lustre <anchor xml:id="dbdoclet.50438211_marker-1291974" xreflabel=""/>Checksums</title>
+ <para>To guard against network data corruption, a Lustre client can perform two types of data checksums: in-memory (for data in client memory) and wire (for data sent over the network). For each checksum type, a 32-bit checksum of the data read or written on both the client and server is computed, to ensure that the data has not been corrupted in transit over the network. The ldiskfs backing file system does NOT do any persistent checksumming, so it does not detect corruption of data in the OST file system.</para>
+ <para>The checksumming feature is enabled, by default, on individual client nodes. If the client or OST detects a checksum mismatch, then an error is logged in the syslog of the form:</para>
+ <screen>LustreError: BAD WRITE CHECKSUM: changed in transit before arrival at OST: \
from 192.168.1.1@tcp inum 8991479/2386814769 object 1127239/0 extent [10240\
0-106495]
</screen>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1292616" xreflabel=""/>If this happens, the client will re-read or re-write the affected data up to five times to get a good copy of the data over the network. If it is still not possible, then an I/O error is returned to the application.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293793" xreflabel=""/>To enable both types of checksums (in-memory and wire), run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293794" xreflabel=""/>echo 1 > /proc/fs/lustre/llite/<emphasis><fsname></emphasis>/checksum_pages
+ <para>If this happens, the client will re-read or re-write the affected data up to five times to get a good copy of the data over the network. If it is still not possible, then an I/O error is returned to the application.</para>
+ <para>To enable both types of checksums (in-memory and wire), run:</para>
+ <screen>echo 1 > /proc/fs/lustre/llite/<emphasis><fsname></emphasis>/checksum_pages
</screen>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293795" xreflabel=""/>To disable both types of checksums (in-memory and wire), run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293796" xreflabel=""/>echo 0 > /proc/fs/lustre/llite/<emphasis><fsname></emphasis>/checksum_pages
+ <para>To disable both types of checksums (in-memory and wire), run:</para>
+ <screen>echo 0 > /proc/fs/lustre/llite/<emphasis><fsname></emphasis>/checksum_pages
</screen>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293797" xreflabel=""/>To check the status of a wire checksum, run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293798" xreflabel=""/>lctl get_param osc.*.checksums
+ <para>To check the status of a wire checksum, run:</para>
+ <screen>lctl get_param osc.*.checksums
</screen>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438211_pgfId-1293158" xreflabel=""/>19.5.1.1 Changing Checksum Algorithms</title>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293209" xreflabel=""/>By default, Lustre uses the adler32 checksum algorithm, because it is robust and has a lower impact on performance than crc32. The Lustre administrator can change the checksum algorithm via /proc, depending on what is supported in the kernel.</para>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293218" xreflabel=""/>To check which checksum algorithm is being used by Lustre, run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293229" xreflabel=""/>$ cat /proc/fs/lustre/osc/<fsname>-OST<index>-osc-*/checksum_type
+ <title>19.5.1.1 Changing Checksum Algorithms</title>
+ <para>By default, Lustre uses the adler32 checksum algorithm, because it is robust and has a lower impact on performance than crc32. The Lustre administrator can change the checksum algorithm via /proc, depending on what is supported in the kernel.</para>
+ <para>To check which checksum algorithm is being used by Lustre, run:</para>
+ <screen>$ cat /proc/fs/lustre/osc/<fsname>-OST<index>-osc-*/checksum_type
</screen>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293260" xreflabel=""/>To change the wire checksum algorithm used by Lustre, run:</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293261" xreflabel=""/>$ echo <algorithm name> /proc/fs/lustre/osc/<fsname>-OST<index>- \osc-*/checksum_\
+ <para>To change the wire checksum algorithm used by Lustre, run:</para>
+ <screen>$ echo <algorithm name> /proc/fs/lustre/osc/<fsname>-OST<index>- \osc-*/checksum_\
type
</screen>
<note><para>The in-memory checksum always uses the adler32 algorithm, if available, and only falls back to crc32 if adler32 cannot be used.</para></note>
- <para><anchor xml:id="dbdoclet.50438211_pgfId-1293876" xreflabel=""/>In the following example, the cat command is used to determine that Lustre is using the adler32 checksum algorithm. Then the echo command is used to change the checksum algorithm to crc32. A second cat command confirms that the crc32 checksum algorithm is now in use.</para>
- <screen><anchor xml:id="dbdoclet.50438211_pgfId-1293284" xreflabel=""/>$ cat /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/checksum_ty\
+ <para>In the following example, the cat command is used to determine that Lustre is using the adler32 checksum algorithm. Then the echo command is used to change the checksum algorithm to crc32. A second cat command confirms that the crc32 checksum algorithm is now in use.</para>
+ <screen>$ cat /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/checksum_ty\
pe
-<anchor xml:id="dbdoclet.50438211_pgfId-1293295" xreflabel=""/>crc32 [adler]
-<anchor xml:id="dbdoclet.50438211_pgfId-1293296" xreflabel=""/>$ echo crc32 > /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/che\
+crc32 [adler]
+$ echo crc32 > /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/che\
cksum_type
-<anchor xml:id="dbdoclet.50438211_pgfId-1293320" xreflabel=""/>$ cat /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/checksum_ty\
+$ cat /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/checksum_ty\
pe
-<anchor xml:id="dbdoclet.50438211_pgfId-1293321" xreflabel=""/>[crc32] adler
+[crc32] adler
</screen>
</section>
</section>
<title xml:id='managinglnet.title'>Managing Lustre Networking (LNET)</title>
</info>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-999824" xreflabel=""/>This chapter describes some tools for managing Lustre Networking (LNET) and includes the following sections:</para>
+ <para>This chapter describes some tools for managing Lustre Networking (LNET) and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438203_51732"/>Updating the Health Status of a Peer or Router</para>
</listitem>
<section xml:id="dbdoclet.50438203_51732">
<title>15.1 Updating the Health Status of a Peer or <anchor xml:id="dbdoclet.50438203_marker-1288828" xreflabel=""/>Router</title>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287380" xreflabel=""/>There are two mechanisms to update the health status of a peer or a router:</para>
+ <para>There are two mechanisms to update the health status of a peer or a router:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287381" xreflabel=""/>LNET can actively check health status of all routers and mark them as dead or alive automatically. By default, this is off. To enable it set auto_down and if desired check_routers_before_use. This initial check may cause a pause equal to router_ping_timeout at system startup, if there are dead routers in the system.</para>
+ <para>LNET can actively check health status of all routers and mark them as dead or alive automatically. By default, this is off. To enable it set auto_down and if desired check_routers_before_use. This initial check may cause a pause equal to router_ping_timeout at system startup, if there are dead routers in the system.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287382" xreflabel=""/>When there is a communication error, all LNDs notify LNET that the peer (not necessarily a router) is down. This mechanism is always on, and there is no parameter to turn it off. However, if you set the LNET module parameter auto_down to 0, LNET ignores all such peer-down notifications.</para>
+ <para>When there is a communication error, all LNDs notify LNET that the peer (not necessarily a router) is down. This mechanism is always on, and there is no parameter to turn it off. However, if you set the LNET module parameter auto_down to 0, LNET ignores all such peer-down notifications.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287383" xreflabel=""/>Several key differences in both mechanisms:</para>
+ <para>Several key differences in both mechanisms:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287384" xreflabel=""/>The router pinger only checks routers for their health, while LNDs notices all dead peers, regardless of whether they are a router or not.</para>
+ <para>The router pinger only checks routers for their health, while LNDs notices all dead peers, regardless of whether they are a router or not.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287385" xreflabel=""/>The router pinger actively checks the router health by sending pings, but LNDs only notice a dead peer when there is network traffic going on.</para>
+ <para>The router pinger actively checks the router health by sending pings, but LNDs only notice a dead peer when there is network traffic going on.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287386" xreflabel=""/> The router pinger can bring a router from alive to dead or vice versa, but LNDs can only bring a peer down.</para>
+ <para> The router pinger can bring a router from alive to dead or vice versa, but LNDs can only bring a peer down.</para>
</listitem>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438203_48703">
<title>15.2 Starting and Stopping LNET</title>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287399" xreflabel=""/>Lustre automatically starts and stops LNET, but it can also be manually started in a standalone manner. This is particularly useful to verify that your networking setup is working correctly before you attempt to start Lustre.</para>
+ <para>Lustre automatically starts and stops LNET, but it can also be manually started in a standalone manner. This is particularly useful to verify that your networking setup is working correctly before you attempt to start Lustre.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438203_pgfId-1287402" xreflabel=""/>15.2.1 Starting <anchor xml:id="dbdoclet.50438203_marker-1287400" xreflabel=""/>LNET</title>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287403" xreflabel=""/>To start LNET, run:</para>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1287404" xreflabel=""/>$ modprobe lnet
-<anchor xml:id="dbdoclet.50438203_pgfId-1287405" xreflabel=""/>$ lctl network up
+ <title>15.2.1 Starting <anchor xml:id="dbdoclet.50438203_marker-1287400" xreflabel=""/>LNET</title>
+ <para>To start LNET, run:</para>
+ <screen>$ modprobe lnet
+$ lctl network up
</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287406" xreflabel=""/>To see the list of local NIDs, run:</para>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1287407" xreflabel=""/>$ lctl list_nids
+ <para>To see the list of local NIDs, run:</para>
+ <screen>$ lctl list_nids
</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289144" xreflabel=""/>This command tells you the network(s) configured to work with Lustre</para>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287409" xreflabel=""/>If the networks are not correctly setup, see the modules.conf "networks=" line and make sure the network layer modules are correctly installed and configured.</para>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287410" xreflabel=""/>To get the best remote NID, run:</para>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1287411" xreflabel=""/>$ lctl which_nid <NID list>
+ <para>This command tells you the network(s) configured to work with Lustre</para>
+ <para>If the networks are not correctly setup, see the modules.conf "networks=" line and make sure the network layer modules are correctly installed and configured.</para>
+ <para>To get the best remote NID, run:</para>
+ <screen>$ lctl which_nid <NID list>
</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287412" xreflabel=""/>where <NID list> is the list of available NIDs.</para>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287413" xreflabel=""/>This command takes the "best" NID from a list of the NIDs of a remote host. The "best" NID is the one that the local node uses when trying to communicate with the remote node.</para>
+ <para>where <NID list> is the list of available NIDs.</para>
+ <para>This command takes the "best" NID from a list of the NIDs of a remote host. The "best" NID is the one that the local node uses when trying to communicate with the remote node.</para>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438203_pgfId-1287415" xreflabel=""/>15.2.1.1 <anchor xml:id="dbdoclet.50438203_46145" xreflabel=""/>Starting Clients</title>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287416" xreflabel=""/>To start a TCP client, run:</para>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1287417" xreflabel=""/>mount -t lustre mdsnode:/mdsA/client /mnt/lustre/
+ <title>15.2.1.1 <anchor xml:id="dbdoclet.50438203_46145" xreflabel=""/>Starting Clients</title>
+ <para>To start a TCP client, run:</para>
+ <screen>mount -t lustre mdsnode:/mdsA/client /mnt/lustre/
</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287418" xreflabel=""/>To start an Elan client, run:</para>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1287419" xreflabel=""/>mount -t lustre 2@elan0:/mdsA/client /mnt/lustre
+ <para>To start an Elan client, run:</para>
+ <screen>mount -t lustre 2@elan0:/mdsA/client /mnt/lustre
</screen>
</section>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438203_pgfId-1287473" xreflabel=""/>15.2.2 Stopping <anchor xml:id="dbdoclet.50438203_marker-1288543" xreflabel=""/>LNET</title>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287480" xreflabel=""/>Before the LNET modules can be removed, LNET references must be removed. In general, these references are removed automatically when Lustre is shut down, but for standalone routers, an explicit step is needed to stop LNET. Run:</para>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1287481" xreflabel=""/>lctl network unconfigure
+ <title>15.2.2 Stopping <anchor xml:id="dbdoclet.50438203_marker-1288543" xreflabel=""/>LNET</title>
+ <para>Before the LNET modules can be removed, LNET references must be removed. In general, these references are removed automatically when Lustre is shut down, but for standalone routers, an explicit step is needed to stop LNET. Run:</para>
+ <screen>lctl network unconfigure
</screen>
<note>
Attempting to remove Lustre modules prior to stopping the network may result in a crash or an LNET hang. if this occurs, the node must be rebooted (in most cases). Make sure that the Lustre network and Lustre are stopped prior to unloading the modules. Be extremely careful using rmmod -f.</para>
</note>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1287486" xreflabel=""/>To unconfigure the LNET network, run:</para>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1287487" xreflabel=""/>modprobe -r <any lnd and the lnet modules>
+ <para>To unconfigure the LNET network, run:</para>
+ <screen>modprobe -r <any lnd and the lnet modules>
</screen>
<note>
</section>
<section xml:id="dbdoclet.50438203_72197">
<title>15.3 <anchor xml:id="dbdoclet.50438203_82542" xreflabel=""/>Multi-Rail Configurations with <anchor xml:id="dbdoclet.50438203_Multi-rail-configurations-with-LNET-LNET" xreflabel=""/>LNET</title>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289933" xreflabel=""/>To aggregate bandwidth across both rails of a dual-rail IB cluster (o2iblnd) <footnote><para><anchor xml:id="dbdoclet.50438203_pgfId-1289932" xreflabel=""/>Multi-rail configurations are only supported by o2iblnd; other IB LNDs do not support multiple interfaces.</para></footnote> using LNET, consider these points:</para>
+ <para>To aggregate bandwidth across both rails of a dual-rail IB cluster (o2iblnd) <footnote><para>Multi-rail configurations are only supported by o2iblnd; other IB LNDs do not support multiple interfaces.</para></footnote> using LNET, consider these points:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289934" xreflabel=""/> LNET can work with multiple rails, however, it does not load balance across them. The actual rail used for any communication is determined by the peer NID.</para>
+ <para> LNET can work with multiple rails, however, it does not load balance across them. The actual rail used for any communication is determined by the peer NID.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289935" xreflabel=""/> Multi-rail LNET configurations do not provide an additional level of network fault tolerance. The configurations described below are for bandwidth aggregation only. Network interface failover is planned as an upcoming Lustre feature.</para>
+ <para> Multi-rail LNET configurations do not provide an additional level of network fault tolerance. The configurations described below are for bandwidth aggregation only. Network interface failover is planned as an upcoming Lustre feature.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289936" xreflabel=""/> A Lustre node always uses the same local NID to communicate with a given peer NID. The criteria used to determine the local NID are:</para>
+ <para> A Lustre node always uses the same local NID to communicate with a given peer NID. The criteria used to determine the local NID are:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289937" xreflabel=""/> Fewest hops (to minimize routing), and</para>
+ <para> Fewest hops (to minimize routing), and</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289938" xreflabel=""/> Appears first in the "networks" or "ip2nets" LNET configuration strings</para>
+ <para> Appears first in the "networks" or "ip2nets" LNET configuration strings</para>
</listitem>
</itemizedlist>
</listitem>
</section>
<section xml:id="dbdoclet.50438203_78227">
<title>15.4 Load Balancing with InfiniBand</title>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1290370" xreflabel=""/>A Lustre file system contains OSSs with two InfiniBand HCAs. Lustre clients have only one InfiniBand HCA using OFED Infiniband ''o2ib'' drivers. Load balancing between the HCAs on the OSS is accomplished through LNET.</para>
+ <para>A Lustre file system contains OSSs with two InfiniBand HCAs. Lustre clients have only one InfiniBand HCA using OFED Infiniband ''o2ib'' drivers. Load balancing between the HCAs on the OSS is accomplished through LNET.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438203_pgfId-1290317" xreflabel=""/>15.4.1 Setting Up modprobe.conf<anchor xml:id="dbdoclet.50438203_77834" xreflabel=""/><anchor xml:id="dbdoclet.50438203_marker-1290316" xreflabel=""/> for Load Balancing</title>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1290318" xreflabel=""/>To configure LNET for load balancing on clients and servers:</para>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1290319" xreflabel=""/> 1. Set the modprobe.conf options.</para>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1290320" xreflabel=""/>Depending on your configuration, set modprobe.conf options as follows:</para>
+ <title>15.4.1 Setting Up modprobe.conf<anchor xml:id="dbdoclet.50438203_77834" xreflabel=""/><anchor xml:id="dbdoclet.50438203_marker-1290316" xreflabel=""/> for Load Balancing</title>
+ <para>To configure LNET for load balancing on clients and servers:</para>
+ <para> 1. Set the modprobe.conf options.</para>
+ <para>Depending on your configuration, set modprobe.conf options as follows:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1290321" xreflabel=""/> Dual HCA OSS server</para>
+ <para> Dual HCA OSS server</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1290322" xreflabel=""/>options lnet networks="o2ib0(ib0),o2ib1(ib1) 192.168.10.1.[101-102]
+ <screen>options lnet networks="o2ib0(ib0),o2ib1(ib1) 192.168.10.1.[101-102]
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1290323" xreflabel=""/> Client with the odd IP address</para>
+ <para> Client with the odd IP address</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1290324" xreflabel=""/>options lnet networks=o2ib0(ib0) 192.168.10.[103-253/2]
+ <screen>options lnet networks=o2ib0(ib0) 192.168.10.[103-253/2]
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1290325" xreflabel=""/> Client with the even IP address</para>
+ <para> Client with the even IP address</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1290326" xreflabel=""/>options lnet networks=o2ib1(ib0) 192.168.10.[102-254/2]
-</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1290327" xreflabel=""/> 2. Run the modprobe lnet command and create a combined MGS/MDT file system.</para>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1290328" xreflabel=""/>The following commands create the MGS/MDT file system and mount the servers (MGS/MDT and OSS).</para>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1290329" xreflabel=""/>modprobe lnet
-<anchor xml:id="dbdoclet.50438203_pgfId-1290330" xreflabel=""/>$ mkfs.lustre --fsname lustre --mgs --mdt <block device name>
-<anchor xml:id="dbdoclet.50438203_pgfId-1290331" xreflabel=""/>$ mkdir -p <mount point>
-<anchor xml:id="dbdoclet.50438203_pgfId-1290332" xreflabel=""/>$ mount -t lustre <block device> <mount point>
-<anchor xml:id="dbdoclet.50438203_pgfId-1290333" xreflabel=""/>$ mount -t lustre <block device> <mount point>
-<anchor xml:id="dbdoclet.50438203_pgfId-1290334" xreflabel=""/>$ mkfs.lustre --fsname lustre --mgs --mdt <block device name>
-<anchor xml:id="dbdoclet.50438203_pgfId-1290335" xreflabel=""/>$ mkdir -p <mount point>
-<anchor xml:id="dbdoclet.50438203_pgfId-1290336" xreflabel=""/>$ mount -t lustre <block device> <mount point>
-<anchor xml:id="dbdoclet.50438203_pgfId-1290337" xreflabel=""/>$ mount -t lustre <block device> <mount point>
-</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1290338" xreflabel=""/>For example:</para>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1290339" xreflabel=""/>modprobe lnet
-<anchor xml:id="dbdoclet.50438203_pgfId-1290340" xreflabel=""/>$ mkfs.lustre --fsname lustre --mdt --mgs /dev/sda
-<anchor xml:id="dbdoclet.50438203_pgfId-1290341" xreflabel=""/>$ mkdir -p /mnt/test/mdt
-<anchor xml:id="dbdoclet.50438203_pgfId-1290342" xreflabel=""/>$ mount -t lustre /dev/sda /mnt/test/mdt
-<anchor xml:id="dbdoclet.50438203_pgfId-1290343" xreflabel=""/>$ mount -t lustre mgs@o2ib0:/lustre /mnt/mdt
-<anchor xml:id="dbdoclet.50438203_pgfId-1290344" xreflabel=""/>$ mkfs.lustre --fsname lustre --ost --mgsnode=mds@o2ib0 /dev/sda
-<anchor xml:id="dbdoclet.50438203_pgfId-1290345" xreflabel=""/>$ mkdir -p /mnt/test/mdt
-<anchor xml:id="dbdoclet.50438203_pgfId-1290346" xreflabel=""/>$ mount -t lustre /dev/sda /mnt/test/ost
-<anchor xml:id="dbdoclet.50438203_pgfId-1290347" xreflabel=""/>$ mount -t lustre mgs@o2ib0:/lustre /mnt/ost
-</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1290348" xreflabel=""/> 3. Mount the clients.</para>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1290349" xreflabel=""/>mount -t lustre <MGS node>:/<fsname> <mount point>
-</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1290350" xreflabel=""/>This example shows an IB client being mounted.</para>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1290351" xreflabel=""/>mount -t lustre
-<anchor xml:id="dbdoclet.50438203_pgfId-1290352" xreflabel=""/>192.168.10.101@o2ib0,192.168.10.102@o2ib1:/mds/client /mnt/lustre
-</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289939" xreflabel=""/>As an example, consider a two-rail IB cluster running the OFA stack (OFED) with these IPoIB address assignments.</para>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1289940" xreflabel=""/> ib0 ib1
-<anchor xml:id="dbdoclet.50438203_pgfId-1289941" xreflabel=""/>Servers 192.168.0.* 192.168.1.*
-<anchor xml:id="dbdoclet.50438203_pgfId-1289942" xreflabel=""/>Clients 192.168.[2-127].* 192.168.[128-253].*
-</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289943" xreflabel=""/>You could create these configurations:</para>
+ <screen>options lnet networks=o2ib1(ib0) 192.168.10.[102-254/2]
+</screen>
+ <para> 2. Run the modprobe lnet command and create a combined MGS/MDT file system.</para>
+ <para>The following commands create the MGS/MDT file system and mount the servers (MGS/MDT and OSS).</para>
+ <screen>modprobe lnet
+$ mkfs.lustre --fsname lustre --mgs --mdt <block device name>
+$ mkdir -p <mount point>
+$ mount -t lustre <block device> <mount point>
+$ mount -t lustre <block device> <mount point>
+$ mkfs.lustre --fsname lustre --mgs --mdt <block device name>
+$ mkdir -p <mount point>
+$ mount -t lustre <block device> <mount point>
+$ mount -t lustre <block device> <mount point>
+</screen>
+ <para>For example:</para>
+ <screen>modprobe lnet
+$ mkfs.lustre --fsname lustre --mdt --mgs /dev/sda
+$ mkdir -p /mnt/test/mdt
+$ mount -t lustre /dev/sda /mnt/test/mdt
+$ mount -t lustre mgs@o2ib0:/lustre /mnt/mdt
+$ mkfs.lustre --fsname lustre --ost --mgsnode=mds@o2ib0 /dev/sda
+$ mkdir -p /mnt/test/mdt
+$ mount -t lustre /dev/sda /mnt/test/ost
+$ mount -t lustre mgs@o2ib0:/lustre /mnt/ost
+</screen>
+ <para> 3. Mount the clients.</para>
+ <screen>mount -t lustre <MGS node>:/<fsname> <mount point>
+</screen>
+ <para>This example shows an IB client being mounted.</para>
+ <screen>mount -t lustre
+192.168.10.101@o2ib0,192.168.10.102@o2ib1:/mds/client /mnt/lustre
+</screen>
+ <para>As an example, consider a two-rail IB cluster running the OFA stack (OFED) with these IPoIB address assignments.</para>
+ <screen> ib0 ib1
+Servers 192.168.0.* 192.168.1.*
+Clients 192.168.[2-127].* 192.168.[128-253].*
+</screen>
+ <para>You could create these configurations:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289944" xreflabel=""/> A cluster with more clients than servers. The fact that an individual client cannot get two rails of bandwidth is unimportant because the servers are the actual bottleneck.</para>
+ <para> A cluster with more clients than servers. The fact that an individual client cannot get two rails of bandwidth is unimportant because the servers are the actual bottleneck.</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1289945" xreflabel=""/>ip2nets="o2ib0(ib0), o2ib1(ib1) 192.168.[0-1].* \
+ <screen>ip2nets="o2ib0(ib0), o2ib1(ib1) 192.168.[0-1].* \
#all servers;\
-<anchor xml:id="dbdoclet.50438203_pgfId-1289946" xreflabel=""/> o2ib0(ib0) 192.168.[2-253].[0-252/2] #even cl\
+ o2ib0(ib0) 192.168.[2-253].[0-252/2] #even cl\
ients;\
-<anchor xml:id="dbdoclet.50438203_pgfId-1289947" xreflabel=""/> o2ib1(ib1) 192.168.[2-253].[1-253/2] #odd cli\
+ o2ib1(ib1) 192.168.[2-253].[1-253/2] #odd cli\
ents"
</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289948" xreflabel=""/>This configuration gives every server two NIDs, one on each network, and statically load-balances clients between the rails.</para>
+ <para>This configuration gives every server two NIDs, one on each network, and statically load-balances clients between the rails.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289949" xreflabel=""/> A single client that must get two rails of bandwidth, and it does not matter if the maximum aggregate bandwidth is only (# servers) * (1 rail).</para>
+ <para> A single client that must get two rails of bandwidth, and it does not matter if the maximum aggregate bandwidth is only (# servers) * (1 rail).</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1289950" xreflabel=""/>ip2nets=" o2ib0(ib0) 192.168.[0-1].[0-252/2] \
+ <screen>ip2nets=" o2ib0(ib0) 192.168.[0-1].[0-252/2] \
#even servers;\
-<anchor xml:id="dbdoclet.50438203_pgfId-1289951" xreflabel=""/> o2ib1(ib1) 192.168.[0-1].[1-253/2] \
+ o2ib1(ib1) 192.168.[0-1].[1-253/2] \
#odd servers;\
-<anchor xml:id="dbdoclet.50438203_pgfId-1289952" xreflabel=""/> o2ib0(ib0),o2ib1(ib1) 192.168.[2-253].* \
+ o2ib0(ib0),o2ib1(ib1) 192.168.[2-253].* \
#clients"
</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289953" xreflabel=""/>This configuration gives every server a single NID on one rail or the other. Clients have a NID on both rails.</para>
+ <para>This configuration gives every server a single NID on one rail or the other. Clients have a NID on both rails.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289954" xreflabel=""/> All clients and all servers must get two rails of bandwidth.</para>
+ <para> All clients and all servers must get two rails of bandwidth.</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438203_pgfId-1289955" xreflabel=""/>ip2nets=†o2ib0(ib0),o2ib2(ib1) 192.168.[0-1].[0-252/2] \
+ <screen>ip2nets=†o2ib0(ib0),o2ib2(ib1) 192.168.[0-1].[0-252/2] \
#even servers;\
-<anchor xml:id="dbdoclet.50438203_pgfId-1289956" xreflabel=""/> o2ib1(ib0),o2ib3(ib1) 192.168.[0-1].[1-253/2] \
+ o2ib1(ib0),o2ib3(ib1) 192.168.[0-1].[1-253/2] \
#odd servers;\
-<anchor xml:id="dbdoclet.50438203_pgfId-1289957" xreflabel=""/> o2ib0(ib0),o2ib3(ib1) 192.168.[2-253].[0-252/2) \
+ o2ib0(ib0),o2ib3(ib1) 192.168.[2-253].[0-252/2) \
#even clients;\
-<anchor xml:id="dbdoclet.50438203_pgfId-1289958" xreflabel=""/> o2ib1(ib0),o2ib2(ib1) 192.168.[2-253].[1-253/2) \
+ o2ib1(ib0),o2ib2(ib1) 192.168.[2-253].[1-253/2) \
#odd clients"
</screen>
- <para><anchor xml:id="dbdoclet.50438203_pgfId-1289959" xreflabel=""/>This configuration includes two additional proxy o2ib networks to work around Lustre's simplistic NID selection algorithm. It connects "even" clients to "even" servers with o2ib0 on rail0, and "odd" servers with o2ib3 on rail1. Similarly, it connects "odd" clients to "odd" servers with o2ib1 on rail0, and "even" servers with o2ib2 on rail1.</para>
+ <para>This configuration includes two additional proxy o2ib networks to work around Lustre's simplistic NID selection algorithm. It connects "even" clients to "even" servers with o2ib0 on rail0, and "odd" servers with o2ib3 on rail1. Similarly, it connects "odd" clients to "odd" servers with o2ib1 on rail0, and "even" servers with o2ib2 on rail1.</para>
</section>
</section>
</chapter>
<info>
<title xml:id='managingsecurity.title'>Managing Lustre Security</title>
</info>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292300" xreflabel=""/>This chapter describes Lustre security and includes the following sections:</para>
+ <para>This chapter describes Lustre security and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438221_16221"/></para>
<section xml:id="dbdoclet.50438221_16221">
<title>22.1 Using <anchor xml:id="dbdoclet.50438221_marker-1292306" xreflabel=""/>ACLs</title>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292308" xreflabel=""/>An access control list (ACL), is a set of data that informs an operating system about permissions or access rights that each user or group has to specific system objects, such as directories or files. Each object has a unique security attribute that identifies users who have access to it. The ACL lists each object and user access privileges such as read, write or execute.</para>
+ <para>An access control list (ACL), is a set of data that informs an operating system about permissions or access rights that each user or group has to specific system objects, such as directories or files. Each object has a unique security attribute that identifies users who have access to it. The ACL lists each object and user access privileges such as read, write or execute.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438221_pgfId-1292309" xreflabel=""/>22.1.1 How ACLs Work</title>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292310" xreflabel=""/>Implementing ACLs varies between operating systems. Systems that support the Portable Operating System Interface (POSIX) family of standards share a simple yet powerful file system permission model, which should be well-known to the Linux/Unix administrator. ACLs add finer-grained permissions to this model, allowing for more complicated permission schemes. For a detailed explanation of ACLs on Linux, refer to the SuSE Labs article, <emphasis>Posix Access Control Lists on Linux</emphasis>:</para>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292312" xreflabel=""/><link xl:href="http://www.suse.de/~agruen/acl/linux-acls/online/">http://www.suse.de/~agruen/acl/linux-acls/online/</link></para>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292313" xreflabel=""/>We have implemented ACLs according to this model. Lustre works with the standard Linux ACL tools, setfacl, getfacl, and the historical chacl, normally installed with the ACL package.</para>
+ <title>22.1.1 How ACLs Work</title>
+ <para>Implementing ACLs varies between operating systems. Systems that support the Portable Operating System Interface (POSIX) family of standards share a simple yet powerful file system permission model, which should be well-known to the Linux/Unix administrator. ACLs add finer-grained permissions to this model, allowing for more complicated permission schemes. For a detailed explanation of ACLs on Linux, refer to the SuSE Labs article, <emphasis>Posix Access Control Lists on Linux</emphasis>:</para>
+ <para><link xl:href="http://www.suse.de/~agruen/acl/linux-acls/online/">http://www.suse.de/~agruen/acl/linux-acls/online/</link></para>
+ <para>We have implemented ACLs according to this model. Lustre works with the standard Linux ACL tools, setfacl, getfacl, and the historical chacl, normally installed with the ACL package.</para>
<note><para>ACL support is a system-range feature, meaning that all clients have ACL enabled or not. You cannot specify which clients should enable ACL.</para></note>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438221_pgfId-1292315" xreflabel=""/>22.1.2 Using <anchor xml:id="dbdoclet.50438221_marker-1292314" xreflabel=""/>ACLs with Lustre</title>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292557" xreflabel=""/>POSIX Access Control Lists (ACLs) can be used with Lustre. An ACL consists of file entries representing permissions based on standard POSIX file system object permissions that define three classes of user (owner, group and other). Each class is associated with a set of permissions [read (r), write (w) and execute (x)].</para>
+ <title>22.1.2 Using <anchor xml:id="dbdoclet.50438221_marker-1292314" xreflabel=""/>ACLs with Lustre</title>
+ <para>POSIX Access Control Lists (ACLs) can be used with Lustre. An ACL consists of file entries representing permissions based on standard POSIX file system object permissions that define three classes of user (owner, group and other). Each class is associated with a set of permissions [read (r), write (w) and execute (x)].</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292560" xreflabel=""/> Owner class permissions define access privileges of the file owner.</para>
+ <para> Owner class permissions define access privileges of the file owner.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292569" xreflabel=""/> Group class permissions define access privileges of the owning group.</para>
+ <para> Group class permissions define access privileges of the owning group.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292572" xreflabel=""/> Other class permissions define access privileges of all users not in the owner or group class.</para>
+ <para> Other class permissions define access privileges of all users not in the owner or group class.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292563" xreflabel=""/>The ls -l command displays the owner, group, and other class permissions in the first column of its output (for example, -rw-r- -- for a regular file with read and write access for the owner class, read access for the group class, and no access for others).</para>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292512" xreflabel=""/>Minimal ACLs have three entries. Extended ACLs have more than the three entries. Extended ACLs also contain a mask entry and may contain any number of named user and named group entries.</para>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292536" xreflabel=""/>The MDS needs to be configured to enable ACLs. Use --mountfsoptions to enable ACLs when creating your configuration:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1292318" xreflabel=""/>$ mkfs.lustre --fsname spfs --mountfsoptions=acl --mdt -mgs /dev/sda
+ <para>The ls -l command displays the owner, group, and other class permissions in the first column of its output (for example, -rw-r- -- for a regular file with read and write access for the owner class, read access for the group class, and no access for others).</para>
+ <para>Minimal ACLs have three entries. Extended ACLs have more than the three entries. Extended ACLs also contain a mask entry and may contain any number of named user and named group entries.</para>
+ <para>The MDS needs to be configured to enable ACLs. Use --mountfsoptions to enable ACLs when creating your configuration:</para>
+ <screen>$ mkfs.lustre --fsname spfs --mountfsoptions=acl --mdt -mgs /dev/sda
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292319" xreflabel=""/>Alternately, you can enable ACLs at run time by using the --acl option with mkfs.lustre:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1292320" xreflabel=""/>$ mount -t lustre -o acl /dev/sda /mnt/mdt
+ <para>Alternately, you can enable ACLs at run time by using the --acl option with mkfs.lustre:</para>
+ <screen>$ mount -t lustre -o acl /dev/sda /mnt/mdt
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292472" xreflabel=""/>To check ACLs on the MDS:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1292484" xreflabel=""/>$ lctl get_param -n mdc.home-MDT0000-mdc-*.connect_flags | grep acl acl
+ <para>To check ACLs on the MDS:</para>
+ <screen>$ lctl get_param -n mdc.home-MDT0000-mdc-*.connect_flags | grep acl acl
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292593" xreflabel=""/>To mount the client with no ACLs:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1292597" xreflabel=""/>$ mount -t lustre -o noacl ibmds2@o2ib:/home /home
+ <para>To mount the client with no ACLs:</para>
+ <screen>$ mount -t lustre -o noacl ibmds2@o2ib:/home /home
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292321" xreflabel=""/>ACLs are enabled in Lustre on a system-wide basis; either all clients enable ACLs or none do. Activating ACLs is controlled by MDS mount options acl / noacl (enable/disableACLs). Client-side mount options acl/noacl are ignored. You do not need to change the client configuration, and the 'acl' string will not appear in the client /etc/mtab. The client acl mount option is no longer needed. If a client is mounted with that option, then this message appears in the MDS syslog:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1292322" xreflabel=""/>...MDS requires ACL support but client does not
+ <para>ACLs are enabled in Lustre on a system-wide basis; either all clients enable ACLs or none do. Activating ACLs is controlled by MDS mount options acl / noacl (enable/disableACLs). Client-side mount options acl/noacl are ignored. You do not need to change the client configuration, and the 'acl' string will not appear in the client /etc/mtab. The client acl mount option is no longer needed. If a client is mounted with that option, then this message appears in the MDS syslog:</para>
+ <screen>...MDS requires ACL support but client does not
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292323" xreflabel=""/>The message is harmless but indicates a configuration issue, which should be corrected.</para>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292324" xreflabel=""/>If ACLs are not enabled on the MDS, then any attempts to reference an ACL on a client return an Operation not supported error.</para>
+ <para>The message is harmless but indicates a configuration issue, which should be corrected.</para>
+ <para>If ACLs are not enabled on the MDS, then any attempts to reference an ACL on a client return an Operation not supported error.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438221_pgfId-1292326" xreflabel=""/>22.1.3 <anchor xml:id="dbdoclet.50438221_marker-1292325" xreflabel=""/>Examples</title>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292327" xreflabel=""/>These examples are taken directly from the POSIX paper referenced above. ACLs on a Lustre file system work exactly like ACLs on any Linux file system. They are manipulated with the standard tools in the standard manner. Below, we create a directory and allow a specific user access.</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1292328" xreflabel=""/>[root@client lustre]# umask 027
-<anchor xml:id="dbdoclet.50438221_pgfId-1292329" xreflabel=""/>[root@client lustre]# mkdir rain
-<anchor xml:id="dbdoclet.50438221_pgfId-1292330" xreflabel=""/>[root@client lustre]# ls -ld rain
-<anchor xml:id="dbdoclet.50438221_pgfId-1292331" xreflabel=""/>drwxr-x--- 2 root root 4096 Feb 20 06:50 rain
-<anchor xml:id="dbdoclet.50438221_pgfId-1292332" xreflabel=""/>[root@client lustre]# getfacl rain
-<anchor xml:id="dbdoclet.50438221_pgfId-1292333" xreflabel=""/># file: rain
-<anchor xml:id="dbdoclet.50438221_pgfId-1292334" xreflabel=""/># owner: root
-<anchor xml:id="dbdoclet.50438221_pgfId-1292335" xreflabel=""/># group: root
-<anchor xml:id="dbdoclet.50438221_pgfId-1292336" xreflabel=""/>user::rwx
-<anchor xml:id="dbdoclet.50438221_pgfId-1292337" xreflabel=""/>group::r-x
-<anchor xml:id="dbdoclet.50438221_pgfId-1292338" xreflabel=""/>other::---
-<anchor xml:id="dbdoclet.50438221_pgfId-1292339" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438221_pgfId-1292340" xreflabel=""/>[root@client lustre]# setfacl -m user:chirag:rwx rain
-<anchor xml:id="dbdoclet.50438221_pgfId-1292341" xreflabel=""/>[root@client lustre]# ls -ld rain
-<anchor xml:id="dbdoclet.50438221_pgfId-1292342" xreflabel=""/>drwxrwx---+ 2 root root 4096 Feb 20 06:50 rain
-<anchor xml:id="dbdoclet.50438221_pgfId-1292343" xreflabel=""/>[root@client lustre]# getfacl --omit-heade rain
-<anchor xml:id="dbdoclet.50438221_pgfId-1292344" xreflabel=""/>user::rwx
-<anchor xml:id="dbdoclet.50438221_pgfId-1292345" xreflabel=""/>user:chirag:rwx
-<anchor xml:id="dbdoclet.50438221_pgfId-1292346" xreflabel=""/>group::r-x
-<anchor xml:id="dbdoclet.50438221_pgfId-1292347" xreflabel=""/>mask::rwx
-<anchor xml:id="dbdoclet.50438221_pgfId-1292348" xreflabel=""/>other::---
+ <title>22.1.3 <anchor xml:id="dbdoclet.50438221_marker-1292325" xreflabel=""/>Examples</title>
+ <para>These examples are taken directly from the POSIX paper referenced above. ACLs on a Lustre file system work exactly like ACLs on any Linux file system. They are manipulated with the standard tools in the standard manner. Below, we create a directory and allow a specific user access.</para>
+ <screen>[root@client lustre]# umask 027
+[root@client lustre]# mkdir rain
+[root@client lustre]# ls -ld rain
+drwxr-x--- 2 root root 4096 Feb 20 06:50 rain
+[root@client lustre]# getfacl rain
+# file: rain
+# owner: root
+# group: root
+user::rwx
+group::r-x
+other::---
+
+[root@client lustre]# setfacl -m user:chirag:rwx rain
+[root@client lustre]# ls -ld rain
+drwxrwx---+ 2 root root 4096 Feb 20 06:50 rain
+[root@client lustre]# getfacl --omit-heade rain
+user::rwx
+user:chirag:rwx
+group::r-x
+mask::rwx
+other::---
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438221_64726">
<title>22.2 Using <anchor xml:id="dbdoclet.50438221_marker-1294644" xreflabel=""/>Root Squash</title>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296220" xreflabel=""/>Lustre 1.6 introduced root squash functionality, a security feature which controls super user access rights to an Lustre file system. Before the root squash feature was added, Lustre users could run rm -rf * as root, and remove data which should not be deleted. Using the root squash feature prevents this outcome.</para>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296221" xreflabel=""/>The root squash feature works by re-mapping the user ID (UID) and group ID (GID) of the root user to a UID and GID specified by the system administrator, via the Lustre configuration management server (MGS). The root squash feature also enables the Lustre administrator to specify a set of client for which UID/GID re-mapping does not apply.</para>
+ <para>Lustre 1.6 introduced root squash functionality, a security feature which controls super user access rights to an Lustre file system. Before the root squash feature was added, Lustre users could run rm -rf * as root, and remove data which should not be deleted. Using the root squash feature prevents this outcome.</para>
+ <para>The root squash feature works by re-mapping the user ID (UID) and group ID (GID) of the root user to a UID and GID specified by the system administrator, via the Lustre configuration management server (MGS). The root squash feature also enables the Lustre administrator to specify a set of client for which UID/GID re-mapping does not apply.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438221_pgfId-1292937" xreflabel=""/>22.2.1 Configuring <anchor xml:id="dbdoclet.50438221_marker-1294610" xreflabel=""/>Root Squash</title>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296229" xreflabel=""/>Root squash functionality is managed by two configuration parameters, root_squash and nosquash_nids.</para>
+ <title>22.2.1 Configuring <anchor xml:id="dbdoclet.50438221_marker-1294610" xreflabel=""/>Root Squash</title>
+ <para>Root squash functionality is managed by two configuration parameters, root_squash and nosquash_nids.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296230" xreflabel=""/> The root_squash parameter specifies the UID and GID with which the root user accesses the Lustre file system.</para>
+ <para> The root_squash parameter specifies the UID and GID with which the root user accesses the Lustre file system.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296231" xreflabel=""/> The nosquash_nids parameter specifies the set of clients to which root squash does not apply. LNET NID range syntax is used for this parameter (see the NID range syntax rules described in <xref linkend="dbdoclet.50438221_48757"/>). For example:</para>
+ <para> The nosquash_nids parameter specifies the set of clients to which root squash does not apply. LNET NID range syntax is used for this parameter (see the NID range syntax rules described in <xref linkend="dbdoclet.50438221_48757"/>). For example:</para>
</listitem>
</itemizedlist>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1296235" xreflabel=""/>nosquash_nids=172.16.245.[0-255/2]@tcp
+ <screen>nosquash_nids=172.16.245.[0-255/2]@tcp
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296236" xreflabel=""/>In this example, root squash does not apply to TCP clients on subnet 172.16.245.0 that have an even number as the last component of their IP address.</para>
+ <para>In this example, root squash does not apply to TCP clients on subnet 172.16.245.0 that have an even number as the last component of their IP address.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438221_pgfId-1292749" xreflabel=""/>22.2.2 <anchor xml:id="dbdoclet.50438221_48757" xreflabel=""/>Enabling and <anchor xml:id="dbdoclet.50438221_marker-1294611" xreflabel=""/>Tuning Root Squash</title>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296245" xreflabel=""/>The default value for nosquash_nids is NULL, which means that root squashing applies to all clients. Setting the root squash UID and GID to 0 turns root squash off.</para>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296246" xreflabel=""/>Root squash parameters can be set when the MDT is created (mkfs.lustre --mdt). For example:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1296247" xreflabel=""/>mkfs.lustre --reformat --fsname=Lustre --mdt --mgs \
-<anchor xml:id="dbdoclet.50438221_pgfId-1296248" xreflabel=""/> --param "mds.root_squash=500:501" \
-<anchor xml:id="dbdoclet.50438221_pgfId-1296249" xreflabel=""/> --param "mds.nosquash_nids='0@elan1 192.168.1.[10,11]'" /dev/sda1
+ <title>22.2.2 <anchor xml:id="dbdoclet.50438221_48757" xreflabel=""/>Enabling and <anchor xml:id="dbdoclet.50438221_marker-1294611" xreflabel=""/>Tuning Root Squash</title>
+ <para>The default value for nosquash_nids is NULL, which means that root squashing applies to all clients. Setting the root squash UID and GID to 0 turns root squash off.</para>
+ <para>Root squash parameters can be set when the MDT is created (mkfs.lustre --mdt). For example:</para>
+ <screen>mkfs.lustre --reformat --fsname=Lustre --mdt --mgs \
+ --param "mds.root_squash=500:501" \
+ --param "mds.nosquash_nids='0@elan1 192.168.1.[10,11]'" /dev/sda1
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296250" xreflabel=""/>Root squash parameters can also be changed on an unmounted device with tunefs.lustre. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1296251" xreflabel=""/>tunefs.lustre --param "mds.root_squash=65534:65534" \
-<anchor xml:id="dbdoclet.50438221_pgfId-1296252" xreflabel=""/>--param "mds.nosquash_nids=192.168.0.13@tcp0" /dev/sda1
+ <para>Root squash parameters can also be changed on an unmounted device with tunefs.lustre. For example:</para>
+ <screen>tunefs.lustre --param "mds.root_squash=65534:65534" \
+--param "mds.nosquash_nids=192.168.0.13@tcp0" /dev/sda1
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296253" xreflabel=""/>Root squash parameters can also be changed with the lctl conf_param command. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1296254" xreflabel=""/>lctl conf_param Lustre.mds.root_squash="1000:100"
-<anchor xml:id="dbdoclet.50438221_pgfId-1296255" xreflabel=""/>lctl conf_param Lustre.mds.nosquash_nids="*@tcp"
+ <para>Root squash parameters can also be changed with the lctl conf_param command. For example:</para>
+ <screen>lctl conf_param Lustre.mds.root_squash="1000:100"
+lctl conf_param Lustre.mds.nosquash_nids="*@tcp"
</screen>
<note><para>When using the lctl conf_param command, keep in mind:</para><para> * lctl conf_param must be run on a live MGS</para><para> * lctl conf_param causes the parameter to change on all MDSs</para><para> * lctl conf_param is to be used once per a parameter</para></note>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296271" xreflabel=""/>The nosquash_nids list can be cleared with:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1296272" xreflabel=""/>lctl conf_param Lustre.mds.nosquash_nids="NONE"
+ <para>The nosquash_nids list can be cleared with:</para>
+ <screen>lctl conf_param Lustre.mds.nosquash_nids="NONE"
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296273" xreflabel=""/>- OR -</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1296274" xreflabel=""/>lctl conf_param Lustre.mds.nosquash_nids="clear"
+ <para>- OR -</para>
+ <screen>lctl conf_param Lustre.mds.nosquash_nids="clear"
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296275" xreflabel=""/>If the nosquash_nids value consists of several NID ranges (e.g. 0@elan, 1@elan1), the list of NID ranges must be quoted with single (') or double ('') quotation marks. List elements must be separated with a space. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1296276" xreflabel=""/>mkfs.lustre ... --param "mds.nosquash_nids='0@elan1 1@elan2'" /dev/sda1
-<anchor xml:id="dbdoclet.50438221_pgfId-1296277" xreflabel=""/>lctl conf_param Lustre.mds.nosquash_nids="24@elan 15@elan1"
+ <para>If the nosquash_nids value consists of several NID ranges (e.g. 0@elan, 1@elan1), the list of NID ranges must be quoted with single (') or double ('') quotation marks. List elements must be separated with a space. For example:</para>
+ <screen>mkfs.lustre ... --param "mds.nosquash_nids='0@elan1 1@elan2'" /dev/sda1
+lctl conf_param Lustre.mds.nosquash_nids="24@elan 15@elan1"
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296278" xreflabel=""/>These are examples of incorrect syntax:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1296279" xreflabel=""/>mkfs.lustre ... --param "mds.nosquash_nids=0@elan1 1@elan2" /dev/sda1
-<anchor xml:id="dbdoclet.50438221_pgfId-1296280" xreflabel=""/>lctl conf_param Lustre.mds.nosquash_nids=24@elan 15@elan1
+ <para>These are examples of incorrect syntax:</para>
+ <screen>mkfs.lustre ... --param "mds.nosquash_nids=0@elan1 1@elan2" /dev/sda1
+lctl conf_param Lustre.mds.nosquash_nids=24@elan 15@elan1
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296281" xreflabel=""/>To check root squash parameters, use the lctl get_param command:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1296282" xreflabel=""/>lctl get_param mds.Lustre-MDT0000.root_squash
-<anchor xml:id="dbdoclet.50438221_pgfId-1296283" xreflabel=""/>lctl get_param mds.Lustre-MDT000*.nosquash_nids
+ <para>To check root squash parameters, use the lctl get_param command:</para>
+ <screen>lctl get_param mds.Lustre-MDT0000.root_squash
+lctl get_param mds.Lustre-MDT000*.nosquash_nids
</screen>
<note><para>An empty nosquash_nids list is reported as NONE.</para></note>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438221_pgfId-1293871" xreflabel=""/>22.2.3 Tips on Using <anchor xml:id="dbdoclet.50438221_marker-1296366" xreflabel=""/>Root Squash</title>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296298" xreflabel=""/>Lustre configuration management limits root squash in several ways.</para>
+ <title>22.2.3 Tips on Using <anchor xml:id="dbdoclet.50438221_marker-1296366" xreflabel=""/>Root Squash</title>
+ <para>Lustre configuration management limits root squash in several ways.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296299" xreflabel=""/> The lctl conf_param value overwrites the parameter's previous value. If the new value uses an incorrect syntax, then the system continues with the old parameters and the previously-correct value is lost on remount. That is, be careful doing root squash tuning.</para>
+ <para> The lctl conf_param value overwrites the parameter's previous value. If the new value uses an incorrect syntax, then the system continues with the old parameters and the previously-correct value is lost on remount. That is, be careful doing root squash tuning.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296300" xreflabel=""/>mkfs.lustre and tunefs.lustre do not perform syntax checking. If the root squash parameters are incorrect, they are ignored on mount and the default values are used instead.</para>
+ <para>mkfs.lustre and tunefs.lustre do not perform syntax checking. If the root squash parameters are incorrect, they are ignored on mount and the default values are used instead.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296301" xreflabel=""/> Root squash parameters are parsed with rigorous syntax checking. The root_squash parameter should be specified as <decnum>':'<decnum>. The nosquash_nids parameter should follow LNET NID range list syntax.</para>
+ <para> Root squash parameters are parsed with rigorous syntax checking. The root_squash parameter should be specified as <decnum>':'<decnum>. The nosquash_nids parameter should follow LNET NID range list syntax.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296302" xreflabel=""/>LNET NID range syntax:</para>
- <screen><anchor xml:id="dbdoclet.50438221_pgfId-1296303" xreflabel=""/><nidlist> :== <nidrange> [ ' ' <nidrange> ]
-<anchor xml:id="dbdoclet.50438221_pgfId-1296304" xreflabel=""/><nidrange> :== <addrrange> '@' <net>
-<anchor xml:id="dbdoclet.50438221_pgfId-1296305" xreflabel=""/><addrrange> :== '*' |
-<anchor xml:id="dbdoclet.50438221_pgfId-1296306" xreflabel=""/> <ipaddr_range> |
-<anchor xml:id="dbdoclet.50438221_pgfId-1296307" xreflabel=""/> <numaddr_range>
-<anchor xml:id="dbdoclet.50438221_pgfId-1296308" xreflabel=""/><ipaddr_range> :==
-<anchor xml:id="dbdoclet.50438221_pgfId-1296309" xreflabel=""/><numaddr_range>.<numaddr_range>.<numaddr_range>.<numaddr_range>
-<anchor xml:id="dbdoclet.50438221_pgfId-1296310" xreflabel=""/><numaddr_range> :== <number> |
-<anchor xml:id="dbdoclet.50438221_pgfId-1296311" xreflabel=""/> <expr_list>
-<anchor xml:id="dbdoclet.50438221_pgfId-1296312" xreflabel=""/><expr_list> :== '[' <range_expr> [ ',' <range_expr>] ']'
-<anchor xml:id="dbdoclet.50438221_pgfId-1296313" xreflabel=""/><range_expr> :== <number> |
-<anchor xml:id="dbdoclet.50438221_pgfId-1296314" xreflabel=""/> <number> '-' <number> |
-<anchor xml:id="dbdoclet.50438221_pgfId-1296315" xreflabel=""/> <number> '-' <number> '/' <number>
-<anchor xml:id="dbdoclet.50438221_pgfId-1296316" xreflabel=""/><net> :== <netname> | <netname><number>
-<anchor xml:id="dbdoclet.50438221_pgfId-1296317" xreflabel=""/><netname> :== "lo" | "tcp" | "o2ib" | "cib" | "openib" | "iib" |
-<anchor xml:id="dbdoclet.50438221_pgfId-1296318" xreflabel=""/> "vib" | "ra" | "elan" | "gm" | "mx" | "ptl"
-<anchor xml:id="dbdoclet.50438221_pgfId-1296319" xreflabel=""/><number> :== <nonnegative decimal> | <hexadecimal>
+ <para>LNET NID range syntax:</para>
+ <screen><nidlist> :== <nidrange> [ ' ' <nidrange> ]
+<nidrange> :== <addrrange> '@' <net>
+<addrrange> :== '*' |
+ <ipaddr_range> |
+ <numaddr_range>
+<ipaddr_range> :==
+<numaddr_range>.<numaddr_range>.<numaddr_range>.<numaddr_range>
+<numaddr_range> :== <number> |
+ <expr_list>
+<expr_list> :== '[' <range_expr> [ ',' <range_expr>] ']'
+<range_expr> :== <number> |
+ <number> '-' <number> |
+ <number> '-' <number> '/' <number>
+<net> :== <netname> | <netname><number>
+<netname> :== "lo" | "tcp" | "o2ib" | "cib" | "openib" | "iib" |
+ "vib" | "ra" | "elan" | "gm" | "mx" | "ptl"
+<number> :== <nonnegative decimal> | <hexadecimal>
</screen>
<note><para>For networks using numeric addresses (e.g. elan), the address range must be specified in the <numaddr_range> syntax. For networks using IP addresses, the address range must be in the <ipaddr_range>. For example, if elan is using numeric addresses, 1.2.3.4@elan is incorrect.</para></note>
</section>
<title xml:id='managingstripingfreespace.title'>Managing File Striping and Free Space</title>
</info>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291802" xreflabel=""/>This chapter describes file striping and I/O options, and includes the following sections:</para>
+ <para>This chapter describes file striping and I/O options, and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438209_79324"/></para>
<section xml:id="dbdoclet.50438209_79324">
<title>18.1 How Lustre Striping Works</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296776" xreflabel=""/>Lustre uses a round-robin algorithm for selecting the next OST to which a stripe is to be written. Normally the usage of OSTs is well balanced. However, if users create a small number of exceptionally large files or incorrectly specify striping parameters, imbalanced OST usage may result.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296839" xreflabel=""/>The MDS allocates objects on seqential OSTs. Periodically, it will adjust the striping layout to eliminate some degenerated cases where applications that create very regular file layouts (striping patterns) would preferentially use a particular OST in the sequence.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297552" xreflabel=""/>Stripes are written to sequential OSTs until free space across the OSTs differs by more than 20%. The MDS will then use weighted random allocations with a preference for allocating objects on OSTs with more free space. This can reduce I/O performance until space usage is rebalanced to within 20% again.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296783" xreflabel=""/>For a more detailed description of stripe assignments, see <xref linkend='dbdoclet.50438209_10424'/>.</para>
+ <para>Lustre uses a round-robin algorithm for selecting the next OST to which a stripe is to be written. Normally the usage of OSTs is well balanced. However, if users create a small number of exceptionally large files or incorrectly specify striping parameters, imbalanced OST usage may result.</para>
+ <para>The MDS allocates objects on seqential OSTs. Periodically, it will adjust the striping layout to eliminate some degenerated cases where applications that create very regular file layouts (striping patterns) would preferentially use a particular OST in the sequence.</para>
+ <para>Stripes are written to sequential OSTs until free space across the OSTs differs by more than 20%. The MDS will then use weighted random allocations with a preference for allocating objects on OSTs with more free space. This can reduce I/O performance until space usage is rebalanced to within 20% again.</para>
+ <para>For a more detailed description of stripe assignments, see <xref linkend='dbdoclet.50438209_10424'/>.</para>
</section>
<section xml:id="dbdoclet.50438209_48033">
<title>18.2 Lustre File <anchor xml:id="dbdoclet.50438209_marker-1291832" xreflabel=""/>Striping Considerations</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296641" xreflabel=""/>Whether you should set up file striping and what parameter values you select depends on your need. A good rule of thumb is to stripe over as few objects as will meet those needs and no more.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296653" xreflabel=""/>Some reasons for using striping include:</para>
+ <para>Whether you should set up file striping and what parameter values you select depends on your need. A good rule of thumb is to stripe over as few objects as will meet those needs and no more.</para>
+ <para>Some reasons for using striping include:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296634" xreflabel=""/><emphasis role="bold">Providing high-bandwidth access</emphasis> - Many applications require high-bandwidth access to a single file - more bandwidth than can be provided by a single OSS. For example, scientific applications that write to a single file from hundreds of nodes, or a binary executable that is loaded by many nodes when an application starts.</para>
+ <para><emphasis role="bold">Providing high-bandwidth access</emphasis> - Many applications require high-bandwidth access to a single file - more bandwidth than can be provided by a single OSS. For example, scientific applications that write to a single file from hundreds of nodes, or a binary executable that is loaded by many nodes when an application starts.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1294264" xreflabel=""/>In cases like these, a file can be striped over as many OSSs as it takes to achieve the required peak aggregate bandwidth for that file. Striping across a larger number of OSSs should only be used when the file size is very large and/or is accessed by many nodes at a time. Currently, Lustre files can be striped across up to 160 OSSs, the maximum stripe count for an ldiskfs file system.</para>
+ <para>In cases like these, a file can be striped over as many OSSs as it takes to achieve the required peak aggregate bandwidth for that file. Striping across a larger number of OSSs should only be used when the file size is very large and/or is accessed by many nodes at a time. Currently, Lustre files can be striped across up to 160 OSSs, the maximum stripe count for an ldiskfs file system.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1294265" xreflabel=""/><emphasis role="bold">Improving performance when OSS bandwidth is exceeded</emphasis> - Striping across many OSSs can improve performance if the aggregate client bandwidth exceeds the server bandwidth and the application reads and writes data fast enough to take advantage of the additional OSS bandwidth. The largest useful stripe count is bounded by the I/O rate of the clients/jobs divided by the performance per OSS.</para>
+ <para><emphasis role="bold">Improving performance when OSS bandwidth is exceeded</emphasis> - Striping across many OSSs can improve performance if the aggregate client bandwidth exceeds the server bandwidth and the application reads and writes data fast enough to take advantage of the additional OSS bandwidth. The largest useful stripe count is bounded by the I/O rate of the clients/jobs divided by the performance per OSS.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1294262" xreflabel=""/><emphasis role="bold">Providing space for very large files.</emphasis> Striping is also useful when a single OST does not have enough free space to hold the entire file.</para>
+ <para><emphasis role="bold">Providing space for very large files.</emphasis> Striping is also useful when a single OST does not have enough free space to hold the entire file.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296431" xreflabel=""/>Some reasons to minimize or avoid striping:</para>
+ <para>Some reasons to minimize or avoid striping:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296706" xreflabel=""/><emphasis role="bold">Increased overhead</emphasis> - Striping results in more locks and extra network operations during common operations such as stat and unlink. Even when these operations are performed in parallel, one network operation takes less time than 100 operations.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291856" xreflabel=""/>Increased overhead also results from server contention. Consider a cluster with 100 clients and 100 OSSs, each with one OST. If each file has exactly one object and the load is distributed evenly, there is no contention and the disks on each server can manage sequential I/O. If each file has 100 objects, then the clients all compete with one another for the attention of the servers, and the disks on each node seek in 100 different directions. In this case, there is needless contention.</para>
+ <para><emphasis role="bold">Increased overhead</emphasis> - Striping results in more locks and extra network operations during common operations such as stat and unlink. Even when these operations are performed in parallel, one network operation takes less time than 100 operations.</para>
+ <para>Increased overhead also results from server contention. Consider a cluster with 100 clients and 100 OSSs, each with one OST. If each file has exactly one object and the load is distributed evenly, there is no contention and the disks on each server can manage sequential I/O. If each file has 100 objects, then the clients all compete with one another for the attention of the servers, and the disks on each node seek in 100 different directions. In this case, there is needless contention.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291857" xreflabel=""/><emphasis role="bold">Increased risk</emphasis> - When a file is striped across all servers and one of the servers breaks down, a small part of each striped file is lost. By comparison, if each file has exactly one stripe, you lose fewer files, but you lose them in their entirety. Many users would prefer to lose some of their files entirely than all of their files partially.</para>
+ <para><emphasis role="bold">Increased risk</emphasis> - When a file is striped across all servers and one of the servers breaks down, a small part of each striped file is lost. By comparison, if each file has exactly one stripe, you lose fewer files, but you lose them in their entirety. Many users would prefer to lose some of their files entirely than all of their files partially.</para>
</listitem>
</itemizedlist>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438209_pgfId-1291860" xreflabel=""/>18.2.1 Choosing a Stripe <anchor xml:id="dbdoclet.50438209_marker-1291859" xreflabel=""/>Size</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296612" xreflabel=""/>Choosing a stripe size is a small balancing act, but there are reasonable defaults.</para>
+ <title>18.2.1 Choosing a Stripe <anchor xml:id="dbdoclet.50438209_marker-1291859" xreflabel=""/>Size</title>
+ <para>Choosing a stripe size is a small balancing act, but there are reasonable defaults.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296443" xreflabel=""/><emphasis role="bold">The stripe size must be a multiple of the page size.</emphasis> Lustre's tools enforce a multiple of 64 KB (the maximum page size on ia64 and PPC64 nodes) so that users on platforms with smaller pages do not accidentally create files that might cause problems for ia64 clients.</para>
+ <para><emphasis role="bold">The stripe size must be a multiple of the page size.</emphasis> Lustre's tools enforce a multiple of 64 KB (the maximum page size on ia64 and PPC64 nodes) so that users on platforms with smaller pages do not accidentally create files that might cause problems for ia64 clients.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291862" xreflabel=""/><emphasis role="bold">The smallest recommended stripe size is 512 KB.</emphasis> Although you can create files with a stripe size of 64 KB, the smallest practical stripe size is 512 KB because Lustre sends 1MB chunks over the network. Choosing a smaller stripe size may result in inefficient I/O to the disks and reduced performance.</para>
+ <para><emphasis role="bold">The smallest recommended stripe size is 512 KB.</emphasis> Although you can create files with a stripe size of 64 KB, the smallest practical stripe size is 512 KB because Lustre sends 1MB chunks over the network. Choosing a smaller stripe size may result in inefficient I/O to the disks and reduced performance.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291864" xreflabel=""/><emphasis role="bold">A good stripe size for sequential I/O using high-speed networks is between 1 MB and 4 MB.</emphasis> In most situations, stripe sizes larger than 4 MB may result in longer lock hold times and contention on shared file access.</para>
+ <para><emphasis role="bold">A good stripe size for sequential I/O using high-speed networks is between 1 MB and 4 MB.</emphasis> In most situations, stripe sizes larger than 4 MB may result in longer lock hold times and contention on shared file access.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297206" xreflabel=""/><emphasis role="bold">The maximum stripe size is 4GB.</emphasis> Using a large stripe size can improve performance when accessing very large files. It allows each client to have exclusive access to its own part of a file. However, it can be counterproductive in some cases if it does not match your I/O pattern.</para>
+ <para><emphasis role="bold">The maximum stripe size is 4GB.</emphasis> Using a large stripe size can improve performance when accessing very large files. It allows each client to have exclusive access to its own part of a file. However, it can be counterproductive in some cases if it does not match your I/O pattern.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291865" xreflabel=""/><emphasis role="bold">Choose a stripe pattern that takes into account your application's write patterns.</emphasis> Writes that cross an object boundary are slightly less efficient than writes that go entirely to one server. If the file is written in a very consistent and aligned way, make the stripe size a multiple of the write() size.</para>
+ <para><emphasis role="bold">Choose a stripe pattern that takes into account your application's write patterns.</emphasis> Writes that cross an object boundary are slightly less efficient than writes that go entirely to one server. If the file is written in a very consistent and aligned way, make the stripe size a multiple of the write() size.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291866" xreflabel=""/><emphasis role="bold">The choice of stripe size has no effect on a single-stripe file.</emphasis></para>
+ <para><emphasis role="bold">The choice of stripe size has no effect on a single-stripe file.</emphasis></para>
</listitem>
</itemizedlist>
</section>
</section>
<section xml:id="dbdoclet.50438209_78664">
<title>18.3 Setting the File Layout/Striping Configuration (lfs setstripe)</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297878" xreflabel=""/>Use the lfs setstripe command to create new files with a specific file layout (stripe pattern) configuration.</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1297879" xreflabel=""/>lfs setstripe [--size|-s stripe_size] [--count|-c stripe_count]
-<anchor xml:id="dbdoclet.50438209_pgfId-1297880" xreflabel=""/>[--index|-i start_ost] [--pool|-p pool_name] <filename|dirname>
+ <para>Use the lfs setstripe command to create new files with a specific file layout (stripe pattern) configuration.</para>
+ <screen>lfs setstripe [--size|-s stripe_size] [--count|-c stripe_count]
+[--index|-i start_ost] [--pool|-p pool_name] <filename|dirname>
</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297881" xreflabel=""/><emphasis role="bold">stripe_size</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297882" xreflabel=""/>The stripe size indicates how much data to write to one OST before moving to the next OST. The default stripe_size is 1 MB, and passing a stripe_size of 0 causes the default stripe size to be used. Otherwise, the stripe_size value must be a multiple of 64 KB.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297883" xreflabel=""/><emphasis role="bold">stripe_count</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297884" xreflabel=""/>The stripe count indicates how many OSTs to use. The default stripe_count value is 1. Setting stripe_count to 0 causes the default stripe count to be used. Setting stripe_count to -1 means stripe over all available OSTs (full OSTs are skipped).</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297885" xreflabel=""/><emphasis role="bold">start_ost</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298131" xreflabel=""/>The start OST is the first OST to which files are written. The default value for start_ost is -1 , which allows the MDS to choose the starting index. This setting is strongly recommended, as it allows space and load balancing to be done by the MDS as needed. Otherwise, the file starts on the specified OST index. The numbering of the OSTs starts at 0.</para>
+ <para><emphasis role="bold">stripe_size</emphasis></para>
+ <para>The stripe size indicates how much data to write to one OST before moving to the next OST. The default stripe_size is 1 MB, and passing a stripe_size of 0 causes the default stripe size to be used. Otherwise, the stripe_size value must be a multiple of 64 KB.</para>
+ <para><emphasis role="bold">stripe_count</emphasis></para>
+ <para>The stripe count indicates how many OSTs to use. The default stripe_count value is 1. Setting stripe_count to 0 causes the default stripe count to be used. Setting stripe_count to -1 means stripe over all available OSTs (full OSTs are skipped).</para>
+ <para><emphasis role="bold">start_ost</emphasis></para>
+ <para>The start OST is the first OST to which files are written. The default value for start_ost is -1 , which allows the MDS to choose the starting index. This setting is strongly recommended, as it allows space and load balancing to be done by the MDS as needed. Otherwise, the file starts on the specified OST index. The numbering of the OSTs starts at 0.</para>
<note><para>If you pass a start_ost value of 0 and a stripe_count value of <emphasis>1</emphasis>, all files are written to OST 0, until space is exhausted. This is probably not what you meant to do. If you only want to adjust the stripe count and keep the other parameters at their default settings, do not specify any of the other parameters:</para><para>lfs setstripe -c <stripe_count> <file></para></note>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297973" xreflabel=""/><emphasis role="bold">pool_name</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297976" xreflabel=""/>Specify the OST pool on which the file will be written. This allows limiting the OSTs used to a subset of all OSTs in the file system. For more details about using OST pools, see <link xl:href="ManagingFileSystemIO.html#50438211_75549">Creating and Managing OST Pools</link>.</para>
+ <para><emphasis role="bold">pool_name</emphasis></para>
+ <para>Specify the OST pool on which the file will be written. This allows limiting the OSTs used to a subset of all OSTs in the file system. For more details about using OST pools, see <link xl:href="ManagingFileSystemIO.html#50438211_75549">Creating and Managing OST Pools</link>.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438209_pgfId-1297896" xreflabel=""/>18.3.1 <anchor xml:id="dbdoclet.50438209_48100" xreflabel=""/>Using a Specific Striping Pattern/File Layout for a Single File</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297948" xreflabel=""/>It is possible to specify the file layout when a new file is created using the command lfssetstripe. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an lfssetstripe command fails if the file already exists.</para>
+ <title>18.3.1 <anchor xml:id="dbdoclet.50438209_48100" xreflabel=""/>Using a Specific Striping Pattern/File Layout for a Single File</title>
+ <para>It is possible to specify the file layout when a new file is created using the command lfssetstripe. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an lfssetstripe command fails if the file already exists.</para>
<section remap="h4">
- <title>
<anchor xml:id="dbdoclet.50438209_pgfId-1298004" xreflabel=""/>18.3.1.1 <anchor xml:id="dbdoclet.50438209_60155" xreflabel=""/>Setting the Stripe Size</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298005" xreflabel=""/>The command to create a new file with a specified stripe size is similar to:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1298006" xreflabel=""/>[client]# lfs setstripe -s 4M /mnt/lustre/new_file
+ <title>18.3.1.1 <anchor xml:id="dbdoclet.50438209_60155" xreflabel=""/>Setting the Stripe Size</title>
+ <para>The command to create a new file with a specified stripe size is similar to:</para>
+ <screen>[client]# lfs setstripe -s 4M /mnt/lustre/new_file
</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298007" xreflabel=""/>This example command creates the new file /mnt/lustre/new_file with a stripe size of 4 MB.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298232" xreflabel=""/>Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1298011" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438209_pgfId-1298012" xreflabel=""/>[client]# lfs getstripe /mnt/lustre/new_file
-<anchor xml:id="dbdoclet.50438209_pgfId-1298013" xreflabel=""/>/mnt/lustre/4mb_file
-<anchor xml:id="dbdoclet.50438209_pgfId-1298014" xreflabel=""/>lmm_stripe_count: 1
-<anchor xml:id="dbdoclet.50438209_pgfId-1298015" xreflabel=""/>lmm_stripe_size: 4194304
-<anchor xml:id="dbdoclet.50438209_pgfId-1298016" xreflabel=""/>lmm_stripe_offset: 1
-<anchor xml:id="dbdoclet.50438209_pgfId-1298017" xreflabel=""/>obdidx objid objid group
-<anchor xml:id="dbdoclet.50438209_pgfId-1298018" xreflabel=""/>1 690550 0xa8976 0
+ <para>This example command creates the new file /mnt/lustre/new_file with a stripe size of 4 MB.</para>
+ <para>Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:</para>
+ <screen>
+[client]# lfs getstripe /mnt/lustre/new_file
+/mnt/lustre/4mb_file
+lmm_stripe_count: 1
+lmm_stripe_size: 4194304
+lmm_stripe_offset: 1
+obdidx objid objid group
+1 690550 0xa8976 0
</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298019" xreflabel=""/>As can be seen, the stripe size is 4 MB.</para>
+ <para>As can be seen, the stripe size is 4 MB.</para>
</section>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1298020" xreflabel=""/>18.3.1.2 Setting the Stripe Count</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298021" xreflabel=""/>The command below creates a new file with a stripe count of -1 to specify striping over all available OSTs:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1298022" xreflabel=""/>[client]# lfs setstripe -c -1 /mnt/lustre/full_stripe
+ <title>18.3.1.2 Setting the Stripe Count</title>
+ <para>The command below creates a new file with a stripe count of -1 to specify striping over all available OSTs:</para>
+ <screen>[client]# lfs setstripe -c -1 /mnt/lustre/full_stripe
</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298028" xreflabel=""/>The example below indicates that the file full_stripe is striped over all six active OSTs in the configuration:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1298029" xreflabel=""/>[client]# lfs getstripe /mnt/lustre/full_stripe
-<anchor xml:id="dbdoclet.50438209_pgfId-1298030" xreflabel=""/>/mnt/lustre/full_stripe
-<anchor xml:id="dbdoclet.50438209_pgfId-1298031" xreflabel=""/>obdidx objid objid group
-<anchor xml:id="dbdoclet.50438209_pgfId-1298032" xreflabel=""/>0 8 0x8 0
-<anchor xml:id="dbdoclet.50438209_pgfId-1298033" xreflabel=""/>1 4 0x4 0
-<anchor xml:id="dbdoclet.50438209_pgfId-1298034" xreflabel=""/>2 5 0x5 0
-<anchor xml:id="dbdoclet.50438209_pgfId-1298035" xreflabel=""/>3 5 0x5 0
-<anchor xml:id="dbdoclet.50438209_pgfId-1298036" xreflabel=""/>4 4 0x4 0
-<anchor xml:id="dbdoclet.50438209_pgfId-1298037" xreflabel=""/>5 2 0x2 0
+ <para>The example below indicates that the file full_stripe is striped over all six active OSTs in the configuration:</para>
+ <screen>[client]# lfs getstripe /mnt/lustre/full_stripe
+/mnt/lustre/full_stripe
+obdidx objid objid group
+0 8 0x8 0
+1 4 0x4 0
+2 5 0x5 0
+3 5 0x5 0
+4 4 0x4 0
+5 2 0x2 0
</screen>
-<para><anchor xml:id="dbdoclet.50438209_pgfId-1298040" xreflabel=""/> This is in contrast to the output in <xref linkend='dbdoclet.50438209_60155'/> that shows only a single object for the file.</para>
+<para> This is in contrast to the output in <xref linkend='dbdoclet.50438209_60155'/> that shows only a single object for the file.</para>
</section>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1297942" xreflabel=""/>18.3.2 Changing Striping for a Directory</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297943" xreflabel=""/>In a directory, the lfssetstripe command sets a default striping configuration for files created in the directory. The usage is the same as lfssetstripe for a regular file, except that the directory must exist prior to setting the default striping configuration. If a file is created in a directory with a default stripe configuration (without otherwise specifying striping), Lustre uses those striping parameters instead of the file system default for the new file.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297944" xreflabel=""/>To change the striping pattern (file layout) for a sub-directory, create a directory with desired file layout as described above. Sub-directories inherit the file layout of the root/parent directory.</para>
+ <title>18.3.2 Changing Striping for a Directory</title>
+ <para>In a directory, the lfssetstripe command sets a default striping configuration for files created in the directory. The usage is the same as lfssetstripe for a regular file, except that the directory must exist prior to setting the default striping configuration. If a file is created in a directory with a default stripe configuration (without otherwise specifying striping), Lustre uses those striping parameters instead of the file system default for the new file.</para>
+ <para>To change the striping pattern (file layout) for a sub-directory, create a directory with desired file layout as described above. Sub-directories inherit the file layout of the root/parent directory.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1298250" xreflabel=""/>18.3.3 Changing Striping for a File System</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297945" xreflabel=""/>Change the striping on the file system root will change the striping for all newly created files that would otherwise have a striping parameter from the parent directory or explicitly on the command line.</para>
+ <title>18.3.3 Changing Striping for a File System</title>
+ <para>Change the striping on the file system root will change the striping for all newly created files that would otherwise have a striping parameter from the parent directory or explicitly on the command line.</para>
<informaltable frame="none">
<tgroup cols="1">
<colspec colname="c1" colwidth="100*"/>
<tbody>
<row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438209_pgfId-1298262" xreflabel=""/>Striping of new files and sub-directories is done per the striping parameter settings of the root directory. Once you set striping on the root directory, then, by default, it applies to any new child directories created in that root directory (unless they have their own striping settings).</para></entry>
+ <entry><para><emphasis role="bold">Note -</emphasis>Striping of new files and sub-directories is done per the striping parameter settings of the root directory. Once you set striping on the root directory, then, by default, it applies to any new child directories created in that root directory (unless they have their own striping settings).</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1297901" xreflabel=""/>18.3.4 Creating a File on a Specific OST</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297902" xreflabel=""/>You can use lfs setstripe to create a file on a specific OST. In the following example, the file "bob" will be created on the first OST (id 0).</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1297903" xreflabel=""/>$ lfs setstripe --count 1 --index 0 bob
-<anchor xml:id="dbdoclet.50438209_pgfId-1297904" xreflabel=""/>$ dd if=/dev/zero of=bob count=1 bs=100M
-<anchor xml:id="dbdoclet.50438209_pgfId-1297905" xreflabel=""/>1+0 records in
-<anchor xml:id="dbdoclet.50438209_pgfId-1297906" xreflabel=""/>1+0 records out
-<anchor xml:id="dbdoclet.50438209_pgfId-1297907" xreflabel=""/>$ lfs getstripe bob
+ <title>18.3.4 Creating a File on a Specific OST</title>
+ <para>You can use lfs setstripe to create a file on a specific OST. In the following example, the file "bob" will be created on the first OST (id 0).</para>
+ <screen>$ lfs setstripe --count 1 --index 0 bob
+$ dd if=/dev/zero of=bob count=1 bs=100M
+1+0 records in
+1+0 records out
+$ lfs getstripe bob
</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297908" xreflabel=""/>OBDS:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1297909" xreflabel=""/>0: home-OST0000_UUID ACTIVE
-<anchor xml:id="dbdoclet.50438209_pgfId-1297910" xreflabel=""/>[...]
-<anchor xml:id="dbdoclet.50438209_pgfId-1297911" xreflabel=""/>bob
-<anchor xml:id="dbdoclet.50438209_pgfId-1297912" xreflabel=""/> obdidx objid objid group
-<anchor xml:id="dbdoclet.50438209_pgfId-1297913" xreflabel=""/> 0 33459243 0x1fe8c2b 0
+ <para>OBDS:</para>
+ <screen>0: home-OST0000_UUID ACTIVE
+[...]
+bob
+ obdidx objid objid group
+ 0 33459243 0x1fe8c2b 0
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438209_44776">
<title>18.4 Retrieving File Layout/Striping Information (getstripe)</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297836" xreflabel=""/>The lfsgetstripe command is used to display information that shows over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along with the OST index and object ID for each stripe in the file. For directories, the default settings for files created in that directory are printed.</para>
+ <para>The lfsgetstripe command is used to display information that shows over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along with the OST index and object ID for each stripe in the file. For directories, the default settings for files created in that directory are printed.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1297837" xreflabel=""/>18.4.1 Displaying the Current Stripe Size</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297838" xreflabel=""/>To see the current stripe size, use the lfsgetstripe command on a Lustre file or directory. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295575" xreflabel=""/>[client]# lfs getstripe /mnt/lustre
+ <title>18.4.1 Displaying the Current Stripe Size</title>
+ <para>To see the current stripe size, use the lfsgetstripe command on a Lustre file or directory. For example:</para>
+ <screen>[client]# lfs getstripe /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296462" xreflabel=""/>This command produces output similar to this:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295583" xreflabel=""/>/mnt/lustre
-<anchor xml:id="dbdoclet.50438209_pgfId-1295584" xreflabel=""/>(Default) stripe_count: 1 stripe_size: 1M stripe_offset: -1
+ <para>This command produces output similar to this:</para>
+ <screen>/mnt/lustre
+(Default) stripe_count: 1 stripe_size: 1M stripe_offset: -1
</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1295585" xreflabel=""/>In this example, the default stripe count is 1 (data blocks are striped over a single OSTs), the default stripe size is 1 MB, and objects are created over all available OSTs.</para>
+ <para>In this example, the default stripe count is 1 (data blocks are striped over a single OSTs), the default stripe size is 1 MB, and objects are created over all available OSTs.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1297424" xreflabel=""/>18.4.2 Inspecting the File Tree</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296903" xreflabel=""/>To inspect an entire tree of files, use the lfs find command:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295699" xreflabel=""/>lfs find [--recursive | -r] <file or directory> ...
+ <title>18.4.2 Inspecting the File Tree</title>
+ <para>To inspect an entire tree of files, use the lfs find command:</para>
+ <screen>lfs find [--recursive | -r] <file or directory> ...
</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1295700" xreflabel=""/>You can also use ls -l /proc/<emphasis><pid></emphasis>/fd/ to find open files using Lustre. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295701" xreflabel=""/>$ lfs getstripe $(readlink /proc/$(pidof cat)/fd/1)
+ <para>You can also use ls -l /proc/<emphasis><pid></emphasis>/fd/ to find open files using Lustre. For example:</para>
+ <screen>$ lfs getstripe $(readlink /proc/$(pidof cat)/fd/1)
</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297751" xreflabel=""/>Typical output is:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295707" xreflabel=""/>/mnt/lustre/foo
-<anchor xml:id="dbdoclet.50438209_pgfId-1295708" xreflabel=""/>obdidx objid objid \
+ <para>Typical output is:</para>
+ <screen>/mnt/lustre/foo
+obdidx objid objid \
group
-<anchor xml:id="dbdoclet.50438209_pgfId-1297465" xreflabel=""/>2 835487 0xcbf9f 0
+2 835487 0xcbf9f 0
</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297466" xreflabel=""/>In this example, the file lives on obdidx 2, which is lustre-OST0002. To see which node is serving that OST, run:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1297482" xreflabel=""/>$ lctl get_param osc.lustre-OST0002-osc.ost_conn_uuid
+ <para>In this example, the file lives on obdidx 2, which is lustre-OST0002. To see which node is serving that OST, run:</para>
+ <screen>$ lctl get_param osc.lustre-OST0002-osc.ost_conn_uuid
</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297761" xreflabel=""/>Typical output is:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295712" xreflabel=""/>osc.lustre-OST0002-osc.ost_conn_uuid=192.168.20.1@tcp
+ <para>Typical output is:</para>
+ <screen>osc.lustre-OST0002-osc.ost_conn_uuid=192.168.20.1@tcp
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438209_10424">
<title>18.5 Managing Free <anchor xml:id="dbdoclet.50438209_marker-1295520" xreflabel=""/>Space</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1293927" xreflabel=""/>The MDT assigns file stripes to OSTs based on location (which OSS) and size considerations (free space) to optimize file system performance. Emptier OSTs are preferentially selected for stripes, and stripes are preferentially spread out between OSSs to increase network bandwidth utilization. The weighting factor between these two optimizations can be adjusted by the user.</para>
+ <para>The MDT assigns file stripes to OSTs based on location (which OSS) and size considerations (free space) to optimize file system performance. Emptier OSTs are preferentially selected for stripes, and stripes are preferentially spread out between OSSs to increase network bandwidth utilization. The weighting factor between these two optimizations can be adjusted by the user.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438209_pgfId-1293929" xreflabel=""/>18.5.1 <anchor xml:id="dbdoclet.50438209_35838" xreflabel=""/>Checking File System Free Space</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1293930" xreflabel=""/>Free space is an important consideration in assigning file stripes. The lfsdf command shows available disk space on the mounted Lustre file system and space consumption per OST. If multiple Lustre file systems are mounted, a path may be specified, but is not required.</para>
+ <title>18.5.1 <anchor xml:id="dbdoclet.50438209_35838" xreflabel=""/>Checking File System Free Space</title>
+ <para>Free space is an important consideration in assigning file stripes. The lfsdf command shows available disk space on the mounted Lustre file system and space consumption per OST. If multiple Lustre file systems are mounted, a path may be specified, but is not required.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438209_pgfId-1293933" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438209_pgfId-1293935" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293937" xreflabel=""/><emphasis role="bold">-h</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293939" xreflabel=""/>Human-readable print sizes in human readable format (for example: 1K, 234M, 5G).</para></entry>
+ <entry><para> <emphasis role="bold">-h</emphasis></para></entry>
+ <entry><para> Human-readable print sizes in human readable format (for example: 1K, 234M, 5G).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293941" xreflabel=""/><emphasis role="bold">-i, --inodes</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293943" xreflabel=""/>Lists inodes instead of block usage.</para></entry>
+ <entry><para> <emphasis role="bold">-i, --inodes</emphasis></para></entry>
+ <entry><para> Lists inodes instead of block usage.</para></entry>
</row>
</tbody>
</tgroup>
<colspec colname="c1" colwidth="100*"/>
<tbody>
<row>
- <entry><para><emphasis role="bold">Note -</emphasis>
<anchor xml:id="dbdoclet.50438209_pgfId-1293944" xreflabel=""/>The df-i and lfsdf-i commands show the minimum number of inodes that can be created in the file system. Depending on the configuration, it may be possible to create more inodes than initially reported by df-i. Later, df-i operations will show the current, estimated free inode count.</para><para> If the underlying file system has fewer free blocks than inodes, then the total inode count for the file system reports only as many inodes as there are free blocks. This is done because Lustre may need to store an external attribute for each new inode, and it is better to report a free inode count that is the guaranteed, minimum number of inodes that can be created.</para></entry>
+ <entry><para><emphasis role="bold">Note -</emphasis>The df-i and lfsdf-i commands show the minimum number of inodes that can be created in the file system. Depending on the configuration, it may be possible to create more inodes than initially reported by df-i. Later, df-i operations will show the current, estimated free inode count.</para><para> If the underlying file system has fewer free blocks than inodes, then the total inode count for the file system reports only as many inodes as there are free blocks. This is done because Lustre may need to store an external attribute for each new inode, and it is better to report a free inode count that is the guaranteed, minimum number of inodes that can be created.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1293952" xreflabel=""/><emphasis role="bold">Examples</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1293953" xreflabel=""/>[lin-cli1] $ lfs df
-<anchor xml:id="dbdoclet.50438209_pgfId-1293954" xreflabel=""/>UUID 1K-blockS Used \
+ <para><emphasis role="bold">Examples</emphasis></para>
+ <screen>[lin-cli1] $ lfs df
+UUID 1K-blockS Used \
Available Use% Mounted on
-<anchor xml:id="dbdoclet.50438209_pgfId-1293955" xreflabel=""/>mds-lustre-0_UUID 9174328 1020024 8154304 \
+mds-lustre-0_UUID 9174328 1020024 8154304 \
11% /mnt/lustre[MDT:0]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293956" xreflabel=""/>ost-lustre-0_UUID 94181368 56330708 37850660\
+ost-lustre-0_UUID 94181368 56330708 37850660\
59% /mnt/lustre[OST:0]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293957" xreflabel=""/>ost-lustre-1_UUID 94181368 56385748 37795620\
+ost-lustre-1_UUID 94181368 56385748 37795620\
59% /mnt/lustre[OST:1]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293958" xreflabel=""/>ost-lustre-2_UUID 94181368 54352012 39829356\
+ost-lustre-2_UUID 94181368 54352012 39829356\
57% /mnt/lustre[OST:2]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293959" xreflabel=""/>filesystem summary: 282544104 167068468 \
+filesystem summary: 282544104 167068468 \
39829356 57% /mnt/lustre
-<anchor xml:id="dbdoclet.50438209_pgfId-1293960" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438209_pgfId-1293961" xreflabel=""/>[lin-cli1] $ lfs df -h
-<anchor xml:id="dbdoclet.50438209_pgfId-1293962" xreflabel=""/>UUID bytes Used \
+
+[lin-cli1] $ lfs df -h
+UUID bytes Used \
Available Use% Mounted on
-<anchor xml:id="dbdoclet.50438209_pgfId-1293963" xreflabel=""/>mds-lustre-0_UUID 8.7G 996.1M 7.8G \
+mds-lustre-0_UUID 8.7G 996.1M 7.8G \
11% /mnt/lustre[MDT:0]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293964" xreflabel=""/>ost-lustre-0_UUID 89.8G 53.7G 36.1G \
+ost-lustre-0_UUID 89.8G 53.7G 36.1G \
59% /mnt/lustre[OST:0]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293965" xreflabel=""/>ost-lustre-1_UUID 89.8G 53.8G 36.0G \
+ost-lustre-1_UUID 89.8G 53.8G 36.0G \
59% /mnt/lustre[OST:1]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293966" xreflabel=""/>ost-lustre-2_UUID 89.8G 51.8G 38.0G \
+ost-lustre-2_UUID 89.8G 51.8G 38.0G \
57% /mnt/lustre[OST:2]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293967" xreflabel=""/>filesystem summary: 269.5G 159.3G \
+filesystem summary: 269.5G 159.3G \
110.1G 59% /mnt/lustre
-<anchor xml:id="dbdoclet.50438209_pgfId-1293968" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438209_pgfId-1293969" xreflabel=""/>[lin-cli1] $ lfs df -i
-<anchor xml:id="dbdoclet.50438209_pgfId-1293970" xreflabel=""/>UUID Inodes IUsed \
+
+[lin-cli1] $ lfs df -i
+UUID Inodes IUsed \
IFree IUse% Mounted on
-<anchor xml:id="dbdoclet.50438209_pgfId-1293971" xreflabel=""/>mds-lustre-0_UUID 2211572 41924 2169648 \
+mds-lustre-0_UUID 2211572 41924 2169648 \
1% /mnt/lustre[MDT:0]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293972" xreflabel=""/>ost-lustre-0_UUID 737280 12183 725097 \
+ost-lustre-0_UUID 737280 12183 725097 \
1% /mnt/lustre[OST:0]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293973" xreflabel=""/>ost-lustre-1_UUID 737280 12232 725048 \
+ost-lustre-1_UUID 737280 12232 725048 \
1% /mnt/lustre[OST:1]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293974" xreflabel=""/>ost-lustre-2_UUID 737280 12214 725066 \
+ost-lustre-2_UUID 737280 12214 725066 \
1% /mnt/lustre[OST:2]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293975" xreflabel=""/>filesystem summary: 2211572 41924 \
+filesystem summary: 2211572 41924 \
2169648 1% /mnt/lustre[OST:2]
</screen>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1293986" xreflabel=""/>18.5.2 Using Stripe Allocations</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1295862" xreflabel=""/>Two stripe allocation methods are provided: <emphasis>round-robin</emphasis> and <emphasis>weighted</emphasis>. By default, the allocation method is determined by the amount of free-space imbalance on the OSTs. The weighted allocator is used when any two OSTs are imbalanced by more than 20%. Otherwise, the faster round-robin allocator is used. (The round-robin order maximizes network balancing.)</para>
+ <title>18.5.2 Using Stripe Allocations</title>
+ <para>Two stripe allocation methods are provided: <emphasis>round-robin</emphasis> and <emphasis>weighted</emphasis>. By default, the allocation method is determined by the amount of free-space imbalance on the OSTs. The weighted allocator is used when any two OSTs are imbalanced by more than 20%. Otherwise, the faster round-robin allocator is used. (The round-robin order maximizes network balancing.)</para>
<itemizedlist><listitem>
- <para><
anchor xml:id="dbdoclet.50438209_pgfId-1293989" xreflabel=""/><emphasis role="bold">Round-robin allocator</emphasis><anchor xml:id="dbdoclet.50438209_marker-1293988" xreflabel=""/> - When OSTs have approximately the same amount of free space (within 20%), an efficient round-robin allocator is used. The round-robin allocator alternates stripes between OSTs on different OSSs, so the OST used for stripe 0 of each file is evenly distributed among OSTs, regardless of the stripe count. Here are several sample round-robin stripe orders (each letter represents a different OST on a single OSS):</para>
+ <para><emphasis role="bold">Round-robin allocator</emphasis><anchor xml:id="dbdoclet.50438209_marker-1293988" xreflabel=""/> - When OSTs have approximately the same amount of free space (within 20%), an efficient round-robin allocator is used. The round-robin allocator alternates stripes between OSTs on different OSSs, so the OST used for stripe 0 of each file is evenly distributed among OSTs, regardless of the stripe count. Here are several sample round-robin stripe orders (each letter represents a different OST on a single OSS):</para>
<informaltable frame="none">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293993" xreflabel=""/>3: AAA</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293995" xreflabel=""/>One 3-OST OSS</para></entry>
+ <entry><para> 3: AAA</para></entry>
+ <entry><para> One 3-OST OSS</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293997" xreflabel=""/>3x3: ABABAB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293999" xreflabel=""/>Two 3-OST OSSs</para></entry>
+ <entry><para> 3x3: ABABAB</para></entry>
+ <entry><para> Two 3-OST OSSs</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1294001" xreflabel=""/>3x4: BBABABA</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1294003" xreflabel=""/>One 3-OST OSS (A) and one 4-OST OSS (B)</para></entry>
+ <entry><para> 3x4: BBABABA</para></entry>
+ <entry><para> One 3-OST OSS (A) and one 4-OST OSS (B)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1294005" xreflabel=""/>3x5: BBABBABA</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1294007" xreflabel=""/>One 3-OST OSS (A) and one 5-OST OSS (B)</para></entry>
+ <entry><para> 3x5: BBABBABA</para></entry>
+ <entry><para> One 3-OST OSS (A) and one 5-OST OSS (B)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1294009" xreflabel=""/>3x3x3: ABCABCABC</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1294011" xreflabel=""/>Three 3-OST OSSs</para></entry>
+ <entry><para> 3x3x3: ABCABCABC</para></entry>
+ <entry><para> Three 3-OST OSSs</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</listitem>
<listitem>
- <para><
anchor xml:id="dbdoclet.50438209_pgfId-1294021" xreflabel=""/><emphasis role="bold">Weighted allocator</emphasis><anchor xml:id="dbdoclet.50438209_marker-1294020" xreflabel=""/> - When the free space difference between the OSTs is significant (by default, 20% of the free space), then a weighting algorithm is used to influence OST ordering based on size and location. Note that these are weightings for a random algorithm, so the OST with the most free space is not necessarily chosen each time. On average, the weighted allocator fills the emptier OSTs faster.</para>
+ <para><emphasis role="bold">Weighted allocator</emphasis><anchor xml:id="dbdoclet.50438209_marker-1294020" xreflabel=""/> - When the free space difference between the OSTs is significant (by default, 20% of the free space), then a weighting algorithm is used to influence OST ordering based on size and location. Note that these are weightings for a random algorithm, so the OST with the most free space is not necessarily chosen each time. On average, the weighted allocator fills the emptier OSTs faster.</para>
</listitem>
</itemizedlist>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438209_pgfId-1294034" xreflabel=""/>18.5.3 Adjusting the <anchor xml:id="dbdoclet.50438209_marker-1294033" xreflabel=""/>Weighting Between Free Space and Location</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1295878" xreflabel=""/>The weighting priority can be adjusted in the /proc file /proc/fs/lustre/lov/lustre-mdtlov/qos_prio_free proc. The default value is 90%. Use this command on the MGS to permanently change this weighting:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295879" xreflabel=""/>lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90
+ <title>18.5.3 Adjusting the <anchor xml:id="dbdoclet.50438209_marker-1294033" xreflabel=""/>Weighting Between Free Space and Location</title>
+ <para>The weighting priority can be adjusted in the /proc file /proc/fs/lustre/lov/lustre-mdtlov/qos_prio_free proc. The default value is 90%. Use this command on the MGS to permanently change this weighting:</para>
+ <screen>lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90
</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1295880" xreflabel=""/>Increasing this value puts more weighting on free space. When the free space priority is set to 100%, then location is no longer used in stripe-ordering calculations and weighting is based entirely on free space.</para>
+ <para>Increasing this value puts more weighting on free space. When the free space priority is set to 100%, then location is no longer used in stripe-ordering calculations and weighting is based entirely on free space.</para>
<note><para>Setting the priority to 100% means that OSS distribution does not count in the weighting, but the stripe assignment is still done via a weighting. For example, if OST2 has twice as much free space as OST1, then OST2 is twice as likely to be used, but it is not guaranteed to be used.</para></note>
</section>
</section>
<colspec colname="c4" colwidth="25*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438248_pgfId-997745" xreflabel=""/>BookTitle</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438248_pgfId-997747" xreflabel=""/>Part Number</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438248_pgfId-997749" xreflabel=""/>Date</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438248_pgfId-997751" xreflabel=""/>Comments</emphasis></para></entry>
+ <entry><para><emphasis role="bold">BookTitle</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Part Number</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Date</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Comments</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438248_pgfId-997756" xreflabel=""/>Lustre 2.0 Operations Manual</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438248_pgfId-997758" xreflabel=""/>821-2076-10</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438248_pgfId-997763" xreflabel=""/>July 2010</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438248_pgfId-997765" xreflabel=""/>First release of Lustre 2.0 manual</para></entry>
+ <entry><para> Lustre 2.0 Operations Manual</para></entry>
+ <entry><para> 821-2076-10</para></entry>
+ <entry><para> July 2010</para></entry>
+ <entry><para> First release of Lustre 2.0 manual</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438248_pgfId-997955" xreflabel=""/>Lustre 2.0 Operations Manual</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438248_pgfId-997957" xreflabel=""/>821-2076-10</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438248_pgfId-997962" xreflabel=""/>January 2011</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438248_pgfId-997964" xreflabel=""/>Second release of Lustre 2.0 manual</para></entry>
+ <entry><para> Lustre 2.0 Operations Manual</para></entry>
+ <entry><para> 821-2076-10</para></entry>
+ <entry><para> January 2011</para></entry>
+ <entry><para> Second release of Lustre 2.0 manual</para></entry>
</row>
</tbody>
</tgroup>
<info>
<title xml:id='settinglustreproperties.title'>Setting Lustre Properties in a C Program (llapi)</title>
</info>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297638" xreflabel=""/>This chapter describes the llapi library of commands used for setting Lustre file properties within a C program running in a cluster environment, such as a data processing or MPI application. The commands described in this chapter are:</para>
+ <para>This chapter describes the llapi library of commands used for setting Lustre file properties within a C program running in a cluster environment, such as a data processing or MPI application. The commands described in this chapter are:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438215_30970"/></para>
</listitem>
<section xml:id="dbdoclet.50438215_30970">
<title>34.1 llapi_file_create</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1296829" xreflabel=""/>Use llapi_file_create to set Lustre properties for a new file.</para>
+ <para>Use llapi_file_create to set Lustre properties for a new file.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1296830" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1296831" xreflabel=""/>#include <lustre/liblustreapi.h>#include <lustre/lustre_user.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1296832" xreflabel=""/>int llapi_file_create(char *name, long stripe_size, int stripe_offset, int \
+ <title>Synopsis</title>
+ <screen>#include <lustre/liblustreapi.h>#include <lustre/lustre_user.h>
+int llapi_file_create(char *name, long stripe_size, int stripe_offset, int \
stripe_count, int stripe_pattern);
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1296833" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1296834" xreflabel=""/>The llapi_file_create() function sets a file descriptor's Lustre striping information. The file descriptor is then accessed with open ().</para>
+ <title>Description</title>
+ <para>The llapi_file_create() function sets a file descriptor's Lustre striping information. The file descriptor is then accessed with open ().</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1296837" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1296839" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1296841" xreflabel=""/><emphasis role="bold">llapi_file_create()</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1296843" xreflabel=""/>If the file already exists, this parameter returns to 'EEXIST'. If the stripe parameters are invalid, this parameter returns to 'EINVAL'.</para></entry>
+ <entry><para> <emphasis role="bold">llapi_file_create()</emphasis></para></entry>
+ <entry><para> If the file already exists, this parameter returns to 'EEXIST'. If the stripe parameters are invalid, this parameter returns to 'EINVAL'.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1296845" xreflabel=""/><emphasis role="bold">stripe_size</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1296847" xreflabel=""/>This value must be an even multiple of system page size, as shown by getpagesize (). The default Lustre stripe size is 4MB.</para></entry>
+ <entry><para> <emphasis role="bold">stripe_size</emphasis></para></entry>
+ <entry><para> This value must be an even multiple of system page size, as shown by getpagesize (). The default Lustre stripe size is 4MB.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1296849" xreflabel=""/><emphasis role="bold">stripe_offset</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1296851" xreflabel=""/>Indicates the starting OST for this file.</para></entry>
+ <entry><para> <emphasis role="bold">stripe_offset</emphasis></para></entry>
+ <entry><para> Indicates the starting OST for this file.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1296853" xreflabel=""/><emphasis role="bold">stripe_count</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1296855" xreflabel=""/>Indicates the number of OSTs that this file will be striped across.</para></entry>
+ <entry><para> <emphasis role="bold">stripe_count</emphasis></para></entry>
+ <entry><para> Indicates the number of OSTs that this file will be striped across.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1296857" xreflabel=""/><emphasis role="bold">stripe_pattern</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1296859" xreflabel=""/> Indicates the RAID pattern.</para></entry>
+ <entry><para> <emphasis role="bold">stripe_pattern</emphasis></para></entry>
+ <entry><para> Indicates the RAID pattern.</para></entry>
</row>
</tbody>
</tgroup>
<note><para>Currently, only RAID 0 is supported. To use the system defaults, set these values: stripe_size = 0, stripe_offset = -1, stripe_count = 0, stripe_pattern = 0</para></note>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1296861" xreflabel=""/>Examples</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1296862" xreflabel=""/>System default size is 4 MB.</para>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1296863" xreflabel=""/>char *tfile = TESTFILE;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296864" xreflabel=""/>int stripe_size = 65536
+ <title>Examples</title>
+ <para>System default size is 4 MB.</para>
+ <screen>char *tfile = TESTFILE;
+int stripe_size = 65536
</screen>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1296865" xreflabel=""/>To start at default, run:</para>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1296866" xreflabel=""/>int stripe_offset = -1
+ <para>To start at default, run:</para>
+ <screen>int stripe_offset = -1
</screen>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1296867" xreflabel=""/>To start at the default, run:</para>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1296868" xreflabel=""/>int stripe_count = 1
+ <para>To start at the default, run:</para>
+ <screen>int stripe_count = 1
</screen>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1296869" xreflabel=""/>To set a single stripe for this example, run:</para>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1296870" xreflabel=""/>int stripe_pattern = 0
+ <para>To set a single stripe for this example, run:</para>
+ <screen>int stripe_pattern = 0
</screen>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1296871" xreflabel=""/>Currently, only RAID 0 is supported.</para>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1296872" xreflabel=""/>int stripe_pattern = 0;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296873" xreflabel=""/>int rc, fd;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296874" xreflabel=""/>rc = llapi_file_create(tfile, stripe_size,stripe_offset, stripe_count,strip\
+ <para>Currently, only RAID 0 is supported.</para>
+ <screen>int stripe_pattern = 0;
+int rc, fd;
+rc = llapi_file_create(tfile, stripe_size,stripe_offset, stripe_count,strip\
e_pattern);
</screen>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1296875" xreflabel=""/>Result code is inverted, you may return with 'EINVAL' or an ioctl error.</para>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1296876" xreflabel=""/>if (rc) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1296877" xreflabel=""/>fprintf(stderr,"llapi_file_create failed: %d (%s) 0, rc, strerror(-rc));retu\
+ <para>Result code is inverted, you may return with 'EINVAL' or an ioctl error.</para>
+ <screen>if (rc) {
+fprintf(stderr,"llapi_file_create failed: %d (%s) 0, rc, strerror(-rc));retu\
rn -1; }
</screen>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1296878" xreflabel=""/>llapi_file_create closes the file descriptor. You must re-open the descriptor. To do this, run:</para>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1296879" xreflabel=""/>fd = open(tfile, O_CREAT | O_RDWR | O_LOV_DELAY_CREATE, 0644); if (fd < 0) \\
+ <para>llapi_file_create closes the file descriptor. You must re-open the descriptor. To do this, run:</para>
+ <screen>fd = open(tfile, O_CREAT | O_RDWR | O_LOV_DELAY_CREATE, 0644); if (fd < 0) \\
{ fprintf(stderr, "Can't open %s file: %s0, tfile,
-<anchor xml:id="dbdoclet.50438215_pgfId-1296880" xreflabel=""/>str-
-<anchor xml:id="dbdoclet.50438215_pgfId-1296881" xreflabel=""/>error(errno));
-<anchor xml:id="dbdoclet.50438215_pgfId-1296882" xreflabel=""/>return -1;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296883" xreflabel=""/>}
+str-
+error(errno));
+return -1;
+}
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438215_50149">
<title>34.2 llapi_file_get_stripe</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1296886" xreflabel=""/>Use llapi_file_get_stripe to get striping information for a file or directory on a Lustre file system.</para>
+ <para>Use llapi_file_get_stripe to get striping information for a file or directory on a Lustre file system.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1296887" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1296888" xreflabel=""/>#include <sys/types.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1296889" xreflabel=""/>#include <sys/stat.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1296890" xreflabel=""/>#include <fcntl.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1296891" xreflabel=""/>#include <liblustre.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1296892" xreflabel=""/>#include <lustre/lustre_idl.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1296893" xreflabel=""/>#include <lustre/liblustreapi.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1296894" xreflabel=""/>#include <lustre/lustre_user.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1296895" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1296896" xreflabel=""/>int llapi_file_get_stripe(const char *<emphasis>path</emphasis>, void *<emphasis>lum</emphasis>);
+ <title>Synopsis</title>
+ <screen>#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <liblustre.h>
+#include <lustre/lustre_idl.h>
+#include <lustre/liblustreapi.h>
+#include <lustre/lustre_user.h>
+
+int llapi_file_get_stripe(const char *<emphasis>path</emphasis>, void *<emphasis>lum</emphasis>);
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1296897" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1296898" xreflabel=""/>The llapi_file_get_stripe() function returns striping information for a file or directory <emphasis>path</emphasis> in <emphasis>lum</emphasis> (which should point to a large enough memory region) in one of the following formats:</para>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1296899" xreflabel=""/>struct lov_user_md_v1 {
-<anchor xml:id="dbdoclet.50438215_pgfId-1296900" xreflabel=""/>__u32 lmm_magic;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296901" xreflabel=""/>__u32 lmm_pattern;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296902" xreflabel=""/>__u64 lmm_object_id;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296903" xreflabel=""/>__u64 lmm_object_seq;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296904" xreflabel=""/>__u32 lmm_stripe_size;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296905" xreflabel=""/>__u16 lmm_stripe_count;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296906" xreflabel=""/>__u16 lmm_stripe_offset;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296907" xreflabel=""/>struct lov_user_ost_data_v1 lmm_objects[0];
-<anchor xml:id="dbdoclet.50438215_pgfId-1296908" xreflabel=""/>} __attribute__((packed));
-<anchor xml:id="dbdoclet.50438215_pgfId-1296909" xreflabel=""/>struct lov_user_md_v3 {
-<anchor xml:id="dbdoclet.50438215_pgfId-1296910" xreflabel=""/>__u32 lmm_magic;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296911" xreflabel=""/>__u32 lmm_pattern;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296912" xreflabel=""/>__u64 lmm_object_id;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296913" xreflabel=""/>__u64 lmm_object_seq;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296914" xreflabel=""/>__u32 lmm_stripe_size;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296915" xreflabel=""/>__u16 lmm_stripe_count;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296916" xreflabel=""/>__u16 lmm_stripe_offset;
-<anchor xml:id="dbdoclet.50438215_pgfId-1296917" xreflabel=""/>char lmm_pool_name[LOV_MAXPOOLNAME];
-<anchor xml:id="dbdoclet.50438215_pgfId-1296918" xreflabel=""/>struct lov_user_ost_data_v1 lmm_objects[0];
-<anchor xml:id="dbdoclet.50438215_pgfId-1297038" xreflabel=""/>} __attribute__((packed));
+ <title>Description</title>
+ <para>The llapi_file_get_stripe() function returns striping information for a file or directory <emphasis>path</emphasis> in <emphasis>lum</emphasis> (which should point to a large enough memory region) in one of the following formats:</para>
+ <screen>struct lov_user_md_v1 {
+__u32 lmm_magic;
+__u32 lmm_pattern;
+__u64 lmm_object_id;
+__u64 lmm_object_seq;
+__u32 lmm_stripe_size;
+__u16 lmm_stripe_count;
+__u16 lmm_stripe_offset;
+struct lov_user_ost_data_v1 lmm_objects[0];
+} __attribute__((packed));
+struct lov_user_md_v3 {
+__u32 lmm_magic;
+__u32 lmm_pattern;
+__u64 lmm_object_id;
+__u64 lmm_object_seq;
+__u32 lmm_stripe_size;
+__u16 lmm_stripe_count;
+__u16 lmm_stripe_offset;
+char lmm_pool_name[LOV_MAXPOOLNAME];
+struct lov_user_ost_data_v1 lmm_objects[0];
+} __attribute__((packed));
</screen>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1298282" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1298284" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298286" xreflabel=""/><emphasis role="bold">lmm_magic</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298288" xreflabel=""/>Specifies the format of the returned striping information. <emphasis role="bold">LOV_MAGIC_V1</emphasis> isused for lov_user_md_v1. <emphasis role="bold">LOV_MAGIC_V3</emphasis> is used for lov_user_md_v3.</para></entry>
+ <entry><para> <emphasis role="bold">lmm_magic</emphasis></para></entry>
+ <entry><para> Specifies the format of the returned striping information. <emphasis role="bold">LOV_MAGIC_V1</emphasis> isused for lov_user_md_v1. <emphasis role="bold">LOV_MAGIC_V3</emphasis> is used for lov_user_md_v3.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298290" xreflabel=""/><emphasis role="bold">lmm_pattern</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298292" xreflabel=""/>Holds the striping pattern. Only <emphasis role="bold">LOV_PATTERN_RAID0</emphasis> is possible in this Lustre version.</para></entry>
+ <entry><para> <emphasis role="bold">lmm_pattern</emphasis></para></entry>
+ <entry><para> Holds the striping pattern. Only <emphasis role="bold">LOV_PATTERN_RAID0</emphasis> is possible in this Lustre version.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298294" xreflabel=""/><emphasis role="bold">lmm_object_id</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298296" xreflabel=""/>Holds the MDS object ID.</para></entry>
+ <entry><para> <emphasis role="bold">lmm_object_id</emphasis></para></entry>
+ <entry><para> Holds the MDS object ID.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298298" xreflabel=""/><emphasis role="bold">lmm_object_gr</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298300" xreflabel=""/>Holds the MDS object group.</para></entry>
+ <entry><para> <emphasis role="bold">lmm_object_gr</emphasis></para></entry>
+ <entry><para> Holds the MDS object group.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298302" xreflabel=""/><emphasis role="bold">lmm_stripe_size</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298304" xreflabel=""/>Holds the stripe size in bytes.</para></entry>
+ <entry><para> <emphasis role="bold">lmm_stripe_size</emphasis></para></entry>
+ <entry><para> Holds the stripe size in bytes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298306" xreflabel=""/><emphasis role="bold">lmm_stripe_count</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298308" xreflabel=""/>Holds the number of OSTs over which the file is striped.</para></entry>
+ <entry><para> <emphasis role="bold">lmm_stripe_count</emphasis></para></entry>
+ <entry><para> Holds the number of OSTs over which the file is striped.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298310" xreflabel=""/><emphasis role="bold">lmm_stripe_offset</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298359" xreflabel=""/>Holds the OST index from which the file starts.</para></entry>
+ <entry><para> <emphasis role="bold">lmm_stripe_offset</emphasis></para></entry>
+ <entry><para> Holds the OST index from which the file starts.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298314" xreflabel=""/><emphasis role="bold">lmm_pool_name</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298374" xreflabel=""/>Holds the OST pool name to which the file belongs.</para></entry>
+ <entry><para> <emphasis role="bold">lmm_pool_name</emphasis></para></entry>
+ <entry><para> Holds the OST pool name to which the file belongs.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298318" xreflabel=""/><emphasis role="bold">lmm_objects</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298390" xreflabel=""/>An array of lmm_stripe_count members containing per OST file information in</para><para><anchor xml:id="dbdoclet.50438215_pgfId-1298391" xreflabel=""/>the following format:</para><para><anchor xml:id="dbdoclet.50438215_pgfId-1298392" xreflabel=""/>struct lov_user_ost_data_v1 {</para><para><anchor xml:id="dbdoclet.50438215_pgfId-1298393" xreflabel=""/>__u64 l_object_id;</para><para><anchor xml:id="dbdoclet.50438215_pgfId-1298394" xreflabel=""/>__u64 l_object_seq;</para><para><anchor xml:id="dbdoclet.50438215_pgfId-1298395" xreflabel=""/>__u32 l_ost_gen;</para><para><anchor xml:id="dbdoclet.50438215_pgfId-1298396" xreflabel=""/>__u32 l_ost_idx;</para><para><anchor xml:id="dbdoclet.50438215_pgfId-1298397" xreflabel=""/>} __attribute__((packed));</para></entry>
+ <entry><para> <emphasis role="bold">lmm_objects</emphasis></para></entry>
+ <entry><para> An array of lmm_stripe_count members containing per OST file information in</para><para>the following format:</para><para>struct lov_user_ost_data_v1 {</para><para>__u64 l_object_id;</para><para>__u64 l_object_seq;</para><para>__u32 l_ost_gen;</para><para>__u32 l_ost_idx;</para><para>} __attribute__((packed));</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298322" xreflabel=""/><emphasis role="bold">l_object_id</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298412" xreflabel=""/>Holds the OST's object ID.</para></entry>
+ <entry><para> <emphasis role="bold">l_object_id</emphasis></para></entry>
+ <entry><para> Holds the OST's object ID.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298326" xreflabel=""/><emphasis role="bold">l_object_seq</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298427" xreflabel=""/>Holds the OST's object group.</para></entry>
+ <entry><para> <emphasis role="bold">l_object_seq</emphasis></para></entry>
+ <entry><para> Holds the OST's object group.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298330" xreflabel=""/><emphasis role="bold">l_ost_gen</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298442" xreflabel=""/>Holds the OST's index generation.</para></entry>
+ <entry><para> <emphasis role="bold">l_ost_gen</emphasis></para></entry>
+ <entry><para> Holds the OST's index generation.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298334" xreflabel=""/><emphasis role="bold">l_ost_idx</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298461" xreflabel=""/>Holds the OST's index in LOV.</para></entry>
+ <entry><para> <emphasis role="bold">l_ost_idx</emphasis></para></entry>
+ <entry><para> Holds the OST's index in LOV.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1297039" xreflabel=""/>Return Values</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297040" xreflabel=""/>llapi_file_get_stripe() returns:</para>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297041" xreflabel=""/><emphasis role="bold">0</emphasis> On success</para>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297042" xreflabel=""/><emphasis role="bold">!= 0</emphasis> On failure, <emphasis>errno</emphasis> is set appropriately</para>
+ <title>Return Values</title>
+ <para>llapi_file_get_stripe() returns:</para>
+ <para><emphasis role="bold">0</emphasis> On success</para>
+ <para><emphasis role="bold">!= 0</emphasis> On failure, <emphasis>errno</emphasis> is set appropriately</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1297043" xreflabel=""/>Errors</title>
+ <title>Errors</title>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1298797" xreflabel=""/>Errors</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1298799" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Errors</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298801" xreflabel=""/><emphasis role="bold">ENOMEM</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298803" xreflabel=""/><emphasis role="bold">Failed to allocate memory</emphasis></para></entry>
+ <entry><para> <emphasis role="bold">ENOMEM</emphasis></para></entry>
+ <entry><para> <emphasis role="bold">Failed to allocate memory</emphasis></para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298805" xreflabel=""/><emphasis role="bold">ENAMETOOLONG</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298868" xreflabel=""/>Path was too long</para></entry>
+ <entry><para> <emphasis role="bold">ENAMETOOLONG</emphasis></para></entry>
+ <entry><para> Path was too long</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298809" xreflabel=""/><emphasis role="bold">ENOENT</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298870" xreflabel=""/>Path does not point to a file or directory</para></entry>
+ <entry><para> <emphasis role="bold">ENOENT</emphasis></para></entry>
+ <entry><para> Path does not point to a file or directory</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298813" xreflabel=""/><emphasis role="bold">ENOTTY</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298872" xreflabel=""/>Path does not point to a Lustre file system</para></entry>
+ <entry><para> <emphasis role="bold">ENOTTY</emphasis></para></entry>
+ <entry><para> Path does not point to a Lustre file system</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298817" xreflabel=""/><emphasis role="bold">EFAULT</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298819" xreflabel=""/>Memory region pointed by lum is not properly mapped</para></entry>
+ <entry><para> <emphasis role="bold">EFAULT</emphasis></para></entry>
+ <entry><para> Memory region pointed by lum is not properly mapped</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1298829" xreflabel=""/>Examples</title>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1297050" xreflabel=""/>#include <sys/vfs.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297051" xreflabel=""/>#include <liblustre.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297052" xreflabel=""/>#include <lnet/lnetctl.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297053" xreflabel=""/>#include <obd.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297054" xreflabel=""/>#include <lustre_lib.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297055" xreflabel=""/>#include <lustre/liblustreapi.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297056" xreflabel=""/>#include <obd_lov.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297057" xreflabel=""/>static inline int maxint(int a, int b)
-<anchor xml:id="dbdoclet.50438215_pgfId-1297058" xreflabel=""/>{
-<anchor xml:id="dbdoclet.50438215_pgfId-1297059" xreflabel=""/>return a > b ? a : b;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297060" xreflabel=""/>}
-<anchor xml:id="dbdoclet.50438215_pgfId-1297061" xreflabel=""/>static void *alloc_lum()
-<anchor xml:id="dbdoclet.50438215_pgfId-1297062" xreflabel=""/>{
-<anchor xml:id="dbdoclet.50438215_pgfId-1297063" xreflabel=""/>int v1, v3, join;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297064" xreflabel=""/>v1 = sizeof(struct lov_user_md_v1) +
-<anchor xml:id="dbdoclet.50438215_pgfId-1297065" xreflabel=""/>LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data_v1);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297066" xreflabel=""/>v3 = sizeof(struct lov_user_md_v3) +
-<anchor xml:id="dbdoclet.50438215_pgfId-1297067" xreflabel=""/>LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data_v1);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297068" xreflabel=""/>return malloc(maxint(v1, v3));
-<anchor xml:id="dbdoclet.50438215_pgfId-1297069" xreflabel=""/>}
-<anchor xml:id="dbdoclet.50438215_pgfId-1297070" xreflabel=""/>int main(int argc, char** argv)
-<anchor xml:id="dbdoclet.50438215_pgfId-1297071" xreflabel=""/>{
-<anchor xml:id="dbdoclet.50438215_pgfId-1297072" xreflabel=""/>struct lov_user_md *lum_file = NULL;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297073" xreflabel=""/>int rc;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297074" xreflabel=""/>int lum_size;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297075" xreflabel=""/>if (argc != 2) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297076" xreflabel=""/>fprintf(stderr, "Usage: %s <filename>\n", argv[0]);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297077" xreflabel=""/>return 1;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297078" xreflabel=""/>}
-<anchor xml:id="dbdoclet.50438215_pgfId-1297079" xreflabel=""/>lum_file = alloc_lum();
-<anchor xml:id="dbdoclet.50438215_pgfId-1297080" xreflabel=""/>if (lum_file == NULL) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297081" xreflabel=""/>rc = ENOMEM;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297082" xreflabel=""/>goto cleanup;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297083" xreflabel=""/>}
-<anchor xml:id="dbdoclet.50438215_pgfId-1297084" xreflabel=""/>rc = llapi_file_get_stripe(argv[1], lum_file);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297085" xreflabel=""/>if (rc) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297086" xreflabel=""/>rc = errno;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297087" xreflabel=""/>goto cleanup;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297088" xreflabel=""/>}
-<anchor xml:id="dbdoclet.50438215_pgfId-1297089" xreflabel=""/>/* stripe_size stripe_count */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297090" xreflabel=""/>printf("%d %d\n",
-<anchor xml:id="dbdoclet.50438215_pgfId-1297091" xreflabel=""/>lum_file->lmm_stripe_size,
-<anchor xml:id="dbdoclet.50438215_pgfId-1297092" xreflabel=""/>lum_file->lmm_stripe_count);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297093" xreflabel=""/>cleanup:
-<anchor xml:id="dbdoclet.50438215_pgfId-1297094" xreflabel=""/>if (lum_file != NULL)
-<anchor xml:id="dbdoclet.50438215_pgfId-1297095" xreflabel=""/>free(lum_file);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297096" xreflabel=""/>return rc;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297097" xreflabel=""/>}
+ <title>Examples</title>
+ <screen>#include <sys/vfs.h>
+#include <liblustre.h>
+#include <lnet/lnetctl.h>
+#include <obd.h>
+#include <lustre_lib.h>
+#include <lustre/liblustreapi.h>
+#include <obd_lov.h>
+static inline int maxint(int a, int b)
+{
+return a > b ? a : b;
+}
+static void *alloc_lum()
+{
+int v1, v3, join;
+v1 = sizeof(struct lov_user_md_v1) +
+LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data_v1);
+v3 = sizeof(struct lov_user_md_v3) +
+LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data_v1);
+return malloc(maxint(v1, v3));
+}
+int main(int argc, char** argv)
+{
+struct lov_user_md *lum_file = NULL;
+int rc;
+int lum_size;
+if (argc != 2) {
+fprintf(stderr, "Usage: %s <filename>\n", argv[0]);
+return 1;
+}
+lum_file = alloc_lum();
+if (lum_file == NULL) {
+rc = ENOMEM;
+goto cleanup;
+}
+rc = llapi_file_get_stripe(argv[1], lum_file);
+if (rc) {
+rc = errno;
+goto cleanup;
+}
+/* stripe_size stripe_count */
+printf("%d %d\n",
+lum_file->lmm_stripe_size,
+lum_file->lmm_stripe_count);
+cleanup:
+if (lum_file != NULL)
+free(lum_file);
+return rc;
+}
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438215_86607">
<title>34.3 llapi_file_open</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1298469" xreflabel=""/>The llapi_file_open command opens (or creates) a file or device on a Lustre filesystem.</para>
+ <para>The llapi_file_open command opens (or creates) a file or device on a Lustre filesystem.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1297361" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1297362" xreflabel=""/>#include <sys/types.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297363" xreflabel=""/>#include <sys/stat.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297364" xreflabel=""/>#include <fcntl.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297365" xreflabel=""/>#include <liblustre.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297366" xreflabel=""/>#include <lustre/lustre_idl.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297367" xreflabel=""/>#include <lustre/liblustreapi.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297368" xreflabel=""/>#include <lustre/lustre_user.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297369" xreflabel=""/>int llapi_file_open(const char *<emphasis>name</emphasis>, int <emphasis>flags</emphasis>, int <emphasis>mode</emphasis>,
-<anchor xml:id="dbdoclet.50438215_pgfId-1297370" xreflabel=""/> unsigned long long <emphasis>stripe_size</emphasis>, int <emphasis>stripe_offset</emphasis>,
-<anchor xml:id="dbdoclet.50438215_pgfId-1297371" xreflabel=""/> int <emphasis>stripe_count</emphasis>, int <emphasis>stripe_pattern</emphasis>);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297372" xreflabel=""/>int llapi_file_create(const char *<emphasis>name</emphasis>, unsigned long long <emphasis>stripe_size</emphasis>,
-<anchor xml:id="dbdoclet.50438215_pgfId-1297373" xreflabel=""/> int <emphasis>stripe_offset</emphasis>, int <emphasis>stripe_count</emphasis>,
-<anchor xml:id="dbdoclet.50438215_pgfId-1297374" xreflabel=""/> int <emphasis>stripe_pattern</emphasis>);
+ <title>Synopsis</title>
+ <screen>#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <liblustre.h>
+#include <lustre/lustre_idl.h>
+#include <lustre/liblustreapi.h>
+#include <lustre/lustre_user.h>
+int llapi_file_open(const char *<emphasis>name</emphasis>, int <emphasis>flags</emphasis>, int <emphasis>mode</emphasis>,
+ unsigned long long <emphasis>stripe_size</emphasis>, int <emphasis>stripe_offset</emphasis>,
+ int <emphasis>stripe_count</emphasis>, int <emphasis>stripe_pattern</emphasis>);
+int llapi_file_create(const char *<emphasis>name</emphasis>, unsigned long long <emphasis>stripe_size</emphasis>,
+ int <emphasis>stripe_offset</emphasis>, int <emphasis>stripe_count</emphasis>,
+ int <emphasis>stripe_pattern</emphasis>);
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1298000" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1298001" xreflabel=""/>The llapi_file_create() call is equivalent to the llapi_file_open call with <emphasis>flags</emphasis> equal to O_CREAT|O_WRONLY and <emphasis>mode</emphasis> equal to 0644, followed by file close.</para>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1298510" xreflabel=""/>llapi_file_open() opens a file with a given name on a Lustre filesystem.</para>
+ <title>Description</title>
+ <para>The llapi_file_create() call is equivalent to the llapi_file_open call with <emphasis>flags</emphasis> equal to O_CREAT|O_WRONLY and <emphasis>mode</emphasis> equal to 0644, followed by file close.</para>
+ <para>llapi_file_open() opens a file with a given name on a Lustre filesystem.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1298598" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1298600" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298602" xreflabel=""/><emphasis role="bold">flags</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298636" xreflabel=""/>Can be a combination of O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_EXCL, O_NOCTTY, O_TRUNC, O_APPEND, O_NONBLOCK, O_SYNC, FASYNC, O_DIRECT, O_LARGEFILE, O_DIRECTORY, O_NOFOLLOW, O_NOATIME.</para></entry>
+ <entry><para> <emphasis role="bold">flags</emphasis></para></entry>
+ <entry><para> Can be a combination of O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_EXCL, O_NOCTTY, O_TRUNC, O_APPEND, O_NONBLOCK, O_SYNC, FASYNC, O_DIRECT, O_LARGEFILE, O_DIRECTORY, O_NOFOLLOW, O_NOATIME.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298606" xreflabel=""/><emphasis role="bold">mode</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298656" xreflabel=""/>Specifies the permission bits to be used for a new file when O_CREAT is used.</para></entry>
+ <entry><para> <emphasis role="bold">mode</emphasis></para></entry>
+ <entry><para> Specifies the permission bits to be used for a new file when O_CREAT is used.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298610" xreflabel=""/><emphasis role="bold">stripe_size</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298669" xreflabel=""/>Specifies stripe size (in bytes). Should be multiple of 64 KB, not exceeding 4 GB.</para></entry>
+ <entry><para> <emphasis role="bold">stripe_size</emphasis></para></entry>
+ <entry><para> Specifies stripe size (in bytes). Should be multiple of 64 KB, not exceeding 4 GB.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298614" xreflabel=""/><emphasis role="bold">stripe_offset</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298678" xreflabel=""/>Specifies an OST index from which the file should start. The default value is -1.</para></entry>
+ <entry><para> <emphasis role="bold">stripe_offset</emphasis></para></entry>
+ <entry><para> Specifies an OST index from which the file should start. The default value is -1.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298618" xreflabel=""/><emphasis role="bold">stripe_count</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298687" xreflabel=""/>Specifies the number of OSTs to stripe the file across. The default value is -1.</para></entry>
+ <entry><para> <emphasis role="bold">stripe_count</emphasis></para></entry>
+ <entry><para> Specifies the number of OSTs to stripe the file across. The default value is -1.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298693" xreflabel=""/><emphasis role="bold">stripe_pattern</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298705" xreflabel=""/>Specifies the striping pattern. In this version of Lustre, only LOV_PATTERN_RAID0 is available. The default value is 0.</para></entry>
+ <entry><para> <emphasis role="bold">stripe_pattern</emphasis></para></entry>
+ <entry><para> Specifies the striping pattern. In this version of Lustre, only LOV_PATTERN_RAID0 is available. The default value is 0.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1298592" xreflabel=""/>Return Values</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297430" xreflabel=""/>llapi_file_open() and llapi_file_create() return:</para>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297431" xreflabel=""/><emphasis role="bold">>=0</emphasis> On success, for llapi_file_open the return value is a file descriptor</para>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297432" xreflabel=""/><emphasis role="bold"><0</emphasis> On failure, the absolute value is an error code</para>
+ <title>Return Values</title>
+ <para>llapi_file_open() and llapi_file_create() return:</para>
+ <para><emphasis role="bold">>=0</emphasis> On success, for llapi_file_open the return value is a file descriptor</para>
+ <para><emphasis role="bold"><0</emphasis> On failure, the absolute value is an error code</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1298750" xreflabel=""/>Errors</title>
+ <title>Errors</title>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1297435" xreflabel=""/>Errors</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1297437" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Errors</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297439" xreflabel=""/><emphasis role="bold">EINVAL</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297441" xreflabel=""/><emphasis role="bold">stripe_size</emphasis> or <emphasis role="bold">stripe_offset</emphasis> or <emphasis role="bold">stripe_count</emphasis> or <emphasis role="bold">stripe_pattern</emphasis> is invalid.</para></entry>
+ <entry><para> <emphasis role="bold">EINVAL</emphasis></para></entry>
+ <entry><para> <emphasis role="bold">stripe_size</emphasis> or <emphasis role="bold">stripe_offset</emphasis> or <emphasis role="bold">stripe_count</emphasis> or <emphasis role="bold">stripe_pattern</emphasis> is invalid.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297443" xreflabel=""/><emphasis role="bold">EEXIST</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298878" xreflabel=""/>Striping information has already been set and cannot be altered; <emphasis role="bold">name</emphasis> already exists.</para></entry>
+ <entry><para> <emphasis role="bold">EEXIST</emphasis></para></entry>
+ <entry><para> Striping information has already been set and cannot be altered; <emphasis role="bold">name</emphasis> already exists.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297447" xreflabel=""/><emphasis role="bold">EALREADY</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297449" xreflabel=""/>Striping information has already been set and cannot be altered</para></entry>
+ <entry><para> <emphasis role="bold">EALREADY</emphasis></para></entry>
+ <entry><para> Striping information has already been set and cannot be altered</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297451" xreflabel=""/><emphasis role="bold">ENOTTY</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297453" xreflabel=""/><emphasis role="bold">name</emphasis> may not point to a Lustre filesystem.</para></entry>
+ <entry><para> <emphasis role="bold">ENOTTY</emphasis></para></entry>
+ <entry><para> <emphasis role="bold">name</emphasis> may not point to a Lustre filesystem.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1297462" xreflabel=""/>Example</title>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1297463" xreflabel=""/>#include <sys/types.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297464" xreflabel=""/>#include <sys/stat.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297465" xreflabel=""/>#include <fcntl.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297466" xreflabel=""/>#include <errno.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297467" xreflabel=""/>#include <stdio.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297468" xreflabel=""/>#include <liblustre.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297469" xreflabel=""/>#include <lustre/lustre_idl.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297470" xreflabel=""/>#include <lustre/liblustreapi.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297471" xreflabel=""/>#include <lustre/lustre_user.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297472" xreflabel=""/>int main(int argc, char *argv[])
-<anchor xml:id="dbdoclet.50438215_pgfId-1297473" xreflabel=""/>{
-<anchor xml:id="dbdoclet.50438215_pgfId-1297474" xreflabel=""/> int rc;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297475" xreflabel=""/> if (argc != 2)
-<anchor xml:id="dbdoclet.50438215_pgfId-1297476" xreflabel=""/> return -1;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297477" xreflabel=""/> rc = llapi_file_create(argv[1], 1048576, 0, 2, LOV_PATTERN_RAID0);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297478" xreflabel=""/> if (rc < 0) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297479" xreflabel=""/> fprintf(stderr, "file creation has failed, %s\n", strerror\
+ <title>Example</title>
+ <screen>#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <stdio.h>
+#include <liblustre.h>
+#include <lustre/lustre_idl.h>
+#include <lustre/liblustreapi.h>
+#include <lustre/lustre_user.h>
+int main(int argc, char *argv[])
+{
+ int rc;
+ if (argc != 2)
+ return -1;
+ rc = llapi_file_create(argv[1], 1048576, 0, 2, LOV_PATTERN_RAID0);
+ if (rc < 0) {
+ fprintf(stderr, "file creation has failed, %s\n", strerror\
(-rc));
-<anchor xml:id="dbdoclet.50438215_pgfId-1297480" xreflabel=""/> return -1;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297481" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297482" xreflabel=""/> printf("%s with stripe size 1048576, striped across 2 OSTs,"
-<anchor xml:id="dbdoclet.50438215_pgfId-1297483" xreflabel=""/> " has been created!\n", argv[1]);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297484" xreflabel=""/> return 0;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297485" xreflabel=""/>}
+ return -1;
+ }
+ printf("%s with stripe size 1048576, striped across 2 OSTs,"
+ " has been created!\n", argv[1]);
+ return 0;
+}
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438215_12433">
<title>34.4 llapi_quotactl</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297144" xreflabel=""/>Use llapi_quotactl to manipulate disk quotas on a Lustre file system.</para>
+ <para>Use llapi_quotactl to manipulate disk quotas on a Lustre file system.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1297145" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1297146" xreflabel=""/>#include <liblustre.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297147" xreflabel=""/>#include <lustre/lustre_idl.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297148" xreflabel=""/>#include <lustre/liblustreapi.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297149" xreflabel=""/>#include <lustre/lustre_user.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297150" xreflabel=""/>int llapi_quotactl(char" " *mnt," " struct if_quotactl" " *qctl)
-<anchor xml:id="dbdoclet.50438215_pgfId-1297151" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297152" xreflabel=""/>struct if_quotactl {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297153" xreflabel=""/> __u32 qc_cmd;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297154" xreflabel=""/> __u32 qc_type;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297155" xreflabel=""/> __u32 qc_id;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297156" xreflabel=""/> __u32 qc_stat;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297157" xreflabel=""/> struct obd_dqinfo qc_dqinfo;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297158" xreflabel=""/> struct obd_dqblk qc_dqblk;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297159" xreflabel=""/> char obd_type[16];
-<anchor xml:id="dbdoclet.50438215_pgfId-1297160" xreflabel=""/> struct obd_uuid obd_uuid;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297161" xreflabel=""/>};
-<anchor xml:id="dbdoclet.50438215_pgfId-1297162" xreflabel=""/>struct obd_dqblk {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297163" xreflabel=""/> __u64 dqb_bhardlimit;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297164" xreflabel=""/> __u64 dqb_bsoftlimit;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297165" xreflabel=""/> __u64 dqb_curspace;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297166" xreflabel=""/> __u64 dqb_ihardlimit;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297167" xreflabel=""/> __u64 dqb_isoftlimit;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297168" xreflabel=""/> __u64 dqb_curinodes;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297169" xreflabel=""/> __u64 dqb_btime;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297170" xreflabel=""/> __u64 dqb_itime;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297171" xreflabel=""/> __u32 dqb_valid;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297172" xreflabel=""/> __u32 padding;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297173" xreflabel=""/>};
-<anchor xml:id="dbdoclet.50438215_pgfId-1297174" xreflabel=""/>struct obd_dqinfo {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297175" xreflabel=""/> __u64 dqi_bgrace;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297176" xreflabel=""/> __u64 dqi_igrace;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297177" xreflabel=""/> __u32 dqi_flags;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297178" xreflabel=""/> __u32 dqi_valid;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297179" xreflabel=""/>};
-<anchor xml:id="dbdoclet.50438215_pgfId-1297180" xreflabel=""/>struct obd_uuid {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297181" xreflabel=""/> char uuid[40];
-<anchor xml:id="dbdoclet.50438215_pgfId-1297182" xreflabel=""/>};
+ <title>Synopsis</title>
+ <screen>#include <liblustre.h>
+#include <lustre/lustre_idl.h>
+#include <lustre/liblustreapi.h>
+#include <lustre/lustre_user.h>
+int llapi_quotactl(char" " *mnt," " struct if_quotactl" " *qctl)
+
+struct if_quotactl {
+ __u32 qc_cmd;
+ __u32 qc_type;
+ __u32 qc_id;
+ __u32 qc_stat;
+ struct obd_dqinfo qc_dqinfo;
+ struct obd_dqblk qc_dqblk;
+ char obd_type[16];
+ struct obd_uuid obd_uuid;
+};
+struct obd_dqblk {
+ __u64 dqb_bhardlimit;
+ __u64 dqb_bsoftlimit;
+ __u64 dqb_curspace;
+ __u64 dqb_ihardlimit;
+ __u64 dqb_isoftlimit;
+ __u64 dqb_curinodes;
+ __u64 dqb_btime;
+ __u64 dqb_itime;
+ __u32 dqb_valid;
+ __u32 padding;
+};
+struct obd_dqinfo {
+ __u64 dqi_bgrace;
+ __u64 dqi_igrace;
+ __u32 dqi_flags;
+ __u32 dqi_valid;
+};
+struct obd_uuid {
+ char uuid[40];
+};
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1297183" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297184" xreflabel=""/>The llapi_quotactl() command manipulates disk quotas on a Lustre file system mount. qc_cmd indicates a command to be applied to UID qc_id or GID qc_id.</para>
+ <title>Description</title>
+ <para>The llapi_quotactl() command manipulates disk quotas on a Lustre file system mount. qc_cmd indicates a command to be applied to UID qc_id or GID qc_id.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1297187" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1297189" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297191" xreflabel=""/><emphasis role="bold">LUSTRE_Q_QUOTAON</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297193" xreflabel=""/>Turns on quotas for a Lustre file system. <emphasis>qc_type</emphasis> is USRQUOTA, GRPQUOTA or UGQUOTA (both user and group quota). The quota files must exist. They are normally created with the llapi_quotacheck call. This call is restricted to the super user privilege.</para></entry>
+ <entry><para> <emphasis role="bold">LUSTRE_Q_QUOTAON</emphasis></para></entry>
+ <entry><para> Turns on quotas for a Lustre file system. <emphasis>qc_type</emphasis> is USRQUOTA, GRPQUOTA or UGQUOTA (both user and group quota). The quota files must exist. They are normally created with the llapi_quotacheck call. This call is restricted to the super user privilege.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297195" xreflabel=""/><emphasis role="bold">LUSTRE_Q_QUOTAOFF</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297197" xreflabel=""/>Turns off quotas for a Lustre file system. <emphasis>qc_type</emphasis> is USRQUOTA, GRPQUOTA or UGQUOTA (both user and group quota). This call is restricted to the super user privilege.</para></entry>
+ <entry><para> <emphasis role="bold">LUSTRE_Q_QUOTAOFF</emphasis></para></entry>
+ <entry><para> Turns off quotas for a Lustre file system. <emphasis>qc_type</emphasis> is USRQUOTA, GRPQUOTA or UGQUOTA (both user and group quota). This call is restricted to the super user privilege.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297199" xreflabel=""/><emphasis role="bold">LUSTRE_Q_GETQUOTA</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297201" xreflabel=""/>Gets disk quota limits and current usage for user or group <emphasis>qc_id</emphasis>. <emphasis>qc_type</emphasis> is USRQUOTA or GRPQUOTA. <emphasis>uuid</emphasis> may be filled with OBD UUID string to query quota information from a specific node. <emphasis>dqb_valid</emphasis> may be set nonzero to query information only from MDS. If <emphasis>uuid</emphasis> is an empty string and <emphasis>dqb_valid</emphasis> is zero then cluster-wide limits and usage are returned. On return, <emphasis>obd_dqblk</emphasis> contains the requested information (block limits unit is kilobyte). Quotas must be turned on before using this command.</para></entry>
+ <entry><para> <emphasis role="bold">LUSTRE_Q_GETQUOTA</emphasis></para></entry>
+ <entry><para> Gets disk quota limits and current usage for user or group <emphasis>qc_id</emphasis>. <emphasis>qc_type</emphasis> is USRQUOTA or GRPQUOTA. <emphasis>uuid</emphasis> may be filled with OBD UUID string to query quota information from a specific node. <emphasis>dqb_valid</emphasis> may be set nonzero to query information only from MDS. If <emphasis>uuid</emphasis> is an empty string and <emphasis>dqb_valid</emphasis> is zero then cluster-wide limits and usage are returned. On return, <emphasis>obd_dqblk</emphasis> contains the requested information (block limits unit is kilobyte). Quotas must be turned on before using this command.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297203" xreflabel=""/><emphasis role="bold">LUSTRE_Q_SETQUOTA</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297205" xreflabel=""/>Sets disk quota limits for user or group <emphasis>qc_id</emphasis>. <emphasis>qc_type</emphasis> is USRQUOTA or GRPQUOTA. <emphasis>dqb_valid</emphasis> must be set to QIF_ILIMITS, QIF_BLIMITS or QIF_LIMITS (both inode limits and block limits) dependent on updating limits. <emphasis>obd_dqblk</emphasis> must be filled with limits values (as set in <emphasis>dqb_valid</emphasis>, block limits unit is kilobyte). Quotas must be turned on before using this command.</para></entry>
+ <entry><para> <emphasis role="bold">LUSTRE_Q_SETQUOTA</emphasis></para></entry>
+ <entry><para> Sets disk quota limits for user or group <emphasis>qc_id</emphasis>. <emphasis>qc_type</emphasis> is USRQUOTA or GRPQUOTA. <emphasis>dqb_valid</emphasis> must be set to QIF_ILIMITS, QIF_BLIMITS or QIF_LIMITS (both inode limits and block limits) dependent on updating limits. <emphasis>obd_dqblk</emphasis> must be filled with limits values (as set in <emphasis>dqb_valid</emphasis>, block limits unit is kilobyte). Quotas must be turned on before using this command.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297207" xreflabel=""/><emphasis role="bold">LUSTRE_Q_GETINFO</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297209" xreflabel=""/>Gets information about quotas. <emphasis>qc_type</emphasis> is either USRQUOTA or GRPQUOTA. On return, <emphasis>dqi_igrace</emphasis> is inode grace time (in seconds), <emphasis>dqi_bgrace</emphasis> is block grace time (in seconds), <emphasis>dqi_flags</emphasis> is not used by the current Lustre version.</para></entry>
+ <entry><para> <emphasis role="bold">LUSTRE_Q_GETINFO</emphasis></para></entry>
+ <entry><para> Gets information about quotas. <emphasis>qc_type</emphasis> is either USRQUOTA or GRPQUOTA. On return, <emphasis>dqi_igrace</emphasis> is inode grace time (in seconds), <emphasis>dqi_bgrace</emphasis> is block grace time (in seconds), <emphasis>dqi_flags</emphasis> is not used by the current Lustre version.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297211" xreflabel=""/><emphasis role="bold">LUSTRE_Q_SETINFO</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297213" xreflabel=""/>Sets quota information (like grace times). <emphasis>qc_type</emphasis> is either USRQUOTA or GRPQUOTA. <emphasis>dqi_igrace</emphasis> is inode grace time (in seconds), <emphasis>dqi_bgrace</emphasis> is block grace time (in seconds), <emphasis>dqi_flags</emphasis> is not used by the current Lustre version and must be zeroed.</para></entry>
+ <entry><para> <emphasis role="bold">LUSTRE_Q_SETINFO</emphasis></para></entry>
+ <entry><para> Sets quota information (like grace times). <emphasis>qc_type</emphasis> is either USRQUOTA or GRPQUOTA. <emphasis>dqi_igrace</emphasis> is inode grace time (in seconds), <emphasis>dqi_bgrace</emphasis> is block grace time (in seconds), <emphasis>dqi_flags</emphasis> is not used by the current Lustre version and must be zeroed.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1297216" xreflabel=""/>Return Values</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1298911" xreflabel=""/>llapi_quotactl() returns:</para>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1298922" xreflabel=""/><emphasis role="bold">0</emphasis> On success</para>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1298923" xreflabel=""/><emphasis role="bold">-1</emphasis> On failure and sets error number (errno) to indicate the error</para>
+ <title>Return Values</title>
+ <para>llapi_quotactl() returns:</para>
+ <para><emphasis role="bold">0</emphasis> On success</para>
+ <para><emphasis role="bold">-1</emphasis> On failure and sets error number (errno) to indicate the error</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1297220" xreflabel=""/>Errors</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297258" xreflabel=""/>llapi_quotactl errors are described below.</para>
+ <title>Errors</title>
+ <para>llapi_quotactl errors are described below.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1297223" xreflabel=""/>Errors</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438215_pgfId-1297225" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Errors</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297227" xreflabel=""/><emphasis role="bold">EFAULT</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298931" xreflabel=""/><emphasis>qctl</emphasis> is invalid.</para></entry>
+ <entry><para> <emphasis role="bold">EFAULT</emphasis></para></entry>
+ <entry><para> <emphasis>qctl</emphasis> is invalid.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297231" xreflabel=""/><emphasis role="bold">ENOSYS</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298933" xreflabel=""/>Kernel or Lustre modules have not been compiled with the QUOTA option.</para></entry>
+ <entry><para> <emphasis role="bold">ENOSYS</emphasis></para></entry>
+ <entry><para> Kernel or Lustre modules have not been compiled with the QUOTA option.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297235" xreflabel=""/><emphasis role="bold">ENOMEM</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298935" xreflabel=""/>Insufficient memory to complete operation.</para></entry>
+ <entry><para> <emphasis role="bold">ENOMEM</emphasis></para></entry>
+ <entry><para> Insufficient memory to complete operation.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297239" xreflabel=""/><emphasis role="bold">ENOTTY</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298937" xreflabel=""/><emphasis>qc_cmd</emphasis> is invalid.</para></entry>
+ <entry><para> <emphasis role="bold">ENOTTY</emphasis></para></entry>
+ <entry><para> <emphasis>qc_cmd</emphasis> is invalid.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297243" xreflabel=""/><emphasis role="bold">EBUSY</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298939" xreflabel=""/> Cannot process during quotacheck.</para></entry>
+ <entry><para> <emphasis role="bold">EBUSY</emphasis></para></entry>
+ <entry><para> Cannot process during quotacheck.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297247" xreflabel=""/><emphasis role="bold">ENOENT</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298941" xreflabel=""/><emphasis>uuid</emphasis> does not correspond to OBD or <emphasis>mnt</emphasis> does not exist.</para></entry>
+ <entry><para> <emphasis role="bold">ENOENT</emphasis></para></entry>
+ <entry><para> <emphasis>uuid</emphasis> does not correspond to OBD or <emphasis>mnt</emphasis> does not exist.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297251" xreflabel=""/><emphasis role="bold">EPERM</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298943" xreflabel=""/> The call is privileged and the caller is not the super user.</para></entry>
+ <entry><para> <emphasis role="bold">EPERM</emphasis></para></entry>
+ <entry><para> The call is privileged and the caller is not the super user.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1297255" xreflabel=""/><emphasis role="bold">ESRCH</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438215_pgfId-1298945" xreflabel=""/> No disk quota is found for the indicated user. Quotas have not been turned on for this file system.</para></entry>
+ <entry><para> <emphasis role="bold">ESRCH</emphasis></para></entry>
+ <entry><para> No disk quota is found for the indicated user. Quotas have not been turned on for this file system.</para></entry>
</row>
</tbody>
</tgroup>
</section>
<section xml:id="dbdoclet.50438215_15718">
<title>34.5 llapi_path2fid</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297262" xreflabel=""/>Use llapi_path2fid to get the FID from the pathname.</para>
+ <para>Use llapi_path2fid to get the FID from the pathname.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1297263" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1297264" xreflabel=""/>#include <lustre/liblustreapi.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297265" xreflabel=""/>#include <lustre/lustre_user.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297266" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297267" xreflabel=""/>int llapi_path2fid(const char *path, unsigned long long *seq, unsigned long\
+ <title>Synopsis</title>
+ <screen>#include <lustre/liblustreapi.h>
+#include <lustre/lustre_user.h>
+
+int llapi_path2fid(const char *path, unsigned long long *seq, unsigned long\
*oid, unsigned long *ver)
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1297268" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297269" xreflabel=""/>The llapi_path2fid function returns the FID (sequence : object ID : version) for the pathname.</para>
+ <title>Description</title>
+ <para>The llapi_path2fid function returns the FID (sequence : object ID : version) for the pathname.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1297270" xreflabel=""/>Return Values</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297271" xreflabel=""/>llapi_path2fid returns:</para>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297272" xreflabel=""/><emphasis role="bold">0</emphasis> On success</para>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297273" xreflabel=""/>non-zero value On failure</para>
+ <title>Return Values</title>
+ <para>llapi_path2fid returns:</para>
+ <para><emphasis role="bold">0</emphasis> On success</para>
+ <para>non-zero value On failure</para>
</section>
</section>
<section xml:id="dbdoclet.50438215_marker-1297700">
<title>34.6 Example Using the llapi Library</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297702" xreflabel=""/>Use llapi_file_create to set Lustre properties for a new file. For a synopsis and description of llapi_file_create and examples of how to use it, see <xref linkend="configurationfilesmoduleparameters"/>.</para>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297709" xreflabel=""/>You can set striping from inside programs like ioctl. To compile the sample program, you need to download libtest.c and liblustreapi.c files from the Lustre source tree.</para>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297710" xreflabel=""/><emphasis role="bold">A simple C program to demonstrate striping API - libtest.c</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1297711" xreflabel=""/>/* -*- mode: c; c-basic-offset: 8; indent-tabs-mode: nil; -*-
-<anchor xml:id="dbdoclet.50438215_pgfId-1297712" xreflabel=""/> * vim:expandtab:shiftwidth=8:tabstop=8:
-<anchor xml:id="dbdoclet.50438215_pgfId-1297713" xreflabel=""/> *
-<anchor xml:id="dbdoclet.50438215_pgfId-1297714" xreflabel=""/> * lustredemo - simple code examples of liblustreapi functions
-<anchor xml:id="dbdoclet.50438215_pgfId-1297715" xreflabel=""/> */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297716" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297717" xreflabel=""/>#include <stdio.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297718" xreflabel=""/>#include <fcntl.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297719" xreflabel=""/>#include <sys/stat.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297720" xreflabel=""/>#include <sys/types.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297721" xreflabel=""/>#include <dirent.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297722" xreflabel=""/>#include <errno.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297723" xreflabel=""/>#include <string.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297724" xreflabel=""/>#include <unistd.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297725" xreflabel=""/>#include <stdlib.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297726" xreflabel=""/>#include <lustre/liblustreapi.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297727" xreflabel=""/>#include <lustre/lustre_user.h>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297728" xreflabel=""/>#define MAX_OSTS 1024
-<anchor xml:id="dbdoclet.50438215_pgfId-1297729" xreflabel=""/>#define LOV_EA_SIZE(lum, num) (sizeof(*lum) + num * sizeof(*lum->lmm_objects\
+ <para>Use llapi_file_create to set Lustre properties for a new file. For a synopsis and description of llapi_file_create and examples of how to use it, see <xref linkend="configurationfilesmoduleparameters"/>.</para>
+ <para>You can set striping from inside programs like ioctl. To compile the sample program, you need to download libtest.c and liblustreapi.c files from the Lustre source tree.</para>
+ <para><emphasis role="bold">A simple C program to demonstrate striping API - libtest.c</emphasis></para>
+ <screen>/* -*- mode: c; c-basic-offset: 8; indent-tabs-mode: nil; -*-
+ * vim:expandtab:shiftwidth=8:tabstop=8:
+ *
+ * lustredemo - simple code examples of liblustreapi functions
+ */
+
+#include <stdio.h>
+#include <fcntl.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <dirent.h>
+#include <errno.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdlib.h>
+#include <lustre/liblustreapi.h>
+#include <lustre/lustre_user.h>
+#define MAX_OSTS 1024
+#define LOV_EA_SIZE(lum, num) (sizeof(*lum) + num * sizeof(*lum->lmm_objects\
))
-<anchor xml:id="dbdoclet.50438215_pgfId-1297730" xreflabel=""/>#define LOV_EA_MAX(lum) LOV_EA_SIZE(lum, MAX_OSTS)
-<anchor xml:id="dbdoclet.50438215_pgfId-1297731" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297732" xreflabel=""/>/*
-<anchor xml:id="dbdoclet.50438215_pgfId-1297733" xreflabel=""/>This program provides crude examples of using the liblustre API functions
-<anchor xml:id="dbdoclet.50438215_pgfId-1297734" xreflabel=""/>*/
-<anchor xml:id="dbdoclet.50438215_pgfId-1297735" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297736" xreflabel=""/>/* Change these definitions to suit */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297737" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297738" xreflabel=""/>#define TESTDIR "/tmp" /* R\
+#define LOV_EA_MAX(lum) LOV_EA_SIZE(lum, MAX_OSTS)
+
+/*
+This program provides crude examples of using the liblustre API functions
+*/
+
+/* Change these definitions to suit */
+
+#define TESTDIR "/tmp" /* R\
esults directory */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297739" xreflabel=""/>#define TESTFILE "lustre_dummy" \
+#define TESTFILE "lustre_dummy" \
/* Name for the file we create/destroy */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297740" xreflabel=""/>#define FILESIZE 262144 \
+#define FILESIZE 262144 \
/* Size of the file in words */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297741" xreflabel=""/>#define DUMWORD "DEADBEEF" \
+#define DUMWORD "DEADBEEF" \
/* Dummy word used to fill files */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297742" xreflabel=""/>#define MY_STRIPE_WIDTH 2 \
+#define MY_STRIPE_WIDTH 2 \
/* Set this to the number of OST required */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297743" xreflabel=""/>#define MY_LUSTRE_DIR "/mnt/lustre/ftest"
-<anchor xml:id="dbdoclet.50438215_pgfId-1297744" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297745" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297746" xreflabel=""/>int close_file(int fd)
-<anchor xml:id="dbdoclet.50438215_pgfId-1297747" xreflabel=""/>{
-<anchor xml:id="dbdoclet.50438215_pgfId-1297748" xreflabel=""/> if (close(fd) < 0) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297749" xreflabel=""/> fprintf(stderr, "File close failed: %d (%s)\n", errno, strerror(er\
+#define MY_LUSTRE_DIR "/mnt/lustre/ftest"
+
+
+int close_file(int fd)
+{
+ if (close(fd) < 0) {
+ fprintf(stderr, "File close failed: %d (%s)\n", errno, strerror(er\
rno));
-<anchor xml:id="dbdoclet.50438215_pgfId-1297750" xreflabel=""/> return -1;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297751" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297752" xreflabel=""/> return 0;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297753" xreflabel=""/>}
-<anchor xml:id="dbdoclet.50438215_pgfId-1297754" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297755" xreflabel=""/>int write_file(int fd)
-<anchor xml:id="dbdoclet.50438215_pgfId-1297756" xreflabel=""/>{
-<anchor xml:id="dbdoclet.50438215_pgfId-1297757" xreflabel=""/> char *stng = DUMWORD;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297758" xreflabel=""/> int cnt = 0;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297759" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297760" xreflabel=""/> for( cnt = 0; cnt < FILESIZE; cnt++) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297761" xreflabel=""/> write(fd, stng, sizeof(stng));
-<anchor xml:id="dbdoclet.50438215_pgfId-1297762" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297763" xreflabel=""/> return 0;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297764" xreflabel=""/>}
-<anchor xml:id="dbdoclet.50438215_pgfId-1297765" xreflabel=""/>/* Open a file, set a specific stripe count, size and starting OST
-<anchor xml:id="dbdoclet.50438215_pgfId-1297766" xreflabel=""/> Adjust the parameters to suit */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297767" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297768" xreflabel=""/>int open_stripe_file()
-<anchor xml:id="dbdoclet.50438215_pgfId-1297769" xreflabel=""/>{
-<anchor xml:id="dbdoclet.50438215_pgfId-1297770" xreflabel=""/> char *tfile = TESTFILE;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297771" xreflabel=""/> int stripe_size = 65536; \
+ return -1;
+ }
+ return 0;
+}
+
+int write_file(int fd)
+{
+ char *stng = DUMWORD;
+ int cnt = 0;
+
+ for( cnt = 0; cnt < FILESIZE; cnt++) {
+ write(fd, stng, sizeof(stng));
+ }
+ return 0;
+}
+/* Open a file, set a specific stripe count, size and starting OST
+ Adjust the parameters to suit */
+
+int open_stripe_file()
+{
+ char *tfile = TESTFILE;
+ int stripe_size = 65536; \
/* System default is 4M */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297772" xreflabel=""/> int stripe_offset = -1; \
+ int stripe_offset = -1; \
/* Start at default */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297773" xreflabel=""/> int stripe_count = MY_STRIPE_WIDTH; \
+ int stripe_count = MY_STRIPE_WIDTH; \
/*Single stripe for this demo*/
-<anchor xml:id="dbdoclet.50438215_pgfId-1297774" xreflabel=""/> int stripe_pattern = 0; \
+ int stripe_pattern = 0; \
/* only RAID 0 at this time */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297775" xreflabel=""/> int rc, fd;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297776" xreflabel=""/> /*
-<anchor xml:id="dbdoclet.50438215_pgfId-1297777" xreflabel=""/> */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297778" xreflabel=""/> rc = llapi_file_create(tfile,
-<anchor xml:id="dbdoclet.50438215_pgfId-1297779" xreflabel=""/>stripe_size,stripe_offset,stripe_count,stripe_pattern);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297780" xreflabel=""/> /* result code is inverted, we may return -EINVAL or an ioctl error.
-<anchor xml:id="dbdoclet.50438215_pgfId-1297781" xreflabel=""/> We borrow an error message from sanity.c
-<anchor xml:id="dbdoclet.50438215_pgfId-1297782" xreflabel=""/> */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297783" xreflabel=""/> if (rc) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297784" xreflabel=""/> fprintf(stderr,"llapi_file_create failed: %d (%s) \n", rc, st\
+ int rc, fd;
+ /*
+ */
+ rc = llapi_file_create(tfile,
+stripe_size,stripe_offset,stripe_count,stripe_pattern);
+ /* result code is inverted, we may return -EINVAL or an ioctl error.
+ We borrow an error message from sanity.c
+ */
+ if (rc) {
+ fprintf(stderr,"llapi_file_create failed: %d (%s) \n", rc, st\
rerror(-rc));
-<anchor xml:id="dbdoclet.50438215_pgfId-1297785" xreflabel=""/> return -1;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297786" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297787" xreflabel=""/> /* llapi_file_create closes the file descriptor, we must re-open */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297788" xreflabel=""/> fd = open(tfile, O_CREAT | O_RDWR | O_LOV_DELAY_CREATE, 0644);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297789" xreflabel=""/> if (fd < 0) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297790" xreflabel=""/> fprintf(stderr, "Can't open %s file: %d (%s)\n", tfile, errno\
+ return -1;
+ }
+ /* llapi_file_create closes the file descriptor, we must re-open */
+ fd = open(tfile, O_CREAT | O_RDWR | O_LOV_DELAY_CREATE, 0644);
+ if (fd < 0) {
+ fprintf(stderr, "Can't open %s file: %d (%s)\n", tfile, errno\
, strerror(errno));
-<anchor xml:id="dbdoclet.50438215_pgfId-1297791" xreflabel=""/> return -1;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297792" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297793" xreflabel=""/> return fd;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297794" xreflabel=""/>}
-<anchor xml:id="dbdoclet.50438215_pgfId-1297795" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297796" xreflabel=""/>/* output a list of uuids for this file */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297797" xreflabel=""/>int get_my_uuids(int fd)
-<anchor xml:id="dbdoclet.50438215_pgfId-1297798" xreflabel=""/>{
-<anchor xml:id="dbdoclet.50438215_pgfId-1297799" xreflabel=""/> struct obd_uuid uuids[1024], *uuidp; \
+ return -1;
+ }
+ return fd;
+}
+
+/* output a list of uuids for this file */
+int get_my_uuids(int fd)
+{
+ struct obd_uuid uuids[1024], *uuidp; \
/* Output var */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297800" xreflabel=""/> int obdcount = 1024;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297801" xreflabel=""/> int rc,i;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297802" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297803" xreflabel=""/> rc = llapi_lov_get_uuids(fd, uuids, &obdcount);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297804" xreflabel=""/> if (rc != 0) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297805" xreflabel=""/> fprintf(stderr, "get uuids failed: %d (%s)\n",errno, strerror(errn\
+ int obdcount = 1024;
+ int rc,i;
+
+ rc = llapi_lov_get_uuids(fd, uuids, &obdcount);
+ if (rc != 0) {
+ fprintf(stderr, "get uuids failed: %d (%s)\n",errno, strerror(errn\
o));
-<anchor xml:id="dbdoclet.50438215_pgfId-1297806" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297807" xreflabel=""/> printf("This file system has %d obds\n", obdcount);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297808" xreflabel=""/> for (i = 0, uuidp = uuids; i < obdcount; i++, uuidp++) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297809" xreflabel=""/> printf("UUID %d is %s\n",i, uuidp->uuid);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297810" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297811" xreflabel=""/> return 0;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297812" xreflabel=""/>}
-<anchor xml:id="dbdoclet.50438215_pgfId-1297813" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297814" xreflabel=""/>/* Print out some LOV attributes. List our objects */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297815" xreflabel=""/>int get_file_info(char *path)
-<anchor xml:id="dbdoclet.50438215_pgfId-1297816" xreflabel=""/>{
-<anchor xml:id="dbdoclet.50438215_pgfId-1297817" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297818" xreflabel=""/> struct lov_user_md *lump;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297819" xreflabel=""/> int rc;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297820" xreflabel=""/> int i;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297821" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297822" xreflabel=""/> lump = malloc(LOV_EA_MAX(lump));
-<anchor xml:id="dbdoclet.50438215_pgfId-1297823" xreflabel=""/> if (lump == NULL) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297824" xreflabel=""/> return -1;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297825" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297826" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297827" xreflabel=""/> rc = llapi_file_get_stripe(path, lump);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297828" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297829" xreflabel=""/> if (rc != 0) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297830" xreflabel=""/> fprintf(stderr, "get_stripe failed: %d (%s)\n",errno, strerror(err\
+ }
+ printf("This file system has %d obds\n", obdcount);
+ for (i = 0, uuidp = uuids; i < obdcount; i++, uuidp++) {
+ printf("UUID %d is %s\n",i, uuidp->uuid);
+ }
+ return 0;
+}
+
+/* Print out some LOV attributes. List our objects */
+int get_file_info(char *path)
+{
+
+ struct lov_user_md *lump;
+ int rc;
+ int i;
+
+ lump = malloc(LOV_EA_MAX(lump));
+ if (lump == NULL) {
+ return -1;
+ }
+
+ rc = llapi_file_get_stripe(path, lump);
+
+ if (rc != 0) {
+ fprintf(stderr, "get_stripe failed: %d (%s)\n",errno, strerror(err\
no));
-<anchor xml:id="dbdoclet.50438215_pgfId-1297831" xreflabel=""/> return -1;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297832" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297833" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297834" xreflabel=""/> printf("Lov magic %u\n", lump->lmm_magic);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297835" xreflabel=""/> printf("Lov pattern %u\n", lump->lmm_pattern);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297836" xreflabel=""/> printf("Lov object id %llu\n", lump->lmm_object_id);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297837" xreflabel=""/> printf("Lov object group %llu\n", lump->lmm_object_gr);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297838" xreflabel=""/> printf("Lov stripe size %u\n", lump->lmm_stripe_size);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297839" xreflabel=""/> printf("Lov stripe count %hu\n", lump->lmm_stripe_count);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297840" xreflabel=""/> printf("Lov stripe offset %u\n", lump->lmm_stripe_offset);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297841" xreflabel=""/> for (i = 0; i < lump->lmm_stripe_count; i++) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297842" xreflabel=""/> printf("Object index %d Objid %llu\n", lump->lmm_objects[i].l_ost_i\
+ return -1;
+ }
+
+ printf("Lov magic %u\n", lump->lmm_magic);
+ printf("Lov pattern %u\n", lump->lmm_pattern);
+ printf("Lov object id %llu\n", lump->lmm_object_id);
+ printf("Lov object group %llu\n", lump->lmm_object_gr);
+ printf("Lov stripe size %u\n", lump->lmm_stripe_size);
+ printf("Lov stripe count %hu\n", lump->lmm_stripe_count);
+ printf("Lov stripe offset %u\n", lump->lmm_stripe_offset);
+ for (i = 0; i < lump->lmm_stripe_count; i++) {
+ printf("Object index %d Objid %llu\n", lump->lmm_objects[i].l_ost_i\
dx, lump->lmm_objects[i].l_object_id);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297843" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297844" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297845" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297846" xreflabel=""/> free(lump);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297847" xreflabel=""/> return rc;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297848" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297849" xreflabel=""/>}
-<anchor xml:id="dbdoclet.50438215_pgfId-1297850" xreflabel=""/>/* Ping all OSTs that belong to this filesysem */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297851" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297852" xreflabel=""/>int ping_osts()
-<anchor xml:id="dbdoclet.50438215_pgfId-1297853" xreflabel=""/>{
-<anchor xml:id="dbdoclet.50438215_pgfId-1297854" xreflabel=""/> DIR *dir;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297855" xreflabel=""/> struct dirent *d;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297856" xreflabel=""/> char osc_dir[100];
-<anchor xml:id="dbdoclet.50438215_pgfId-1297857" xreflabel=""/> int rc;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297858" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297859" xreflabel=""/> sprintf(osc_dir, "/proc/fs/lustre/osc");
-<anchor xml:id="dbdoclet.50438215_pgfId-1297860" xreflabel=""/> dir = opendir(osc_dir);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297861" xreflabel=""/> if (dir == NULL) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297862" xreflabel=""/> printf("Can't open dir\n");
-<anchor xml:id="dbdoclet.50438215_pgfId-1297863" xreflabel=""/> return -1;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297864" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297865" xreflabel=""/> while((d = readdir(dir)) != NULL) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297866" xreflabel=""/> if ( d->d_type == DT_DIR ) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297867" xreflabel=""/> if (! strncmp(d->d_name, "OSC", 3)) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297868" xreflabel=""/> printf("Pinging OSC %s ", d->d_name);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297869" xreflabel=""/> rc = llapi_ping("osc", d->d_name);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297870" xreflabel=""/> if (rc) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297871" xreflabel=""/> printf(" bad\n");
-<anchor xml:id="dbdoclet.50438215_pgfId-1297872" xreflabel=""/> } else {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297873" xreflabel=""/> printf(" good\n");
-<anchor xml:id="dbdoclet.50438215_pgfId-1297874" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297875" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297876" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297877" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297878" xreflabel=""/> return 0;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297879" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297880" xreflabel=""/>}
-<anchor xml:id="dbdoclet.50438215_pgfId-1297881" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297882" xreflabel=""/>int main()
-<anchor xml:id="dbdoclet.50438215_pgfId-1297883" xreflabel=""/>{
-<anchor xml:id="dbdoclet.50438215_pgfId-1297884" xreflabel=""/> int file;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297885" xreflabel=""/> int rc;
-<anchor xml:id="dbdoclet.50438215_pgfId-1297886" xreflabel=""/> char filename[100];
-<anchor xml:id="dbdoclet.50438215_pgfId-1297887" xreflabel=""/> char sys_cmd[100];
-<anchor xml:id="dbdoclet.50438215_pgfId-1297888" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297889" xreflabel=""/> sprintf(filename, "%s/%s",MY_LUSTRE_DIR, TESTFILE);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297890" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297891" xreflabel=""/> printf("Open a file with striping\n");
-<anchor xml:id="dbdoclet.50438215_pgfId-1297892" xreflabel=""/> file = open_stripe_file();
-<anchor xml:id="dbdoclet.50438215_pgfId-1297893" xreflabel=""/> if ( file < 0 ) {
-<anchor xml:id="dbdoclet.50438215_pgfId-1297894" xreflabel=""/> printf("Exiting\n");
-<anchor xml:id="dbdoclet.50438215_pgfId-1297895" xreflabel=""/> exit(1);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297896" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297897" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297898" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297899" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297900" xreflabel=""/> }
-<anchor xml:id="dbdoclet.50438215_pgfId-1297901" xreflabel=""/> printf("Getting uuid list\n");
-<anchor xml:id="dbdoclet.50438215_pgfId-1297902" xreflabel=""/> rc = get_my_uuids(file);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297903" xreflabel=""/> rintf("Write to the file\n");
-<anchor xml:id="dbdoclet.50438215_pgfId-1297904" xreflabel=""/> rc = write_file(file);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297905" xreflabel=""/> rc = close_file(file);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297906" xreflabel=""/> printf("Listing LOV data\n");
-<anchor xml:id="dbdoclet.50438215_pgfId-1297907" xreflabel=""/> rc = get_file_info(filename);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297908" xreflabel=""/> printf("Ping our OSTs\n");
-<anchor xml:id="dbdoclet.50438215_pgfId-1297909" xreflabel=""/> rc = ping_osts();
-<anchor xml:id="dbdoclet.50438215_pgfId-1297910" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297911" xreflabel=""/> /* the results should match lfs getstripe */
-<anchor xml:id="dbdoclet.50438215_pgfId-1297912" xreflabel=""/> printf("Confirming our results with lfs getsrtipe\n");
-<anchor xml:id="dbdoclet.50438215_pgfId-1297913" xreflabel=""/> sprintf(sys_cmd, "/usr/bin/lfs getstripe %s/%s", MY_LUSTRE_DIR, TESTFILE);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297914" xreflabel=""/> system(sys_cmd);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297915" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297916" xreflabel=""/> printf("All done\n");
-<anchor xml:id="dbdoclet.50438215_pgfId-1297917" xreflabel=""/> exit(rc);
-<anchor xml:id="dbdoclet.50438215_pgfId-1297918" xreflabel=""/>}
+ }
+
+
+ free(lump);
+ return rc;
+
+}
+/* Ping all OSTs that belong to this filesysem */
+
+int ping_osts()
+{
+ DIR *dir;
+ struct dirent *d;
+ char osc_dir[100];
+ int rc;
+
+ sprintf(osc_dir, "/proc/fs/lustre/osc");
+ dir = opendir(osc_dir);
+ if (dir == NULL) {
+ printf("Can't open dir\n");
+ return -1;
+ }
+ while((d = readdir(dir)) != NULL) {
+ if ( d->d_type == DT_DIR ) {
+ if (! strncmp(d->d_name, "OSC", 3)) {
+ printf("Pinging OSC %s ", d->d_name);
+ rc = llapi_ping("osc", d->d_name);
+ if (rc) {
+ printf(" bad\n");
+ } else {
+ printf(" good\n");
+ }
+ }
+ }
+ }
+ return 0;
+
+}
+
+int main()
+{
+ int file;
+ int rc;
+ char filename[100];
+ char sys_cmd[100];
+
+ sprintf(filename, "%s/%s",MY_LUSTRE_DIR, TESTFILE);
+
+ printf("Open a file with striping\n");
+ file = open_stripe_file();
+ if ( file < 0 ) {
+ printf("Exiting\n");
+ exit(1);
+
+
+
+
+ }
+ printf("Getting uuid list\n");
+ rc = get_my_uuids(file);
+ rintf("Write to the file\n");
+ rc = write_file(file);
+ rc = close_file(file);
+ printf("Listing LOV data\n");
+ rc = get_file_info(filename);
+ printf("Ping our OSTs\n");
+ rc = ping_osts();
+
+ /* the results should match lfs getstripe */
+ printf("Confirming our results with lfs getsrtipe\n");
+ sprintf(sys_cmd, "/usr/bin/lfs getstripe %s/%s", MY_LUSTRE_DIR, TESTFILE);
+ system(sys_cmd);
+
+ printf("All done\n");
+ exit(rc);
+}
</screen>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297919" xreflabel=""/><emphasis role="bold">Makefile for sample application:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438215_pgfId-1297920" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438215_pgfId-1297921" xreflabel=""/>gcc -g -O2 -Wall -o lustredemo libtest.c -llustreapi
-<anchor xml:id="dbdoclet.50438215_pgfId-1297922" xreflabel=""/>clean:
-<anchor xml:id="dbdoclet.50438215_pgfId-1297923" xreflabel=""/>rm -f core lustredemo *.o
-<anchor xml:id="dbdoclet.50438215_pgfId-1297924" xreflabel=""/>run:
-<anchor xml:id="dbdoclet.50438215_pgfId-1297925" xreflabel=""/>make
-<anchor xml:id="dbdoclet.50438215_pgfId-1297926" xreflabel=""/>rm -f /mnt/lustre/ftest/lustredemo
-<anchor xml:id="dbdoclet.50438215_pgfId-1297927" xreflabel=""/>rm -f /mnt/lustre/ftest/lustre_dummy
-<anchor xml:id="dbdoclet.50438215_pgfId-1297928" xreflabel=""/>cp lustredemo /mnt/lustre/ftest/
+ <para><emphasis role="bold">Makefile for sample application:</emphasis></para>
+ <screen>
+gcc -g -O2 -Wall -o lustredemo libtest.c -llustreapi
+clean:
+rm -f core lustredemo *.o
+run:
+make
+rm -f /mnt/lustre/ftest/lustredemo
+rm -f /mnt/lustre/ftest/lustre_dummy
+cp lustredemo /mnt/lustre/ftest/
</screen>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438215_pgfId-1297929" xreflabel=""/>See Also</title>
- <para><anchor xml:id="dbdoclet.50438215_pgfId-1297942" xreflabel=""/>
+ <title>See Also</title>
+ <para>
<xref linkend="dbdoclet.50438215_30970"/>llapi_file_create,
<xref linkend="dbdoclet.50438215_50149"/>llapi_file_get_stripe,
<xref linkend="dbdoclet.50438215_86607"/>llapi_file_open,
<title xml:id='settingupbonding.title'>Setting Up Network Interface Bonding</title>
</info>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301676" xreflabel=""/>This chapter describes how to use multiple network interfaces in parallel to increase bandwidth and/or redundancy. Topics include:</para>
+ <para>This chapter describes how to use multiple network interfaces in parallel to increase bandwidth and/or redundancy. Topics include:</para>
<itemizedlist>
<listitem>
<section xml:id="dbdoclet.50438258_95824">
<title>7.1 Network Interface Bonding Overview</title>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301680" xreflabel=""/>Bonding, also known as link aggregation, trunking and port trunking, is a method of aggregating multiple physical network links into a single logical link for increased bandwidth.</para>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301681" xreflabel=""/>Several different types of bonding are available in Linux. All these types are referred to as 'modes', and use the bonding kernel module.</para>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301682" xreflabel=""/>Modes 0 to 3 allow load balancing and fault tolerance by using multiple interfaces. Mode 4 aggregates a group of interfaces into a single virtual interface where all members of the group share the same speed and duplex settings. This mode is described under IEEE spec 802.3ad, and it is referred to as either 'mode 4' or '802.3ad.'</para>
+ <para>Bonding, also known as link aggregation, trunking and port trunking, is a method of aggregating multiple physical network links into a single logical link for increased bandwidth.</para>
+ <para>Several different types of bonding are available in Linux. All these types are referred to as 'modes', and use the bonding kernel module.</para>
+ <para>Modes 0 to 3 allow load balancing and fault tolerance by using multiple interfaces. Mode 4 aggregates a group of interfaces into a single virtual interface where all members of the group share the same speed and duplex settings. This mode is described under IEEE spec 802.3ad, and it is referred to as either 'mode 4' or '802.3ad.'</para>
</section>
<section xml:id="dbdoclet.50438258_29107">
<title>7.2 Requirements</title>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301686" xreflabel=""/>The most basic requirement for successful bonding is that both endpoints of the connection must be capable of bonding. In a normal case, the non-server endpoint is a switch. (Two systems connected via crossover cables can also use bonding.) Any switch used must explicitly handle 802.3ad Dynamic Link Aggregation.</para>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301687" xreflabel=""/>The kernel must also be configured with bonding. All supported Lustre kernels have bonding functionality. The network driver for the interfaces to be bonded must have the ethtool functionality to determine slave speed and duplex settings. All recent network drivers implement it.</para>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301688" xreflabel=""/>To verify that your interface works with ethtool, run:</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301689" xreflabel=""/># which ethtool
-<anchor xml:id="dbdoclet.50438258_pgfId-1301690" xreflabel=""/>/sbin/ethtool
-<anchor xml:id="dbdoclet.50438258_pgfId-1301691" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301692" xreflabel=""/># ethtool eth0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301693" xreflabel=""/>Settings for eth0:
-<anchor xml:id="dbdoclet.50438258_pgfId-1301694" xreflabel=""/> Supported ports: [ TP MII ]
-<anchor xml:id="dbdoclet.50438258_pgfId-1301695" xreflabel=""/> Supported link modes: 10baseT/Half 10baseT/Full/
-<anchor xml:id="dbdoclet.50438258_pgfId-1301696" xreflabel=""/> 100baseT/Half 100baseT/Full
-<anchor xml:id="dbdoclet.50438258_pgfId-1301697" xreflabel=""/> Supports auto-negotiation: Yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301698" xreflabel=""/> Advertised link modes: 10baseT/Half 10baseT/Full
-<anchor xml:id="dbdoclet.50438258_pgfId-1301699" xreflabel=""/> 100baseT/Half 100baseT/Full
-<anchor xml:id="dbdoclet.50438258_pgfId-1301700" xreflabel=""/> Advertised auto-negotiation: Yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301701" xreflabel=""/> Speed: 100Mb/s
-<anchor xml:id="dbdoclet.50438258_pgfId-1301702" xreflabel=""/> Duplex: Full
-<anchor xml:id="dbdoclet.50438258_pgfId-1301703" xreflabel=""/> Port: MII
-<anchor xml:id="dbdoclet.50438258_pgfId-1301704" xreflabel=""/> PHYAD: 1
-<anchor xml:id="dbdoclet.50438258_pgfId-1301705" xreflabel=""/> Transceiver: internal
-<anchor xml:id="dbdoclet.50438258_pgfId-1301706" xreflabel=""/> Auto-negotiation: on
-<anchor xml:id="dbdoclet.50438258_pgfId-1301707" xreflabel=""/> Supports Wake-on: pumbg
-<anchor xml:id="dbdoclet.50438258_pgfId-1301708" xreflabel=""/> Wake-on: d
-<anchor xml:id="dbdoclet.50438258_pgfId-1301709" xreflabel=""/> Current message level: 0x00000001 (1)
-<anchor xml:id="dbdoclet.50438258_pgfId-1301710" xreflabel=""/> Link detected: yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301711" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301712" xreflabel=""/># ethtool eth1
-<anchor xml:id="dbdoclet.50438258_pgfId-1301713" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301714" xreflabel=""/>Settings for eth1:
-<anchor xml:id="dbdoclet.50438258_pgfId-1301715" xreflabel=""/> Supported ports: [ TP MII ]
-<anchor xml:id="dbdoclet.50438258_pgfId-1301716" xreflabel=""/> Supported link modes: 10baseT/Half 10baseT/Full
-<anchor xml:id="dbdoclet.50438258_pgfId-1301717" xreflabel=""/> 100baseT/Half 100baseT/Full
-<anchor xml:id="dbdoclet.50438258_pgfId-1301718" xreflabel=""/> Supports auto-negotiation: Yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301719" xreflabel=""/> Advertised link modes: 10baseT/Half 10baseT/Full
-<anchor xml:id="dbdoclet.50438258_pgfId-1301720" xreflabel=""/> 100baseT/Half 100baseT/Full
-<anchor xml:id="dbdoclet.50438258_pgfId-1301721" xreflabel=""/> Advertised auto-negotiation: Yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301722" xreflabel=""/> Speed: 100Mb/s
-<anchor xml:id="dbdoclet.50438258_pgfId-1301723" xreflabel=""/> Duplex: Full
-<anchor xml:id="dbdoclet.50438258_pgfId-1301724" xreflabel=""/> Port: MII
-<anchor xml:id="dbdoclet.50438258_pgfId-1301725" xreflabel=""/> PHYAD: 32
-<anchor xml:id="dbdoclet.50438258_pgfId-1301726" xreflabel=""/> Transceiver: internal
-<anchor xml:id="dbdoclet.50438258_pgfId-1301727" xreflabel=""/> Auto-negotiation: on
-<anchor xml:id="dbdoclet.50438258_pgfId-1301728" xreflabel=""/> Supports Wake-on: pumbg
-<anchor xml:id="dbdoclet.50438258_pgfId-1301729" xreflabel=""/> Wake-on: d
-<anchor xml:id="dbdoclet.50438258_pgfId-1301730" xreflabel=""/> Current message level: 0x00000007 (7)
-<anchor xml:id="dbdoclet.50438258_pgfId-1301731" xreflabel=""/> Link detected: yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301732" xreflabel=""/> To quickly check whether your kernel supports bonding, run:
-<anchor xml:id="dbdoclet.50438258_pgfId-1301733" xreflabel=""/> # grep ifenslave /sbin/ifup
-<anchor xml:id="dbdoclet.50438258_pgfId-1301734" xreflabel=""/> # which ifenslave
-<anchor xml:id="dbdoclet.50438258_pgfId-1301735" xreflabel=""/> /sbin/ifenslave
+ <para>The most basic requirement for successful bonding is that both endpoints of the connection must be capable of bonding. In a normal case, the non-server endpoint is a switch. (Two systems connected via crossover cables can also use bonding.) Any switch used must explicitly handle 802.3ad Dynamic Link Aggregation.</para>
+ <para>The kernel must also be configured with bonding. All supported Lustre kernels have bonding functionality. The network driver for the interfaces to be bonded must have the ethtool functionality to determine slave speed and duplex settings. All recent network drivers implement it.</para>
+ <para>To verify that your interface works with ethtool, run:</para>
+ <screen># which ethtool
+/sbin/ethtool
+
+# ethtool eth0
+Settings for eth0:
+ Supported ports: [ TP MII ]
+ Supported link modes: 10baseT/Half 10baseT/Full/
+ 100baseT/Half 100baseT/Full
+ Supports auto-negotiation: Yes
+ Advertised link modes: 10baseT/Half 10baseT/Full
+ 100baseT/Half 100baseT/Full
+ Advertised auto-negotiation: Yes
+ Speed: 100Mb/s
+ Duplex: Full
+ Port: MII
+ PHYAD: 1
+ Transceiver: internal
+ Auto-negotiation: on
+ Supports Wake-on: pumbg
+ Wake-on: d
+ Current message level: 0x00000001 (1)
+ Link detected: yes
+
+# ethtool eth1
+
+Settings for eth1:
+ Supported ports: [ TP MII ]
+ Supported link modes: 10baseT/Half 10baseT/Full
+ 100baseT/Half 100baseT/Full
+ Supports auto-negotiation: Yes
+ Advertised link modes: 10baseT/Half 10baseT/Full
+ 100baseT/Half 100baseT/Full
+ Advertised auto-negotiation: Yes
+ Speed: 100Mb/s
+ Duplex: Full
+ Port: MII
+ PHYAD: 32
+ Transceiver: internal
+ Auto-negotiation: on
+ Supports Wake-on: pumbg
+ Wake-on: d
+ Current message level: 0x00000007 (7)
+ Link detected: yes
+ To quickly check whether your kernel supports bonding, run:
+ # grep ifenslave /sbin/ifup
+ # which ifenslave
+ /sbin/ifenslave
</screen>
</section>
<section xml:id="dbdoclet.50438258_25353">
<title>7.3 Bonding Module Parameters</title>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301749" xreflabel=""/>Bonding module parameters control various aspects of bonding.</para>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301750" xreflabel=""/>Outgoing traffic is mapped across the slave interfaces according to the transmit hash policy. For Lustre, we recommend that you set the xmit_hash_policy option to the layer3+4 option for bonding. This policy uses upper layer protocol information if available to generate the hash. This allows traffic to a particular network peer to span multiple slaves, although a single connection does not span multiple slaves.</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301751" xreflabel=""/>$ xmit_hash_policy=layer3+4
+ <para>Bonding module parameters control various aspects of bonding.</para>
+ <para>Outgoing traffic is mapped across the slave interfaces according to the transmit hash policy. For Lustre, we recommend that you set the xmit_hash_policy option to the layer3+4 option for bonding. This policy uses upper layer protocol information if available to generate the hash. This allows traffic to a particular network peer to span multiple slaves, although a single connection does not span multiple slaves.</para>
+ <screen>$ xmit_hash_policy=layer3+4
</screen>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301752" xreflabel=""/>The miimon option enables users to monitor the link status. (The parameter is a time interval in milliseconds.) It makes an interface failure transparent to avoid serious network degradation during link failures. A reasonable default setting is 100 milliseconds; run:</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301753" xreflabel=""/>$ miimon=100
+ <para>The miimon option enables users to monitor the link status. (The parameter is a time interval in milliseconds.) It makes an interface failure transparent to avoid serious network degradation during link failures. A reasonable default setting is 100 milliseconds; run:</para>
+ <screen>$ miimon=100
</screen>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301754" xreflabel=""/>For a busy network, increase the timeout.</para>
+ <para>For a busy network, increase the timeout.</para>
</section>
<section xml:id="dbdoclet.50438258_26899">
<title>7.4 Setting Up Bonding</title>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301759" xreflabel=""/>To set up bonding:</para>
+ <para>To set up bonding:</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301760" xreflabel=""/>Create a virtual 'bond' interface by creating a configuration file in:</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301761" xreflabel=""/>/etc/sysconfig/network-scripts/ # vi /etc/sysconfig/ network-scripts/ifcfg-\
+ <para>Create a virtual 'bond' interface by creating a configuration file in:</para>
+ <screen>/etc/sysconfig/network-scripts/ # vi /etc/sysconfig/ network-scripts/ifcfg-\
bond0
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301762" xreflabel=""/>Append the following lines to the file.</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301763" xreflabel=""/>DEVICE=bond0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301764" xreflabel=""/>IPADDR=192.168.10.79 # Use the free IP Address of your network
-<anchor xml:id="dbdoclet.50438258_pgfId-1301765" xreflabel=""/>NETWORK=192.168.10.0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301766" xreflabel=""/>NETMASK=255.255.255.0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301767" xreflabel=""/>USERCTL=no
-<anchor xml:id="dbdoclet.50438258_pgfId-1301768" xreflabel=""/>BOOTPROTO=none
-<anchor xml:id="dbdoclet.50438258_pgfId-1301769" xreflabel=""/>ONBOOT=yes
+ <para>Append the following lines to the file.</para>
+ <screen>DEVICE=bond0
+IPADDR=192.168.10.79 # Use the free IP Address of your network
+NETWORK=192.168.10.0
+NETMASK=255.255.255.0
+USERCTL=no
+BOOTPROTO=none
+ONBOOT=yes
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301770" xreflabel=""/> Attach one or more slave interfaces to the bond interface. Modify the eth0 and eth1 configuration files (using a VI text editor).</para>
+ <para> Attach one or more slave interfaces to the bond interface. Modify the eth0 and eth1 configuration files (using a VI text editor).</para>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301771" xreflabel=""/>Use the VI text editor to open the eth0 configuration file.</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301772" xreflabel=""/># vi /etc/sysconfig/network-scripts/ifcfg-eth0
+ <para>Use the VI text editor to open the eth0 configuration file.</para>
+ <screen># vi /etc/sysconfig/network-scripts/ifcfg-eth0
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301773" xreflabel=""/>Modify/append the eth0 file as follows:</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301774" xreflabel=""/>DEVICE=eth0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301775" xreflabel=""/>USERCTL=no
-<anchor xml:id="dbdoclet.50438258_pgfId-1301776" xreflabel=""/>ONBOOT=yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301777" xreflabel=""/>MASTER=bond0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301778" xreflabel=""/>SLAVE=yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301779" xreflabel=""/>BOOTPROTO=none
+ <para>Modify/append the eth0 file as follows:</para>
+ <screen>DEVICE=eth0
+USERCTL=no
+ONBOOT=yes
+MASTER=bond0
+SLAVE=yes
+BOOTPROTO=none
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301780" xreflabel=""/>Use the VI text editor to open the eth1 configuration file.</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301781" xreflabel=""/># vi /etc/sysconfig/network-scripts/ifcfg-eth1
+ <para>Use the VI text editor to open the eth1 configuration file.</para>
+ <screen># vi /etc/sysconfig/network-scripts/ifcfg-eth1
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301782" xreflabel=""/>Modify/append the eth1 file as follows:</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301783" xreflabel=""/>DEVICE=eth1
-<anchor xml:id="dbdoclet.50438258_pgfId-1301784" xreflabel=""/>USERCTL=no
-<anchor xml:id="dbdoclet.50438258_pgfId-1301785" xreflabel=""/>ONBOOT=yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301786" xreflabel=""/>MASTER=bond0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301787" xreflabel=""/>SLAVE=yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301788" xreflabel=""/>BOOTPROTO=none
+ <para>Modify/append the eth1 file as follows:</para>
+ <screen>DEVICE=eth1
+USERCTL=no
+ONBOOT=yes
+MASTER=bond0
+SLAVE=yes
+BOOTPROTO=none
</screen>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301789" xreflabel=""/>Set up the bond interface and its options in /etc/modprobe.conf. Start the slave interfaces by your normal network method.</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301790" xreflabel=""/># vi /etc/modprobe.conf
+ <para>Set up the bond interface and its options in /etc/modprobe.conf. Start the slave interfaces by your normal network method.</para>
+ <screen># vi /etc/modprobe.conf
</screen>
<orderedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301791" xreflabel=""/>Append the following lines to the file.</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301792" xreflabel=""/>alias bond0 bonding
-<anchor xml:id="dbdoclet.50438258_pgfId-1301793" xreflabel=""/>options bond0 mode=balance-alb miimon=100
+ <para>Append the following lines to the file.</para>
+ <screen>alias bond0 bonding
+options bond0 mode=balance-alb miimon=100
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301794" xreflabel=""/>Load the bonding module.</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301795" xreflabel=""/># modprobe bonding
-<anchor xml:id="dbdoclet.50438258_pgfId-1301796" xreflabel=""/># ifconfig bond0 up
-<anchor xml:id="dbdoclet.50438258_pgfId-1301797" xreflabel=""/># ifenslave bond0 eth0 eth1
+ <para>Load the bonding module.</para>
+ <screen># modprobe bonding
+# ifconfig bond0 up
+# ifenslave bond0 eth0 eth1
</screen>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301798" xreflabel=""/>Start/restart the slave interfaces (using your normal network method).</para>
+ <para>Start/restart the slave interfaces (using your normal network method).</para>
<note>
<para>You must modprobe the bonding module for each bonded interface. If you wish to create bond0 and bond1, two entries in modprobe.conf are required.</para></note>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301800" xreflabel=""/>The examples below are from RedHat systems. For setup use: /etc/sysconfig/networking-scripts/ifcfg-* The website referenced below includes detailed instructions for other configuration methods, instructions to use DHCP with bonding, and other setup details. We strongly recommend you use this website.</para>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1302123" xreflabel=""/><link xl:href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding">http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding</link></para>
+ <para>The examples below are from RedHat systems. For setup use: /etc/sysconfig/networking-scripts/ifcfg-* The website referenced below includes detailed instructions for other configuration methods, instructions to use DHCP with bonding, and other setup details. We strongly recommend you use this website.</para>
+ <para><link xl:href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding">http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding</link></para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301803" xreflabel=""/>Check /proc/net/bonding to determine status on bonding. There should be a file there for each bond interface.</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301804" xreflabel=""/># cat /proc/net/bonding/bond0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301805" xreflabel=""/>Ethernet Channel Bonding Driver: v3.0.3 (March 23, 2006)
-<anchor xml:id="dbdoclet.50438258_pgfId-1301806" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301807" xreflabel=""/>Bonding Mode: load balancing (round-robin)
-<anchor xml:id="dbdoclet.50438258_pgfId-1301808" xreflabel=""/>MII Status: up
-<anchor xml:id="dbdoclet.50438258_pgfId-1301809" xreflabel=""/>MII Polling Interval (ms): 0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301810" xreflabel=""/>Up Delay (ms): 0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301811" xreflabel=""/>Down Delay (ms): 0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301812" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301813" xreflabel=""/>Slave Interface: eth0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301814" xreflabel=""/>MII Status: up
-<anchor xml:id="dbdoclet.50438258_pgfId-1301815" xreflabel=""/>Link Failure Count: 0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301816" xreflabel=""/>Permanent HW addr: 4c:00:10:ac:61:e0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301817" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301818" xreflabel=""/>Slave Interface: eth1
-<anchor xml:id="dbdoclet.50438258_pgfId-1301819" xreflabel=""/>MII Status: up
-<anchor xml:id="dbdoclet.50438258_pgfId-1301820" xreflabel=""/>Link Failure Count: 0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301821" xreflabel=""/>Permanent HW addr: 00:14:2a:7c:40:1d
+ <para>Check /proc/net/bonding to determine status on bonding. There should be a file there for each bond interface.</para>
+ <screen># cat /proc/net/bonding/bond0
+Ethernet Channel Bonding Driver: v3.0.3 (March 23, 2006)
+
+Bonding Mode: load balancing (round-robin)
+MII Status: up
+MII Polling Interval (ms): 0
+Up Delay (ms): 0
+Down Delay (ms): 0
+
+Slave Interface: eth0
+MII Status: up
+Link Failure Count: 0
+Permanent HW addr: 4c:00:10:ac:61:e0
+
+Slave Interface: eth1
+MII Status: up
+Link Failure Count: 0
+Permanent HW addr: 00:14:2a:7c:40:1d
</screen>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301822" xreflabel=""/>Use ethtool or ifconfig to check the interface state. ifconfig lists the first bonded interface as 'bond0.'</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301823" xreflabel=""/>ifconfig
-<anchor xml:id="dbdoclet.50438258_pgfId-1301824" xreflabel=""/>bond0 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301825" xreflabel=""/> inet addr:192.168.10.79 Bcast:192.168.10.255 \ Mask:255.255.255.0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301826" xreflabel=""/> inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link
-<anchor xml:id="dbdoclet.50438258_pgfId-1301827" xreflabel=""/> UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
-<anchor xml:id="dbdoclet.50438258_pgfId-1301828" xreflabel=""/> RX packets:3091 errors:0 dropped:0 overruns:0 frame:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301829" xreflabel=""/> TX packets:880 errors:0 dropped:0 overruns:0 carrier:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301830" xreflabel=""/> collisions:0 txqueuelen:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301831" xreflabel=""/> RX bytes:314203 (306.8 KiB) TX bytes:129834 (126.7 KiB)
-<anchor xml:id="dbdoclet.50438258_pgfId-1301832" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301833" xreflabel=""/>eth0 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301834" xreflabel=""/> inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link
-<anchor xml:id="dbdoclet.50438258_pgfId-1301835" xreflabel=""/> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
-<anchor xml:id="dbdoclet.50438258_pgfId-1301836" xreflabel=""/> RX packets:1581 errors:0 dropped:0 overruns:0 frame:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301837" xreflabel=""/> TX packets:448 errors:0 dropped:0 overruns:0 carrier:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301838" xreflabel=""/> collisions:0 txqueuelen:1000
-<anchor xml:id="dbdoclet.50438258_pgfId-1301839" xreflabel=""/> RX bytes:162084 (158.2 KiB) TX bytes:67245 (65.6 KiB)
-<anchor xml:id="dbdoclet.50438258_pgfId-1301840" xreflabel=""/> Interrupt:193 Base address:0x8c00
-<anchor xml:id="dbdoclet.50438258_pgfId-1301841" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301842" xreflabel=""/>eth1 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301843" xreflabel=""/> inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link
-<anchor xml:id="dbdoclet.50438258_pgfId-1301844" xreflabel=""/> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
-<anchor xml:id="dbdoclet.50438258_pgfId-1301845" xreflabel=""/> RX packets:1513 errors:0 dropped:0 overruns:0 frame:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301846" xreflabel=""/> TX packets:444 errors:0 dropped:0 overruns:0 carrier:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301847" xreflabel=""/> collisions:0 txqueuelen:1000
-<anchor xml:id="dbdoclet.50438258_pgfId-1301848" xreflabel=""/> RX bytes:152299 (148.7 KiB) TX bytes:64517 (63.0 KiB)
-<anchor xml:id="dbdoclet.50438258_pgfId-1301849" xreflabel=""/> Interrupt:185 Base address:0x6000
+ <para>Use ethtool or ifconfig to check the interface state. ifconfig lists the first bonded interface as 'bond0.'</para>
+ <screen>ifconfig
+bond0 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0
+ inet addr:192.168.10.79 Bcast:192.168.10.255 \ Mask:255.255.255.0
+ inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link
+ UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
+ RX packets:3091 errors:0 dropped:0 overruns:0 frame:0
+ TX packets:880 errors:0 dropped:0 overruns:0 carrier:0
+ collisions:0 txqueuelen:0
+ RX bytes:314203 (306.8 KiB) TX bytes:129834 (126.7 KiB)
+
+eth0 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0
+ inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link
+ UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
+ RX packets:1581 errors:0 dropped:0 overruns:0 frame:0
+ TX packets:448 errors:0 dropped:0 overruns:0 carrier:0
+ collisions:0 txqueuelen:1000
+ RX bytes:162084 (158.2 KiB) TX bytes:67245 (65.6 KiB)
+ Interrupt:193 Base address:0x8c00
+
+eth1 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0
+ inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link
+ UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
+ RX packets:1513 errors:0 dropped:0 overruns:0 frame:0
+ TX packets:444 errors:0 dropped:0 overruns:0 carrier:0
+ collisions:0 txqueuelen:1000
+ RX bytes:152299 (148.7 KiB) TX bytes:64517 (63.0 KiB)
+ Interrupt:185 Base address:0x6000
</screen>
</listitem>
</orderedlist>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438258_pgfId-1301850" xreflabel=""/>7.4.1 Examples</title>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301851" xreflabel=""/>This is an example showing modprobe.conf entries for bonding Ethernet interfaces eth1 and eth2 to bond0:</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301852" xreflabel=""/># cat /etc/modprobe.conf
-<anchor xml:id="dbdoclet.50438258_pgfId-1301853" xreflabel=""/>alias eth0 8139too
-<anchor xml:id="dbdoclet.50438258_pgfId-1301854" xreflabel=""/>alias scsi_hostadapter sata_via
-<anchor xml:id="dbdoclet.50438258_pgfId-1301855" xreflabel=""/>alias scsi_hostadapter1 usb-storage
-<anchor xml:id="dbdoclet.50438258_pgfId-1301856" xreflabel=""/>alias snd-card-0 snd-via82xx
-<anchor xml:id="dbdoclet.50438258_pgfId-1301857" xreflabel=""/>options snd-card-0 index=0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301858" xreflabel=""/>options snd-via82xx index=0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301859" xreflabel=""/>alias bond0 bonding
-<anchor xml:id="dbdoclet.50438258_pgfId-1301860" xreflabel=""/>options bond0 mode=balance-alb miimon=100
-<anchor xml:id="dbdoclet.50438258_pgfId-1301861" xreflabel=""/>options lnet networks=tcp
-<anchor xml:id="dbdoclet.50438258_pgfId-1301862" xreflabel=""/>alias eth1 via-rhine
-<anchor xml:id="dbdoclet.50438258_pgfId-1301863" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301864" xreflabel=""/># cat /etc/sysconfig/network-scripts/ifcfg-bond0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301865" xreflabel=""/>DEVICE=bond0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301866" xreflabel=""/>BOOTPROTO=none
-<anchor xml:id="dbdoclet.50438258_pgfId-1301867" xreflabel=""/>NETMASK=255.255.255.0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301868" xreflabel=""/>IPADDR=192.168.10.79 # (Assign here the IP of the bonded interface.)
-<anchor xml:id="dbdoclet.50438258_pgfId-1301869" xreflabel=""/>ONBOOT=yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301870" xreflabel=""/>USERCTL=no
-<anchor xml:id="dbdoclet.50438258_pgfId-1301871" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301872" xreflabel=""/>ifcfg-ethx
-<anchor xml:id="dbdoclet.50438258_pgfId-1301873" xreflabel=""/># cat /etc/sysconfig/network-scripts/ifcfg-eth0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301874" xreflabel=""/>TYPE=Ethernet
-<anchor xml:id="dbdoclet.50438258_pgfId-1301875" xreflabel=""/>DEVICE=eth0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301876" xreflabel=""/>HWADDR=4c:00:10:ac:61:e0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301877" xreflabel=""/>BOOTPROTO=none
-<anchor xml:id="dbdoclet.50438258_pgfId-1301878" xreflabel=""/>ONBOOT=yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301879" xreflabel=""/>USERCTL=no
-<anchor xml:id="dbdoclet.50438258_pgfId-1301880" xreflabel=""/>IPV6INIT=no
-<anchor xml:id="dbdoclet.50438258_pgfId-1301881" xreflabel=""/>PEERDNS=yes
-<anchor xml:id="dbdoclet.50438258_pgfId-1301882" xreflabel=""/>MASTER=bond0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301883" xreflabel=""/>SLAVE=yes
+ <title>7.4.1 Examples</title>
+ <para>This is an example showing modprobe.conf entries for bonding Ethernet interfaces eth1 and eth2 to bond0:</para>
+ <screen># cat /etc/modprobe.conf
+alias eth0 8139too
+alias scsi_hostadapter sata_via
+alias scsi_hostadapter1 usb-storage
+alias snd-card-0 snd-via82xx
+options snd-card-0 index=0
+options snd-via82xx index=0
+alias bond0 bonding
+options bond0 mode=balance-alb miimon=100
+options lnet networks=tcp
+alias eth1 via-rhine
+
+# cat /etc/sysconfig/network-scripts/ifcfg-bond0
+DEVICE=bond0
+BOOTPROTO=none
+NETMASK=255.255.255.0
+IPADDR=192.168.10.79 # (Assign here the IP of the bonded interface.)
+ONBOOT=yes
+USERCTL=no
+
+ifcfg-ethx
+# cat /etc/sysconfig/network-scripts/ifcfg-eth0
+TYPE=Ethernet
+DEVICE=eth0
+HWADDR=4c:00:10:ac:61:e0
+BOOTPROTO=none
+ONBOOT=yes
+USERCTL=no
+IPV6INIT=no
+PEERDNS=yes
+MASTER=bond0
+SLAVE=yes
</screen>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301884" xreflabel=""/>In the following example, the bond0 interface is the master (MASTER) while eth0 and eth1 are slaves (SLAVE).</para>
+ <para>In the following example, the bond0 interface is the master (MASTER) while eth0 and eth1 are slaves (SLAVE).</para>
<informaltable frame="none">
<tgroup cols="1">
<colspec colname="c1" colwidth="100*"/>
<tbody>
<row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438258_pgfId-1301885" xreflabel=""/>All slaves of bond0 have the same MAC address (Hwaddr) - bond0. All modes, except TLB and ALB, have this MAC address. TLB and ALB require a unique MAC address for each slave.</para></entry>
+ <entry><para><emphasis role="bold">Note -</emphasis>All slaves of bond0 have the same MAC address (Hwaddr) - bond0. All modes, except TLB and ALB, have this MAC address. TLB and ALB require a unique MAC address for each slave.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301886" xreflabel=""/>$ /sbin/ifconfig
-<anchor xml:id="dbdoclet.50438258_pgfId-1301887" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301888" xreflabel=""/>bond0Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4
-<anchor xml:id="dbdoclet.50438258_pgfId-1301889" xreflabel=""/>inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301890" xreflabel=""/>UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
-<anchor xml:id="dbdoclet.50438258_pgfId-1301891" xreflabel=""/>RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301892" xreflabel=""/>TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301893" xreflabel=""/>collisions:0 txqueuelen:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301894" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301895" xreflabel=""/>eth0Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4
-<anchor xml:id="dbdoclet.50438258_pgfId-1301896" xreflabel=""/>inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301897" xreflabel=""/>UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
-<anchor xml:id="dbdoclet.50438258_pgfId-1301898" xreflabel=""/>RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301899" xreflabel=""/>TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301900" xreflabel=""/>collisions:0 txqueuelen:100
-<anchor xml:id="dbdoclet.50438258_pgfId-1301901" xreflabel=""/>Interrupt:10 Base address:0x1080
-<anchor xml:id="dbdoclet.50438258_pgfId-1301902" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438258_pgfId-1301903" xreflabel=""/>eth1Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4
-<anchor xml:id="dbdoclet.50438258_pgfId-1301904" xreflabel=""/>inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301905" xreflabel=""/>UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
-<anchor xml:id="dbdoclet.50438258_pgfId-1301906" xreflabel=""/>RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301907" xreflabel=""/>TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
-<anchor xml:id="dbdoclet.50438258_pgfId-1301908" xreflabel=""/>collisions:0 txqueuelen:100
-<anchor xml:id="dbdoclet.50438258_pgfId-1301909" xreflabel=""/>Interrupt:9 Base address:0x1400
+ <screen>$ /sbin/ifconfig
+
+bond0Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4
+inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
+UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
+RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0
+TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0
+collisions:0 txqueuelen:0
+
+eth0Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4
+inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
+UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
+RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0
+TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0
+collisions:0 txqueuelen:100
+Interrupt:10 Base address:0x1080
+
+eth1Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4
+inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
+UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
+RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
+TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
+collisions:0 txqueuelen:100
+Interrupt:9 Base address:0x1400
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438258_54769">
<title>7.5 Configuring Lustre with Bonding</title>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301913" xreflabel=""/>Lustre uses the IP address of the bonded interfaces and requires no special configuration. It treats the bonded interface as a regular TCP/IP interface. If needed, specify bond0 using the Lustre networks parameter in /etc/modprobe.</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301914" xreflabel=""/>options lnet networks=tcp(bond0)
+ <para>Lustre uses the IP address of the bonded interfaces and requires no special configuration. It treats the bonded interface as a regular TCP/IP interface. If needed, specify bond0 using the Lustre networks parameter in /etc/modprobe.</para>
+ <screen>options lnet networks=tcp(bond0)
</screen>
</section>
<section xml:id="dbdoclet.50438258_92244">
<title>7.6 Bonding References</title>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301917" xreflabel=""/>We recommend the following bonding references:</para>
+ <para>We recommend the following bonding references:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301918" xreflabel=""/> In the Linux kernel source tree, see documentation/networking/bonding.txt</para>
+ <para> In the Linux kernel source tree, see documentation/networking/bonding.txt</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301920" xreflabel=""/><link xl:href="http://linux-ip.net/html/ether-bonding.html">http://linux-ip.net/html/ether-bonding.html</link></para>
+ <para><link xl:href="http://linux-ip.net/html/ether-bonding.html">http://linux-ip.net/html/ether-bonding.html</link></para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301922" xreflabel=""/><link xl:href="http://www.sourceforge.net/projects/bonding">http://www.sourceforge.net/projects/bonding</link></para>
+ <para><link xl:href="http://www.sourceforge.net/projects/bonding">http://www.sourceforge.net/projects/bonding</link></para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301923" xreflabel=""/> Linux Foundation bonding website: <link xl:href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding">http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding</link></para>
+ <para> Linux Foundation bonding website: <link xl:href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding">http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding</link></para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438258_pgfId-1301924" xreflabel=""/>This is the most extensive reference and we highly recommend it. This website includes explanations of more complicated setups, including the use of DHCP with bonding.</para>
- <screen><anchor xml:id="dbdoclet.50438258_pgfId-1301673" xreflabel=""/>
+ <para>This is the most extensive reference and we highly recommend it. This website includes explanations of more complicated setups, including the use of DHCP with bonding.</para>
+ <screen>
</screen>
</section>
</chapter>
<info>
<title xml:id="settinguplustresystem.title">Setting Up a Lustre File System</title>
</info>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290293" xreflabel=""/>This chapter describes hardware configuration requirements for a Lustre file system including:</para>
+ <para>This chapter describes hardware configuration requirements for a Lustre file system including:</para>
<itemizedlist>
<listitem>
<para>
</itemizedlist>
<section xml:id="dbdoclet.50438256_49017">
<title>5.1 Hardware Considerations</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292812" xreflabel=""/>Lustre can work with any kind of block storage device such as single disks, software RAID, hardware RAID, or a logical volume manager. In contrast to some networked file systems, the block devices are only attached to the MDS and OSS nodes in Lustre and are not accessed by the clients directly.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292817" xreflabel=""/>Since the block devices are accessed by only one or two server nodes, a storage area network (SAN) that is accessible from all the servers is not required. Expensive switches are not needed because point-to-point connections between the servers and the storage arrays normally provide the simplest and best attachments. (If failover capability is desired, the storage must be attached to multiple servers.)</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290297" xreflabel=""/>For a production environment, it is preferable that the MGS have separate storage to allow future expansion to multiple file systems. However, it is possible to run the MDS and MGS on the same machine and have them share the same storage device.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292437" xreflabel=""/>For best performance in a production environment, dedicated clients are required. For a non-production Lustre environment or for testing, a Lustre client and server can run on the same machine. However, dedicated clients are the only supported configuration.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292435" xreflabel=""/>Performance and other issues can occur when an MDS or OSS and a client are running on the same machine:</para>
+ <para>Lustre can work with any kind of block storage device such as single disks, software RAID, hardware RAID, or a logical volume manager. In contrast to some networked file systems, the block devices are only attached to the MDS and OSS nodes in Lustre and are not accessed by the clients directly.</para>
+ <para>Since the block devices are accessed by only one or two server nodes, a storage area network (SAN) that is accessible from all the servers is not required. Expensive switches are not needed because point-to-point connections between the servers and the storage arrays normally provide the simplest and best attachments. (If failover capability is desired, the storage must be attached to multiple servers.)</para>
+ <para>For a production environment, it is preferable that the MGS have separate storage to allow future expansion to multiple file systems. However, it is possible to run the MDS and MGS on the same machine and have them share the same storage device.</para>
+ <para>For best performance in a production environment, dedicated clients are required. For a non-production Lustre environment or for testing, a Lustre client and server can run on the same machine. However, dedicated clients are the only supported configuration.</para>
+ <para>Performance and other issues can occur when an MDS or OSS and a client are running on the same machine:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292457" xreflabel=""/> Running the MDS and a client on the same machine can cause recovery and deadlock issues and impact the performance of other Lustre clients.</para>
+ <para> Running the MDS and a client on the same machine can cause recovery and deadlock issues and impact the performance of other Lustre clients.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292461" xreflabel=""/> Running the OSS and a client on the same machine can cause issues with low memory and memory pressure. If the client consumes all the memory and then tries to write data to the file system, the OSS will need to allocate pages to receive data from the client but will not be able to perform this operation due to low memory. This can cause the client to hang.</para>
+ <para> Running the OSS and a client on the same machine can cause issues with low memory and memory pressure. If the client consumes all the memory and then tries to write data to the file system, the OSS will need to allocate pages to receive data from the client but will not be able to perform this operation due to low memory. This can cause the client to hang.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292132" xreflabel=""/>Only servers running on 64-bit CPUs are tested and supported. 64-bit CPU clients are typically used for testing to match expected customer usage and avoid limitations due to the 4 GB limit for RAM size, 1 GB low-memory limitation, and 16 TB file size limit of 32-bit CPUs. Also, due to kernel API limitations, performing backups of Lustre 2.x. filesystems on 32-bit clients may cause backup tools to confuse files that have the same 32-bit inode number.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292472" xreflabel=""/>The storage attached to the servers typically uses RAID to provide fault tolerance and can optionally be organized with logical volume management (LVM). It is then formatted by Lustre as a file system. Lustre OSS and MDS servers read, write and modify data in the format imposed by the file system.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292545" xreflabel=""/>Lustre uses journaling file system technology on both the MDTs and OSTs. For a MDT, as much as a 20 percent performance gain can be obtained by placing the journal on a separate device.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292546" xreflabel=""/>The MDS can effectively utilize a lot of CPU cycles. A minimium of four processor cores are recommended. More are advisable for files systems with many clients.</para>
+ <para>Only servers running on 64-bit CPUs are tested and supported. 64-bit CPU clients are typically used for testing to match expected customer usage and avoid limitations due to the 4 GB limit for RAM size, 1 GB low-memory limitation, and 16 TB file size limit of 32-bit CPUs. Also, due to kernel API limitations, performing backups of Lustre 2.x. filesystems on 32-bit clients may cause backup tools to confuse files that have the same 32-bit inode number.</para>
+ <para>The storage attached to the servers typically uses RAID to provide fault tolerance and can optionally be organized with logical volume management (LVM). It is then formatted by Lustre as a file system. Lustre OSS and MDS servers read, write and modify data in the format imposed by the file system.</para>
+ <para>Lustre uses journaling file system technology on both the MDTs and OSTs. For a MDT, as much as a 20 percent performance gain can be obtained by placing the journal on a separate device.</para>
+ <para>The MDS can effectively utilize a lot of CPU cycles. A minimium of four processor cores are recommended. More are advisable for files systems with many clients.</para>
<note>
<para>Lustre clients running on architectures with different endianness are supported. One limitation is that the PAGE_SIZE kernel macro on the client must be as large as the PAGE_SIZE of the server. In particular, ia64 or PPC clients with large pages (up to 64kB pages) can run with x86 servers (4kB pages). If you are running x86 clients with ia64 or PPC servers, you must compile the ia64 kernel with a 4kB PAGE_SIZE (so the server page size is not larger than the client page size). <anchor xml:id="dbdoclet.50438256_51943" xreflabel=""/></para>
</note>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438256_pgfId-1293428" xreflabel=""/>5.1.1 MDT Storage Hardware Considerations</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293438" xreflabel=""/>The data access pattern for MDS storage is a database-like access pattern with many seeks and read-and-writes of small amounts of data. High throughput to MDS storage is not important. Storage types that provide much lower seek times, such as high-RPM SAS or SSD drives can be used for the MDT.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1295314" xreflabel=""/>For maximum performance, the MDT should be configured as RAID1 with an internal journal and two disks from different controllers.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1295315" xreflabel=""/>If you need a larger MDT, create multiple RAID1 devices from pairs of disks, and then make a RAID0 array of the RAID1 devices. This ensures maximum reliability because multiple disk failures only have a small chance of hitting both disks in the same RAID1 device.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1295316" xreflabel=""/>Doing the opposite (RAID1 of a pair of RAID0 devices) has a 50% chance that even two disk failures can cause the loss of the whole MDT device. The first failure disables an entire half of the mirror and the second failure has a 50% chance of disabling the remaining mirror.</para>
+ <title>5.1.1 MDT Storage Hardware Considerations</title>
+ <para>The data access pattern for MDS storage is a database-like access pattern with many seeks and read-and-writes of small amounts of data. High throughput to MDS storage is not important. Storage types that provide much lower seek times, such as high-RPM SAS or SSD drives can be used for the MDT.</para>
+ <para>For maximum performance, the MDT should be configured as RAID1 with an internal journal and two disks from different controllers.</para>
+ <para>If you need a larger MDT, create multiple RAID1 devices from pairs of disks, and then make a RAID0 array of the RAID1 devices. This ensures maximum reliability because multiple disk failures only have a small chance of hitting both disks in the same RAID1 device.</para>
+ <para>Doing the opposite (RAID1 of a pair of RAID0 devices) has a 50% chance that even two disk failures can cause the loss of the whole MDT device. The first failure disables an entire half of the mirror and the second failure has a 50% chance of disabling the remaining mirror.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438256_pgfId-1295312" xreflabel=""/>5.1.2 OST Storage Hardware Considerations</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293429" xreflabel=""/>The data access pattern for the OSS storage is a streaming I/O pattern that is dependent on the access patterns of applications being used. Each OSS can manage multiple object storage targets (OSTs), one for each volume with I/O traffic load-balanced between servers and targets. An OSS should be configured to have a balance between the network bandwidth and the attached storage bandwidth to prevent bottlenecks in the I/O path. Depending on the server hardware, an OSS typically serves between 2 and 8 targets, with each target up to 16 terabytes (TBs) in size.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293431" xreflabel=""/>Lustre file system capacity is the sum of the capacities provided by the targets. For example, 64 OSSs, each with two 8 TB targets, provide a file system with a capacity of nearly 1 PB. If each OST uses ten 1 TB SATA disks (8 data disks plus 2 parity disks in a RAID 6 configuration), it may be possible to get 50 MB/sec from each drive, providing up to 400 MB/sec of disk bandwidth per OST. If this system is used as storage backend with a system network like InfiniBand that provides a similar bandwidth, then each OSS could provide 800 MB/sec of end-to-end I/O throughput. (Although the architectural constraints described here are simple, in practice it takes careful hardware selection, benchmarking and integration to obtain such results.)</para>
+ <title>5.1.2 OST Storage Hardware Considerations</title>
+ <para>The data access pattern for the OSS storage is a streaming I/O pattern that is dependent on the access patterns of applications being used. Each OSS can manage multiple object storage targets (OSTs), one for each volume with I/O traffic load-balanced between servers and targets. An OSS should be configured to have a balance between the network bandwidth and the attached storage bandwidth to prevent bottlenecks in the I/O path. Depending on the server hardware, an OSS typically serves between 2 and 8 targets, with each target up to 16 terabytes (TBs) in size.</para>
+ <para>Lustre file system capacity is the sum of the capacities provided by the targets. For example, 64 OSSs, each with two 8 TB targets, provide a file system with a capacity of nearly 1 PB. If each OST uses ten 1 TB SATA disks (8 data disks plus 2 parity disks in a RAID 6 configuration), it may be possible to get 50 MB/sec from each drive, providing up to 400 MB/sec of disk bandwidth per OST. If this system is used as storage backend with a system network like InfiniBand that provides a similar bandwidth, then each OSS could provide 800 MB/sec of end-to-end I/O throughput. (Although the architectural constraints described here are simple, in practice it takes careful hardware selection, benchmarking and integration to obtain such results.)</para>
</section>
</section>
<section xml:id="dbdoclet.50438256_31079">
<title>5.2 Determining Space Requirements</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293293" xreflabel=""/>The desired performance characteristics of the backing file systems on the MDT and OSTs are independent of one another. The size of the MDT backing file system depends on the number of inodes needed in the total Lustre file system, while the aggregate OST space depends on the total amount of data stored on the file system.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293404" xreflabel=""/>Each time a file is created on a Lustre file system, it consumes one inode on the MDT and one inode for each OST object over which the file is striped. Normally, each file's stripe count is based on the system-wide default stripe count. However, this can be changed for individual files using the lfssetstripe option. For more details, see <xref linkend="managingstripingfreespace"/>.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293400" xreflabel=""/>In a Lustre ldiskfs file system, all the inodes are allocated on the MDT and OSTs when the file system is first formatted. The total number of inodes on a formatted MDT or OST cannot be easily changed, although it is possible to add OSTs with additional space and corresponding inodes. Thus, the number of inodes created at format time should be generous enough to anticipate future expansion.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293818" xreflabel=""/>When the file system is in use and a file is created, the metadata associated with that file is stored in one of the pre-allocated inodes and does not consume any of the free space used to store file data.</para>
+ <para>The desired performance characteristics of the backing file systems on the MDT and OSTs are independent of one another. The size of the MDT backing file system depends on the number of inodes needed in the total Lustre file system, while the aggregate OST space depends on the total amount of data stored on the file system.</para>
+ <para>Each time a file is created on a Lustre file system, it consumes one inode on the MDT and one inode for each OST object over which the file is striped. Normally, each file's stripe count is based on the system-wide default stripe count. However, this can be changed for individual files using the lfssetstripe option. For more details, see <xref linkend="managingstripingfreespace"/>.</para>
+ <para>In a Lustre ldiskfs file system, all the inodes are allocated on the MDT and OSTs when the file system is first formatted. The total number of inodes on a formatted MDT or OST cannot be easily changed, although it is possible to add OSTs with additional space and corresponding inodes. Thus, the number of inodes created at format time should be generous enough to anticipate future expansion.</para>
+ <para>When the file system is in use and a file is created, the metadata associated with that file is stored in one of the pre-allocated inodes and does not consume any of the free space used to store file data.</para>
<note>
<para>By default, the ldiskfs file system used by Lustre servers to store user-data objects and system data reserves 5% of space that cannot be used by Lustre. Additionally, Lustre reserves up to 400 MB on each OST for journal use and a small amount of space outside the journal to store accounting data for Lustre. This reserved space is unusable for general storage. Thus, at least 400 MB of space is used on each OST before any file object data is saved.<anchor xml:id="dbdoclet.50438256_74070" xreflabel=""/></para>
</note>
<section xml:id="dbdoclet.50438256_87676">
<title>5.2.1 Determining MDS/MDT Space Requirements</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293355" xreflabel=""/>When calculating the MDT size, the important factor to consider is the number of files to be stored in the file system. This determines the number of inodes needed, which drives the MDT sizing. To be on the safe side, plan for 4 KB per inode on the MDT, which is the default value. Attached storage required for Lustre metadata is typically 1-2 percent of the file system capacity depending upon file size.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293134" xreflabel=""/>For example, if the average file size is 5 MB and you have 100 TB of usable OST space, then you can calculate the minimum number of inodes as follows:</para>
- <example><anchor xml:id="dbdoclet.50438256_pgfId-1293526" xreflabel=""/>(100 TB * 1024 GB/TB * 1024 MB/GB) / 5 MB/inode = 20 million inodes</example>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293527" xreflabel=""/>We recommend that you use at least twice the minimum number of inodes to allow for future expansion and allow for an average file size smaller than expected. Thus, the required space is:</para>
+ <para>When calculating the MDT size, the important factor to consider is the number of files to be stored in the file system. This determines the number of inodes needed, which drives the MDT sizing. To be on the safe side, plan for 4 KB per inode on the MDT, which is the default value. Attached storage required for Lustre metadata is typically 1-2 percent of the file system capacity depending upon file size.</para>
+ <para>For example, if the average file size is 5 MB and you have 100 TB of usable OST space, then you can calculate the minimum number of inodes as follows:</para>
+ <example>(100 TB * 1024 GB/TB * 1024 MB/GB) / 5 MB/inode = 20 million inodes</example>
+ <para>We recommend that you use at least twice the minimum number of inodes to allow for future expansion and allow for an average file size smaller than expected. Thus, the required space is:</para>
<example>4 KB/inode * 40 million inodes = 160 GB</example>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293135" xreflabel=""/>If the average file size is small, 4 KB for example, Lustre is not very efficient as the MDT uses as much space as the OSTs. However, this is not a common configuration for Lustre.</para>
+ <para>If the average file size is small, 4 KB for example, Lustre is not very efficient as the MDT uses as much space as the OSTs. However, this is not a common configuration for Lustre.</para>
<note>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293142" xreflabel=""/>If the MDT is too small, this can cause all the space on the OSTs to be unusable. Be sure to determine the appropriate size of the MDT needed to support the file system before formatting the file system. It is difficult to increase the number of inodes after the file system is formatted.</para>
+ <para>If the MDT is too small, this can cause all the space on the OSTs to be unusable. Be sure to determine the appropriate size of the MDT needed to support the file system before formatting the file system. It is difficult to increase the number of inodes after the file system is formatted.</para>
</note>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438256_pgfId-1293490" xreflabel=""/>5.2.2 Determining OSS/OST Space Requirements</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293491" xreflabel=""/>For the OST, the amount of space taken by each object depends on the usage pattern of the users/applications running on the system. Lustre defaults to a conservative estimate for the object size (16 KB per object). If you are confident that the average file size for your applications will be larger than this, you can specify a larger average file size (fewer total inodes) to reduce file system overhead and minimize file system check time. See <xref linkend="dbdoclet.50438256_53886"/> for more details.</para>
+ <title>5.2.2 Determining OSS/OST Space Requirements</title>
+ <para>For the OST, the amount of space taken by each object depends on the usage pattern of the users/applications running on the system. Lustre defaults to a conservative estimate for the object size (16 KB per object). If you are confident that the average file size for your applications will be larger than this, you can specify a larger average file size (fewer total inodes) to reduce file system overhead and minimize file system check time. See <xref linkend="dbdoclet.50438256_53886"/> for more details.</para>
</section>
</section>
<section xml:id="dbdoclet.50438256_84701">
<title>5.3 Setting File System Formatting Options</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293227" xreflabel=""/>To override the default formatting options for any of the Lustre backing file systems, use this argument to <literal>mkfs.lustre</literal> to pass formatting options to the backing <literal>mkfs</literal>:</para>
- <screen><anchor xml:id="dbdoclet.50438256_pgfId-1293982" xreflabel=""/>--mkfsoptions='backing fs options'</screen>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293480" xreflabel=""/>For other options to format backing ldiskfs filesystems, see the Linux man page for <literal>mke2fs(8)</literal>.</para>
+ <para>To override the default formatting options for any of the Lustre backing file systems, use this argument to <literal>mkfs.lustre</literal> to pass formatting options to the backing <literal>mkfs</literal>:</para>
+ <screen>--mkfsoptions='backing fs options'</screen>
+ <para>For other options to format backing ldiskfs filesystems, see the Linux man page for <literal>mke2fs(8)</literal>.</para>
<section xml:id="dbdoclet.50438256_pgfId-1293228">
<title>5.3.1 Setting the Number of Inodes for the MDS</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1294133" xreflabel=""/>The number of inodes on the MDT is determined at format time based on the total size of the file system to be created. The default MDT inode ratio is one inode for every 4096 bytes of file system space. To override the inode ratio, use the following option:</para>
- <screen><anchor xml:id="dbdoclet.50438256_pgfId-1294002" xreflabel=""/>-i <emphasis><bytes per inode></emphasis></screen>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1294004" xreflabel=""/>For example, use the following option to create one inode per 2048 bytes of file system space.</para>
- <screen><anchor xml:id="dbdoclet.50438256_pgfId-1294011" xreflabel=""/>--mkfsoptions="-i 2048"
+ <para>The number of inodes on the MDT is determined at format time based on the total size of the file system to be created. The default MDT inode ratio is one inode for every 4096 bytes of file system space. To override the inode ratio, use the following option:</para>
+ <screen>-i <emphasis><bytes per inode></emphasis></screen>
+ <para>For example, use the following option to create one inode per 2048 bytes of file system space.</para>
+ <screen>--mkfsoptions="-i 2048"
</screen>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1294027" xreflabel=""/>To avoid mke2fs creating an unusable file system, do not specify the -i option with an inode ratio below one inode per 1024 bytes. Instead, specify an absolute number of inodes, using this option:</para>
- <screen><anchor xml:id="dbdoclet.50438256_pgfId-1294044" xreflabel=""/>-N<emphasis> <number of inodes></emphasis></screen>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293230" xreflabel=""/>For example, by default, a 2 TB MDT will have 512M inodes. The largest currently-supported file system size is 16 TB, which would hold 4B inodes, the maximum possible number of inodes in a ldiskfs file system. With an MDS inode ratio of 1024 bytes per inode, a 2 TB MDT would hold 2B inodes, and a 4 TB MDT would hold 4B inodes.</para>
+ <para>To avoid mke2fs creating an unusable file system, do not specify the -i option with an inode ratio below one inode per 1024 bytes. Instead, specify an absolute number of inodes, using this option:</para>
+ <screen>-N<emphasis> <number of inodes></emphasis></screen>
+ <para>For example, by default, a 2 TB MDT will have 512M inodes. The largest currently-supported file system size is 16 TB, which would hold 4B inodes, the maximum possible number of inodes in a ldiskfs file system. With an MDS inode ratio of 1024 bytes per inode, a 2 TB MDT would hold 2B inodes, and a 4 TB MDT would hold 4B inodes.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438256_pgfId-1294109" xreflabel=""/>5.3.2 <anchor xml:id="dbdoclet.50438256_34593" xreflabel=""/>Setting the Inode Size for the <anchor xml:id="dbdoclet.50438256_marker-1293231" xreflabel=""/>MDT</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293233" xreflabel=""/>Lustre uses "large" inodes on backing file systems to efficiently store Lustre metadata with each file. On the MDT, each inode is at least 512 bytes in size (by default), while on the OST each inode is 256 bytes in size.</para>
+ <title>5.3.2 <anchor xml:id="dbdoclet.50438256_34593" xreflabel=""/>Setting the Inode Size for the <anchor xml:id="dbdoclet.50438256_marker-1293231" xreflabel=""/>MDT</title>
+ <para>Lustre uses "large" inodes on backing file systems to efficiently store Lustre metadata with each file. On the MDT, each inode is at least 512 bytes in size (by default), while on the OST each inode is 256 bytes in size.</para>
<para> The backing ldiskfs file system also needs sufficient space for other metadata like the journal (up to 400 MB), bitmaps and directories and a few files that Lustre uses to maintain cluster consistency.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293234" xreflabel=""/>To specify a larger inode size, use the <literal>-I <inodesize></literal> option. We recommend you do NOT specify a smaller-than-default inode size, as this can lead to serious performance problems; and you cannot change this parameter after formatting the file system. The inode ratio must always be larger than the inode size.</para>
+ <para>To specify a larger inode size, use the <literal>-I <inodesize></literal> option. We recommend you do NOT specify a smaller-than-default inode size, as this can lead to serious performance problems; and you cannot change this parameter after formatting the file system. The inode ratio must always be larger than the inode size.</para>
</section>
<section xml:id="dbdoclet.50438256_53886">
<title>5.3.3 Setting the Number of Inodes for an <anchor xml:id="dbdoclet.50438256_marker-1293235" xreflabel=""/>OST</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293237" xreflabel=""/>When formatting OST file systems, it is normally advantageous to take local file system usage into account. Try to minimize the number of inodes on each OST, while keeping enough margin for potential variance in future usage. This helps reduce the format and file system check time, and makes more space available for data.</para>
+ <para>When formatting OST file systems, it is normally advantageous to take local file system usage into account. Try to minimize the number of inodes on each OST, while keeping enough margin for potential variance in future usage. This helps reduce the format and file system check time, and makes more space available for data.</para>
<para> The current default is to create one inode per 16 KB of space in the OST file system, but in many environments, this is far too many inodes for the average file size. As a good rule of thumb, the OSTs should have at least:</para>
- <screen><anchor xml:id="dbdoclet.50438256_pgfId-1293238" xreflabel=""/>num_ost_inodes = <anchor xml:id="dbdoclet.50438256_pgfId-1294224" xreflabel=""/>4 * <emphasis><num_mds_inodes></emphasis> * <emphasis><default_stripe_count></emphasis> / <emphasis><number_osts></emphasis></screen>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293239" xreflabel=""/>You can specify the number of inodes on the OST file systems using the following option to the <literal>--mkfs</literal> option:</para>
- <screen><anchor xml:id="dbdoclet.50438256_pgfId-1294232" xreflabel=""/>-N <emphasis><num_inodes></emphasis></screen>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1294248" xreflabel=""/></para>
+ <screen>num_ost_inodes = 4 * <emphasis><num_mds_inodes></emphasis> * <emphasis><default_stripe_count></emphasis> / <emphasis><number_osts></emphasis></screen>
+ <para>You can specify the number of inodes on the OST file systems using the following option to the <literal>--mkfs</literal> option:</para>
+ <screen>-N <emphasis><num_inodes></emphasis></screen>
+ <para></para>
<para> Alternately, if you know the average file size, then you can specify the OST inode count for the OST file systems using:</para>
- <screen><anchor xml:id="dbdoclet.50438256_pgfId-1294260" xreflabel=""/>-i <emphasis><average_file_size</emphasis> / (<emphasis>number_of_stripes</emphasis> * 4)<emphasis>></emphasis></screen>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1294261" xreflabel=""/>For example, if the average file size is 16 MB and there are, by default 4 stripes per file, then <literal>--mkfsoptions='-i 1048576'</literal> would be appropriate.</para>
+ <screen>-i <emphasis><average_file_size</emphasis> / (<emphasis>number_of_stripes</emphasis> * 4)<emphasis>></emphasis></screen>
+ <para>For example, if the average file size is 16 MB and there are, by default 4 stripes per file, then <literal>--mkfsoptions='-i 1048576'</literal> would be appropriate.</para>
<note>
<para>In addition to the number of inodes, file system check time on OSTs is affected by a number of other variables: size of the file system, number of allocated blocks, distribution of allocated blocks on the disk, disk speed, CPU speed, and amount of RAM on the server. Reasonable file system check times (without serious file system problems), are expected to take five and thirty minutes per TB.</para>
</note>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293243" xreflabel=""/>For more details on formatting MDT and OST file systems, see <xref linkend="dbdoclet.50438208_51921"/>.</para>
+ <para>For more details on formatting MDT and OST file systems, see <xref linkend="dbdoclet.50438208_51921"/>.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438256_pgfId-1290698" xreflabel=""/>5.3.4 File and File System Limits</title>
+ <title>5.3.4 File and File System Limits</title>
<para><xref linkend="settinguplustresystem.tab1"/> describes file and file system size limits. These limits are imposed by either the Lustre architecture or the Linux virtual file system (VFS) and virtual memory subsystems. In a few cases, a limit is defined within the code and can be changed by re-compiling Lustre (see <xref linkend="installinglustrefromsourcecode"/>). In these cases, the indicated limit was used for Lustre testing. </para>
<table frame="all">
<title xml:id="settinguplustresystem.tab1">File and file system limits</title>
<thead>
<row>
<entry>
- <para><emphasis role="bold"><anchor xml:id="dbdoclet.50438256_pgfId-1290708" xreflabel=""/>Limit</emphasis></para>
+ <para><emphasis role="bold">Limit</emphasis></para>
</entry>
<entry>
- <para><emphasis role="bold"><anchor xml:id="dbdoclet.50438256_pgfId-1290710" xreflabel=""/>Value</emphasis></para>
+ <para><emphasis role="bold">Value</emphasis></para>
</entry>
<entry>
- <para><emphasis role="bold"><anchor xml:id="dbdoclet.50438256_pgfId-1290712" xreflabel=""/>Description</emphasis></para>
+ <para><emphasis role="bold">Description</emphasis></para>
</entry>
</row>
</thead>
<tbody>
<row>
<entry>
- <para>
<anchor xml:id="dbdoclet.50438256_pgfId-1290715" xreflabel=""/>Maximum <anchor xml:id="dbdoclet.50438256_marker-1290714" xreflabel=""/>stripe count</para>
+ <para> Maximum <anchor xml:id="dbdoclet.50438256_marker-1290714" xreflabel=""/>stripe count</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290718" xreflabel=""/>160</para>
+ <para> 160</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290720" xreflabel=""/>This limit is hard-coded, but is near the upper limit imposed by the underlying ldiskfs file system.</para>
+ <para> This limit is hard-coded, but is near the upper limit imposed by the underlying ldiskfs file system.</para>
</entry>
</row>
<row>
<entry>
- <para>
<anchor xml:id="dbdoclet.50438256_pgfId-1290723" xreflabel=""/>Maximum <anchor xml:id="dbdoclet.50438256_marker-1290722" xreflabel=""/>stripe size</para>
+ <para> Maximum <anchor xml:id="dbdoclet.50438256_marker-1290722" xreflabel=""/>stripe size</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290726" xreflabel=""/>< 4 GB</para>
+ <para> < 4 GB</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290728" xreflabel=""/>The amount of data written to each object before moving on to next object.</para>
+ <para> The amount of data written to each object before moving on to next object.</para>
</entry>
</row>
<row>
<entry>
- <para>
<anchor xml:id="dbdoclet.50438256_pgfId-1290731" xreflabel=""/>Minimum <anchor xml:id="dbdoclet.50438256_marker-1290730" xreflabel=""/>stripe size</para>
+ <para> Minimum <anchor xml:id="dbdoclet.50438256_marker-1290730" xreflabel=""/>stripe size</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290734" xreflabel=""/>64 KB</para>
+ <para> 64 KB</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290736" xreflabel=""/>Due to the 64 KB PAGE_SIZE on some 64-bit machines, the minimum stripe size is set to 64 KB.</para>
+ <para> Due to the 64 KB PAGE_SIZE on some 64-bit machines, the minimum stripe size is set to 64 KB.</para>
</entry>
</row>
<row>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1294306" xreflabel=""/>Maximum object size</para>
+ <para> Maximum object size</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1294308" xreflabel=""/>2 TB</para>
+ <para> 2 TB</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1294310" xreflabel=""/>The amount of data that can be stored in a single object. The ldiskfs limit of 2TB for a single file applies. Lustre allows 160 stripes of 2 TB each.</para>
+ <para> The amount of data that can be stored in a single object. The ldiskfs limit of 2TB for a single file applies. Lustre allows 160 stripes of 2 TB each.</para>
</entry>
</row>
<row>
<entry>
- <para>
<anchor xml:id="dbdoclet.50438256_pgfId-1290739" xreflabel=""/>Maximum <anchor xml:id="dbdoclet.50438256_marker-1290738" xreflabel=""/>number of OSTs</para>
+ <para> Maximum <anchor xml:id="dbdoclet.50438256_marker-1290738" xreflabel=""/>number of OSTs</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290741" xreflabel=""/>8150</para>
+ <para> 8150</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290743" xreflabel=""/>The maximum number of OSTs is a constant that can be changed at compile time. Lustre has been tested with up to 4000 OSTs.</para>
+ <para> The maximum number of OSTs is a constant that can be changed at compile time. Lustre has been tested with up to 4000 OSTs.</para>
</entry>
</row>
<row>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1291931" xreflabel=""/>Maximum number of MDTs</para>
+ <para> Maximum number of MDTs</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1291933" xreflabel=""/>1</para>
+ <para> 1</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1291935" xreflabel=""/>Maximum of 1 MDT per file system, but a single MDS can host multiple MDTs, each one for a separate file system.</para>
+ <para> Maximum of 1 MDT per file system, but a single MDS can host multiple MDTs, each one for a separate file system.</para>
</entry>
</row>
<row>
<entry>
- <para>
<anchor xml:id="dbdoclet.50438256_pgfId-1290747" xreflabel=""/>Maximum <anchor xml:id="dbdoclet.50438256_marker-1290746" xreflabel=""/>number of clients</para>
+ <para> Maximum <anchor xml:id="dbdoclet.50438256_marker-1290746" xreflabel=""/>number of clients</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290749" xreflabel=""/>131072</para>
+ <para> 131072</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290751" xreflabel=""/>The number of clients is a constant that can be changed at compile time.</para>
+ <para> The number of clients is a constant that can be changed at compile time.</para>
</entry>
</row>
<row>
<entry>
- <para>
<anchor xml:id="dbdoclet.50438256_pgfId-1290754" xreflabel=""/>Maximum size of a <anchor xml:id="dbdoclet.50438256_marker-1290753" xreflabel=""/>file system</para>
+ <para> Maximum size of a <anchor xml:id="dbdoclet.50438256_marker-1290753" xreflabel=""/>file system</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290756" xreflabel=""/>64 PB</para>
+ <para> 64 PB</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290758" xreflabel=""/>Each OST or MDT can have a file system up to 16 TB, regardless of whether 32-bit or 64-bit kernels are on the server.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290759" xreflabel=""/>You can have multiple OST file systems on a single OSS node.</para>
+ <para> Each OST or MDT can have a file system up to 16 TB, regardless of whether 32-bit or 64-bit kernels are on the server.</para>
+ <para>You can have multiple OST file systems on a single OSS node.</para>
</entry>
</row>
<row>
<entry>
- <para>
<anchor xml:id="dbdoclet.50438256_pgfId-1290762" xreflabel=""/>Maximum <anchor xml:id="dbdoclet.50438256_marker-1290761" xreflabel=""/>file size</para>
+ <para> Maximum <anchor xml:id="dbdoclet.50438256_marker-1290761" xreflabel=""/>file size</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290764" xreflabel=""/>16 TB on 32-bit systems</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290765" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290766" xreflabel=""/>320 TB on 64-bit systems</para>
+ <para> 16 TB on 32-bit systems</para>
+ <para> </para>
+ <para>320 TB on 64-bit systems</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290768" xreflabel=""/>Individual files have a hard limit of nearly 16 TB on 32-bit systems imposed by the kernel memory subsystem. On 64-bit systems this limit does not exist. Hence, files can be 64-bits in size. Lustre imposes an additional size limit of up to the number of stripes, where each stripe is 2 TB.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1294622" xreflabel=""/>A single file can have a maximum of 160 stripes, which gives an upper single file limit of 320 TB for 64-bit systems. The actual amount of data that can be stored in a file depends upon the amount of free space in each OST on which the file is striped.</para>
+ <para> Individual files have a hard limit of nearly 16 TB on 32-bit systems imposed by the kernel memory subsystem. On 64-bit systems this limit does not exist. Hence, files can be 64-bits in size. Lustre imposes an additional size limit of up to the number of stripes, where each stripe is 2 TB.</para>
+ <para>A single file can have a maximum of 160 stripes, which gives an upper single file limit of 320 TB for 64-bit systems. The actual amount of data that can be stored in a file depends upon the amount of free space in each OST on which the file is striped.</para>
</entry>
</row>
<row>
<entry>
- <para>
<anchor xml:id="dbdoclet.50438256_pgfId-1290771" xreflabel=""/>Maximum number of <anchor xml:id="dbdoclet.50438256_marker-1290770" xreflabel=""/>files or subdirectories in a single directory</para>
+ <para> Maximum number of <anchor xml:id="dbdoclet.50438256_marker-1290770" xreflabel=""/>files or subdirectories in a single directory</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290773" xreflabel=""/>10 million files</para>
+ <para> 10 million files</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290775" xreflabel=""/>Lustre uses the ldiskfs hashed directory code, which has a limit of about 10 million files depending on the length of the file name. The limit on subdirectories is the same as the limit on regular files.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290776" xreflabel=""/>Lustre is tested with ten million files in a single directory.</para>
+ <para> Lustre uses the ldiskfs hashed directory code, which has a limit of about 10 million files depending on the length of the file name. The limit on subdirectories is the same as the limit on regular files.</para>
+ <para>Lustre is tested with ten million files in a single directory.</para>
</entry>
</row>
<row>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290779" xreflabel=""/>Maximum number of files in the file system</para>
+ <para> Maximum number of files in the file system</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290781" xreflabel=""/>4 billion</para>
+ <para> 4 billion</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290783" xreflabel=""/>The ldiskfs file system imposes an upper limit of 4 billion inodes. By default, the MDS file system is formatted with 4 KB of space per inode, meaning 512 million inodes per file system of 2 TB.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1291983" xreflabel=""/>This can be increased initially, at the time of MDS file system creation. For more information, see <xref linkend="settinguplustresystem"/>.</para>
+ <para> The ldiskfs file system imposes an upper limit of 4 billion inodes. By default, the MDS file system is formatted with 4 KB of space per inode, meaning 512 million inodes per file system of 2 TB.</para>
+ <para>This can be increased initially, at the time of MDS file system creation. For more information, see <xref linkend="settinguplustresystem"/>.</para>
</entry>
</row>
<row>
<entry>
- <para>
<anchor xml:id="dbdoclet.50438256_pgfId-1290792" xreflabel=""/>Maximum length of a <anchor xml:id="dbdoclet.50438256_marker-1290791" xreflabel=""/>filename</para>
+ <para> Maximum length of a <anchor xml:id="dbdoclet.50438256_marker-1290791" xreflabel=""/>filename</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290794" xreflabel=""/>255 bytes (filename)</para>
+ <para> 255 bytes (filename)</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290798" xreflabel=""/>This limit is 255 bytes for a single filename, the same as in an ldiskfs file system.</para>
+ <para> This limit is 255 bytes for a single filename, the same as in an ldiskfs file system.</para>
</entry>
</row>
<row>
<entry>
- <para>
<anchor xml:id="dbdoclet.50438256_pgfId-1292013" xreflabel=""/>Maximum length of a <anchor xml:id="dbdoclet.50438256_marker-1292019" xreflabel=""/>pathname</para>
+ <para> Maximum length of a <anchor xml:id="dbdoclet.50438256_marker-1292019" xreflabel=""/>pathname</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1292015" xreflabel=""/>4096 bytes (pathname)</para>
+ <para> 4096 bytes (pathname)</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1292017" xreflabel=""/>The Linux VFS imposes a full pathname length of 4096 bytes.</para>
+ <para> The Linux VFS imposes a full pathname length of 4096 bytes.</para>
</entry>
</row>
<row>
<entry>
- <para>
<anchor xml:id="dbdoclet.50438256_pgfId-1290801" xreflabel=""/>Maximum number of <anchor xml:id="dbdoclet.50438256_marker-1290800" xreflabel=""/>open files for Lustre file systems</para>
+ <para> Maximum number of <anchor xml:id="dbdoclet.50438256_marker-1290800" xreflabel=""/>open files for Lustre file systems</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290803" xreflabel=""/>None</para>
+ <para> None</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438256_pgfId-1290805" xreflabel=""/>Lustre does not impose a maximum for the number of open files, but the practical limit depends on the amount of RAM on the MDS. No "tables" for open files exist on the MDS, as they are only linked in a list to a given client's export. Each client process probably has a limit of several thousands of open files which depends on the ulimit.</para>
+ <para> Lustre does not impose a maximum for the number of open files, but the practical limit depends on the amount of RAM on the MDS. No "tables" for open files exist on the MDS, as they are only linked in a list to a given client's export. Each client process probably has a limit of several thousands of open files which depends on the ulimit.</para>
</entry>
</row>
</tbody>
</tgroup>
</table>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290676" xreflabel=""/> </para>
+ <para> </para>
</section>
</section>
<section xml:id="dbdoclet.50438256_26456">
<title>5.4 Determining Memory<anchor xml:id="dbdoclet.50438256_marker-1292212" xreflabel=""/> Requirements</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292214" xreflabel=""/>This section describes the memory requirements for each Lustre file system component.</para>
+ <para>This section describes the memory requirements for each Lustre file system component.</para>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438256_pgfId-1292221" xreflabel=""/>5.4.1 Client Memory Requirements</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292222" xreflabel=""/>A minimum of 2 GB RAM is recommended for clients.</para>
+ <title>5.4.1 Client Memory Requirements</title>
+ <para>A minimum of 2 GB RAM is recommended for clients.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438256_pgfId-1292223" xreflabel=""/>5.4.2 <anchor xml:id="dbdoclet.50438256_78447" xreflabel=""/>MDS Memory Requirements</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292225" xreflabel=""/>MDS memory requirements are determined by the following factors:</para>
+ <title>5.4.2 <anchor xml:id="dbdoclet.50438256_78447" xreflabel=""/>MDS Memory Requirements</title>
+ <para>MDS memory requirements are determined by the following factors:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292226" xreflabel=""/> Number of clients</para>
+ <para> Number of clients</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292227" xreflabel=""/> Size of the directories</para>
+ <para> Size of the directories</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292228" xreflabel=""/> Load placed on server</para>
+ <para> Load placed on server</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292229" xreflabel=""/>The amount of memory used by the MDS is a function of how many clients are on the system, and how many files they are using in their working set. This is driven, primarily, by the number of locks a client can hold at one time. The number of locks held by clients varies by load and memory availability on the server. Interactive clients can hold in excess of 10,000 locks at times. On the MDS, memory usage is approximately 2 KB per file, including the Lustre distributed lock manager (DLM) lock and kernel data structures for the files currently in use. Having file data in cache can improve metadata performance by a factor of 10x or more compared to reading it from disk.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292230" xreflabel=""/>MDS memory requirements include:</para>
+ <para>The amount of memory used by the MDS is a function of how many clients are on the system, and how many files they are using in their working set. This is driven, primarily, by the number of locks a client can hold at one time. The number of locks held by clients varies by load and memory availability on the server. Interactive clients can hold in excess of 10,000 locks at times. On the MDS, memory usage is approximately 2 KB per file, including the Lustre distributed lock manager (DLM) lock and kernel data structures for the files currently in use. Having file data in cache can improve metadata performance by a factor of 10x or more compared to reading it from disk.</para>
+ <para>MDS memory requirements include:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1294632" xreflabel=""/><emphasis role="bold">File system metadata</emphasis> : A reasonable amount of RAM needs to be available for file system metadata. While no hard limit can be placed on the amount of file system metadata, if more RAM is available, then the disk I/O is needed less often to retrieve the metadata.</para>
+ <para><emphasis role="bold">File system metadata</emphasis> : A reasonable amount of RAM needs to be available for file system metadata. While no hard limit can be placed on the amount of file system metadata, if more RAM is available, then the disk I/O is needed less often to retrieve the metadata.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1294633" xreflabel=""/><emphasis role="bold">Network transport</emphasis> : If you are using TCP or other network transport that uses system memory for send/receive buffers, this memory requirement must also be taken into consideration.</para>
+ <para><emphasis role="bold">Network transport</emphasis> : If you are using TCP or other network transport that uses system memory for send/receive buffers, this memory requirement must also be taken into consideration.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1294399" xreflabel=""/><emphasis role="bold">Journal size</emphasis> : By default, the journal size is 400 MB for each Lustre ldiskfs file system. This can pin up to an equal amount of RAM on the MDS node per file system.</para>
+ <para><emphasis role="bold">Journal size</emphasis> : By default, the journal size is 400 MB for each Lustre ldiskfs file system. This can pin up to an equal amount of RAM on the MDS node per file system.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292234" xreflabel=""/><emphasis role="bold">Failover configuration</emphasis> : If the MDS node will be used for failover from another node, then the RAM for each journal should be doubled, so the backup server can handle the additional load if the primary server fails.</para>
+ <para><emphasis role="bold">Failover configuration</emphasis> : If the MDS node will be used for failover from another node, then the RAM for each journal should be doubled, so the backup server can handle the additional load if the primary server fails.</para>
</listitem>
</itemizedlist>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438256_pgfId-1292235" xreflabel=""/>5.4.2.1 Calculating MDS Memory Requirements</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292236" xreflabel=""/>By default, 400 MB are used for the file system journal. Additional RAM is used for caching file data for the larger working set, which is not actively in use by clients but should be kept "hot" for improved access times. Approximately 1.5 KB per file is needed to keep a file in cache without a lock.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292237" xreflabel=""/>For example, for a single MDT on an MDS with 1,000 clients, 16 interactive nodes, and a 2 million file working set (of which 400,000 files are cached on the clients):</para>
+ <title>5.4.2.1 Calculating MDS Memory Requirements</title>
+ <para>By default, 400 MB are used for the file system journal. Additional RAM is used for caching file data for the larger working set, which is not actively in use by clients but should be kept "hot" for improved access times. Approximately 1.5 KB per file is needed to keep a file in cache without a lock.</para>
+ <para>For example, for a single MDT on an MDS with 1,000 clients, 16 interactive nodes, and a 2 million file working set (of which 400,000 files are cached on the clients):</para>
<example>
<para>Operating system overhead = 512 MB</para>
<para>File system journal = 400 MB</para>
<para>16 interactive clients * 10,000 files * 2kB = 320 MB</para>
<para>1,600,000 file extra working set * 1.5kB/file = 2400 MB</para>
</example>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292242" xreflabel=""/>Thus, the minimum requirement for a system with this configuration is at least 4 GB of RAM. However, additional memory may significantly improve performance.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292243" xreflabel=""/>For directories containing 1 million or more files, more memory may provide a significant benefit. For example, in an environment where clients randomly access one of 10 million files, having extra memory for the cache significantly improves performance.</para>
+ <para>Thus, the minimum requirement for a system with this configuration is at least 4 GB of RAM. However, additional memory may significantly improve performance.</para>
+ <para>For directories containing 1 million or more files, more memory may provide a significant benefit. For example, in an environment where clients randomly access one of 10 million files, having extra memory for the cache significantly improves performance.</para>
</section>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438256_pgfId-1292246" xreflabel=""/>5.4.3 <anchor xml:id="dbdoclet.50438256_84553" xreflabel=""/>OSS Memory Requirements</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292248" xreflabel=""/>When planning the hardware for an OSS node, consider the memory usage of several components in the Lustre system (i.e., journal, service threads, file system metadata, etc.). Also, consider the effect of the OSS read cache feature, which consumes memory as it caches data on the OSS node.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292249" xreflabel=""/>In addition to the MDS memory requirements mentioned in <xref linkend="dbdoclet.50438256_87676"/>, the OSS requirements include:</para>
+ <title>5.4.3 <anchor xml:id="dbdoclet.50438256_84553" xreflabel=""/>OSS Memory Requirements</title>
+ <para>When planning the hardware for an OSS node, consider the memory usage of several components in the Lustre system (i.e., journal, service threads, file system metadata, etc.). Also, consider the effect of the OSS read cache feature, which consumes memory as it caches data on the OSS node.</para>
+ <para>In addition to the MDS memory requirements mentioned in <xref linkend="dbdoclet.50438256_87676"/>, the OSS requirements include:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292250" xreflabel=""/><emphasis role="bold">Service threads</emphasis> : The service threads on the OSS node pre-allocate a 1 MB I/O buffer for each ost_io service thread, so these buffers do not need to be allocated and freed for each I/O request.</para>
+ <para><emphasis role="bold">Service threads</emphasis> : The service threads on the OSS node pre-allocate a 1 MB I/O buffer for each ost_io service thread, so these buffers do not need to be allocated and freed for each I/O request.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292251" xreflabel=""/><emphasis role="bold">OSS read cache</emphasis> : OSS read cache provides read-only caching of data on an OSS, using the regular Linux page cache to store the data. Just like caching from a regular file system in Linux, OSS read cache uses as much physical memory as is available.</para>
+ <para><emphasis role="bold">OSS read cache</emphasis> : OSS read cache provides read-only caching of data on an OSS, using the regular Linux page cache to store the data. Just like caching from a regular file system in Linux, OSS read cache uses as much physical memory as is available.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292252" xreflabel=""/>The same calculation applies to files accessed from the OSS as for the MDS, but the load is distributed over many more OSSs nodes, so the amount of memory required for locks, inode cache, etc. listed under MDS is spread out over the OSS nodes.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292253" xreflabel=""/>Because of these memory requirements, the following calculations should be taken as determining the absolute minimum RAM required in an OSS node.</para>
+ <para>The same calculation applies to files accessed from the OSS as for the MDS, but the load is distributed over many more OSSs nodes, so the amount of memory required for locks, inode cache, etc. listed under MDS is spread out over the OSS nodes.</para>
+ <para>Because of these memory requirements, the following calculations should be taken as determining the absolute minimum RAM required in an OSS node.</para>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438256_pgfId-1292254" xreflabel=""/>5.4.3.1 Calculating OSS Memory Requirements</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292255" xreflabel=""/>The minimum recommended RAM size for an OSS with two OSTs is computed below:</para>
+ <title>5.4.3.1 Calculating OSS Memory Requirements</title>
+ <para>The minimum recommended RAM size for an OSS with two OSTs is computed below:</para>
<example>
<para>Ethernet/TCP send/receive buffers (1 MB * 512 threads) = 512 MB</para>
<para>400 MB journal size * 2 OST devices = 800 MB</para>
<para>Per OSS RAM minimum requirement = 4096MB (approx.)</para>
</example>
<para>This consumes about 1,400 MB just for the pre-allocated buffers, and an additional 2 GB for minimal file system and kernel usage. Therefore, for a non-failover configuration, the minimum RAM would be 4 GB for an OSS node with two OSTs. Adding additional memory on the OSS will improve the performance of reading smaller, frequently-accessed files.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292269" xreflabel=""/>For a failover configuration, the minimum RAM would be at least 6 GB. For 4 OSTs on each OSS in a failover configuration 10GB of RAM is reasonable. When the OSS is not handling any failed-over OSTs the extra RAM will be used as a read cache.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292270" xreflabel=""/>As a reasonable rule of thumb, about 2 GB of base memory plus 1 GB per OST can be used. In failover configurations, about 2 GB per OST is needed.</para>
+ <para>For a failover configuration, the minimum RAM would be at least 6 GB. For 4 OSTs on each OSS in a failover configuration 10GB of RAM is reasonable. When the OSS is not handling any failed-over OSTs the extra RAM will be used as a read cache.</para>
+ <para>As a reasonable rule of thumb, about 2 GB of base memory plus 1 GB per OST can be used. In failover configurations, about 2 GB per OST is needed.</para>
</section>
</section>
</section>
<section xml:id="dbdoclet.50438256_78272">
<title>5.5 Implementing Networks To Be Used by Lustre</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290491" xreflabel=""/>As a high performance file system, Lustre places heavy loads on networks. Thus, a network interface in each Lustre server and client is commonly dedicated to Lustre traffic. This is often a dedicated TCP/IP subnet, although other network hardware can also be used.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290944" xreflabel=""/>A typical Lustre implementation may include the following:</para>
+ <para>As a high performance file system, Lustre places heavy loads on networks. Thus, a network interface in each Lustre server and client is commonly dedicated to Lustre traffic. This is often a dedicated TCP/IP subnet, although other network hardware can also be used.</para>
+ <para>A typical Lustre implementation may include the following:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290945" xreflabel=""/> A high-performance backend network for the Lustre servers, typically an InfiniBand (IB) network.</para>
+ <para> A high-performance backend network for the Lustre servers, typically an InfiniBand (IB) network.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290946" xreflabel=""/> A larger client network.</para>
+ <para> A larger client network.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290947" xreflabel=""/> Lustre routers to connect the two networks.</para>
+ <para> Lustre routers to connect the two networks.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1291003" xreflabel=""/>Lustre networks and routing are configured and managed by specifying parameters to the Lustre Networking (lnet) module in <literal>/etc/modprobe.conf</literal> or <literal>/etc/modprobe.conf.local</literal> (depending on your Linux distribution).</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290498" xreflabel=""/>To prepare to configure Lustre Networking, complete the following steps:</para>
+ <para>Lustre networks and routing are configured and managed by specifying parameters to the Lustre Networking (lnet) module in <literal>/etc/modprobe.conf</literal> or <literal>/etc/modprobe.conf.local</literal> (depending on your Linux distribution).</para>
+ <para>To prepare to configure Lustre Networking, complete the following steps:</para>
<orderedlist>
<listitem>
<para><emphasis role="bold">Identify all machines that will be running Lustre and the network interfaces they will use to run Lustre traffic. These machines will form the Lustre network.</emphasis></para>
<title xml:id='systemconfigurationutilities.title'>System Configuration Utilities</title>
</info>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294056" xreflabel=""/>This chapter includes system configuration utilities and includes the following sections:</para>
+ <para>This chapter includes system configuration utilities and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438219_55923"/></para>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438219_55923">
<title>36.1 e2<anchor xml:id="dbdoclet.50438219_marker-1317398" xreflabel=""/>scan</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317400" xreflabel=""/>The e2scan utility is an ext2 file system-modified inode scan program. The e2scan program uses libext2fs to find inodes with ctime or mtime newer than a given time and prints out their pathname. Use e2scan to efficiently generate lists of files that have been modified. The e2scan tool is included in the e2fsprogs package, located at:</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317402" xreflabel=""/><link xl:href="http://downloads.lustre.org/public/tools/e2fsprogs/">http://downloads.lustre.org/public/tools/e2fsprogs/</link></para>
+ <para>The e2scan utility is an ext2 file system-modified inode scan program. The e2scan program uses libext2fs to find inodes with ctime or mtime newer than a given time and prints out their pathname. Use e2scan to efficiently generate lists of files that have been modified. The e2scan tool is included in the e2fsprogs package, located at:</para>
+ <para><link xl:href="http://downloads.lustre.org/public/tools/e2fsprogs/">http://downloads.lustre.org/public/tools/e2fsprogs/</link></para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317403" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317404" xreflabel=""/>e2scan [options] [-f file] block_device
+ <title>Synopsis</title>
+ <screen>e2scan [options] [-f file] block_device
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317405" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317406" xreflabel=""/>When invoked, the e2scan utility iterates all inodes on the block device, finds modified inodes, and prints their inode numbers. A similar iterator, using libext2fs(5), builds a table (called parent database) which lists the parent node for each inode. With a lookup function, you can reconstruct modified pathnames from root.</para>
+ <title>Description</title>
+ <para>When invoked, the e2scan utility iterates all inodes on the block device, finds modified inodes, and prints their inode numbers. A similar iterator, using libext2fs(5), builds a table (called parent database) which lists the parent node for each inode. With a lookup function, you can reconstruct modified pathnames from root.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317430" xreflabel=""/>Options</title>
+ <title>Options</title>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317409" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317411" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317413" xreflabel=""/><emphasis role="bold">-b inode buffer blocks</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317415" xreflabel=""/>Sets the readahead inode blocks to get excellent performance when scanning the block device.</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317416" xreflabel=""/> </para></entry>
+ <entry><para> <emphasis role="bold">-b inode buffer blocks</emphasis></para></entry>
+ <entry><para> Sets the readahead inode blocks to get excellent performance when scanning the block device.</para><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317418" xreflabel=""/><emphasis role="bold">-o output file</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317420" xreflabel=""/>If an output file is specified, modified pathnames are written to this file. Otherwise, modified parameters are written to stdout.</para></entry>
+ <entry><para> <emphasis role="bold">-o output file</emphasis></para></entry>
+ <entry><para> If an output file is specified, modified pathnames are written to this file. Otherwise, modified parameters are written to stdout.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317422" xreflabel=""/><emphasis role="bold">-t inode | pathname</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317424" xreflabel=""/>Sets the e2scan type if type is inode. The e2scan utility prints modified inode numbers to stdout. By default, the type is set as pathname.</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317425" xreflabel=""/>The e2scan utility lists modified pathnames based on modified inode numbers.</para></entry>
+ <entry><para> <emphasis role="bold">-t inode | pathname</emphasis></para></entry>
+ <entry><para> Sets the e2scan type if type is inode. The e2scan utility prints modified inode numbers to stdout. By default, the type is set as pathname.</para><para>The e2scan utility lists modified pathnames based on modified inode numbers.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317427" xreflabel=""/><emphasis role="bold">-u</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317429" xreflabel=""/>Rebuilds the parent database from scratch. Otherwise, the current parent database is used.</para></entry>
+ <entry><para> <emphasis role="bold">-u</emphasis></para></entry>
+ <entry><para> Rebuilds the parent database from scratch. Otherwise, the current parent database is used.</para></entry>
</row>
</tbody>
</tgroup>
</section>
<section xml:id="dbdoclet.50438219_76969">
<title>36.2 l_<anchor xml:id="dbdoclet.50438219_marker-1318227" xreflabel=""/>getidentity</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318229" xreflabel=""/>The l_getidentity utility handles Lustre user / group cache upcall.</para>
+ <para>The l_getidentity utility handles Lustre user / group cache upcall.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318230" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318231" xreflabel=""/>l_getidentity {mdtname} {uid}
+ <title>Synopsis</title>
+ <screen>l_getidentity {mdtname} {uid}
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318232" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318233" xreflabel=""/>The group upcall file contains the path to an executable file that, when properly installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the mds_grp_downcall_data structure and write it to the /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info pseudo-file.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318234" xreflabel=""/>The l_getidentity utility is the reference implementation of the user or group cache upcall.</para>
+ <title>Description</title>
+ <para>The group upcall file contains the path to an executable file that, when properly installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the mds_grp_downcall_data structure and write it to the /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info pseudo-file.</para>
+ <para>The l_getidentity utility is the reference implementation of the user or group cache upcall.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318248" xreflabel=""/>Options</title>
+ <title>Options</title>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1318237" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1318239" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318241" xreflabel=""/><emphasis role="bold">mdtname</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318243" xreflabel=""/>Metadata server target name</para></entry>
+ <entry><para> <emphasis role="bold">mdtname</emphasis></para></entry>
+ <entry><para> Metadata server target name</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318245" xreflabel=""/><emphasis role="bold">uid</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318247" xreflabel=""/>User identifier</para></entry>
+ <entry><para> <emphasis role="bold">uid</emphasis></para></entry>
+ <entry><para> User identifier</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318249" xreflabel=""/>Files</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318250" xreflabel=""/>The l_getidentity files are located at:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318251" xreflabel=""/>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall
+ <title>Files</title>
+ <para>The l_getidentity files are located at:</para>
+ <screen>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438219_38274">
<title>36.3 lc<anchor xml:id="dbdoclet.50438219_marker-1302539" xreflabel=""/>tl</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1302541" xreflabel=""/>The lctl utility is used for root control and configuration. With lctl you can directly control Lustre via an ioctl interface, allowing various configuration, maintenance and debugging features to be accessed.</para>
+ <para>The lctl utility is used for root control and configuration. With lctl you can directly control Lustre via an ioctl interface, allowing various configuration, maintenance and debugging features to be accessed.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1294259" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1314608" xreflabel=""/>lctl
-<anchor xml:id="dbdoclet.50438219_pgfId-1314609" xreflabel=""/>lctl --device <devno><emphasis/><command [args]>
+ <title>Synopsis</title>
+ <screen>lctl
+lctl --device <devno><emphasis/><command [args]>
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1314610" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1314611" xreflabel=""/>The lctl utility can be invoked in interactive mode by issuing the lctl command. After that, commands are issued as shown below. The most common lctl commands are:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1314612" xreflabel=""/>dl
-<anchor xml:id="dbdoclet.50438219_pgfId-1314613" xreflabel=""/>dk
-<anchor xml:id="dbdoclet.50438219_pgfId-1314614" xreflabel=""/>device
-<anchor xml:id="dbdoclet.50438219_pgfId-1314615" xreflabel=""/>network <<emphasis>up/down</emphasis>>
-<anchor xml:id="dbdoclet.50438219_pgfId-1314616" xreflabel=""/>list_nids
-<anchor xml:id="dbdoclet.50438219_pgfId-1314617" xreflabel=""/>ping <emphasis>nid</emphasis><anchor xml:id="dbdoclet.50438219_pgfId-1314618" xreflabel=""/>help
-<anchor xml:id="dbdoclet.50438219_pgfId-1314619" xreflabel=""/>quit
+ <title>Description</title>
+ <para>The lctl utility can be invoked in interactive mode by issuing the lctl command. After that, commands are issued as shown below. The most common lctl commands are:</para>
+ <screen>dl
+dk
+device
+network <<emphasis>up/down</emphasis>>
+list_nids
+ping <emphasis>nid</emphasis>help
+quit
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1314620" xreflabel=""/>For a complete list of available commands, type help at the lctl prompt. To get basic help on command meaning and syntax, type help<emphasis>command</emphasis>. Command completion is activated with the TAB key, and command history is available via the up- and down-arrow keys.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1314621" xreflabel=""/>For non-interactive use, use the second invocation, which runs the command after connecting to the device.</para>
+ <para>For a complete list of available commands, type help at the lctl prompt. To get basic help on command meaning and syntax, type help<emphasis>command</emphasis>. Command completion is activated with the TAB key, and command history is available via the up- and down-arrow keys.</para>
+ <para>For non-interactive use, use the second invocation, which runs the command after connecting to the device.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1311076" xreflabel=""/>Setting Parameters with lctl</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1314629" xreflabel=""/>Lustre parameters are not always accessible using the procfs interface, as it is platform-specific. As a solution, lctl {get,set}_param has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to /proc/{fs,sys}/{lustre,lnet}. For future portability, use lctl {get,set}_param .</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1314630" xreflabel=""/>When the file system is running, use the lctl set_param command to set temporary parameters (mapping to items in /proc/{fs,sys}/{lnet,lustre}). The lctl set_param command uses this syntax:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1314631" xreflabel=""/>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value>
+ <title>Setting Parameters with lctl</title>
+ <para>Lustre parameters are not always accessible using the procfs interface, as it is platform-specific. As a solution, lctl {get,set}_param has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to /proc/{fs,sys}/{lustre,lnet}. For future portability, use lctl {get,set}_param .</para>
+ <para>When the file system is running, use the lctl set_param command to set temporary parameters (mapping to items in /proc/{fs,sys}/{lnet,lustre}). The lctl set_param command uses this syntax:</para>
+ <screen>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value>
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1314632" xreflabel=""/>For example:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1314633" xreflabel=""/>$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
+ <para>For example:</para>
+ <screen>$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1314634" xreflabel=""/>Many permanent parameters can be set with lctl conf_param. In general, lctl conf_param can be used to specify any parameter settable in a /proc/fs/lustre file, with its own OBD device. The lctl conf_param command uses this syntax:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1314635" xreflabel=""/><obd|fsname>.<obdtype>.<proc_file_name>=<value>)
+ <para>Many permanent parameters can be set with lctl conf_param. In general, lctl conf_param can be used to specify any parameter settable in a /proc/fs/lustre file, with its own OBD device. The lctl conf_param command uses this syntax:</para>
+ <screen><obd|fsname>.<obdtype>.<proc_file_name>=<value>)
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1314636" xreflabel=""/>For example:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1314637" xreflabel=""/>$ lctl conf_param testfs-MDT0000.mdt.group_upcall=NONE
-<anchor xml:id="dbdoclet.50438219_pgfId-1314638" xreflabel=""/>$ lctl conf_param testfs.llite.max_read_ahead_mb=16
+ <para>For example:</para>
+ <screen>$ lctl conf_param testfs-MDT0000.mdt.group_upcall=NONE
+$ lctl conf_param testfs.llite.max_read_ahead_mb=16
</screen>
<informaltable frame="none">
<tgroup cols="2">
<imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
</imageobject>
</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis><anchor xml:id="dbdoclet.50438219_pgfId-1314639" xreflabel=""/>The lctlconf_param command permanently sets parameters in the file system configuration.</para></entry>
+ <entry><para><emphasis role="bold">Caution -</emphasis>The lctlconf_param command permanently sets parameters in the file system configuration.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1314640" xreflabel=""/>To get current Lustre parameter settings, use the lctl get_param command with this syntax:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1314641" xreflabel=""/>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name>
+ <para>To get current Lustre parameter settings, use the lctl get_param command with this syntax:</para>
+ <screen>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name>
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1314642" xreflabel=""/>For example:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1314643" xreflabel=""/>$ lctl get_param -n ost.*.ost_io.timeouts
+ <para>For example:</para>
+ <screen>$ lctl get_param -n ost.*.ost_io.timeouts
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1314644" xreflabel=""/>To list Lustre parameters that are available to set, use the lctl list_param command, with this syntax:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1311097" xreflabel=""/>lctl list_param [-n] <obdtype>.<obdname>
+ <para>To list Lustre parameters that are available to set, use the lctl list_param command, with this syntax:</para>
+ <screen>lctl list_param [-n] <obdtype>.<obdname>
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311098" xreflabel=""/>For example:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1314652" xreflabel=""/>$ lctl list_param obdfilter.lustre-OST0000
+ <para>For example:</para>
+ <screen>$ lctl list_param obdfilter.lustre-OST0000
</screen>
-<para><anchor xml:id="dbdoclet.50438219_pgfId-1314653" xreflabel=""/>For more information on using lctl to set temporary and permanent parameters, see <xref linkend="dbdoclet.50438194_51490"/> (Setting Parameters with lctl).</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1302704" xreflabel=""/><emphasis role="bold">Network Configuration</emphasis></para>
+<para>For more information on using lctl to set temporary and permanent parameters, see <xref linkend="dbdoclet.50438194_51490"/> (Setting Parameters with lctl).</para>
+ <para><emphasis role="bold">Network Configuration</emphasis></para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1304902" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1304907" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314664" xreflabel=""/><emphasis role="bold">network</emphasis> <<emphasis>up/down</emphasis>>|<<emphasis>tcp/elan/myrinet</emphasis>></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314668" xreflabel=""/>Starts or stops LNET, or selects a network type for other <emphasis role="bold">lctl</emphasis> LNET commands.</para></entry>
+ <entry><para> <emphasis role="bold">network</emphasis> <<emphasis>up/down</emphasis>>|<<emphasis>tcp/elan/myrinet</emphasis>></para></entry>
+ <entry><para> Starts or stops LNET, or selects a network type for other <emphasis role="bold">lctl</emphasis> LNET commands.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314670" xreflabel=""/><emphasis role="bold">list_nids</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314674" xreflabel=""/>Prints all NIDs on the local node. LNET must be running.</para></entry>
+ <entry><para> <emphasis role="bold">list_nids</emphasis></para></entry>
+ <entry><para> Prints all NIDs on the local node. LNET must be running.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314676" xreflabel=""/><emphasis role="bold">which_nid</emphasis> <<emphasis>nidlist</emphasis>></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314680" xreflabel=""/>From a list of NIDs for a remote node, identifies the NID on which interface communication will occur.</para></entry>
+ <entry><para> <emphasis role="bold">which_nid</emphasis> <<emphasis>nidlist</emphasis>></para></entry>
+ <entry><para> From a list of NIDs for a remote node, identifies the NID on which interface communication will occur.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314682" xreflabel=""/><emphasis role="bold">ping</emphasis> <<emphasis>nid</emphasis>></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314686" xreflabel=""/>Checks LNET connectivity via an LNET ping. This uses the fabric appropriate to the specified NID.</para></entry>
+ <entry><para> <emphasis role="bold">ping</emphasis> <<emphasis>nid</emphasis>></para></entry>
+ <entry><para> Checks LNET connectivity via an LNET ping. This uses the fabric appropriate to the specified NID.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314692" xreflabel=""/><emphasis role="bold">interface_list</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314696" xreflabel=""/>Prints the network interface information for a given <emphasis role="bold">network</emphasis> type.</para></entry>
+ <entry><para> <emphasis role="bold">interface_list</emphasis></para></entry>
+ <entry><para> Prints the network interface information for a given <emphasis role="bold">network</emphasis> type.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314698" xreflabel=""/><emphasis role="bold">peer_list</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314702" xreflabel=""/>Prints the known peers for a given <emphasis role="bold">network</emphasis> type.</para></entry>
+ <entry><para> <emphasis role="bold">peer_list</emphasis></para></entry>
+ <entry><para> Prints the known peers for a given <emphasis role="bold">network</emphasis> type.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314704" xreflabel=""/><emphasis role="bold">conn_list</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314708" xreflabel=""/>Prints all the connected remote NIDs for a given <emphasis role="bold">network</emphasis> type.</para></entry>
+ <entry><para> <emphasis role="bold">conn_list</emphasis></para></entry>
+ <entry><para> Prints all the connected remote NIDs for a given <emphasis role="bold">network</emphasis> type.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314710" xreflabel=""/><emphasis role="bold">active_tx</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314714" xreflabel=""/>This command prints active transmits. It is only used for the Elan <emphasis role="bold">network</emphasis> type.</para></entry>
+ <entry><para> <emphasis role="bold">active_tx</emphasis></para></entry>
+ <entry><para> This command prints active transmits. It is only used for the Elan <emphasis role="bold">network</emphasis> type.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314716" xreflabel=""/><emphasis role="bold">route_list</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314720" xreflabel=""/>Prints the complete routing table.</para></entry>
+ <entry><para> <emphasis role="bold">route_list</emphasis></para></entry>
+ <entry><para> Prints the complete routing table.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1312839" xreflabel=""/><emphasis role="bold">Device Selection</emphasis></para>
+ <para><emphasis role="bold">Device Selection</emphasis></para>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
<colspec colname="c3" colwidth="33*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1312822" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1312824" xreflabel=""/> </emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1312826" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold"> </emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314726" xreflabel=""/><emphasis role="bold">device</emphasis> <<emphasis>devname</emphasis>></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314728" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314730" xreflabel=""/>This selects the specified OBD device. All other commands depend on the device being set.</para></entry>
+ <entry><para> <emphasis role="bold">device</emphasis> <<emphasis>devname</emphasis>></para></entry>
+ <entry><para> </para></entry>
+ <entry><para> This selects the specified OBD device. All other commands depend on the device being set.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314732" xreflabel=""/><emphasis role="bold">device_list</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314734" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314736" xreflabel=""/>Shows the local Lustre OBDs, a/k/a <emphasis role="bold">dl</emphasis>.</para></entry>
+ <entry><para> <emphasis role="bold">device_list</emphasis></para></entry>
+ <entry><para> </para></entry>
+ <entry><para> Shows the local Lustre OBDs, a/k/a <emphasis role="bold">dl</emphasis>.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1312818" xreflabel=""/><emphasis role="bold">Device Operations</emphasis></para>
+ <para><emphasis role="bold">Device Operations</emphasis></para>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
<colspec colname="c3" colwidth="33*"/>
<thead>
<row>
- <entry nameend="c2" namest="c1"><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1313502" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1313506" xreflabel=""/>Description</emphasis></para></entry>
+ <entry nameend="c2" namest="c1"><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314742" xreflabel=""/><emphasis role="bold">list_param</emphasis><emphasis>[-F|-R]</emphasis> <<emphasis>param_path ...</emphasis>></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314746" xreflabel=""/>Lists the Lustre or LNET parameter name.</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1314747" xreflabel=""/> </para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">list_param</emphasis><emphasis>[-F|-R]</emphasis> <<emphasis>param_path ...</emphasis>></para></entry>
+ <entry><para> Lists the Lustre or LNET parameter name.</para><para> </para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314749" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314751" xreflabel=""/>-<emphasis role="bold">F</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314753" xreflabel=""/>Adds '/', '@' or '=' for directories, symlinks and writeable files, respectively.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> -<emphasis role="bold">F</emphasis></para></entry>
+ <entry><para> Adds '/', '@' or '=' for directories, symlinks and writeable files, respectively.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314755" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314757" xreflabel=""/>-<emphasis role="bold">R</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314759" xreflabel=""/>Recursively lists all parameters under the specified path. If <emphasis>param_path</emphasis> is unspecified, all parameters are shown.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> -<emphasis role="bold">R</emphasis></para></entry>
+ <entry><para> Recursively lists all parameters under the specified path. If <emphasis>param_path</emphasis> is unspecified, all parameters are shown.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314765" xreflabel=""/><emphasis role="bold">get_param</emphasis><emphasis>[-n|-N|-F]</emphasis> <<emphasis>param_path ...</emphasis>></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314769" xreflabel=""/>Gets the value of a Lustre or LNET parameter from the specified path.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">get_param</emphasis><emphasis>[-n|-N|-F]</emphasis> <<emphasis>param_path ...</emphasis>></para></entry>
+ <entry><para> Gets the value of a Lustre or LNET parameter from the specified path.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314771" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314773" xreflabel=""/><emphasis role="bold">-n</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314775" xreflabel=""/>Prints only the parameter value and not the parameter name.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">-n</emphasis></para></entry>
+ <entry><para> Prints only the parameter value and not the parameter name.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314777" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314779" xreflabel=""/><emphasis role="bold">-N</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314781" xreflabel=""/>Prints only matched parameter names and not the values; especially useful when using patterns.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">-N</emphasis></para></entry>
+ <entry><para> Prints only matched parameter names and not the values; especially useful when using patterns.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314783" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314785" xreflabel=""/><emphasis role="bold">-F</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314787" xreflabel=""/>When <emphasis role="bold">-N</emphasis> is specified, adds '/', '@' or '=' for directories, symlinks and writeable files, respectively.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">-F</emphasis></para></entry>
+ <entry><para> When <emphasis role="bold">-N</emphasis> is specified, adds '/', '@' or '=' for directories, symlinks and writeable files, respectively.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314789" xreflabel=""/><emphasis role="bold">set_param</emphasis><emphasis>[-n]</emphasis><emphasis><param_path=value...></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314793" xreflabel=""/>Sets the value of a Lustre or LNET parameter from the specified path.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">set_param</emphasis><emphasis>[-n]</emphasis><emphasis><param_path=value...></emphasis></para></entry>
+ <entry><para> Sets the value of a Lustre or LNET parameter from the specified path.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314812" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314814" xreflabel=""/><emphasis role="bold">-n</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314816" xreflabel=""/>Disables printing of the key name when printing values.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">-n</emphasis></para></entry>
+ <entry><para> Disables printing of the key name when printing values.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314801" xreflabel=""/><emphasis role="bold">conf_param</emphasis><emphasis>[-d] <device|fsname>.<parameter>=<value></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314805" xreflabel=""/>Sets a permanent configuration parameter for any device via the MGS. This command must be run on the MGS node.</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1314806" xreflabel=""/>All writeable parameters under <emphasis role="bold">lctl list_param</emphasis> (e.g. <emphasis>lctl list_param -F osc.*.* | grep</emphasis> =) can be permanently set using <emphasis role="bold">lctl conf_param</emphasis>, but the format is slightly different. For <emphasis role="bold">conf_param</emphasis>, the device is specified first, then the obdtype. Wildcards are not supported. Additionally, failover nodes may be added (or removed), and some system-wide parameters may be set as well (sys.at_max, sys.at_min, sys.at_extra, sys.at_early_margin, sys.at_history, sys.timeout, sys.ldlm_timeout). For system-wide parameters, <device> is ignored.</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1314807" xreflabel=""/>For more information on setting permanent parameters and <emphasis role="bold">lctl conf_param</emphasis> command examples, see <xref linkend="dbdoclet.50438194_64195"/> (Setting Permanent Parameters).</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">conf_param</emphasis><emphasis>[-d] <device|fsname>.<parameter>=<value></emphasis></para></entry>
+ <entry><para> Sets a permanent configuration parameter for any device via the MGS. This command must be run on the MGS node.</para><para>All writeable parameters under <emphasis role="bold">lctl list_param</emphasis> (e.g. <emphasis>lctl list_param -F osc.*.* | grep</emphasis> =) can be permanently set using <emphasis role="bold">lctl conf_param</emphasis>, but the format is slightly different. For <emphasis role="bold">conf_param</emphasis>, the device is specified first, then the obdtype. Wildcards are not supported. Additionally, failover nodes may be added (or removed), and some system-wide parameters may be set as well (sys.at_max, sys.at_min, sys.at_extra, sys.at_early_margin, sys.at_history, sys.timeout, sys.ldlm_timeout). For system-wide parameters, <device> is ignored.</para><para>For more information on setting permanent parameters and <emphasis role="bold">lctl conf_param</emphasis> command examples, see <xref linkend="dbdoclet.50438194_64195"/> (Setting Permanent Parameters).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314848" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314850" xreflabel=""/><emphasis role="bold">-d <device|fsname>.<parameter></emphasis></para><para><anchor xml:id="dbdoclet.50438219_pgfId-1314851" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314853" xreflabel=""/>Deletes a parameter setting (use the default value at the next restart). A null value for <emphasis role="bold"><value></emphasis> also deletes the parameter setting.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">-d <device|fsname>.<parameter></emphasis></para><para> </para></entry>
+ <entry><para> Deletes a parameter setting (use the default value at the next restart). A null value for <emphasis role="bold"><value></emphasis> also deletes the parameter setting.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314842" xreflabel=""/><emphasis role="bold">activate</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314846" xreflabel=""/>Re-activates an import after the deactivate operation. This setting is only effective until the next restart (see <emphasis role="bold">conf_param</emphasis>).</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">activate</emphasis></para></entry>
+ <entry><para> Re-activates an import after the deactivate operation. This setting is only effective until the next restart (see <emphasis role="bold">conf_param</emphasis>).</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314836" xreflabel=""/><emphasis role="bold">deactivate</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314840" xreflabel=""/>Deactivates an import, in particular meaning do not assign new file stripes to an OSC. Running lctl deactivate on the MDS stops new objects from being allocated on the OST. Running lctl deactivate on Lustre clients causes them to return -EIO when accessing objects on the OST instead of waiting for recovery.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">deactivate</emphasis></para></entry>
+ <entry><para> Deactivates an import, in particular meaning do not assign new file stripes to an OSC. Running lctl deactivate on the MDS stops new objects from being allocated on the OST. Running lctl deactivate on Lustre clients causes them to return -EIO when accessing objects on the OST instead of waiting for recovery.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314830" xreflabel=""/><emphasis role="bold">abort_recovery</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314834" xreflabel=""/>Aborts the recovery process on a re-starting MDT or OST.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">abort_recovery</emphasis></para></entry>
+ <entry><para> Aborts the recovery process on a re-starting MDT or OST.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
<note><para>Lustre tunables are not always accessible using the procfs interface, as it is platform-specific. As a solution, lctl {get,set,list}_param has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to /proc/{fs,sys}/{lustre,lnet}. For future portability, use lctl {get,set,list}_param instead.</para></note>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1312745" xreflabel=""/><emphasis role="bold">Virtual Block Device Operations</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1302906" xreflabel=""/>Lustre can emulate a virtual block device upon a regular file. This emulation is needed when you are trying to set up a swap space via the file.</para>
+ <para><emphasis role="bold">Virtual Block Device Operations</emphasis></para>
+ <para>Lustre can emulate a virtual block device upon a regular file. This emulation is needed when you are trying to set up a swap space via the file.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1307850" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1307854" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314859" xreflabel=""/><emphasis role="bold">blockdev_attach</emphasis><emphasis><file name> <device node></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314863" xreflabel=""/>Attaches a regular Lustre file to a block device. If the device node does not exist, <emphasis role="bold">lctl</emphasis> creates it. We recommend that you create the device node by <emphasis role="bold">lctl</emphasis> since the emulator uses a dynamical major number.</para></entry>
+ <entry><para> <emphasis role="bold">blockdev_attach</emphasis><emphasis><file name> <device node></emphasis></para></entry>
+ <entry><para> Attaches a regular Lustre file to a block device. If the device node does not exist, <emphasis role="bold">lctl</emphasis> creates it. We recommend that you create the device node by <emphasis role="bold">lctl</emphasis> since the emulator uses a dynamical major number.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314865" xreflabel=""/><emphasis role="bold">blockdev_detach</emphasis><emphasis><device node></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314869" xreflabel=""/>Detaches the virtual block device.</para></entry>
+ <entry><para> <emphasis role="bold">blockdev_detach</emphasis><emphasis><device node></emphasis></para></entry>
+ <entry><para> Detaches the virtual block device.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314871" xreflabel=""/><emphasis role="bold">blockdev_info</emphasis><emphasis><device node></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314875" xreflabel=""/>Provides information about the Lustre file attached to the device node.</para></entry>
+ <entry><para> <emphasis role="bold">blockdev_info</emphasis><emphasis><device node></emphasis></para></entry>
+ <entry><para> Provides information about the Lustre file attached to the device node.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1312907" xreflabel=""/><emphasis role="bold">Changelogs</emphasis></para>
+ <para><emphasis role="bold">Changelogs</emphasis></para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1314964" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1314966" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314995" xreflabel=""/><emphasis role="bold">changelog_register</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1314999" xreflabel=""/>Registers a new changelog user for a particular device. Changelog entries are not purged beyond a registered user's set point (see <emphasis role="bold">lfs changelog_clear</emphasis>).</para></entry>
+ <entry><para> <emphasis role="bold">changelog_register</emphasis></para></entry>
+ <entry><para> Registers a new changelog user for a particular device. Changelog entries are not purged beyond a registered user's set point (see <emphasis role="bold">lfs changelog_clear</emphasis>).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315001" xreflabel=""/><emphasis role="bold">changelog_deregister</emphasis><emphasis><id></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315005" xreflabel=""/>Unregisters an existing changelog user. If the user's "clear" record number is the minimum for the device, changelog records are purged until the next minimum.</para></entry>
+ <entry><para> <emphasis role="bold">changelog_deregister</emphasis><emphasis><id></emphasis></para></entry>
+ <entry><para> Unregisters an existing changelog user. If the user's "clear" record number is the minimum for the device, changelog records are purged until the next minimum.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1312864" xreflabel=""/><emphasis role="bold">Debug</emphasis></para>
+ <para><emphasis role="bold">Debug</emphasis></para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1307958" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1307962" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315007" xreflabel=""/><emphasis role="bold">debug_daemon</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315011" xreflabel=""/>Starts and stops the debug daemon, and controls the output filename and size.</para></entry>
+ <entry><para> <emphasis role="bold">debug_daemon</emphasis></para></entry>
+ <entry><para> Starts and stops the debug daemon, and controls the output filename and size.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315013" xreflabel=""/><emphasis role="bold">debug_kernel</emphasis><emphasis>[file] [raw]</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315017" xreflabel=""/>Dumps the kernel debug buffer to stdout or a file.</para></entry>
+ <entry><para> <emphasis role="bold">debug_kernel</emphasis><emphasis>[file] [raw]</emphasis></para></entry>
+ <entry><para> Dumps the kernel debug buffer to stdout or a file.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315019" xreflabel=""/><emphasis role="bold">debug_file</emphasis><emphasis><input> [output]</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315023" xreflabel=""/>Converts the kernel-dumped debug log from binary to plain text format.</para></entry>
+ <entry><para> <emphasis role="bold">debug_file</emphasis><emphasis><input> [output]</emphasis></para></entry>
+ <entry><para> Converts the kernel-dumped debug log from binary to plain text format.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315025" xreflabel=""/><emphasis role="bold">clear</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315029" xreflabel=""/>Clears the kernel debug buffer.</para></entry>
+ <entry><para> <emphasis role="bold">clear</emphasis></para></entry>
+ <entry><para> Clears the kernel debug buffer.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315035" xreflabel=""/><emphasis role="bold">mark</emphasis><emphasis><text></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315039" xreflabel=""/>Inserts marker text in the kernel debug buffer.</para></entry>
+ <entry><para> <emphasis role="bold">mark</emphasis><emphasis><text></emphasis></para></entry>
+ <entry><para> Inserts marker text in the kernel debug buffer.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315041" xreflabel=""/><emphasis role="bold">filter</emphasis><emphasis><subsystem id/debug mask></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315045" xreflabel=""/>Filters kernel debug messages by subsystem or mask.</para></entry>
+ <entry><para> <emphasis role="bold">filter</emphasis><emphasis><subsystem id/debug mask></emphasis></para></entry>
+ <entry><para> Filters kernel debug messages by subsystem or mask.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315047" xreflabel=""/><emphasis role="bold">show</emphasis><emphasis><subsystem id/debug mask></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315051" xreflabel=""/>Shows specific types of messages.</para></entry>
+ <entry><para> <emphasis role="bold">show</emphasis><emphasis><subsystem id/debug mask></emphasis></para></entry>
+ <entry><para> Shows specific types of messages.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315053" xreflabel=""/><emphasis role="bold">debug_list</emphasis><emphasis><subs/types></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315057" xreflabel=""/>Lists all subsystem and debug types.</para></entry>
+ <entry><para> <emphasis role="bold">debug_list</emphasis><emphasis><subs/types></emphasis></para></entry>
+ <entry><para> Lists all subsystem and debug types.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315059" xreflabel=""/><emphasis role="bold">modules</emphasis><emphasis><path></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315063" xreflabel=""/>Provides GDB-friendly module information.</para></entry>
+ <entry><para> <emphasis role="bold">modules</emphasis><emphasis><path></emphasis></para></entry>
+ <entry><para> Provides GDB-friendly module information.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1302703" xreflabel=""/>Options</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1303112" xreflabel=""/>Use the following options to invoke lctl.</para>
+ <title>Options</title>
+ <para>Use the following options to invoke lctl.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1308105" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1308109" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1308144" xreflabel=""/><emphasis role="bold">--device</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1308153" xreflabel=""/>Device to be used for the operation (specified by name or number). See device_list.</para></entry>
+ <entry><para> <emphasis role="bold">--device</emphasis></para></entry>
+ <entry><para> Device to be used for the operation (specified by name or number). See device_list.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1308162" xreflabel=""/><emphasis role="bold">--ignore_errors | ignore_errors</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1308171" xreflabel=""/>Ignores errors during script processing.</para></entry>
+ <entry><para> <emphasis role="bold">--ignore_errors | ignore_errors</emphasis></para></entry>
+ <entry><para> Ignores errors during script processing.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1303154" xreflabel=""/>Examples</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1315069" xreflabel=""/><emphasis role="bold">lctl</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1315070" xreflabel=""/>$ lctl
-<anchor xml:id="dbdoclet.50438219_pgfId-1315071" xreflabel=""/>lctl > dl
-<anchor xml:id="dbdoclet.50438219_pgfId-1315072" xreflabel=""/> 0 UP mgc MGC192.168.0.20@tcp btbb24e3-7deb-2ffa-eab0-44dffe00f692 5
-<anchor xml:id="dbdoclet.50438219_pgfId-1315073" xreflabel=""/> 1 UP ost OSS OSS_uuid 3
-<anchor xml:id="dbdoclet.50438219_pgfId-1315074" xreflabel=""/> 2 UP obdfilter testfs-OST0000 testfs-OST0000_UUID 3
-<anchor xml:id="dbdoclet.50438219_pgfId-1315075" xreflabel=""/>lctl > dk /tmp/log Debug log: 87 lines, 87 kept, 0 dropped.
-<anchor xml:id="dbdoclet.50438219_pgfId-1315076" xreflabel=""/>lctl > quit
+ <title>Examples</title>
+ <para><emphasis role="bold">lctl</emphasis></para>
+ <screen>$ lctl
+lctl > dl
+ 0 UP mgc MGC192.168.0.20@tcp btbb24e3-7deb-2ffa-eab0-44dffe00f692 5
+ 1 UP ost OSS OSS_uuid 3
+ 2 UP obdfilter testfs-OST0000 testfs-OST0000_UUID 3
+lctl > dk /tmp/log Debug log: 87 lines, 87 kept, 0 dropped.
+lctl > quit
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1315077" xreflabel=""/>See Also</title>
+ <title>See Also</title>
<itemizedlist>
<listitem>
<para> <xref linkend="dbdoclet.50438219_75432"/> </para>
</section>
<section xml:id="dbdoclet.50438219_58217">
<title>36.4 ll_decode_filter_fid</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318305" xreflabel=""/>The ll_decode_filter_fid utility displays the Lustre object ID and MDT parent FID.</para>
+ <para>The ll_decode_filter_fid utility displays the Lustre object ID and MDT parent FID.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318306" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318307" xreflabel=""/>ll_decode_filter_fid object_file [object_file ...]
+ <title>Synopsis</title>
+ <screen>ll_decode_filter_fid object_file [object_file ...]
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318308" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318309" xreflabel=""/>The ll_decode_filter_fid utility decodes and prints the Lustre OST object ID, MDT FID, stripe index for the specified OST object(s), which is stored in the "trusted.fid" attribute on each OST object. This is accessible to ll_decode_filter_fid when the OST filesystem is mounted locally as type ldiskfs for maintenance.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318310" xreflabel=""/>The "trusted.fid" extended attribute is stored on each OST object when it is first modified (data written or attributes set), and is not accessed or modified by Lustre after that time.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318311" xreflabel=""/>The OST object ID (objid) is useful in case of OST directory corruption, though normally the ll_recover_lost_found_objs(8) utility is able to reconstruct the entire OST object directory hierarchy. The MDS FID can be useful to determine which MDS inode an OST object is (or was) used by. The stripe index can be used in conjunction with other OST objects to reconstruct the layout of a file even if the MDT inode was lost.</para>
+ <title>Description</title>
+ <para>The ll_decode_filter_fid utility decodes and prints the Lustre OST object ID, MDT FID, stripe index for the specified OST object(s), which is stored in the "trusted.fid" attribute on each OST object. This is accessible to ll_decode_filter_fid when the OST filesystem is mounted locally as type ldiskfs for maintenance.</para>
+ <para>The "trusted.fid" extended attribute is stored on each OST object when it is first modified (data written or attributes set), and is not accessed or modified by Lustre after that time.</para>
+ <para>The OST object ID (objid) is useful in case of OST directory corruption, though normally the ll_recover_lost_found_objs(8) utility is able to reconstruct the entire OST object directory hierarchy. The MDS FID can be useful to determine which MDS inode an OST object is (or was) used by. The stripe index can be used in conjunction with other OST objects to reconstruct the layout of a file even if the MDT inode was lost.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318312" xreflabel=""/>Examples</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318313" xreflabel=""/>root@oss1# cd /mnt/ost/lost+found
-<anchor xml:id="dbdoclet.50438219_pgfId-1318314" xreflabel=""/>root@oss1# ll_decode_filter_fid #12345[4,5,8]
-<anchor xml:id="dbdoclet.50438219_pgfId-1318315" xreflabel=""/>#123454: objid=690670 seq=0 parent=[0x751c5:0xfce6e605:0x0]
-<anchor xml:id="dbdoclet.50438219_pgfId-1318316" xreflabel=""/>#123455: objid=614725 seq=0 parent=[0x18d11:0xebba84eb:0x1]
-<anchor xml:id="dbdoclet.50438219_pgfId-1318317" xreflabel=""/>#123458: objid=533088 seq=0 parent=[0x21417:0x19734d61:0x0]
+ <title>Examples</title>
+ <screen>root@oss1# cd /mnt/ost/lost+found
+root@oss1# ll_decode_filter_fid #12345[4,5,8]
+#123454: objid=690670 seq=0 parent=[0x751c5:0xfce6e605:0x0]
+#123455: objid=614725 seq=0 parent=[0x18d11:0xebba84eb:0x1]
+#123458: objid=533088 seq=0 parent=[0x21417:0x19734d61:0x0]
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318319" xreflabel=""/>This shows that the three files in lost+found have decimal object IDs - 690670, 614725, and 533088, respectively. The object sequence number (formerly object group) is 0 for all current OST objects.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318320" xreflabel=""/>The MDT parent inode FIDs are hexdecimal numbers of the form sequence:oid:idx. Since the sequence number is below 0x100000000 in all these cases, the FIDs are in the legacy Inode and Generation In FID (IGIF) namespace and are mapped directly to the MDT inode = seq and generation = oid values; the MDT inodes are 0x751c5, 0x18d11, and 0x21417 respectively. For objects with MDT parent sequence numbers above 0x200000000, this indicates that the FID needs to be mapped via the MDT Object Index (OI) file on the MDT to determine the internal inode number.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318321" xreflabel=""/>The idx field shows the stripe number of this OST object in the Lustre RAID-0 striped file.</para>
+ <para>This shows that the three files in lost+found have decimal object IDs - 690670, 614725, and 533088, respectively. The object sequence number (formerly object group) is 0 for all current OST objects.</para>
+ <para>The MDT parent inode FIDs are hexdecimal numbers of the form sequence:oid:idx. Since the sequence number is below 0x100000000 in all these cases, the FIDs are in the legacy Inode and Generation In FID (IGIF) namespace and are mapped directly to the MDT inode = seq and generation = oid values; the MDT inodes are 0x751c5, 0x18d11, and 0x21417 respectively. For objects with MDT parent sequence numbers above 0x200000000, this indicates that the FID needs to be mapped via the MDT Object Index (OI) file on the MDT to determine the internal inode number.</para>
+ <para>The idx field shows the stripe number of this OST object in the Lustre RAID-0 striped file.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318322" xreflabel=""/>See Also</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318326" xreflabel=""/><xref linkend="dbdoclet.50438219_44971"/></para>
+ <title>See Also</title>
+ <para><xref linkend="dbdoclet.50438219_44971"/></para>
</section>
</section>
<section xml:id="dbdoclet.50438219_44971">
<title>36.5 ll_recover_lost_<anchor xml:id="dbdoclet.50438219_marker-1317865" xreflabel=""/>found_objs</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317867" xreflabel=""/>The ll_recover_lost_found_objs utility helps recover Lustre OST objects (file data) from a lost and found directory and return them to their correct locations.</para>
+ <para>The ll_recover_lost_found_objs utility helps recover Lustre OST objects (file data) from a lost and found directory and return them to their correct locations.</para>
<note><para>Running the ll_recover_lost_found_objs tool is not strictly necessary to bring an OST back online, it just avoids losing access to objects that were moved to the lost and found directory due to directory corruption.</para></note>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317869" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317870" xreflabel=""/>$ ll_recover_lost_found_objs [-hv] -d directory
+ <title>Synopsis</title>
+ <screen>$ ll_recover_lost_found_objs [-hv] -d directory
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317871" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317872" xreflabel=""/>The first time Lustre writes to an object, it saves the MDS inode number and the objid as an extended attribute on the object, so in case of directory corruption of the OST, it is possible to recover the objects. Running e2fsck fixes the corrupted OST directory, but it puts all of the objects into a lost and found directory, where they are inaccessible to Lustre. Use the ll_recover_lost_found_objs utility to recover all (or at least most) objects from a lost and found directory and return them to the O/0/d* directories.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317873" xreflabel=""/>To use ll_recover_lost_found_objs, mount the file system locally (using the -t ldiskfs command), run the utility and then unmount it again. The OST must not be mounted by Lustre when ll_recover_lost_found_objs is run.</para>
+ <title>Description</title>
+ <para>The first time Lustre writes to an object, it saves the MDS inode number and the objid as an extended attribute on the object, so in case of directory corruption of the OST, it is possible to recover the objects. Running e2fsck fixes the corrupted OST directory, but it puts all of the objects into a lost and found directory, where they are inaccessible to Lustre. Use the ll_recover_lost_found_objs utility to recover all (or at least most) objects from a lost and found directory and return them to the O/0/d* directories.</para>
+ <para>To use ll_recover_lost_found_objs, mount the file system locally (using the -t ldiskfs command), run the utility and then unmount it again. The OST must not be mounted by Lustre when ll_recover_lost_found_objs is run.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317892" xreflabel=""/>Options</title>
+ <title>Options</title>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317876" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317878" xreflabel=""/> <anchor xml:id="dbdoclet.50438219_pgfId-1317879" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold"> Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317881" xreflabel=""/><emphasis role="bold">-h</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317883" xreflabel=""/>Prints a help message</para></entry>
+ <entry><para> <emphasis role="bold">-h</emphasis></para></entry>
+ <entry><para> Prints a help message</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317885" xreflabel=""/><emphasis role="bold">-v</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317887" xreflabel=""/>Increases verbosity</para></entry>
+ <entry><para> <emphasis role="bold">-v</emphasis></para></entry>
+ <entry><para> Increases verbosity</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317889" xreflabel=""/><emphasis role="bold">-d</emphasis><emphasis>directory</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317891" xreflabel=""/>Sets the lost and found directory path</para></entry>
+ <entry><para> <emphasis role="bold">-d</emphasis><emphasis>directory</emphasis></para></entry>
+ <entry><para> Sets the lost and found directory path</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317893" xreflabel=""/>Example</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317894" xreflabel=""/>ll_recover_lost_found_objs -d /mnt/ost/lost+found
+ <title>Example</title>
+ <screen>ll_recover_lost_found_objs -d /mnt/ost/lost+found
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438219_84890">
<title>36.6 llobdstat</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1316535" xreflabel=""/>The llobdstat utility displays OST statistics.</para>
+ <para>The llobdstat utility displays OST statistics.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1316536" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1316537" xreflabel=""/>llobdstat ost_name [interval]
+ <title>Synopsis</title>
+ <screen>llobdstat ost_name [interval]
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1316538" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1316539" xreflabel=""/>The llobdstat utility displays a line of OST statistics for the given ost_name every interval seconds. It should be run directly on an OSS node. Type <emphasis>CTRL-C</emphasis> to stop statistics printing.</para>
+ <title>Description</title>
+ <para>The llobdstat utility displays a line of OST statistics for the given ost_name every interval seconds. It should be run directly on an OSS node. Type <emphasis>CTRL-C</emphasis> to stop statistics printing.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1316540" xreflabel=""/>Example</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1316541" xreflabel=""/># llobdstat liane-OST0002 1
-<anchor xml:id="dbdoclet.50438219_pgfId-1316542" xreflabel=""/>/usr/bin/llobdstat on /proc/fs/lustre/obdfilter/liane-OST0002/stats
-<anchor xml:id="dbdoclet.50438219_pgfId-1316543" xreflabel=""/>Processor counters run at 2800.189 MHz
-<anchor xml:id="dbdoclet.50438219_pgfId-1316544" xreflabel=""/>Read: 1.21431e+07, Write: 9.93363e+08, create/destroy: 24/1499, stat: 34, p\
+ <title>Example</title>
+ <screen># llobdstat liane-OST0002 1
+/usr/bin/llobdstat on /proc/fs/lustre/obdfilter/liane-OST0002/stats
+Processor counters run at 2800.189 MHz
+Read: 1.21431e+07, Write: 9.93363e+08, create/destroy: 24/1499, stat: 34, p\
unch: 18
-<anchor xml:id="dbdoclet.50438219_pgfId-1316545" xreflabel=""/>[NOTE: cx: create, dx: destroy, st: statfs, pu: punch ]
-<anchor xml:id="dbdoclet.50438219_pgfId-1316546" xreflabel=""/>Timestamp Read-delta ReadRate Write-delta WriteRate
-<anchor xml:id="dbdoclet.50438219_pgfId-1316547" xreflabel=""/>--------------------------------------------------------
-<anchor xml:id="dbdoclet.50438219_pgfId-1316548" xreflabel=""/>1217026053 0.00MB 0.00MB/s 0.00MB 0.00MB/s
-<anchor xml:id="dbdoclet.50438219_pgfId-1316549" xreflabel=""/>1217026054 0.00MB 0.00MB/s 0.00MB 0.00MB/s
-<anchor xml:id="dbdoclet.50438219_pgfId-1316550" xreflabel=""/>1217026055 0.00MB 0.00MB/s 0.00MB 0.00MB/s
-<anchor xml:id="dbdoclet.50438219_pgfId-1316551" xreflabel=""/>1217026056 0.00MB 0.00MB/s 0.00MB 0.00MB/s
-<anchor xml:id="dbdoclet.50438219_pgfId-1316552" xreflabel=""/>1217026057 0.00MB 0.00MB/s 0.00MB 0.00MB/s
-<anchor xml:id="dbdoclet.50438219_pgfId-1316553" xreflabel=""/>1217026058 0.00MB 0.00MB/s 0.00MB 0.00MB/s
-<anchor xml:id="dbdoclet.50438219_pgfId-1316554" xreflabel=""/>1217026059 0.00MB 0.00MB/s 0.00MB 0.00MB/s st:1
+[NOTE: cx: create, dx: destroy, st: statfs, pu: punch ]
+Timestamp Read-delta ReadRate Write-delta WriteRate
+--------------------------------------------------------
+1217026053 0.00MB 0.00MB/s 0.00MB 0.00MB/s
+1217026054 0.00MB 0.00MB/s 0.00MB 0.00MB/s
+1217026055 0.00MB 0.00MB/s 0.00MB 0.00MB/s
+1217026056 0.00MB 0.00MB/s 0.00MB 0.00MB/s
+1217026057 0.00MB 0.00MB/s 0.00MB 0.00MB/s
+1217026058 0.00MB 0.00MB/s 0.00MB 0.00MB/s
+1217026059 0.00MB 0.00MB/s 0.00MB 0.00MB/s st:1
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1316555" xreflabel=""/>Files</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1316556" xreflabel=""/>/proc/fs/lustre/obdfilter/<ostname>/stats
+ <title>Files</title>
+ <screen>/proc/fs/lustre/obdfilter/<ostname>/stats
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438219_90386">
<title>36.7 llog_reader</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318200" xreflabel=""/>The llog_reader utility parses Lustre's on-disk configuration logs.</para>
+ <para>The llog_reader utility parses Lustre's on-disk configuration logs.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318201" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318202" xreflabel=""/>llog_reader filename
+ <title>Synopsis</title>
+ <screen>llog_reader filename
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318203" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318204" xreflabel=""/>The llog_reader utility parses the binary format of Lustre's on-disk configuration logs. Llog_reader can only read logs; use tunefs.lustre to write to them.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318205" xreflabel=""/>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs, then use llog_reader to dump the log file's contents, for example:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318206" xreflabel=""/>mount -t ldiskfs /dev/sda /mnt/mgs
-<anchor xml:id="dbdoclet.50438219_pgfId-1318207" xreflabel=""/>llog_reader /mnt/mgs/CONFIGS/tfs-client
+ <title>Description</title>
+ <para>The llog_reader utility parses the binary format of Lustre's on-disk configuration logs. Llog_reader can only read logs; use tunefs.lustre to write to them.</para>
+ <para>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs, then use llog_reader to dump the log file's contents, for example:</para>
+ <screen>mount -t ldiskfs /dev/sda /mnt/mgs
+llog_reader /mnt/mgs/CONFIGS/tfs-client
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318208" xreflabel=""/>To examine the same log file on a running Lustre server, use the ldiskfs-enabled debugfs utility (called debug.ldiskfs on some distributions) to extract the file, for example:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318209" xreflabel=""/>debugfs -c -R 'dump CONFIGS/tfs-client /tmp/tfs-client' /dev/sda
-<anchor xml:id="dbdoclet.50438219_pgfId-1318210" xreflabel=""/>llog_reader /tmp/tfs-client
+ <para>To examine the same log file on a running Lustre server, use the ldiskfs-enabled debugfs utility (called debug.ldiskfs on some distributions) to extract the file, for example:</para>
+ <screen>debugfs -c -R 'dump CONFIGS/tfs-client /tmp/tfs-client' /dev/sda
+llog_reader /tmp/tfs-client
</screen>
<informaltable frame="none">
<tgroup cols="2">
<imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
</imageobject>
</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis><anchor xml:id="dbdoclet.50438219_pgfId-1318211" xreflabel=""/>Although they are stored in the CONFIGS directory, mountdata files do not use the configuration log format and will confuse the llog_reader utility.</para></entry>
+ <entry><para><emphasis role="bold">Caution -</emphasis>Although they are stored in the CONFIGS directory, mountdata files do not use the configuration log format and will confuse the llog_reader utility.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318212" xreflabel=""/>See Also</title>
+ <title>See Also</title>
<para><xref linkend="dbdoclet.50438219_39574"/></para>
</section>
</section>
<section xml:id="dbdoclet.50438219_23232">
<title>36.8 ll<anchor xml:id="dbdoclet.50438219_marker-1318052" xreflabel=""/>stat</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318054" xreflabel=""/>The llstat utility displays Lustre statistics.</para>
+ <para>The llstat utility displays Lustre statistics.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318055" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318056" xreflabel=""/>llstat [-c] [-g] [-i interval] stats_file
+ <title>Synopsis</title>
+ <screen>llstat [-c] [-g] [-i interval] stats_file
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318057" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318058" xreflabel=""/>The llstat utility displays statistics from any of the Lustre statistics files that share a common format and are updated at <emphasis>interval</emphasis> seconds. To stop statistics printing, use <emphasis>ctrl</emphasis>-<emphasis>c.</emphasis></para>
+ <title>Description</title>
+ <para>The llstat utility displays statistics from any of the Lustre statistics files that share a common format and are updated at <emphasis>interval</emphasis> seconds. To stop statistics printing, use <emphasis>ctrl</emphasis>-<emphasis>c.</emphasis></para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318085" xreflabel=""/>Options</title>
+ <title>Options</title>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1318061" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1318063" xreflabel=""/> <anchor xml:id="dbdoclet.50438219_pgfId-1318064" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold"> Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318066" xreflabel=""/><emphasis role="bold">-c</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318068" xreflabel=""/>Clears the statistics file.</para></entry>
+ <entry><para> <emphasis role="bold">-c</emphasis></para></entry>
+ <entry><para> Clears the statistics file.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318070" xreflabel=""/><emphasis role="bold">-i</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318072" xreflabel=""/>Specifies the polling period (in seconds).</para></entry>
+ <entry><para> <emphasis role="bold">-i</emphasis></para></entry>
+ <entry><para> Specifies the polling period (in seconds).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318074" xreflabel=""/><emphasis role="bold">-g</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318076" xreflabel=""/>Specifies graphable output format.</para></entry>
+ <entry><para> <emphasis role="bold">-g</emphasis></para></entry>
+ <entry><para> Specifies graphable output format.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318078" xreflabel=""/><emphasis role="bold">-h</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318080" xreflabel=""/>Displays help information.</para></entry>
+ <entry><para> <emphasis role="bold">-h</emphasis></para></entry>
+ <entry><para> Displays help information.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318082" xreflabel=""/><emphasis role="bold">stats_file</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318084" xreflabel=""/>Specifies either the full path to a statistics file or the shorthand reference, <emphasis role="bold">mds</emphasis> or <emphasis role="bold">ost</emphasis></para></entry>
+ <entry><para> <emphasis role="bold">stats_file</emphasis></para></entry>
+ <entry><para> Specifies either the full path to a statistics file or the shorthand reference, <emphasis role="bold">mds</emphasis> or <emphasis role="bold">ost</emphasis></para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318086" xreflabel=""/>Example</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318087" xreflabel=""/>To monitor /proc/fs/lustre/ost/OSS/ost/stats at 1 second intervals, run;</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318088" xreflabel=""/>llstat -i 1 ost
+ <title>Example</title>
+ <para>To monitor /proc/fs/lustre/ost/OSS/ost/stats at 1 second intervals, run;</para>
+ <screen>llstat -i 1 ost
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318089" xreflabel=""/>Files</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318090" xreflabel=""/>The llstat files are located at:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318091" xreflabel=""/>/proc/fs/lustre/mdt/MDS/*/stats
-<anchor xml:id="dbdoclet.50438219_pgfId-1318092" xreflabel=""/>/proc/fs/lustre/mds/*/exports/*/stats
-<anchor xml:id="dbdoclet.50438219_pgfId-1318093" xreflabel=""/>/proc/fs/lustre/mdc/*/stats
-<anchor xml:id="dbdoclet.50438219_pgfId-1318094" xreflabel=""/>/proc/fs/lustre/ldlm/services/*/stats
-<anchor xml:id="dbdoclet.50438219_pgfId-1318095" xreflabel=""/>/proc/fs/lustre/ldlm/namespaces/*/pool/stats
-<anchor xml:id="dbdoclet.50438219_pgfId-1318096" xreflabel=""/>/proc/fs/lustre/mgs/MGS/exports/*/stats
-<anchor xml:id="dbdoclet.50438219_pgfId-1318097" xreflabel=""/>/proc/fs/lustre/ost/OSS/*/stats
-<anchor xml:id="dbdoclet.50438219_pgfId-1318098" xreflabel=""/>/proc/fs/lustre/osc/*/stats
-<anchor xml:id="dbdoclet.50438219_pgfId-1318099" xreflabel=""/>/proc/fs/lustre/obdfilter/*/exports/*/stats
-<anchor xml:id="dbdoclet.50438219_pgfId-1318100" xreflabel=""/>/proc/fs/lustre/obdfilter/*/stats
-<anchor xml:id="dbdoclet.50438219_pgfId-1318101" xreflabel=""/>/proc/fs/lustre/llite/*/stats
+ <title>Files</title>
+ <para>The llstat files are located at:</para>
+ <screen>/proc/fs/lustre/mdt/MDS/*/stats
+/proc/fs/lustre/mds/*/exports/*/stats
+/proc/fs/lustre/mdc/*/stats
+/proc/fs/lustre/ldlm/services/*/stats
+/proc/fs/lustre/ldlm/namespaces/*/pool/stats
+/proc/fs/lustre/mgs/MGS/exports/*/stats
+/proc/fs/lustre/ost/OSS/*/stats
+/proc/fs/lustre/osc/*/stats
+/proc/fs/lustre/obdfilter/*/exports/*/stats
+/proc/fs/lustre/obdfilter/*/stats
+/proc/fs/lustre/llite/*/stats
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438219_23648">
<title>36.9 llverdev<anchor xml:id="dbdoclet.50438219_marker-1317750" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317752" xreflabel=""/>The llverdev verifies a block device is functioning properly over its full size.</para>
+ <para>The llverdev verifies a block device is functioning properly over its full size.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317753" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317754" xreflabel=""/>llverdev [-c <emphasis>chunksize</emphasis>] [-f] [-h] [-o <emphasis>offset</emphasis>] [-l] [-p] [-r] [-t <emphasis>timestamp</emphasis>] [-v] [-w] <emphasis>device</emphasis></screen>
+ <title>Synopsis</title>
+ <screen>llverdev [-c <emphasis>chunksize</emphasis>] [-f] [-h] [-o <emphasis>offset</emphasis>] [-l] [-p] [-r] [-t <emphasis>timestamp</emphasis>] [-v] [-w] <emphasis>device</emphasis></screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317755" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317756" xreflabel=""/>Sometimes kernel drivers or hardware devices have bugs that prevent them from accessing the full device size correctly, or possibly have bad sectors on disk or other problems which prevent proper data storage. There are often defects associated with major system boundaries such as 2^32 bytes, 2^31 sectors, 2^31 blocks, 2^32 blocks, etc.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317757" xreflabel=""/>The llverdev utility writes and verifies a unique test pattern across the entire device to ensure that data is accessible after it was written, and that data written to one part of the disk is not overwriting data on another part of the disk.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317758" xreflabel=""/>It is expected that llverdev will be run on large size devices (TB). It is always better to run llverdev in verbose mode, so that device testing can be easily restarted from the point where it was stopped.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317759" xreflabel=""/>Running a full verification can be time-consuming for very large devices. We recommend starting with a partial verification to ensure that the device is minimally sane before investing in a full verification.</para>
+ <title>Description</title>
+ <para>Sometimes kernel drivers or hardware devices have bugs that prevent them from accessing the full device size correctly, or possibly have bad sectors on disk or other problems which prevent proper data storage. There are often defects associated with major system boundaries such as 2^32 bytes, 2^31 sectors, 2^31 blocks, 2^32 blocks, etc.</para>
+ <para>The llverdev utility writes and verifies a unique test pattern across the entire device to ensure that data is accessible after it was written, and that data written to one part of the disk is not overwriting data on another part of the disk.</para>
+ <para>It is expected that llverdev will be run on large size devices (TB). It is always better to run llverdev in verbose mode, so that device testing can be easily restarted from the point where it was stopped.</para>
+ <para>Running a full verification can be time-consuming for very large devices. We recommend starting with a partial verification to ensure that the device is minimally sane before investing in a full verification.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317827" xreflabel=""/>Options</title>
+ <title>Options</title>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
<colspec colname="c3" colwidth="33*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317762" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317764" xreflabel=""/> </emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317766" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold"> </emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317768" xreflabel=""/><emphasis role="bold">-c|--chunksize</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317772" xreflabel=""/>I/O chunk size in bytes (default value is 1048576).</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-c|--chunksize</emphasis></para></entry>
+ <entry><para> I/O chunk size in bytes (default value is 1048576).</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317774" xreflabel=""/><emphasis role="bold">-f|--force</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317778" xreflabel=""/>Forces the test to run without a confirmation that the device will be overwritten and all data will be permanently destroyed.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-f|--force</emphasis></para></entry>
+ <entry><para> Forces the test to run without a confirmation that the device will be overwritten and all data will be permanently destroyed.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317780" xreflabel=""/><emphasis role="bold">-h|--help</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317784" xreflabel=""/>Displays a brief help message.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-h|--help</emphasis></para></entry>
+ <entry><para> Displays a brief help message.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317786" xreflabel=""/><emphasis role="bold">-o</emphasis><emphasis>offset</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317790" xreflabel=""/>Offset (in kilobytes) of the start of the test (default value is 0).</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-o</emphasis><emphasis>offset</emphasis></para></entry>
+ <entry><para> Offset (in kilobytes) of the start of the test (default value is 0).</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317792" xreflabel=""/><emphasis role="bold">-l|--long</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317796" xreflabel=""/>Runs a full check, writing and then reading and verifying every block on the disk.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-l|--long</emphasis></para></entry>
+ <entry><para> Runs a full check, writing and then reading and verifying every block on the disk.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317798" xreflabel=""/><emphasis role="bold">-p|--partial</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317802" xreflabel=""/>Runs a partial check, only doing periodic checks across the device (1 GB steps).</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-p|--partial</emphasis></para></entry>
+ <entry><para> Runs a partial check, only doing periodic checks across the device (1 GB steps).</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317804" xreflabel=""/><emphasis role="bold">-r|--read</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317808" xreflabel=""/>Runs the test in read (verify) mode only, after having previously run the test in <emphasis role="bold">-w</emphasis> mode.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-r|--read</emphasis></para></entry>
+ <entry><para> Runs the test in read (verify) mode only, after having previously run the test in <emphasis role="bold">-w</emphasis> mode.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317810" xreflabel=""/><emphasis role="bold">-t</emphasis><emphasis>timestamp</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317814" xreflabel=""/>Sets the test start time as printed at the start of a previously-interrupted test to ensure that validation data is the same across the entire filesystem (default value is the current time()).</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-t</emphasis><emphasis>timestamp</emphasis></para></entry>
+ <entry><para> Sets the test start time as printed at the start of a previously-interrupted test to ensure that validation data is the same across the entire filesystem (default value is the current time()).</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317816" xreflabel=""/><emphasis role="bold">-v|--verbose</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317820" xreflabel=""/>Runs the test in verbose mode, listing each read and write operation.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-v|--verbose</emphasis></para></entry>
+ <entry><para> Runs the test in verbose mode, listing each read and write operation.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317822" xreflabel=""/><emphasis role="bold">-w|--write</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317826" xreflabel=""/>Runs the test in write (test-pattern) mode (default runs both read and write).</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">-w|--write</emphasis></para></entry>
+ <entry><para> Runs the test in write (test-pattern) mode (default runs both read and write).</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317828" xreflabel=""/>Examples</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317829" xreflabel=""/>Runs a partial device verification on /dev/sda:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317830" xreflabel=""/>llverdev -v -p /dev/sda
-<anchor xml:id="dbdoclet.50438219_pgfId-1317831" xreflabel=""/>llverdev: permanently overwrite all data on /dev/sda (yes/no)? y
-<anchor xml:id="dbdoclet.50438219_pgfId-1317832" xreflabel=""/>llverdev: /dev/sda is 4398046511104 bytes (4096.0 GB) in size
-<anchor xml:id="dbdoclet.50438219_pgfId-1317833" xreflabel=""/>Timestamp: 1009839028
-<anchor xml:id="dbdoclet.50438219_pgfId-1317834" xreflabel=""/>Current write offset: 4096 kB
+ <title>Examples</title>
+ <para>Runs a partial device verification on /dev/sda:</para>
+ <screen>llverdev -v -p /dev/sda
+llverdev: permanently overwrite all data on /dev/sda (yes/no)? y
+llverdev: /dev/sda is 4398046511104 bytes (4096.0 GB) in size
+Timestamp: 1009839028
+Current write offset: 4096 kB
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317835" xreflabel=""/>Continues an interrupted verification at offset 4096kB from the start of the device, using the same timestamp as the previous run:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317836" xreflabel=""/>llverdev -f -v -p --offset=4096 --timestamp=1009839028 /dev/sda
-<anchor xml:id="dbdoclet.50438219_pgfId-1317837" xreflabel=""/>llverdev: /dev/sda is 4398046511104 bytes (4096.0 GB) in size
-<anchor xml:id="dbdoclet.50438219_pgfId-1317838" xreflabel=""/>Timestamp: 1009839028
-<anchor xml:id="dbdoclet.50438219_pgfId-1317839" xreflabel=""/>write complete
-<anchor xml:id="dbdoclet.50438219_pgfId-1317840" xreflabel=""/>read complete
+ <para>Continues an interrupted verification at offset 4096kB from the start of the device, using the same timestamp as the previous run:</para>
+ <screen>llverdev -f -v -p --offset=4096 --timestamp=1009839028 /dev/sda
+llverdev: /dev/sda is 4398046511104 bytes (4096.0 GB) in size
+Timestamp: 1009839028
+write complete
+read complete
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438219_64286">
<title>36.10 lshowmount<anchor xml:id="dbdoclet.50438219_marker-1317690" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317692" xreflabel=""/>The lshowmount utility shows Lustre exports.</para>
+ <para>The lshowmount utility shows Lustre exports.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317693" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317694" xreflabel=""/>lshowmount [-ehlv]
+ <title>Synopsis</title>
+ <screen>lshowmount [-ehlv]
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317695" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317696" xreflabel=""/>The lshowmount utility shows the hosts that have Lustre mounted to a server. Ths utility looks for exports from the MGS, MDS, and obdfilter.</para>
+ <title>Description</title>
+ <para>The lshowmount utility shows the hosts that have Lustre mounted to a server. Ths utility looks for exports from the MGS, MDS, and obdfilter.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317728" xreflabel=""/>Options</title>
+ <title>Options</title>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317699" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317703" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317705" xreflabel=""/><emphasis role="bold">-e|--enumerate</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317709" xreflabel=""/>Causes lshowmount to list each client mounted on a separate line instead of trying to compress the list of clients into a hostrange string.</para></entry>
+ <entry><para> <emphasis role="bold">-e|--enumerate</emphasis></para></entry>
+ <entry><para> Causes lshowmount to list each client mounted on a separate line instead of trying to compress the list of clients into a hostrange string.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317711" xreflabel=""/><emphasis role="bold">-h|--help</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317715" xreflabel=""/>Causes lshowmount to print out a usage message.</para></entry>
+ <entry><para> <emphasis role="bold">-h|--help</emphasis></para></entry>
+ <entry><para> Causes lshowmount to print out a usage message.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317717" xreflabel=""/><emphasis role="bold">-l|--lookup</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317721" xreflabel=""/>Causes lshowmount to try to look up the hostname for NIDs that look like IP addresses.</para></entry>
+ <entry><para> <emphasis role="bold">-l|--lookup</emphasis></para></entry>
+ <entry><para> Causes lshowmount to try to look up the hostname for NIDs that look like IP addresses.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317723" xreflabel=""/><emphasis role="bold">-v|--verbose</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317727" xreflabel=""/>Causes lshowmount to output export information for each service instead of only displaying the aggregate information for all Lustre services on the server.</para></entry>
+ <entry><para> <emphasis role="bold">-v|--verbose</emphasis></para></entry>
+ <entry><para> Causes lshowmount to output export information for each service instead of only displaying the aggregate information for all Lustre services on the server.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317729" xreflabel=""/>Files</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317730" xreflabel=""/>/proc/fs/lustre/mgs/<server>/exports/<uuid>/nid /proc/fs/lustre/mds/<server>/expo\
+ <title>Files</title>
+ <screen>/proc/fs/lustre/mgs/<server>/exports/<uuid>/nid /proc/fs/lustre/mds/<server>/expo\
rts/<uuid>/nid /proc/fs/lustre/obdfilter/<server>/exports/<uuid>/nid
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438219_90218">
<title>36.11 l<anchor xml:id="dbdoclet.50438219_marker-1318121" xreflabel=""/><anchor xml:id="dbdoclet.50438219_marker-1318122" xreflabel=""/>st</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318124" xreflabel=""/>The lst utility starts LNET self-test.</para>
+ <para>The lst utility starts LNET self-test.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318125" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318126" xreflabel=""/>lst
+ <title>Synopsis</title>
+ <screen>lst
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318127" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318128" xreflabel=""/>LNET self-test helps site administrators confirm that Lustre Networking (LNET) has been properly installed and configured. The self-test also confirms that LNET and the network software and hardware underlying it are performing as expected.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318129" xreflabel=""/>Each LNET self-test runs in the context of a session. A node can be associated with only one session at a time, to ensure that the session has exclusive use of the nodes on which it is running. A session is create, controlled and monitored from a single node; this is referred to as the self-test console.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318130" xreflabel=""/>Any node may act as the self-test console. Nodes are named and allocated to a self-test session in groups. This allows all nodes in a group to be referenced by a single name.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318131" xreflabel=""/>Test configurations are built by describing and running test batches. A test batch is a named collection of tests, with each test composed of a number of individual point-to-point tests running in parallel. These individual point-to-point tests are instantiated according to the test type, source group, target group and distribution specified when the test is added to the test batch.</para>
+ <title>Description</title>
+ <para>LNET self-test helps site administrators confirm that Lustre Networking (LNET) has been properly installed and configured. The self-test also confirms that LNET and the network software and hardware underlying it are performing as expected.</para>
+ <para>Each LNET self-test runs in the context of a session. A node can be associated with only one session at a time, to ensure that the session has exclusive use of the nodes on which it is running. A session is create, controlled and monitored from a single node; this is referred to as the self-test console.</para>
+ <para>Any node may act as the self-test console. Nodes are named and allocated to a self-test session in groups. This allows all nodes in a group to be referenced by a single name.</para>
+ <para>Test configurations are built by describing and running test batches. A test batch is a named collection of tests, with each test composed of a number of individual point-to-point tests running in parallel. These individual point-to-point tests are instantiated according to the test type, source group, target group and distribution specified when the test is added to the test batch.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318132" xreflabel=""/>Modules</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318133" xreflabel=""/>To run LNET self-test, load these modules: libcfs, lnet, lnet_selftest and any one of the klnds (ksocklnd, ko2iblnd...). To load all necessary modules, run modprobe lnet_selftest, which recursively loads the modules on which lnet_selftest depends.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318134" xreflabel=""/>There are two types of nodes for LNET self-test: the console node and test nodes. Both node types require all previously-specified modules to be loaded. (The userspace test node does not require these modules).</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318135" xreflabel=""/>Test nodes can be in either kernel or in userspace. A console user can invite a kernel test node to join the test session by running lst add_group NID, but the user cannot actively add a userspace test node to the test session. However, the console user can passively accept a test node to the test session while the test node runs lst client to connect to the console.</para>
+ <title>Modules</title>
+ <para>To run LNET self-test, load these modules: libcfs, lnet, lnet_selftest and any one of the klnds (ksocklnd, ko2iblnd...). To load all necessary modules, run modprobe lnet_selftest, which recursively loads the modules on which lnet_selftest depends.</para>
+ <para>There are two types of nodes for LNET self-test: the console node and test nodes. Both node types require all previously-specified modules to be loaded. (The userspace test node does not require these modules).</para>
+ <para>Test nodes can be in either kernel or in userspace. A console user can invite a kernel test node to join the test session by running lst add_group NID, but the user cannot actively add a userspace test node to the test session. However, the console user can passively accept a test node to the test session while the test node runs lst client to connect to the console.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318136" xreflabel=""/>Utilities</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318137" xreflabel=""/>LNET self-test includes two user utilities, lst and lstclient.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318138" xreflabel=""/>lst is the user interface for the self-test console (run on the console node). It provides a list of commands to control the entire test system, such as create session, create test groups, etc.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318139" xreflabel=""/>lstclient is the userspace self-test program which is linked with userspace LNDs and LNET. A user can invoke lstclient to join a self-test session:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318140" xreflabel=""/>lstclient -sesid CONSOLE_NID group NAME
+ <title>Utilities</title>
+ <para>LNET self-test includes two user utilities, lst and lstclient.</para>
+ <para>lst is the user interface for the self-test console (run on the console node). It provides a list of commands to control the entire test system, such as create session, create test groups, etc.</para>
+ <para>lstclient is the userspace self-test program which is linked with userspace LNDs and LNET. A user can invoke lstclient to join a self-test session:</para>
+ <screen>lstclient -sesid CONSOLE_NID group NAME
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318141" xreflabel=""/>Example Script</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318142" xreflabel=""/>This is a sample LNET self-test script which simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an IB network (connected via LNET routers), with half the clients reading and half the clients writing.</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318143" xreflabel=""/>#!/bin/bash
-<anchor xml:id="dbdoclet.50438219_pgfId-1318144" xreflabel=""/>export LST_SESSION=$$
-<anchor xml:id="dbdoclet.50438219_pgfId-1318145" xreflabel=""/>lst new_session read/write
-<anchor xml:id="dbdoclet.50438219_pgfId-1318146" xreflabel=""/>lst add_group servers 192.168.10.[8,10,12-16]@tcp
-<anchor xml:id="dbdoclet.50438219_pgfId-1318147" xreflabel=""/>lst add_group readers 192.168.1.[1-253/2]@o2ib
-<anchor xml:id="dbdoclet.50438219_pgfId-1318148" xreflabel=""/>lst add_group writers 192.168.1.[2-254/2]@o2ib
-<anchor xml:id="dbdoclet.50438219_pgfId-1318149" xreflabel=""/>lst add_batch bulk_rw
-<anchor xml:id="dbdoclet.50438219_pgfId-1318150" xreflabel=""/>lst add_test --batch bulk_rw --from readers --to servers brw read check\
+ <title>Example Script</title>
+ <para>This is a sample LNET self-test script which simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an IB network (connected via LNET routers), with half the clients reading and half the clients writing.</para>
+ <screen>#!/bin/bash
+export LST_SESSION=$$
+lst new_session read/write
+lst add_group servers 192.168.10.[8,10,12-16]@tcp
+lst add_group readers 192.168.1.[1-253/2]@o2ib
+lst add_group writers 192.168.1.[2-254/2]@o2ib
+lst add_batch bulk_rw
+lst add_test --batch bulk_rw --from readers --to servers brw read check\
=simple size=1M
-<anchor xml:id="dbdoclet.50438219_pgfId-1318151" xreflabel=""/>lst add_test --batch bulk_rw --from writers --to servers brw write chec\
+lst add_test --batch bulk_rw --from writers --to servers brw write chec\
k=full size=4K
-<anchor xml:id="dbdoclet.50438219_pgfId-1318152" xreflabel=""/># start running
-<anchor xml:id="dbdoclet.50438219_pgfId-1318153" xreflabel=""/>lst run bulk_rw
-<anchor xml:id="dbdoclet.50438219_pgfId-1318154" xreflabel=""/># display server stats for 30 seconds
-<anchor xml:id="dbdoclet.50438219_pgfId-1318155" xreflabel=""/>lst stat servers & sleep 30; kill $!
-<anchor xml:id="dbdoclet.50438219_pgfId-1318156" xreflabel=""/># tear down
-<anchor xml:id="dbdoclet.50438219_pgfId-1318157" xreflabel=""/>lst end_session
+# start running
+lst run bulk_rw
+# display server stats for 30 seconds
+lst stat servers & sleep 30; kill $!
+# tear down
+lst end_session
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438219_54734">
<title>36.12 lustre_<anchor xml:id="dbdoclet.50438219_marker-1317379" xreflabel=""/>rmmod.sh</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317381" xreflabel=""/>The lustre_rmmod.sh utility removes all Lustre and LNET modules (assuming no Lustre services are running). It is located in /usr/bin.</para>
+ <para>The lustre_rmmod.sh utility removes all Lustre and LNET modules (assuming no Lustre services are running). It is located in /usr/bin.</para>
<note><para>The lustre_rmmod.sh utility does not work if Lustre modules are being used or if you have manually run the lctl network up command.</para></note>
</section>
<section xml:id="dbdoclet.50438219_63667">
<title>36.13 lustre_<anchor xml:id="dbdoclet.50438219_marker-1317224" xreflabel=""/>rsync</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317226" xreflabel=""/>The lustre_rsync utility synchronizes (replicates) a Lustre file system to a target file system.</para>
+ <para>The lustre_rsync utility synchronizes (replicates) a Lustre file system to a target file system.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317227" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317228" xreflabel=""/>lustre_rsync --source|-s <src> --target|-t <tgt>
-<anchor xml:id="dbdoclet.50438219_pgfId-1317229" xreflabel=""/> --mdt|-m <mdt> [--user|-u <user id>]
-<anchor xml:id="dbdoclet.50438219_pgfId-1317230" xreflabel=""/> [--xattr|-x <yes|no>] [--verbose|-v]
-<anchor xml:id="dbdoclet.50438219_pgfId-1317231" xreflabel=""/> [--statuslog|-l <log>] [--dry-run] [--abort-on-err]
-<anchor xml:id="dbdoclet.50438219_pgfId-1317232" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438219_pgfId-1317233" xreflabel=""/>lustre_rsync --statuslog|-l <log>
-<anchor xml:id="dbdoclet.50438219_pgfId-1317234" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438219_pgfId-1317235" xreflabel=""/>lustre_rsync --statuslog|-l <log> --source|-s <source>
-<anchor xml:id="dbdoclet.50438219_pgfId-1317236" xreflabel=""/> --target|-t <tgt> --mdt|-m <mdt>
+ <title>Synopsis</title>
+ <screen>lustre_rsync --source|-s <src> --target|-t <tgt>
+ --mdt|-m <mdt> [--user|-u <user id>]
+ [--xattr|-x <yes|no>] [--verbose|-v]
+ [--statuslog|-l <log>] [--dry-run] [--abort-on-err]
+
+lustre_rsync --statuslog|-l <log>
+
+lustre_rsync --statuslog|-l <log> --source|-s <source>
+ --target|-t <tgt> --mdt|-m <mdt>
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317237" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317238" xreflabel=""/>The lustre_rsync utility is designed to synchronize (replicate) a Lustre file system (source) to another file system (target). The target can be a Lustre file system or any other type, and is a normal, usable file system. The synchronization operation is efficient and does not require directory walking, as lustre_rsync uses Lustre MDT changelogs to identify changes in the Lustre file system.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317239" xreflabel=""/>Before using lustre_rsync:</para>
+ <title>Description</title>
+ <para>The lustre_rsync utility is designed to synchronize (replicate) a Lustre file system (source) to another file system (target). The target can be a Lustre file system or any other type, and is a normal, usable file system. The synchronization operation is efficient and does not require directory walking, as lustre_rsync uses Lustre MDT changelogs to identify changes in the Lustre file system.</para>
+ <para>Before using lustre_rsync:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317240" xreflabel=""/> A changelog user must be registered (see lctl (8) changelog_register)</para>
+ <para> A changelog user must be registered (see lctl (8) changelog_register)</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317241" xreflabel=""/>- AND -</para>
+ <para>- AND -</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317242" xreflabel=""/> Verify that the Lustre file system (source) and the replica file system (target) are identical before the changelog user is registered. If the file systems are discrepant, use a utility, e.g. regular rsync (not lustre_rsync) to make them identical.</para>
+ <para> Verify that the Lustre file system (source) and the replica file system (target) are identical before the changelog user is registered. If the file systems are discrepant, use a utility, e.g. regular rsync (not lustre_rsync) to make them identical.</para>
</listitem>
</itemizedlist>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317305" xreflabel=""/>Options</title>
+ <title>Options</title>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317245" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317249" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317251" xreflabel=""/><emphasis role="bold">--source=</emphasis><emphasis><src></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317255" xreflabel=""/>The path to the root of the Lustre file system (source) which will be synchronized. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
+ <entry><para> <emphasis role="bold">--source=</emphasis><emphasis><src></emphasis></para></entry>
+ <entry><para> The path to the root of the Lustre file system (source) which will be synchronized. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317257" xreflabel=""/><emphasis role="bold">--target=</emphasis><emphasis><tgt></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317261" xreflabel=""/>The path to the root where the source file system will be synchronized (target). This is a mandatory option if the status log created during a previous synchronization operation (--statuslog) is not specified. This option can be repeated if multiple synchronization targets are desired.</para></entry>
+ <entry><para> <emphasis role="bold">--target=</emphasis><emphasis><tgt></emphasis></para></entry>
+ <entry><para> The path to the root where the source file system will be synchronized (target). This is a mandatory option if the status log created during a previous synchronization operation (--statuslog) is not specified. This option can be repeated if multiple synchronization targets are desired.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317263" xreflabel=""/><emphasis role="bold">--mdt=</emphasis><emphasis><mdt></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317267" xreflabel=""/>The metadata device to be synchronized. A changelog user must be registered for this device. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
+ <entry><para> <emphasis role="bold">--mdt=</emphasis><emphasis><mdt></emphasis></para></entry>
+ <entry><para> The metadata device to be synchronized. A changelog user must be registered for this device. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317269" xreflabel=""/><emphasis role="bold">--user=</emphasis><emphasis><user id></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317273" xreflabel=""/>The changelog user ID for the specified MDT. To use lustre_rsync, the changelog user must be registered. For details, see the changelog_register parameter in the lctl man page. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
+ <entry><para> <emphasis role="bold">--user=</emphasis><emphasis><user id></emphasis></para></entry>
+ <entry><para> The changelog user ID for the specified MDT. To use lustre_rsync, the changelog user must be registered. For details, see the changelog_register parameter in the lctl man page. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317275" xreflabel=""/><emphasis role="bold">--statuslog=</emphasis><emphasis><log></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317279" xreflabel=""/>A log file to which synchronization status is saved. When lustre_rsync starts, the state of a previous replication is read from here. If the status log from a previous synchronization operation is specified, otherwise mandatory options like --source, --target and --mdt options may be skipped. By specifying options like --source, --target and/or --mdt in addition to the --statuslog option, parameters in the status log can be overriden. Command line options take precedence over options in the status log.</para></entry>
+ <entry><para> <emphasis role="bold">--statuslog=</emphasis><emphasis><log></emphasis></para></entry>
+ <entry><para> A log file to which synchronization status is saved. When lustre_rsync starts, the state of a previous replication is read from here. If the status log from a previous synchronization operation is specified, otherwise mandatory options like --source, --target and --mdt options may be skipped. By specifying options like --source, --target and/or --mdt in addition to the --statuslog option, parameters in the status log can be overriden. Command line options take precedence over options in the status log.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317281" xreflabel=""/><emphasis role="bold">--xattr</emphasis><emphasis><yes|no></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317285" xreflabel=""/>Specifies whether extended attributes (xattrs) are synchronized or not. The default is to synchronize extended attributes.</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317286" xreflabel=""/>NOTE: Disabling xattrs causes Lustre striping information not to be synchronized.</para></entry>
+ <entry><para> <emphasis role="bold">--xattr</emphasis><emphasis><yes|no></emphasis></para></entry>
+ <entry><para> Specifies whether extended attributes (xattrs) are synchronized or not. The default is to synchronize extended attributes.</para><para>NOTE: Disabling xattrs causes Lustre striping information not to be synchronized.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317288" xreflabel=""/><emphasis role="bold">--verbose</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317292" xreflabel=""/>Produces a verbose output.</para></entry>
+ <entry><para> <emphasis role="bold">--verbose</emphasis></para></entry>
+ <entry><para> Produces a verbose output.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317294" xreflabel=""/><emphasis role="bold">--dry-run</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317298" xreflabel=""/>Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para></entry>
+ <entry><para> <emphasis role="bold">--dry-run</emphasis></para></entry>
+ <entry><para> Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317300" xreflabel=""/><emphasis role="bold">--abort-on-err</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317304" xreflabel=""/>Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para></entry>
+ <entry><para> <emphasis role="bold">--abort-on-err</emphasis></para></entry>
+ <entry><para> Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317306" xreflabel=""/>Examples</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317307" xreflabel=""/>Register a changelog user for an MDT (e.g., MDT lustre-MDT0000).</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317308" xreflabel=""/>$ ssh
-<anchor xml:id="dbdoclet.50438219_pgfId-1317309" xreflabel=""/>$ MDS lctl changelog_register \
-<anchor xml:id="dbdoclet.50438219_pgfId-1317310" xreflabel=""/> --device lustre-MDT0000 -n
-<anchor xml:id="dbdoclet.50438219_pgfId-1317311" xreflabel=""/>cl1
-</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317312" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317313" xreflabel=""/>Synchronize/replicate a Lustre file system (/mnt/lustre) to a target file system (/mnt/target).</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317314" xreflabel=""/>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target \
-<anchor xml:id="dbdoclet.50438219_pgfId-1317315" xreflabel=""/> --mdt=lustre-MDT0000 --user=cl1 \
-<anchor xml:id="dbdoclet.50438219_pgfId-1317316" xreflabel=""/> --statuslog replicate.log --verbose
-<anchor xml:id="dbdoclet.50438219_pgfId-1317317" xreflabel=""/>Lustre filesystem: lustre
-<anchor xml:id="dbdoclet.50438219_pgfId-1317318" xreflabel=""/>MDT device: lustre-MDT0000
-<anchor xml:id="dbdoclet.50438219_pgfId-1317319" xreflabel=""/>Source: /mnt/lustre
-<anchor xml:id="dbdoclet.50438219_pgfId-1317320" xreflabel=""/>Target: /mnt/target
-<anchor xml:id="dbdoclet.50438219_pgfId-1317321" xreflabel=""/>Statuslog: sync.log
-<anchor xml:id="dbdoclet.50438219_pgfId-1317322" xreflabel=""/>Changelog registration: cl1
-<anchor xml:id="dbdoclet.50438219_pgfId-1317323" xreflabel=""/>Starting changelog record: 0
-<anchor xml:id="dbdoclet.50438219_pgfId-1317324" xreflabel=""/>Errors: 0
-<anchor xml:id="dbdoclet.50438219_pgfId-1317325" xreflabel=""/>lustre_rsync took 1 seconds
-<anchor xml:id="dbdoclet.50438219_pgfId-1317326" xreflabel=""/>Changelog records consumed: 22
-</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317327" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317328" xreflabel=""/>After the file system undergoes changes, synchronize the changes with the target file system. Only the statuslog name needs to be specified, as it has all the parameters passed earlier.</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317329" xreflabel=""/>$ lustre_rsync --statuslog replicate.log --verbose
-<anchor xml:id="dbdoclet.50438219_pgfId-1317330" xreflabel=""/>Replicating Lustre filesystem: lustre
-<anchor xml:id="dbdoclet.50438219_pgfId-1317331" xreflabel=""/>MDT device: lustre-MDT0000
-<anchor xml:id="dbdoclet.50438219_pgfId-1317332" xreflabel=""/>Source: /mnt/lustre
-<anchor xml:id="dbdoclet.50438219_pgfId-1317333" xreflabel=""/>Target: /mnt/target
-<anchor xml:id="dbdoclet.50438219_pgfId-1317334" xreflabel=""/>Statuslog: replicate.log
-<anchor xml:id="dbdoclet.50438219_pgfId-1317335" xreflabel=""/>Changelog registration: cl1
-<anchor xml:id="dbdoclet.50438219_pgfId-1317336" xreflabel=""/>Starting changelog record: 22
-<anchor xml:id="dbdoclet.50438219_pgfId-1317337" xreflabel=""/>Errors: 0
-<anchor xml:id="dbdoclet.50438219_pgfId-1317338" xreflabel=""/>lustre_rsync took 2 seconds
-<anchor xml:id="dbdoclet.50438219_pgfId-1317339" xreflabel=""/>Changelog records consumed: 42
-</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317340" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317341" xreflabel=""/>Synchronize a Lustre file system (/mnt/lustre) to two target file systems (/mnt/target1 and /mnt/target2).</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317342" xreflabel=""/>$ lustre_rsync --source=/mnt/lustre \
-<anchor xml:id="dbdoclet.50438219_pgfId-1317343" xreflabel=""/> --target=/mnt/target1 --target=/mnt/target2 \
-<anchor xml:id="dbdoclet.50438219_pgfId-1317344" xreflabel=""/> --mdt=lustre-MDT0000 --user=cl1
-<anchor xml:id="dbdoclet.50438219_pgfId-1317345" xreflabel=""/> --statuslog replicate.log
-</screen>
- </section>
- <section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317346" xreflabel=""/>See Also</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317353" xreflabel=""/><xref linkend="dbdoclet.50438206_94597"/></para>
+ <title>Examples</title>
+ <para>Register a changelog user for an MDT (e.g., MDT lustre-MDT0000).</para>
+ <screen>$ ssh
+$ MDS lctl changelog_register \
+ --device lustre-MDT0000 -n
+cl1
+</screen>
+ <para> </para>
+ <para>Synchronize/replicate a Lustre file system (/mnt/lustre) to a target file system (/mnt/target).</para>
+ <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target \
+ --mdt=lustre-MDT0000 --user=cl1 \
+ --statuslog replicate.log --verbose
+Lustre filesystem: lustre
+MDT device: lustre-MDT0000
+Source: /mnt/lustre
+Target: /mnt/target
+Statuslog: sync.log
+Changelog registration: cl1
+Starting changelog record: 0
+Errors: 0
+lustre_rsync took 1 seconds
+Changelog records consumed: 22
+</screen>
+ <para> </para>
+ <para>After the file system undergoes changes, synchronize the changes with the target file system. Only the statuslog name needs to be specified, as it has all the parameters passed earlier.</para>
+ <screen>$ lustre_rsync --statuslog replicate.log --verbose
+Replicating Lustre filesystem: lustre
+MDT device: lustre-MDT0000
+Source: /mnt/lustre
+Target: /mnt/target
+Statuslog: replicate.log
+Changelog registration: cl1
+Starting changelog record: 22
+Errors: 0
+lustre_rsync took 2 seconds
+Changelog records consumed: 42
+</screen>
+ <para> </para>
+ <para>Synchronize a Lustre file system (/mnt/lustre) to two target file systems (/mnt/target1 and /mnt/target2).</para>
+ <screen>$ lustre_rsync --source=/mnt/lustre \
+ --target=/mnt/target1 --target=/mnt/target2 \
+ --mdt=lustre-MDT0000 --user=cl1
+ --statuslog replicate.log
+</screen>
+ </section>
+ <section remap="h5">
+ <title>See Also</title>
+ <para><xref linkend="dbdoclet.50438206_94597"/></para>
</section>
</section>
<section xml:id="dbdoclet.50438219_75432">
<title>36.14 mkfs.<anchor xml:id="dbdoclet.50438219_marker-1316695" xreflabel=""/>lustre</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1316697" xreflabel=""/>The mkfs.lustre utility formats a disk for a Lustre service.</para>
+ <para>The mkfs.lustre utility formats a disk for a Lustre service.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1316927" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1316698" xreflabel=""/>mkfs.lustre <target_type> [options] device
+ <title>Synopsis</title>
+ <screen>mkfs.lustre <target_type> [options] device
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1316734" xreflabel=""/>where <target_type> is one of the following:</para>
+ <para>where <target_type> is one of the following:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1316715" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1316717" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316719" xreflabel=""/><emphasis role="bold">--ost</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316721" xreflabel=""/>Object Storage Target (OST)</para></entry>
+ <entry><para> <emphasis role="bold">--ost</emphasis></para></entry>
+ <entry><para> Object Storage Target (OST)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316723" xreflabel=""/><emphasis role="bold">--mdt</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316725" xreflabel=""/>Metadata Storage Target (MDT)</para></entry>
+ <entry><para> <emphasis role="bold">--mdt</emphasis></para></entry>
+ <entry><para> Metadata Storage Target (MDT)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316727" xreflabel=""/><emphasis role="bold">--network=</emphasis><emphasis>net,...</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316729" xreflabel=""/>Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para></entry>
+ <entry><para> <emphasis role="bold">--network=</emphasis><emphasis>net,...</emphasis></para></entry>
+ <entry><para> Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316731" xreflabel=""/><emphasis role="bold">--mgs</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316733" xreflabel=""/>Configuration Management Service (MGS), one per site. This service can be combined with one <emphasis role="bold">--mdt</emphasis> service by specifying both types.</para></entry>
+ <entry><para> <emphasis role="bold">--mgs</emphasis></para></entry>
+ <entry><para> Configuration Management Service (MGS), one per site. This service can be combined with one <emphasis role="bold">--mdt</emphasis> service by specifying both types.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1316744" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1316745" xreflabel=""/>mkfs.lustre is used to format a disk device for use as part of a Lustre file system. After formatting, a disk can be mounted to start the Lustre service defined by this command.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1316746" xreflabel=""/>When the file system is created, parameters can simply be added as a --param option to the mkfs.lustre command. See <xref linkend="dbdoclet.50438194_17237"/>.</para>
+ <title>Description</title>
+ <para>mkfs.lustre is used to format a disk device for use as part of a Lustre file system. After formatting, a disk can be mounted to start the Lustre service defined by this command.</para>
+ <para>When the file system is created, parameters can simply be added as a --param option to the mkfs.lustre command. See <xref linkend="dbdoclet.50438194_17237"/>.</para>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
<colspec colname="c3" colwidth="33*"/>
<thead>
<row>
- <entry nameend="c2" namest="c1"><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1316749" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1316753" xreflabel=""/>Description</emphasis></para></entry>
+ <entry nameend="c2" namest="c1"><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316755" xreflabel=""/><emphasis role="bold">--backfstype</emphasis>=<emphasis>fstype</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316759" xreflabel=""/>Forces a particular format for the backing file system (such as ext3, ldiskfs).</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--backfstype</emphasis>=<emphasis>fstype</emphasis></para></entry>
+ <entry><para> Forces a particular format for the backing file system (such as ext3, ldiskfs).</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316761" xreflabel=""/><emphasis role="bold">--comment</emphasis>=<emphasis>comment</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316765" xreflabel=""/>Sets a user comment about this disk, ignored by Lustre.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--comment</emphasis>=<emphasis>comment</emphasis></para></entry>
+ <entry><para> Sets a user comment about this disk, ignored by Lustre.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316767" xreflabel=""/><emphasis role="bold">--device-size</emphasis>=<emphasis>KB</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316771" xreflabel=""/>Sets the device size for loop devices.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--device-size</emphasis>=<emphasis>KB</emphasis></para></entry>
+ <entry><para> Sets the device size for loop devices.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316773" xreflabel=""/><emphasis role="bold">--dryrun</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316777" xreflabel=""/>Only prints what would be done; it does not affect the disk.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--dryrun</emphasis></para></entry>
+ <entry><para> Only prints what would be done; it does not affect the disk.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316779" xreflabel=""/><emphasis role="bold">--failnode</emphasis>=<emphasis>nid</emphasis>,...</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316783" xreflabel=""/>Sets the NID(s) of a failover partner. This option can be repeated as needed.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--failnode</emphasis>=<emphasis>nid</emphasis>,...</para></entry>
+ <entry><para> Sets the NID(s) of a failover partner. This option can be repeated as needed.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316785" xreflabel=""/><emphasis role="bold">--fsname</emphasis>=<emphasis>filesystem_name</emphasis></para></entry>
- <entry><para>
<anchor xml:id="dbdoclet.50438219_pgfId-1316789" xreflabel=""/>The Lustre file system of which this service/node will be a part. The default file system name is 'lustreâ€.</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1316790" xreflabel=""/> </para><note><para>The file system name is limited to 8 characters.</para></note></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--fsname</emphasis>=<emphasis>filesystem_name</emphasis></para></entry>
+ <entry><para>
The Lustre file system of which this service/node will be a part. The default file system name is 'lustreâ€.</para><para> </para><note><para>The file system name is limited to 8 characters.</para></note></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316793" xreflabel=""/><emphasis role="bold">--index</emphasis>=<emphasis>index</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316797" xreflabel=""/> Forces a particular OST or MDT index.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--index</emphasis>=<emphasis>index</emphasis></para></entry>
+ <entry><para> Forces a particular OST or MDT index.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316799" xreflabel=""/><emphasis role="bold">--mkfsoptions</emphasis>=<emphasis>opts</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316803" xreflabel=""/> Formats options for the backing file system. For example, ext3 options could be set here.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--mkfsoptions</emphasis>=<emphasis>opts</emphasis></para></entry>
+ <entry><para> Formats options for the backing file system. For example, ext3 options could be set here.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316805" xreflabel=""/><emphasis role="bold">--mountfsoptions</emphasis>=<emphasis>opts</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316809" xreflabel=""/> Sets the mount options used when the backing file system is mounted.</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1316810" xreflabel=""/><emphasis role="bold">CAUTION</emphasis>: Unlike earlier versions of mkfs.lustre, this version completely replaces the default mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1316811" xreflabel=""/>The defaults for ldiskfs are:</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1316812" xreflabel=""/>OST: <emphasis>errors=remount-ro,mballoc,extents</emphasis>;</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1316813" xreflabel=""/>MGS/MDT: <emphasis>errors=remount-ro,iopen_nopriv,user_xattr</emphasis></para><para><anchor xml:id="dbdoclet.50438219_pgfId-1316814" xreflabel=""/>Do not alter the default mount options unless you know what you are doing.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--mountfsoptions</emphasis>=<emphasis>opts</emphasis></para></entry>
+ <entry><para> Sets the mount options used when the backing file system is mounted.</para><para><emphasis role="bold">CAUTION</emphasis>: Unlike earlier versions of mkfs.lustre, this version completely replaces the default mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para><para>The defaults for ldiskfs are:</para><para>OST: <emphasis>errors=remount-ro,mballoc,extents</emphasis>;</para><para>MGS/MDT: <emphasis>errors=remount-ro,iopen_nopriv,user_xattr</emphasis></para><para>Do not alter the default mount options unless you know what you are doing.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316816" xreflabel=""/><emphasis role="bold">--network</emphasis>=<emphasis>net</emphasis>,...</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1316817" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316821" xreflabel=""/> Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--network</emphasis>=<emphasis>net</emphasis>,...</para><para> </para></entry>
+ <entry><para> Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316823" xreflabel=""/><emphasis role="bold">--mgsnode</emphasis>=<emphasis>nid</emphasis>,...</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316827" xreflabel=""/>Sets the NIDs of the MGS node, required for all targets other than the MGS.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--mgsnode</emphasis>=<emphasis>nid</emphasis>,...</para></entry>
+ <entry><para> Sets the NIDs of the MGS node, required for all targets other than the MGS.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316829" xreflabel=""/><emphasis role="bold">--param</emphasis><emphasis>key</emphasis>=<emphasis>value</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316833" xreflabel=""/>Sets the permanent parameter <emphasis>key</emphasis> to value <emphasis>value</emphasis>. This option can be repeated as necessary. Typical options might include:</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--param</emphasis><emphasis>key</emphasis>=<emphasis>value</emphasis></para></entry>
+ <entry><para> Sets the permanent parameter <emphasis>key</emphasis> to value <emphasis>value</emphasis>. This option can be repeated as necessary. Typical options might include:</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316835" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316837" xreflabel=""/>--<emphasis>param sys.timeout=40</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316839" xreflabel=""/>System obd timeout.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> --<emphasis>param sys.timeout=40</emphasis></para></entry>
+ <entry><para> System obd timeout.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316841" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316843" xreflabel=""/>--<emphasis>param lov.stripesize=2M</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316845" xreflabel=""/>Default stripe size.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> --<emphasis>param lov.stripesize=2M</emphasis></para></entry>
+ <entry><para> Default stripe size.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316847" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316849" xreflabel=""/>--<emphasis>param lov.stripecount=2</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316851" xreflabel=""/>Default stripe count.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> --<emphasis>param lov.stripecount=2</emphasis></para></entry>
+ <entry><para> Default stripe count.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316853" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316855" xreflabel=""/>--<emphasis>param failover.mode=failout</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316857" xreflabel=""/>Returns errors instead of waiting for recovery.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> --<emphasis>param failover.mode=failout</emphasis></para></entry>
+ <entry><para> Returns errors instead of waiting for recovery.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316859" xreflabel=""/><emphasis role="bold">--quiet</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316863" xreflabel=""/>Prints less information.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--quiet</emphasis></para></entry>
+ <entry><para> Prints less information.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316865" xreflabel=""/><emphasis role="bold">--reformat</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316869" xreflabel=""/>Reformats an existing Lustre disk.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--reformat</emphasis></para></entry>
+ <entry><para> Reformats an existing Lustre disk.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316871" xreflabel=""/><emphasis role="bold">--stripe-count-hint=stripes</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316875" xreflabel=""/>Used to optimize the MDT's inode size.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--stripe-count-hint=stripes</emphasis></para></entry>
+ <entry><para> Used to optimize the MDT's inode size.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316877" xreflabel=""/><emphasis role="bold">--verbose</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1316881" xreflabel=""/>Prints more information.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">--verbose</emphasis></para></entry>
+ <entry><para> Prints more information.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1316890" xreflabel=""/>Examples</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1316891" xreflabel=""/>Creates a combined MGS and MDT for file system <emphasis role="bold">testfs</emphasis> on, e.g., node <emphasis role="bold">cfs21</emphasis>:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1316892" xreflabel=""/>mkfs.lustre --fsname=testfs --mdt --mgs /dev/sda1
+ <title>Examples</title>
+ <para>Creates a combined MGS and MDT for file system <emphasis role="bold">testfs</emphasis> on, e.g., node <emphasis role="bold">cfs21</emphasis>:</para>
+ <screen>mkfs.lustre --fsname=testfs --mdt --mgs /dev/sda1
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1316893" xreflabel=""/>Creates an OST for file system <emphasis role="bold">testfs</emphasis> on any node (using the above MGS):</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1316894" xreflabel=""/>mkfs.lustre --fsname=testfs --ost --mgsnode=cfs21@tcp0 /dev/sdb
+ <para>Creates an OST for file system <emphasis role="bold">testfs</emphasis> on any node (using the above MGS):</para>
+ <screen>mkfs.lustre --fsname=testfs --ost --mgsnode=cfs21@tcp0 /dev/sdb
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1316895" xreflabel=""/>Creates a standalone MGS on, e.g., node <emphasis role="bold">cfs22</emphasis>:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1316896" xreflabel=""/>mkfs.lustre --mgs /dev/sda1
+ <para>Creates a standalone MGS on, e.g., node <emphasis role="bold">cfs22</emphasis>:</para>
+ <screen>mkfs.lustre --mgs /dev/sda1
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1316897" xreflabel=""/>Creates an MDT for file system <emphasis role="bold">myfs1</emphasis> on any node (using the above MGS):</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1316898" xreflabel=""/>mkfs.lustre --fsname=myfs1 --mdt --mgsnode=cfs22@tcp0 /dev/sda2
+ <para>Creates an MDT for file system <emphasis role="bold">myfs1</emphasis> on any node (using the above MGS):</para>
+ <screen>mkfs.lustre --fsname=myfs1 --mdt --mgsnode=cfs22@tcp0 /dev/sda2
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1316899" xreflabel=""/>See Also</title>
+ <title>See Also</title>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438219_75432"/>mkfs.lustre, </para>
</section>
<section xml:id="dbdoclet.50438219_12635">
<title>36.15 mount.<anchor xml:id="dbdoclet.50438219_marker-1310637" xreflabel=""/>lustre</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1310639" xreflabel=""/>The mount.lustre utility starts a Lustre client or target service.</para>
+ <para>The mount.lustre utility starts a Lustre client or target service.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1310640" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1294724" xreflabel=""/>mount -t lustre [-o options] directory
+ <title>Synopsis</title>
+ <screen>mount -t lustre [-o options] directory
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1294725" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1315116" xreflabel=""/>The mount.lustre utility starts a Lustre client or target service. This program should not be called directly; rather, it is a helper program invoked through mount(8), as shown above. Use the umount command to stop Lustre clients and targets.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294727" xreflabel=""/>There are two forms for the device option, depending on whether a client or a target service is started:</para>
+ <title>Description</title>
+ <para>The mount.lustre utility starts a Lustre client or target service. This program should not be called directly; rather, it is a helper program invoked through mount(8), as shown above. Use the umount command to stop Lustre clients and targets.</para>
+ <para>There are two forms for the device option, depending on whether a client or a target service is started:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1308672" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1308676" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315124" xreflabel=""/><emphasis><mgsspec>:/<fsname></emphasis></para><para><anchor xml:id="dbdoclet.50438219_pgfId-1315125" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315129" xreflabel=""/>Mounts the Lustre file system named <emphasis>fsname</emphasis> on the client by contacting the Management Service at <emphasis>mgsspec</emphasis> on the pathname given by <emphasis>directory</emphasis>. The format for <emphasis>mgsspec</emphasis> is defined below. A mounted client file system appears in fstab(5) and is usable, like any local file system, and provides a full POSIX-compliant interface.</para></entry>
+ <entry><para> <emphasis><mgsspec>:/<fsname></emphasis></para><para> </para></entry>
+ <entry><para> Mounts the Lustre file system named <emphasis>fsname</emphasis> on the client by contacting the Management Service at <emphasis>mgsspec</emphasis> on the pathname given by <emphasis>directory</emphasis>. The format for <emphasis>mgsspec</emphasis> is defined below. A mounted client file system appears in fstab(5) and is usable, like any local file system, and provides a full POSIX-compliant interface.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315131" xreflabel=""/><emphasis><disk_device></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315135" xreflabel=""/>Starts the target service defined by the mkfs.lustre command on the physical disk <emphasis>disk_device</emphasis>. A mounted target service file system is only useful for df(1) operations and appears in fstab(5) to show the device is in use.</para></entry>
+ <entry><para> <emphasis><disk_device></emphasis></para></entry>
+ <entry><para> Starts the target service defined by the mkfs.lustre command on the physical disk <emphasis>disk_device</emphasis>. A mounted target service file system is only useful for df(1) operations and appears in fstab(5) to show the device is in use.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1301179" xreflabel=""/>Options</title>
+ <title>Options</title>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1309271" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1309273" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315144" xreflabel=""/><emphasis role="bold"><mgsspec>:</emphasis>=<emphasis><mgsnode>[:<mgsnode>]</emphasis></para><para><anchor xml:id="dbdoclet.50438219_pgfId-1315145" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315147" xreflabel=""/>The MGS specification may be a colon-separated list of nodes.</para></entry>
+ <entry><para> <emphasis role="bold"><mgsspec>:</emphasis>=<emphasis><mgsnode>[:<mgsnode>]</emphasis></para><para> </para></entry>
+ <entry><para> The MGS specification may be a colon-separated list of nodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315149" xreflabel=""/><emphasis role="bold"><mgsnode>:</emphasis>=<emphasis><mgsnid>[,<mgsnid>]</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315151" xreflabel=""/>Each node may be specified by a comma-separated list of NIDs.</para></entry>
+ <entry><para> <emphasis role="bold"><mgsnode>:</emphasis>=<emphasis><mgsnid>[,<mgsnid>]</emphasis></para></entry>
+ <entry><para> Each node may be specified by a comma-separated list of NIDs.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1308731" xreflabel=""/>In addition to the standard mount options, Lustre understands the following client-specific options:</para>
+ <para>In addition to the standard mount options, Lustre understands the following client-specific options:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1309286" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1309288" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315157" xreflabel=""/><emphasis role="bold">flock</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315159" xreflabel=""/>Enables full flock support, coherent across all client nodes.</para></entry>
+ <entry><para> <emphasis role="bold">flock</emphasis></para></entry>
+ <entry><para> Enables full flock support, coherent across all client nodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315161" xreflabel=""/><emphasis role="bold">localflock</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315163" xreflabel=""/>Enables local flock support, using only client-local flock (faster, for applications that require flock, but do not run on multiple nodes).</para></entry>
+ <entry><para> <emphasis role="bold">localflock</emphasis></para></entry>
+ <entry><para> Enables local flock support, using only client-local flock (faster, for applications that require flock, but do not run on multiple nodes).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315165" xreflabel=""/><emphasis role="bold">noflock</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315167" xreflabel=""/>Disables flock support entirely. Applications calling flock get an error. It is up to the administrator to choose either <emphasis role="bold">localflock</emphasis> (fastest, low impact, not coherent between nodes) or <emphasis role="bold">flock</emphasis> (slower, performance impact for use, coherent between nodes).</para></entry>
+ <entry><para> <emphasis role="bold">noflock</emphasis></para></entry>
+ <entry><para> Disables flock support entirely. Applications calling flock get an error. It is up to the administrator to choose either <emphasis role="bold">localflock</emphasis> (fastest, low impact, not coherent between nodes) or <emphasis role="bold">flock</emphasis> (slower, performance impact for use, coherent between nodes).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315169" xreflabel=""/><emphasis role="bold">user_xattr</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315171" xreflabel=""/>Enables get/set of extended attributes by regular users. See the attr(5) manual page.</para></entry>
+ <entry><para> <emphasis role="bold">user_xattr</emphasis></para></entry>
+ <entry><para> Enables get/set of extended attributes by regular users. See the attr(5) manual page.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315177" xreflabel=""/><emphasis role="bold">nouser_xattr</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315179" xreflabel=""/>Disables use of extended attributes by regular users. Root and system processes can still use extended attributes.</para></entry>
+ <entry><para> <emphasis role="bold">nouser_xattr</emphasis></para></entry>
+ <entry><para> Disables use of extended attributes by regular users. Root and system processes can still use extended attributes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315181" xreflabel=""/><emphasis role="bold">acl</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315183" xreflabel=""/>Enables POSIX Access Control List support. See the acl(5) manual page.</para></entry>
+ <entry><para> <emphasis role="bold">acl</emphasis></para></entry>
+ <entry><para> Enables POSIX Access Control List support. See the acl(5) manual page.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315185" xreflabel=""/><emphasis role="bold">noacl</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315187" xreflabel=""/>Disables Access Control List support.</para></entry>
+ <entry><para> <emphasis role="bold">noacl</emphasis></para></entry>
+ <entry><para> Disables Access Control List support.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1310887" xreflabel=""/>In addition to the standard mount options and backing disk type (e.g. ext3) options, Lustre understands the following server-specific options:</para>
+ <para>In addition to the standard mount options and backing disk type (e.g. ext3) options, Lustre understands the following server-specific options:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1310890" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1310894" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1310896" xreflabel=""/><emphasis role="bold">nosvc</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1310900" xreflabel=""/> Starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para></entry>
+ <entry><para> <emphasis role="bold">nosvc</emphasis></para></entry>
+ <entry><para> Starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1310902" xreflabel=""/><emphasis role="bold">nomsgs</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1310906" xreflabel=""/> Starts only the MDT (with a co-located MGS), without starting the MGS.</para></entry>
+ <entry><para> <emphasis role="bold">nomsgs</emphasis></para></entry>
+ <entry><para> Starts only the MDT (with a co-located MGS), without starting the MGS.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1310908" xreflabel=""/><emphasis role="bold">exclude=<ostlist></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1310912" xreflabel=""/> Starts a client or MDT with a colon-separated list of known inactive OSTs.</para></entry>
+ <entry><para> <emphasis role="bold">exclude=<ostlist></emphasis></para></entry>
+ <entry><para> Starts a client or MDT with a colon-separated list of known inactive OSTs.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315189" xreflabel=""/><emphasis role="bold">nosvc</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315193" xreflabel=""/> Only starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para></entry>
+ <entry><para> <emphasis role="bold">nosvc</emphasis></para></entry>
+ <entry><para> Only starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315195" xreflabel=""/><emphasis role="bold">nomsgs</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315199" xreflabel=""/> Starts a MDT with a co-located MGS, without starting the MGS.</para></entry>
+ <entry><para> <emphasis role="bold">nomsgs</emphasis></para></entry>
+ <entry><para> Starts a MDT with a co-located MGS, without starting the MGS.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315201" xreflabel=""/><emphasis role="bold">exclude</emphasis>=<emphasis>ostlist</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315205" xreflabel=""/> Starts a client or MDT with a (colon-separated) list of known inactive OSTs.</para></entry>
+ <entry><para> <emphasis role="bold">exclude</emphasis>=<emphasis>ostlist</emphasis></para></entry>
+ <entry><para> Starts a client or MDT with a (colon-separated) list of known inactive OSTs.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315211" xreflabel=""/><emphasis role="bold">abort_recov</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315215" xreflabel=""/> Aborts client recovery and starts the target service immediately.</para></entry>
+ <entry><para> <emphasis role="bold">abort_recov</emphasis></para></entry>
+ <entry><para> Aborts client recovery and starts the target service immediately.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315217" xreflabel=""/><emphasis role="bold">md_stripe_cache_size</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315221" xreflabel=""/> Sets the stripe cache size for server-side disk with a striped RAID configuration.</para></entry>
+ <entry><para> <emphasis role="bold">md_stripe_cache_size</emphasis></para></entry>
+ <entry><para> Sets the stripe cache size for server-side disk with a striped RAID configuration.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315223" xreflabel=""/><emphasis role="bold">recovery_time_soft</emphasis>=<emphasis>timeout</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315227" xreflabel=""/> Allows <emphasis>timeout</emphasis> seconds for clients to reconnect for recovery after a server crash. This timeout is incrementally extended if it is about to expire and the server is still handling new connections from recoverable clients. The default soft recovery timeout is 300 seconds (5 minutes).</para></entry>
+ <entry><para> <emphasis role="bold">recovery_time_soft</emphasis>=<emphasis>timeout</emphasis></para></entry>
+ <entry><para> Allows <emphasis>timeout</emphasis> seconds for clients to reconnect for recovery after a server crash. This timeout is incrementally extended if it is about to expire and the server is still handling new connections from recoverable clients. The default soft recovery timeout is 300 seconds (5 minutes).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315229" xreflabel=""/><emphasis role="bold">recovery_time_hard</emphasis>=<emphasis>timeout</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1315233" xreflabel=""/> The server is allowed to incrementally extend its timeout up to a hard maximum of <emphasis>timeout</emphasis> seconds. The default hard recovery timeout is set to 900 seconds (15 minutes).</para></entry>
+ <entry><para> <emphasis role="bold">recovery_time_hard</emphasis>=<emphasis>timeout</emphasis></para></entry>
+ <entry><para> The server is allowed to incrementally extend its timeout up to a hard maximum of <emphasis>timeout</emphasis> seconds. The default hard recovery timeout is set to 900 seconds (15 minutes).</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1310925" xreflabel=""/>Examples</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1315242" xreflabel=""/>Starts a client for the Lustre file system testfs at mount point /mnt/myfilesystem. The Management Service is running on a node reachable from this client via the cfs21@tcp0 NID.</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1315243" xreflabel=""/>mount -t lustre cfs21@tcp0:/testfs /mnt/myfilesystem
+ <title>Examples</title>
+ <para>Starts a client for the Lustre file system testfs at mount point /mnt/myfilesystem. The Management Service is running on a node reachable from this client via the cfs21@tcp0 NID.</para>
+ <screen>mount -t lustre cfs21@tcp0:/testfs /mnt/myfilesystem
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1315244" xreflabel=""/>Starts the Lustre metadata target service from /dev/sda1 on mount point /mnt/test/mdt.</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1315245" xreflabel=""/>mount -t lustre /dev/sda1 /mnt/test/mdt
+ <para>Starts the Lustre metadata target service from /dev/sda1 on mount point /mnt/test/mdt.</para>
+ <screen>mount -t lustre /dev/sda1 /mnt/test/mdt
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1315246" xreflabel=""/>Starts the testfs-MDT0000 service (using the disk label), but aborts the recovery process.</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1315247" xreflabel=""/>mount -t lustre -L testfs-MDT0000 -o abort_recov /mnt/test/mdt
+ <para>Starts the testfs-MDT0000 service (using the disk label), but aborts the recovery process.</para>
+ <screen>mount -t lustre -L testfs-MDT0000 -o abort_recov /mnt/test/mdt
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1315248" xreflabel=""/>See Also</title>
+ <title>See Also</title>
<itemizedlist><listitem>
<para> <xref linkend="dbdoclet.50438219_75432"/></para>
</section>
<section xml:id="dbdoclet.50438219_82679">
<title>36.16 plot-<anchor xml:id="dbdoclet.50438219_marker-1317989" xreflabel=""/>llstat</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317991" xreflabel=""/>The plot-llstat utility plots Lustre statistics.</para>
+ <para>The plot-llstat utility plots Lustre statistics.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317992" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317993" xreflabel=""/>plot-llstat results_filename [parameter_index]
+ <title>Synopsis</title>
+ <screen>plot-llstat results_filename [parameter_index]
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317994" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317995" xreflabel=""/>The plot-llstat utility generates a CSV file and instruction files for gnuplot from the output of llstat. Since llstat is generic in nature, plot-llstat is also a generic script. The value of parameter_index can be 1 for count per interval, 2 for count per second (default setting) or 3 for total count.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317996" xreflabel=""/>The plot-llstat utility creates a .dat (CSV) file using the number of operations specified by the user. The number of operations equals the number of columns in the CSV file. The values in those columns are equal to the corresponding value of parameter_index in the output file.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317997" xreflabel=""/>The plot-llstat utility also creates a .scr file that contains instructions for gnuplot to plot the graph. After generating the .dat and .scr files, the plot-llstat tool invokes gnuplot to display the graph.</para>
+ <title>Description</title>
+ <para>The plot-llstat utility generates a CSV file and instruction files for gnuplot from the output of llstat. Since llstat is generic in nature, plot-llstat is also a generic script. The value of parameter_index can be 1 for count per interval, 2 for count per second (default setting) or 3 for total count.</para>
+ <para>The plot-llstat utility creates a .dat (CSV) file using the number of operations specified by the user. The number of operations equals the number of columns in the CSV file. The values in those columns are equal to the corresponding value of parameter_index in the output file.</para>
+ <para>The plot-llstat utility also creates a .scr file that contains instructions for gnuplot to plot the graph. After generating the .dat and .scr files, the plot-llstat tool invokes gnuplot to display the graph.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318018" xreflabel=""/>Options</title>
+ <title>Options</title>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1318000" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1318004" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318006" xreflabel=""/><emphasis role="bold">results_filename</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318010" xreflabel=""/>Output generated by plot-llstat</para></entry>
+ <entry><para> <emphasis role="bold">results_filename</emphasis></para></entry>
+ <entry><para> Output generated by plot-llstat</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1318012" xreflabel=""/><emphasis role="bold">parameter_index</emphasis></para><para><anchor xml:id="dbdoclet.50438219_pgfId-1318013" xreflabel=""/> </para></entry>
- <entry><para>
<anchor xml:id="dbdoclet.50438219_pgfId-1318017" xreflabel=""/>Value of parameter_index can be:</para><para> 1 - count per interval</para><para> 2 - count per second (default setting)</para><para> 3 - total count</para></entry>
+ <entry><para> <emphasis role="bold">parameter_index</emphasis></para><para> </para></entry>
+ <entry><para> Value of parameter_index can be:</para><para> 1 - count per interval</para><para> 2 - count per second (default setting)</para><para> 3 - total count</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318019" xreflabel=""/>Example</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1318020" xreflabel=""/>llstat -i2 -g -c lustre-OST0000 > log
-<anchor xml:id="dbdoclet.50438219_pgfId-1318021" xreflabel=""/>plot-llstat log 3
+ <title>Example</title>
+ <screen>llstat -i2 -g -c lustre-OST0000 > log
+plot-llstat log 3
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438219_51496">
<title>36.17 router<anchor xml:id="dbdoclet.50438219_marker-1317921" xreflabel=""/>stat</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317923" xreflabel=""/>The routerstat utility prints Lustre router statistics.</para>
+ <para>The routerstat utility prints Lustre router statistics.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317924" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317925" xreflabel=""/>routerstat [interval]
+ <title>Synopsis</title>
+ <screen>routerstat [interval]
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317926" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317927" xreflabel=""/>The routerstat utility watches LNET router statistics. If no <emphasis>interval</emphasis> is specified, then statistics are sampled and printed only one time. Otherwise, statistics are sampled and printed at the specified <emphasis>interval</emphasis> (in seconds).</para>
+ <title>Description</title>
+ <para>The routerstat utility watches LNET router statistics. If no <emphasis>interval</emphasis> is specified, then statistics are sampled and printed only one time. Otherwise, statistics are sampled and printed at the specified <emphasis>interval</emphasis> (in seconds).</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317928" xreflabel=""/>Options</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317958" xreflabel=""/>The routerstat output includes the following fields:</para>
+ <title>Options</title>
+ <para>The routerstat output includes the following fields:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317931" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317933" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317935" xreflabel=""/><emphasis role="bold">M</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317937" xreflabel=""/>msgs_alloc(msgs_max)</para></entry>
+ <entry><para> <emphasis role="bold">M</emphasis></para></entry>
+ <entry><para> msgs_alloc(msgs_max)</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317939" xreflabel=""/><emphasis role="bold">E</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317941" xreflabel=""/>errors</para></entry>
+ <entry><para> <emphasis role="bold">E</emphasis></para></entry>
+ <entry><para> errors</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317943" xreflabel=""/><emphasis role="bold">S</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317945" xreflabel=""/>send_count/send_length</para></entry>
+ <entry><para> <emphasis role="bold">S</emphasis></para></entry>
+ <entry><para> send_count/send_length</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317947" xreflabel=""/><emphasis role="bold">R</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317949" xreflabel=""/>recv_count/recv_length</para></entry>
+ <entry><para> <emphasis role="bold">R</emphasis></para></entry>
+ <entry><para> recv_count/recv_length</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317951" xreflabel=""/><emphasis role="bold">F</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317953" xreflabel=""/>route_count/route_length</para></entry>
+ <entry><para> <emphasis role="bold">F</emphasis></para></entry>
+ <entry><para> route_count/route_length</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317955" xreflabel=""/><emphasis role="bold">D</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317957" xreflabel=""/>drop_count/drop_length</para></entry>
+ <entry><para> <emphasis role="bold">D</emphasis></para></entry>
+ <entry><para> drop_count/drop_length</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317959" xreflabel=""/>Files</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317960" xreflabel=""/>The routerstat utility extracts statistics data from:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317961" xreflabel=""/>/proc/sys/lnet/stats
+ <title>Files</title>
+ <para>The routerstat utility extracts statistics data from:</para>
+ <screen>/proc/sys/lnet/stats
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438219_39574">
<title>36.18 tunefs.<anchor xml:id="dbdoclet.50438219_marker-1317076" xreflabel=""/>lustre</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317078" xreflabel=""/>The tunefs.lustre utility modifies configuration information on a Lustre target disk.</para>
+ <para>The tunefs.lustre utility modifies configuration information on a Lustre target disk.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317079" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317080" xreflabel=""/>tunefs.lustre [options] <device>
+ <title>Synopsis</title>
+ <screen>tunefs.lustre [options] <device>
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317081" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317082" xreflabel=""/>tunefs.lustre is used to modify configuration information on a Lustre target disk. This includes upgrading old (pre-Lustre 1.6) disks. This does not reformat the disk or erase the target information, but modifying the configuration information can result in an unusable file system.</para>
+ <title>Description</title>
+ <para>tunefs.lustre is used to modify configuration information on a Lustre target disk. This includes upgrading old (pre-Lustre 1.6) disks. This does not reformat the disk or erase the target information, but modifying the configuration information can result in an unusable file system.</para>
<informaltable frame="none">
<tgroup cols="2">
<colspec colname="c1" colwidth="5*"/>
<imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
</imageobject>
</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis><anchor xml:id="dbdoclet.50438219_pgfId-1317083" xreflabel=""/>Changes made here affect a file system only when the target is mounted the next time.</para></entry>
+ <entry><para><emphasis role="bold">Caution -</emphasis>Changes made here affect a file system only when the target is mounted the next time.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317084" xreflabel=""/>With tunefs.lustre, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old tunefs.lustre parameters and just use newly-specified parameters, run:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317085" xreflabel=""/>$ tunefs.lustre --erase-params --param=<new parameters> </screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317086" xreflabel=""/>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <obd|fsname>.<obdtype>.<proc_file_name>=<value>. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317087" xreflabel=""/>$ tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1
+ <para>With tunefs.lustre, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old tunefs.lustre parameters and just use newly-specified parameters, run:</para>
+ <screen>$ tunefs.lustre --erase-params --param=<new parameters> </screen>
+ <para>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <obd|fsname>.<obdtype>.<proc_file_name>=<value>. For example:</para>
+ <screen>$ tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317089" xreflabel=""/>Options</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317193" xreflabel=""/>The tunefs.lustre options are listed and explained below.</para>
+ <title>Options</title>
+ <para>The tunefs.lustre options are listed and explained below.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317092" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438219_pgfId-1317096" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317098" xreflabel=""/><emphasis role="bold">--comment</emphasis>=<emphasis>comment</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317102" xreflabel=""/>Sets a user comment about this disk, ignored by Lustre.</para></entry>
+ <entry><para> <emphasis role="bold">--comment</emphasis>=<emphasis>comment</emphasis></para></entry>
+ <entry><para> Sets a user comment about this disk, ignored by Lustre.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317104" xreflabel=""/><emphasis role="bold">--dryrun</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317108" xreflabel=""/>Only prints what would be done; does not affect the disk.</para></entry>
+ <entry><para> <emphasis role="bold">--dryrun</emphasis></para></entry>
+ <entry><para> Only prints what would be done; does not affect the disk.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317110" xreflabel=""/><emphasis role="bold">--erase-params</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317114" xreflabel=""/>Removes all previous parameter information.</para></entry>
+ <entry><para> <emphasis role="bold">--erase-params</emphasis></para></entry>
+ <entry><para> Removes all previous parameter information.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317116" xreflabel=""/><emphasis role="bold">--failnode</emphasis>=<emphasis>nid</emphasis>,...</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317120" xreflabel=""/>Sets the NID(s) of a failover partner. This option can be repeated as needed.</para></entry>
+ <entry><para> <emphasis role="bold">--failnode</emphasis>=<emphasis>nid</emphasis>,...</para></entry>
+ <entry><para> Sets the NID(s) of a failover partner. This option can be repeated as needed.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317122" xreflabel=""/><emphasis role="bold">--fsname</emphasis>=<emphasis>filesystem_name</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317126" xreflabel=""/>The Lustre file system of which this service will be a part. The default file system name is 'lustreâ€.</para></entry>
+ <entry><para> <emphasis role="bold">--fsname</emphasis>=<emphasis>filesystem_name</emphasis></para></entry>
+ <entry><para> The Lustre file system of which this service will be a part. The default file system name is 'lustreâ€.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317128" xreflabel=""/><emphasis role="bold">--index</emphasis>=index</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317132" xreflabel=""/>Forces a particular OST or MDT index.</para></entry>
+ <entry><para> <emphasis role="bold">--index</emphasis>=index</para></entry>
+ <entry><para> Forces a particular OST or MDT index.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317134" xreflabel=""/><emphasis role="bold">--mountfsoptions</emphasis>=opts</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317138" xreflabel=""/>Sets the mount options used when the backing file system is mounted.</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317139" xreflabel=""/><emphasis role="bold">CAUTION</emphasis>: Unlike earlier versions of tunefs.lustre, this version completely replaces the existing mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317140" xreflabel=""/>The defaults for ldiskfs are:</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317141" xreflabel=""/>OST: <emphasis>errors=remount-ro,mballoc,extents</emphasis>;</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317142" xreflabel=""/>MGS/MDT: <emphasis>errors=remount-ro,iopen_nopriv,user_xattr</emphasis></para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317143" xreflabel=""/>Do not alter the default mount options unless you know what you are doing.</para></entry>
+ <entry><para> <emphasis role="bold">--mountfsoptions</emphasis>=opts</para></entry>
+ <entry><para> Sets the mount options used when the backing file system is mounted.</para><para><emphasis role="bold">CAUTION</emphasis>: Unlike earlier versions of tunefs.lustre, this version completely replaces the existing mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para><para>The defaults for ldiskfs are:</para><para>OST: <emphasis>errors=remount-ro,mballoc,extents</emphasis>;</para><para>MGS/MDT: <emphasis>errors=remount-ro,iopen_nopriv,user_xattr</emphasis></para><para>Do not alter the default mount options unless you know what you are doing.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317145" xreflabel=""/><emphasis role="bold">--network</emphasis>=net,...</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317149" xreflabel=""/>Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para></entry>
+ <entry><para> <emphasis role="bold">--network</emphasis>=net,...</para></entry>
+ <entry><para> Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317151" xreflabel=""/><emphasis role="bold">--mgs</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317155" xreflabel=""/>Adds a configuration management service to this target.</para></entry>
+ <entry><para> <emphasis role="bold">--mgs</emphasis></para></entry>
+ <entry><para> Adds a configuration management service to this target.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317157" xreflabel=""/><emphasis role="bold">--msgnode</emphasis>=<emphasis>nid</emphasis>,...</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317161" xreflabel=""/>Sets the NID(s) of the MGS node; required for all targets other than the MGS.</para></entry>
+ <entry><para> <emphasis role="bold">--msgnode</emphasis>=<emphasis>nid</emphasis>,...</para></entry>
+ <entry><para> Sets the NID(s) of the MGS node; required for all targets other than the MGS.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317163" xreflabel=""/><emphasis role="bold">--nomgs</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317167" xreflabel=""/>Removes a configuration management service to this target.</para></entry>
+ <entry><para> <emphasis role="bold">--nomgs</emphasis></para></entry>
+ <entry><para> Removes a configuration management service to this target.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317169" xreflabel=""/><emphasis role="bold">--quiet</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317173" xreflabel=""/>Prints less information.</para></entry>
+ <entry><para> <emphasis role="bold">--quiet</emphasis></para></entry>
+ <entry><para> Prints less information.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317175" xreflabel=""/><emphasis role="bold">--verbose</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317179" xreflabel=""/>Prints more information.</para></entry>
+ <entry><para> <emphasis role="bold">--verbose</emphasis></para></entry>
+ <entry><para> Prints more information.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317181" xreflabel=""/><emphasis role="bold">--writeconf</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1317185" xreflabel=""/>Erases all configuration logs for the file system to which this MDT belongs, and regenerates them. This is dangerous operation. All clients must be unmounted and servers for this file system should be stopped. All targets (OSTs/MDTs) must then be restarted to regenerate the logs. No clients should be started until all targets have restarted.</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317186" xreflabel=""/> </para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317187" xreflabel=""/>The correct order of operations is:</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317188" xreflabel=""/>* Unmount all clients on the file system</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317189" xreflabel=""/>* Unmount the MDT and all OSTs on the file system</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317190" xreflabel=""/>* Run <emphasis role="bold">tunefs.lustre --writeconf <device></emphasis> on every server</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317191" xreflabel=""/>* Mount the MDT and OSTs</para><para><anchor xml:id="dbdoclet.50438219_pgfId-1317192" xreflabel=""/>* Mount the clients</para></entry>
+ <entry><para> <emphasis role="bold">--writeconf</emphasis></para></entry>
+ <entry><para> Erases all configuration logs for the file system to which this MDT belongs, and regenerates them. This is dangerous operation. All clients must be unmounted and servers for this file system should be stopped. All targets (OSTs/MDTs) must then be restarted to regenerate the logs. No clients should be started until all targets have restarted.</para><para> </para><para>The correct order of operations is:</para><para>* Unmount all clients on the file system</para><para>* Unmount the MDT and all OSTs on the file system</para><para>* Run <emphasis role="bold">tunefs.lustre --writeconf <device></emphasis> on every server</para><para>* Mount the MDT and OSTs</para><para>* Mount the clients</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317194" xreflabel=""/>Examples</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317195" xreflabel=""/>Change the MGS's NID address. (This should be done on each target disk, since they should all contact the same MGS.)</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317196" xreflabel=""/>tunefs.lustre --erase-param --mgsnode=<new_nid> --writeconf /dev/sda
+ <title>Examples</title>
+ <para>Change the MGS's NID address. (This should be done on each target disk, since they should all contact the same MGS.)</para>
+ <screen>tunefs.lustre --erase-param --mgsnode=<new_nid> --writeconf /dev/sda
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1317197" xreflabel=""/>Add a failover NID location for this target.</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1317198" xreflabel=""/>tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda
+ <para>Add a failover NID location for this target.</para>
+ <screen>tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1317199" xreflabel=""/>See Also</title>
+ <title>See Also</title>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438219_75432"/></para>
</section>
<section xml:id="dbdoclet.50438219_99928">
<title>36.19 Additional System <anchor xml:id="dbdoclet.50438219_marker-1313048" xreflabel=""/>Configuration Utilities</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294808" xreflabel=""/>This section describes additional system configuration utilities for Lustre.</para>
+ <para>This section describes additional system configuration utilities for Lustre.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438219_pgfId-1294829" xreflabel=""/>36.19.1 Application <anchor xml:id="dbdoclet.50438219_marker-1313051" xreflabel=""/>Profiling Utilities</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294830" xreflabel=""/>The following utilities are located in /usr/bin.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294832" xreflabel=""/><emphasis role="bold">lustre_req_history.sh</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294833" xreflabel=""/>The lustre_req_history.sh utility (run from a client), assembles as much Lustre RPC request history as possible from the local node and from the servers that were contacted, providing a better picture of the coordinated network activity.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294835" xreflabel=""/><emphasis role="bold">llstat.sh</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294836" xreflabel=""/>The llstat.sh utility (improved in Lustre 1.6), handles a wider range of /proc files, and has command line switches to produce more graphable output.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294838" xreflabel=""/><emphasis role="bold">plot-llstat.sh</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294839" xreflabel=""/>The plot-llstat.sh utility plots the output from llstat.sh using gnuplot.</para>
+ <title>36.19.1 Application <anchor xml:id="dbdoclet.50438219_marker-1313051" xreflabel=""/>Profiling Utilities</title>
+ <para>The following utilities are located in /usr/bin.</para>
+ <para><emphasis role="bold">lustre_req_history.sh</emphasis></para>
+ <para>The lustre_req_history.sh utility (run from a client), assembles as much Lustre RPC request history as possible from the local node and from the servers that were contacted, providing a better picture of the coordinated network activity.</para>
+ <para><emphasis role="bold">llstat.sh</emphasis></para>
+ <para>The llstat.sh utility (improved in Lustre 1.6), handles a wider range of /proc files, and has command line switches to produce more graphable output.</para>
+ <para><emphasis role="bold">plot-llstat.sh</emphasis></para>
+ <para>The plot-llstat.sh utility plots the output from llstat.sh using gnuplot.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438219_pgfId-1294840" xreflabel=""/>36.19.2 More /proc Statistics for <anchor xml:id="dbdoclet.50438219_marker-1313052" xreflabel=""/>Application Profiling</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294841" xreflabel=""/>The following utilities provide additional statistics.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294843" xreflabel=""/><emphasis role="bold">vfs_ops_stats</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294844" xreflabel=""/>The client vfs_ops_stats utility tracks Linux VFS operation calls into Lustre for a single PID, PPID, GID or everything.</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1294845" xreflabel=""/>/proc/fs/lustre/llite/*/vfs_ops_stats
-<anchor xml:id="dbdoclet.50438219_pgfId-1294846" xreflabel=""/>/proc/fs/lustre/llite/*/vfs_track_[pid|ppid|gid]
-</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294848" xreflabel=""/><emphasis role="bold">extents_stats</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294849" xreflabel=""/>The client extents_stats utility shows the size distribution of I/O calls from the client (cumulative and by process).</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1294850" xreflabel=""/>/proc/fs/lustre/llite/*/extents_stats, extents_stats_per_process
-</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294852" xreflabel=""/><emphasis role="bold">offset_stats</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294853" xreflabel=""/>The client offset_stats utility shows the read/write seek activity of a client by offsets and ranges.</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1294854" xreflabel=""/>/proc/fs/lustre/llite/*/offset_stats
-</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294855" xreflabel=""/>Lustre 1.6 included per-client and improved MDT statistics:</para>
+ <title>36.19.2 More /proc Statistics for <anchor xml:id="dbdoclet.50438219_marker-1313052" xreflabel=""/>Application Profiling</title>
+ <para>The following utilities provide additional statistics.</para>
+ <para><emphasis role="bold">vfs_ops_stats</emphasis></para>
+ <para>The client vfs_ops_stats utility tracks Linux VFS operation calls into Lustre for a single PID, PPID, GID or everything.</para>
+ <screen>/proc/fs/lustre/llite/*/vfs_ops_stats
+/proc/fs/lustre/llite/*/vfs_track_[pid|ppid|gid]
+</screen>
+ <para><emphasis role="bold">extents_stats</emphasis></para>
+ <para>The client extents_stats utility shows the size distribution of I/O calls from the client (cumulative and by process).</para>
+ <screen>/proc/fs/lustre/llite/*/extents_stats, extents_stats_per_process
+</screen>
+ <para><emphasis role="bold">offset_stats</emphasis></para>
+ <para>The client offset_stats utility shows the read/write seek activity of a client by offsets and ranges.</para>
+ <screen>/proc/fs/lustre/llite/*/offset_stats
+</screen>
+ <para>Lustre 1.6 included per-client and improved MDT statistics:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294856" xreflabel=""/> Per-client statistics tracked on the servers</para>
+ <para> Per-client statistics tracked on the servers</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294857" xreflabel=""/>Each MDT and OST now tracks LDLM and operations statistics for every connected client, for comparisons and simpler collection of distributed job statistics.</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1294858" xreflabel=""/>/proc/fs/lustre/mds|obdfilter/*/exports/
+ <para>Each MDT and OST now tracks LDLM and operations statistics for every connected client, for comparisons and simpler collection of distributed job statistics.</para>
+ <screen>/proc/fs/lustre/mds|obdfilter/*/exports/
</screen>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294859" xreflabel=""/> Improved MDT statistics</para>
+ <para> Improved MDT statistics</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294860" xreflabel=""/>More detailed MDT operations statistics are collected for better profiling.</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1294861" xreflabel=""/>/proc/fs/lustre/mds/*/stats
+ <para>More detailed MDT operations statistics are collected for better profiling.</para>
+ <screen>/proc/fs/lustre/mds/*/stats
</screen>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438219_pgfId-1294862" xreflabel=""/>36.19.3 Testing / Debugging <anchor xml:id="dbdoclet.50438219_marker-1313053" xreflabel=""/>Utilities</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294863" xreflabel=""/>Lustre offers the following test and debugging utilities.</para>
+ <title>36.19.3 Testing / Debugging <anchor xml:id="dbdoclet.50438219_marker-1313053" xreflabel=""/>Utilities</title>
+ <para>Lustre offers the following test and debugging utilities.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1311299" xreflabel=""/>loadgen</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311300" xreflabel=""/>The Load Generator (loadgen) is a test program designed to simulate large numbers of Lustre clients connecting and writing to an OST. The loadgen utility is located at lustre/utils/loadgen (in a build directory) or at /usr/sbin/loadgen (from an RPM).</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1312081" xreflabel=""/>Loadgen offers the ability to run this test:</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311221" xreflabel=""/> 1. Start an arbitrary number of (echo) clients.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311232" xreflabel=""/> 2. Start and connect to an echo server, instead of a real OST.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311226" xreflabel=""/> 3. Create/bulk_write/delete objects on any number of echo clients simultaneously.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311244" xreflabel=""/>Currently, the maximum number of clients is limited by MAX_OBD_DEVICES and the amount of memory available.</para>
+ <title>loadgen</title>
+ <para>The Load Generator (loadgen) is a test program designed to simulate large numbers of Lustre clients connecting and writing to an OST. The loadgen utility is located at lustre/utils/loadgen (in a build directory) or at /usr/sbin/loadgen (from an RPM).</para>
+ <para>Loadgen offers the ability to run this test:</para>
+ <para> 1. Start an arbitrary number of (echo) clients.</para>
+ <para> 2. Start and connect to an echo server, instead of a real OST.</para>
+ <para> 3. Create/bulk_write/delete objects on any number of echo clients simultaneously.</para>
+ <para>Currently, the maximum number of clients is limited by MAX_OBD_DEVICES and the amount of memory available.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1311320" xreflabel=""/>Usage</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311311" xreflabel=""/>The loadgen utility can be run locally on the OST server machine or remotely from any LNET host. The device command can take an optional NID as a parameter; if unspecified, the first local NID found is used.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311331" xreflabel=""/>The obdecho module must be loaded by hand before running loadgen.</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1311341" xreflabel=""/># cd lustre/utils/
-<anchor xml:id="dbdoclet.50438219_pgfId-1311420" xreflabel=""/># insmod ../obdecho/obdecho.ko
-<anchor xml:id="dbdoclet.50438219_pgfId-1311423" xreflabel=""/># ./loadgen
-<anchor xml:id="dbdoclet.50438219_pgfId-1311426" xreflabel=""/>loadgen> h
-<anchor xml:id="dbdoclet.50438219_pgfId-1311429" xreflabel=""/>This is a test program used to simulate large numbers of clients. The echo \
+ <title>Usage</title>
+ <para>The loadgen utility can be run locally on the OST server machine or remotely from any LNET host. The device command can take an optional NID as a parameter; if unspecified, the first local NID found is used.</para>
+ <para>The obdecho module must be loaded by hand before running loadgen.</para>
+ <screen># cd lustre/utils/
+# insmod ../obdecho/obdecho.ko
+# ./loadgen
+loadgen> h
+This is a test program used to simulate large numbers of clients. The echo \
obds are used, so the obdecho module must be loaded.
-<anchor xml:id="dbdoclet.50438219_pgfId-1311436" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438219_pgfId-1311437" xreflabel=""/>Typical usage would be:
-<anchor xml:id="dbdoclet.50438219_pgfId-1311440" xreflabel=""/>loadgen> dev lustre-OST0000 set the target device
-<anchor xml:id="dbdoclet.50438219_pgfId-1311443" xreflabel=""/>loadgen> start 20 start 20 echo clients
-<anchor xml:id="dbdoclet.50438219_pgfId-1311446" xreflabel=""/>loadgen> wr 10 5 have 10 clients do simultaneous brw_write \
+
+Typical usage would be:
+loadgen> dev lustre-OST0000 set the target device
+loadgen> start 20 start 20 echo clients
+loadgen> wr 10 5 have 10 clients do simultaneous brw_write \
tests 5 times each
-<anchor xml:id="dbdoclet.50438219_pgfId-1311451" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438219_pgfId-1311452" xreflabel=""/>Available commands are:
-<anchor xml:id="dbdoclet.50438219_pgfId-1311462" xreflabel=""/> device
-<anchor xml:id="dbdoclet.50438219_pgfId-1311472" xreflabel=""/> dl
-<anchor xml:id="dbdoclet.50438219_pgfId-1311482" xreflabel=""/> echosrv
-<anchor xml:id="dbdoclet.50438219_pgfId-1311492" xreflabel=""/> start
-<anchor xml:id="dbdoclet.50438219_pgfId-1311502" xreflabel=""/> verbose
-<anchor xml:id="dbdoclet.50438219_pgfId-1311512" xreflabel=""/> wait
-<anchor xml:id="dbdoclet.50438219_pgfId-1311522" xreflabel=""/> write
-<anchor xml:id="dbdoclet.50438219_pgfId-1311532" xreflabel=""/> help
-<anchor xml:id="dbdoclet.50438219_pgfId-1311542" xreflabel=""/> exit
-<anchor xml:id="dbdoclet.50438219_pgfId-1311552" xreflabel=""/> quit
-<anchor xml:id="dbdoclet.50438219_pgfId-1311557" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438219_pgfId-1311558" xreflabel=""/>For more help type: help command-name
-<anchor xml:id="dbdoclet.50438219_pgfId-1311561" xreflabel=""/>loadgen>
-<anchor xml:id="dbdoclet.50438219_pgfId-1311564" xreflabel=""/>loadgen> device lustre-OST0000 192.168.0.21@tcp
-<anchor xml:id="dbdoclet.50438219_pgfId-1311567" xreflabel=""/>Added uuid OSS_UUID: 192.168.0.21@tcp
-<anchor xml:id="dbdoclet.50438219_pgfId-1311570" xreflabel=""/>Target OST name is 'lustre-OST0000'
-<anchor xml:id="dbdoclet.50438219_pgfId-1311573" xreflabel=""/>loadgen>
-<anchor xml:id="dbdoclet.50438219_pgfId-1311576" xreflabel=""/>loadgen> st 4
-<anchor xml:id="dbdoclet.50438219_pgfId-1311579" xreflabel=""/>start 0 to 4
-<anchor xml:id="dbdoclet.50438219_pgfId-1311582" xreflabel=""/>./loadgen: running thread #1
-<anchor xml:id="dbdoclet.50438219_pgfId-1311585" xreflabel=""/>./loadgen: running thread #2
-<anchor xml:id="dbdoclet.50438219_pgfId-1311588" xreflabel=""/>./loadgen: running thread #3
-<anchor xml:id="dbdoclet.50438219_pgfId-1311591" xreflabel=""/>./loadgen: running thread #4
-<anchor xml:id="dbdoclet.50438219_pgfId-1311594" xreflabel=""/>loadgen> wr 4 5
-<anchor xml:id="dbdoclet.50438219_pgfId-1311597" xreflabel=""/>Estimate 76 clients before we run out of grant space (155872K / 2097152)
-<anchor xml:id="dbdoclet.50438219_pgfId-1311600" xreflabel=""/>1: i0
-<anchor xml:id="dbdoclet.50438219_pgfId-1311603" xreflabel=""/>2: i0
-<anchor xml:id="dbdoclet.50438219_pgfId-1311606" xreflabel=""/>4: i0
-<anchor xml:id="dbdoclet.50438219_pgfId-1311609" xreflabel=""/>3: i0
-<anchor xml:id="dbdoclet.50438219_pgfId-1311612" xreflabel=""/>1: done (0)
-<anchor xml:id="dbdoclet.50438219_pgfId-1311615" xreflabel=""/>2: done (0)
-<anchor xml:id="dbdoclet.50438219_pgfId-1311618" xreflabel=""/>4: done (0)
-<anchor xml:id="dbdoclet.50438219_pgfId-1311621" xreflabel=""/>3: done (0)
-<anchor xml:id="dbdoclet.50438219_pgfId-1311624" xreflabel=""/>wrote 25MB in 1.419s (17.623 MB/s)
-<anchor xml:id="dbdoclet.50438219_pgfId-1311627" xreflabel=""/>loadgen>
-</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311630" xreflabel=""/>The loadgen utility prints periodic status messages; message output can be controlled with the verbose command.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311657" xreflabel=""/>To insure that a file can be written to (a requirement of write cache), OSTs reserve ("grants"), chunks of space for each newly-created file. A grant may cause an OST to report that it is out of space, even though there is plenty of space on the disk, because the space is "reserved" by other files. The loadgen utility estimates the number of simultaneous open files as the disk size divided by the grant size and reports that number when the write tests are first started.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311332" xreflabel=""/><emphasis role="bold">Echo Server</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311333" xreflabel=""/>The loadgen utility can start an echo server. On another node, loadgen can specify the echo server as the device, thus creating a network-only test environment.</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1311700" xreflabel=""/>loadgen> echosrv
-<anchor xml:id="dbdoclet.50438219_pgfId-1311723" xreflabel=""/>loadgen> dl
-<anchor xml:id="dbdoclet.50438219_pgfId-1311730" xreflabel=""/> 0 UP obdecho echosrv echosrv 3
-<anchor xml:id="dbdoclet.50438219_pgfId-1311734" xreflabel=""/> 1 UP ost OSS OSS 3
-</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311335" xreflabel=""/>On another node:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1311752" xreflabel=""/>loadgen> device echosrv cfs21@tcp
-<anchor xml:id="dbdoclet.50438219_pgfId-1311785" xreflabel=""/>Added uuid OSS_UUID: 192.168.0.21@tcp
-<anchor xml:id="dbdoclet.50438219_pgfId-1311788" xreflabel=""/>Target OST name is 'echosrv'
-<anchor xml:id="dbdoclet.50438219_pgfId-1311791" xreflabel=""/>loadgen> st 1
-<anchor xml:id="dbdoclet.50438219_pgfId-1311794" xreflabel=""/>start 0 to 1
-<anchor xml:id="dbdoclet.50438219_pgfId-1311797" xreflabel=""/>./loadgen: running thread #1
-<anchor xml:id="dbdoclet.50438219_pgfId-1311800" xreflabel=""/>loadgen> wr 1
-<anchor xml:id="dbdoclet.50438219_pgfId-1311803" xreflabel=""/>start a test_brw write test on X clients for Y iterations
-<anchor xml:id="dbdoclet.50438219_pgfId-1311806" xreflabel=""/>usage: write <num_clients> <num_iter> [<delay>]
-<anchor xml:id="dbdoclet.50438219_pgfId-1311809" xreflabel=""/>loadgen> wr 1 1
-<anchor xml:id="dbdoclet.50438219_pgfId-1311812" xreflabel=""/>loadgen>
-<anchor xml:id="dbdoclet.50438219_pgfId-1311815" xreflabel=""/>1: i0
-<anchor xml:id="dbdoclet.50438219_pgfId-1311818" xreflabel=""/>1: done (0)
-<anchor xml:id="dbdoclet.50438219_pgfId-1311821" xreflabel=""/>wrote 1MB in 0.029s (34.023 MB/s)
-</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311750" xreflabel=""/>Scripting</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311747" xreflabel=""/>The threads all perform their actions in non-blocking mode; use the wait command to block for the idle state. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1311833" xreflabel=""/>#!/bin/bash
-<anchor xml:id="dbdoclet.50438219_pgfId-1311852" xreflabel=""/>./loadgen << EOF
-<anchor xml:id="dbdoclet.50438219_pgfId-1311855" xreflabel=""/>device lustre-OST0000
-<anchor xml:id="dbdoclet.50438219_pgfId-1311858" xreflabel=""/>st 1
-<anchor xml:id="dbdoclet.50438219_pgfId-1311864" xreflabel=""/>wr 1 10
-<anchor xml:id="dbdoclet.50438219_pgfId-1311863" xreflabel=""/>wait
-<anchor xml:id="dbdoclet.50438219_pgfId-1311867" xreflabel=""/>quit
-<anchor xml:id="dbdoclet.50438219_pgfId-1311870" xreflabel=""/>EOF
-</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311749" xreflabel=""/><emphasis role="bold">Feature Requests</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311881" xreflabel=""/>The loadgen utility is intended to grow into a more comprehensive test tool; feature requests are encouraged. The current feature requests include:</para>
+
+Available commands are:
+ device
+ dl
+ echosrv
+ start
+ verbose
+ wait
+ write
+ help
+ exit
+ quit
+
+For more help type: help command-name
+loadgen>
+loadgen> device lustre-OST0000 192.168.0.21@tcp
+Added uuid OSS_UUID: 192.168.0.21@tcp
+Target OST name is 'lustre-OST0000'
+loadgen>
+loadgen> st 4
+start 0 to 4
+./loadgen: running thread #1
+./loadgen: running thread #2
+./loadgen: running thread #3
+./loadgen: running thread #4
+loadgen> wr 4 5
+Estimate 76 clients before we run out of grant space (155872K / 2097152)
+1: i0
+2: i0
+4: i0
+3: i0
+1: done (0)
+2: done (0)
+4: done (0)
+3: done (0)
+wrote 25MB in 1.419s (17.623 MB/s)
+loadgen>
+</screen>
+ <para>The loadgen utility prints periodic status messages; message output can be controlled with the verbose command.</para>
+ <para>To insure that a file can be written to (a requirement of write cache), OSTs reserve ("grants"), chunks of space for each newly-created file. A grant may cause an OST to report that it is out of space, even though there is plenty of space on the disk, because the space is "reserved" by other files. The loadgen utility estimates the number of simultaneous open files as the disk size divided by the grant size and reports that number when the write tests are first started.</para>
+ <para><emphasis role="bold">Echo Server</emphasis></para>
+ <para>The loadgen utility can start an echo server. On another node, loadgen can specify the echo server as the device, thus creating a network-only test environment.</para>
+ <screen>loadgen> echosrv
+loadgen> dl
+ 0 UP obdecho echosrv echosrv 3
+ 1 UP ost OSS OSS 3
+</screen>
+ <para>On another node:</para>
+ <screen>loadgen> device echosrv cfs21@tcp
+Added uuid OSS_UUID: 192.168.0.21@tcp
+Target OST name is 'echosrv'
+loadgen> st 1
+start 0 to 1
+./loadgen: running thread #1
+loadgen> wr 1
+start a test_brw write test on X clients for Y iterations
+usage: write <num_clients> <num_iter> [<delay>]
+loadgen> wr 1 1
+loadgen>
+1: i0
+1: done (0)
+wrote 1MB in 0.029s (34.023 MB/s)
+</screen>
+ <para>Scripting</para>
+ <para>The threads all perform their actions in non-blocking mode; use the wait command to block for the idle state. For example:</para>
+ <screen>#!/bin/bash
+./loadgen << EOF
+device lustre-OST0000
+st 1
+wr 1 10
+wait
+quit
+EOF
+</screen>
+ <para><emphasis role="bold">Feature Requests</emphasis></para>
+ <para>The loadgen utility is intended to grow into a more comprehensive test tool; feature requests are encouraged. The current feature requests include:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311883" xreflabel=""/> Locking simulation</para>
+ <para> Locking simulation</para>
</listitem>
<listitem>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311917" xreflabel=""/> Many (echo) clients cache locks for the specified resource at the same time.</para>
+ <para> Many (echo) clients cache locks for the specified resource at the same time.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311927" xreflabel=""/> Many (echo) clients enqueue locks for the specified resource simultaneously.</para>
+ <para> Many (echo) clients enqueue locks for the specified resource simultaneously.</para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311888" xreflabel=""/> obdsurvey functionality</para>
+ <para> obdsurvey functionality</para>
</listitem>
<listitem>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1311941" xreflabel=""/> Fold the Lustre I/O kit's obdsurvey script functionality into loadgen</para>
+ <para> Fold the Lustre I/O kit's obdsurvey script functionality into loadgen</para>
</listitem>
</itemizedlist>
</itemizedlist>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1315452" xreflabel=""/>llog_reader</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1315453" xreflabel=""/>The llog_reader utility translates a Lustre configuration log into human-readable form.</para>
+ <title>llog_reader</title>
+ <para>The llog_reader utility translates a Lustre configuration log into human-readable form.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1315454" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1315455" xreflabel=""/>llog_reader filename
+ <title>Synopsis</title>
+ <screen>llog_reader filename
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1315458" xreflabel=""/>Description</title>
+ <title>Description</title>
<para>llog_reader parses the binary format of Lustre's on-disk configuration logs. It can only read the logs. Use tunefs.lustre to write to them.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1315459" xreflabel=""/>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs, then use llog_reader to dump the log file's contents. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1315460" xreflabel=""/>mount -t ldiskfs /dev/sda /mnt/mgs
-<anchor xml:id="dbdoclet.50438219_pgfId-1315461" xreflabel=""/>llog_reader /mnt/mgs/CONFIGS/tfs-client
+ <para>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs, then use llog_reader to dump the log file's contents. For example:</para>
+ <screen>mount -t ldiskfs /dev/sda /mnt/mgs
+llog_reader /mnt/mgs/CONFIGS/tfs-client
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1315462" xreflabel=""/>To examine the same log file on a running Lustre server, use the ldiskfs-enabled debugfs utility (called debug.ldiskfs on some distributions) to extract the file. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1315463" xreflabel=""/>debugfs -c -R 'dump CONFIGS/tfs-client /tmp/tfs-client' /dev/sda
-<anchor xml:id="dbdoclet.50438219_pgfId-1315464" xreflabel=""/>llog_reader /tmp/tfs-client
+ <para>To examine the same log file on a running Lustre server, use the ldiskfs-enabled debugfs utility (called debug.ldiskfs on some distributions) to extract the file. For example:</para>
+ <screen>debugfs -c -R 'dump CONFIGS/tfs-client /tmp/tfs-client' /dev/sda
+llog_reader /tmp/tfs-client
</screen>
<informaltable frame="none">
<tgroup cols="2">
<imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
</imageobject>
</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis><anchor xml:id="dbdoclet.50438219_pgfId-1315476" xreflabel=""/>Although they are stored in the CONFIGS directory, mountdata files do not use the config log format and will confuse llog_reader.</para></entry>
+ <entry><para><emphasis role="bold">Caution -</emphasis>Although they are stored in the CONFIGS directory, mountdata files do not use the config log format and will confuse llog_reader.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1315477" xreflabel=""/><emphasis role="bold">See Also</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1315478" xreflabel=""/><xref linkend="dbdoclet.50438219_39574"/></para>
+ <para><emphasis role="bold">See Also</emphasis></para>
+ <para><xref linkend="dbdoclet.50438219_39574"/></para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1294873" xreflabel=""/>lr_reader</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1294874" xreflabel=""/>The lr_reader utility translates a last received (last_rcvd) file into human-readable form.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318389" xreflabel=""/>The following utilites are part of the Lustre I/O kit. For more information, see <xref linkend='benchmarkingtests'/>.</para>
+ <title>lr_reader</title>
+ <para>The lr_reader utility translates a last received (last_rcvd) file into human-readable form.</para>
+ <para>The following utilites are part of the Lustre I/O kit. For more information, see <xref linkend='benchmarkingtests'/>.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318396" xreflabel=""/>sgpdd_survey</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318397" xreflabel=""/>The sgpdd_survey utility tests 'bare metal' performance, bypassing as much of the kernel as possible. The sgpdd_survey tool does not require Lustre, but it does require the sgp_dd package.</para>
+ <title>sgpdd_survey</title>
+ <para>The sgpdd_survey utility tests 'bare metal' performance, bypassing as much of the kernel as possible. The sgpdd_survey tool does not require Lustre, but it does require the sgp_dd package.</para>
<informaltable frame="none">
<tgroup cols="2">
<colspec colname="c1" colwidth="5*"/>
<imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
</imageobject>
</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis><anchor xml:id="dbdoclet.50438219_pgfId-1318398" xreflabel=""/>The sgpdd_survey utility erases all data on the device.</para></entry>
+ <entry><para><emphasis role="bold">Caution -</emphasis>The sgpdd_survey utility erases all data on the device.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318399" xreflabel=""/>obdfilter_survey</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318400" xreflabel=""/>The obdfilter_survey utility is a shell script that tests performance of isolated OSTS, the network via echo clients, and an end-to-end test.</para>
+ <title>obdfilter_survey</title>
+ <para>The obdfilter_survey utility is a shell script that tests performance of isolated OSTS, the network via echo clients, and an end-to-end test.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318401" xreflabel=""/>ior-survey</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318402" xreflabel=""/>The ior-survey utility is a script used to run the IOR benchmark. Lustre includes IOR version 2.8.6.</para>
+ <title>ior-survey</title>
+ <para>The ior-survey utility is a script used to run the IOR benchmark. Lustre includes IOR version 2.8.6.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318403" xreflabel=""/>ost_survey</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318404" xreflabel=""/>The ost_survey utility is an OST performance survey that tests client-to-disk performance of the individual OSTs in a Lustre file system.</para>
+ <title>ost_survey</title>
+ <para>The ost_survey utility is an OST performance survey that tests client-to-disk performance of the individual OSTs in a Lustre file system.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1318405" xreflabel=""/>stats-collect</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1318406" xreflabel=""/>The stats-collect utility contains scripts used to collect application profiling information from Lustre clients and servers.</para>
+ <title>stats-collect</title>
+ <para>The stats-collect utility contains scripts used to collect application profiling information from Lustre clients and servers.</para>
</section>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438219_pgfId-1295755" xreflabel=""/>36.19.4 Flock <anchor xml:id="dbdoclet.50438219_marker-1295910" xreflabel=""/>Feature</title>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1295756" xreflabel=""/>Lustre now includes the flock feature, which provides file locking support. Flock describes classes of file locks known as 'flocks'. Flock can apply or remove a lock on an open file as specified by the user. However, a single file may not, simultaneously, have both shared and exclusive locks.</para>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1295757" xreflabel=""/>By default, the flock utility is disabled on Lustre. Two modes are available.</para>
+ <title>36.19.4 Flock <anchor xml:id="dbdoclet.50438219_marker-1295910" xreflabel=""/>Feature</title>
+ <para>Lustre now includes the flock feature, which provides file locking support. Flock describes classes of file locks known as 'flocks'. Flock can apply or remove a lock on an open file as specified by the user. However, a single file may not, simultaneously, have both shared and exclusive locks.</para>
+ <para>By default, the flock utility is disabled on Lustre. Two modes are available.</para>
<informaltable frame="none">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1295805" xreflabel=""/><emphasis role="bold">local mode</emphasis></para></entry>
- <entry><para>
<anchor xml:id="dbdoclet.50438219_pgfId-1295807" xreflabel=""/> In this mode, locks are coherent on one node (a single-node flock), but not across all clients. To enable it, use -o localflock. This is a client-mount option.</para><note><para>This mode does not impact performance and is appropriate for single-node databases.</para></note></entry>
+ <entry><para> <emphasis role="bold">local mode</emphasis></para></entry>
+ <entry><para> In this mode, locks are coherent on one node (a single-node flock), but not across all clients. To enable it, use -o localflock. This is a client-mount option.</para><note><para>This mode does not impact performance and is appropriate for single-node databases.</para></note></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438219_pgfId-1295809" xreflabel=""/><emphasis role="bold">consistent mode</emphasis></para></entry>
- <entry><para>
<anchor xml:id="dbdoclet.50438219_pgfId-1312474" xreflabel=""/>In this mode, locks are coherent across all clients.</para><para> To enable it, use the -o flock. This is a client-mount option.</para><para><emphasis role="bold">CAUTION</emphasis>: This mode affects the performance of the file being flocked and may affect stability, depending on the Lustre version used. Consider using a newer Lustre version which is more stable. If the consistent mode is enabled and no applications are using flock, then it has no effect.</para></entry>
+ <entry><para> <emphasis role="bold">consistent mode</emphasis></para></entry>
+ <entry><para> In this mode, locks are coherent across all clients.</para><para> To enable it, use the -o flock. This is a client-mount option.</para><para><emphasis role="bold">CAUTION</emphasis>: This mode affects the performance of the file being flocked and may affect stability, depending on the Lustre version used. Consider using a newer Lustre version which is more stable. If the consistent mode is enabled and no applications are using flock, then it has no effect.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1295883" xreflabel=""/>A call to use flock may be blocked if another process is holding an incompatible lock. Locks created using flock are applicable for an open file table entry. Therefore, a single process may hold only one type of lock (shared or exclusive) on a single file. Subsequent flock calls on a file that is already locked converts the existing lock to the new lock mode.</para>
+ <para>A call to use flock may be blocked if another process is holding an incompatible lock. Locks created using flock are applicable for an open file table entry. Therefore, a single process may hold only one type of lock (shared or exclusive) on a single file. Subsequent flock calls on a file that is already locked converts the existing lock to the new lock mode.</para>
<section remap="h4">
- <title><anchor xml:id="dbdoclet.50438219_pgfId-1295763" xreflabel=""/>36.19.4.1 Example</title>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1295764" xreflabel=""/>$ mount -t lustre -o flock mds@tcp0:/lustre /mnt/client
+ <title>36.19.4.1 Example</title>
+ <screen>$ mount -t lustre -o flock mds@tcp0:/lustre /mnt/client
</screen>
- <para><anchor xml:id="dbdoclet.50438219_pgfId-1295765" xreflabel=""/>You can check it in /etc/mtab. It should look like,</para>
- <screen><anchor xml:id="dbdoclet.50438219_pgfId-1295766" xreflabel=""/>mds@tcp0:/lustre /mnt/client lustre rw,flock 0 0
+ <para>You can check it in /etc/mtab. It should look like,</para>
+ <screen>mds@tcp0:/lustre /mnt/client lustre rw,flock 0 0
</screen>
</section>
</section>
<info>
<title xml:id='troubleshootingrecovery.title'>Troubleshooting Recovery</title>
</info>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1292105" xreflabel=""/>This chapter describes what to do if something goes wrong during recovery. It describes:</para>
+ <para>This chapter describes what to do if something goes wrong during recovery. It describes:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438225_71141"/></para>
<section xml:id="dbdoclet.50438225_71141">
<title>27.1 Recovering from Errors or <anchor xml:id="dbdoclet.50438225_marker-1292184" xreflabel=""/>Corruption on a Backing File System</title>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1291241" xreflabel=""/>When an OSS, MDS, or MGS server crash occurs, it is not necessary to run e2fsck on the file system. ldiskfs journaling ensures that the file system remains coherent. The backing file systems are never accessed directly from the client, so client crashes are not relevant.</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1291243" xreflabel=""/>The only time it is REQUIRED that e2fsck be run on a device is when an event causes problems that ldiskfs journaling is unable to handle, such as a hardware device failure or I/O error. If the ldiskfs kernel code detects corruption on the disk, it mounts the file system as read-only to prevent further corruption, but still allows read access to the device. This appears as error "-30" (EROFS) in the syslogs on the server, e.g.:</para>
- <screen><anchor xml:id="dbdoclet.50438225_pgfId-1291304" xreflabel=""/>Dec 29 14:11:32 mookie kernel: LDISKFS-fs error (device sdz): ldiskfs_looku\
+ <para>When an OSS, MDS, or MGS server crash occurs, it is not necessary to run e2fsck on the file system. ldiskfs journaling ensures that the file system remains coherent. The backing file systems are never accessed directly from the client, so client crashes are not relevant.</para>
+ <para>The only time it is REQUIRED that e2fsck be run on a device is when an event causes problems that ldiskfs journaling is unable to handle, such as a hardware device failure or I/O error. If the ldiskfs kernel code detects corruption on the disk, it mounts the file system as read-only to prevent further corruption, but still allows read access to the device. This appears as error "-30" (EROFS) in the syslogs on the server, e.g.:</para>
+ <screen>Dec 29 14:11:32 mookie kernel: LDISKFS-fs error (device sdz): ldiskfs_looku\
p: unlinked inode 5384166 in dir #145170469
</screen>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1291325" xreflabel=""/>Dec 29 14:11:32 mookie kernel: Remounting filesystem read-only</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1291305" xreflabel=""/>In such a situation, it is normally required that e2fsck only be run on the bad device before placing the device back into service.</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1291245" xreflabel=""/>In the vast majority of cases, Lustre can cope with any inconsistencies it finds on the disk and between other devices in the file system.</para>
+ <para>Dec 29 14:11:32 mookie kernel: Remounting filesystem read-only</para>
+ <para>In such a situation, it is normally required that e2fsck only be run on the bad device before placing the device back into service.</para>
+ <para>In the vast majority of cases, Lustre can cope with any inconsistencies it finds on the disk and between other devices in the file system.</para>
<note><para>lfsck is rarely required for Lustre operation.</para></note>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1291249" xreflabel=""/>For problem analysis, it is strongly recommended that e2fsck be run under a logger, like script, to record all of the output and changes that are made to the file system in case this information is needed later.</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1291251" xreflabel=""/>If time permits, it is also a good idea to first run e2fsck in non-fixing mode (-n option) to assess the type and extent of damage to the file system. The drawback is that in this mode, e2fsck does not recover the file system journal, so there may appear to be file system corruption when none really exists.</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1291253" xreflabel=""/>To address concern about whether corruption is real or only due to the journal not being replayed, you can briefly mount and unmount the ldiskfs filesystem directly on the node with Lustre stopped (NOT via Lustre), using a command similar to:</para>
- <screen><anchor xml:id="dbdoclet.50438225_pgfId-1291255" xreflabel=""/>mount -t ldiskfs /dev/{ostdev} /mnt/ost; umount /mnt/ost
+ <para>For problem analysis, it is strongly recommended that e2fsck be run under a logger, like script, to record all of the output and changes that are made to the file system in case this information is needed later.</para>
+ <para>If time permits, it is also a good idea to first run e2fsck in non-fixing mode (-n option) to assess the type and extent of damage to the file system. The drawback is that in this mode, e2fsck does not recover the file system journal, so there may appear to be file system corruption when none really exists.</para>
+ <para>To address concern about whether corruption is real or only due to the journal not being replayed, you can briefly mount and unmount the ldiskfs filesystem directly on the node with Lustre stopped (NOT via Lustre), using a command similar to:</para>
+ <screen>mount -t ldiskfs /dev/{ostdev} /mnt/ost; umount /mnt/ost
</screen>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1291257" xreflabel=""/>This causes the journal to be recovered.</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1291259" xreflabel=""/>The e2fsck utility works well when fixing file system corruption (better than similar file system recovery tools and a primary reason why ldiskfs was chosen over other file systems for Lustre). However, it is often useful to identify the type of damage that has occurred so an ldiskfs expert can make intelligent decisions about what needs fixing, in place of e2fsck.</para>
- <screen><anchor xml:id="dbdoclet.50438225_pgfId-1291261" xreflabel=""/>root# {stop lustre services for this device, if running}
-<anchor xml:id="dbdoclet.50438225_pgfId-1291375" xreflabel=""/>root# script /tmp/e2fsck.sda
-<anchor xml:id="dbdoclet.50438225_pgfId-1291378" xreflabel=""/>Script started, file is /tmp/e2fsck.sda
-<anchor xml:id="dbdoclet.50438225_pgfId-1291381" xreflabel=""/>root# mount -t ldiskfs /dev/sda /mnt/ost
-<anchor xml:id="dbdoclet.50438225_pgfId-1291384" xreflabel=""/>root# umount /mnt/ost
-<anchor xml:id="dbdoclet.50438225_pgfId-1291387" xreflabel=""/>root# e2fsck -fn /dev/sda # don't fix file system, just check for corrupt\
+ <para>This causes the journal to be recovered.</para>
+ <para>The e2fsck utility works well when fixing file system corruption (better than similar file system recovery tools and a primary reason why ldiskfs was chosen over other file systems for Lustre). However, it is often useful to identify the type of damage that has occurred so an ldiskfs expert can make intelligent decisions about what needs fixing, in place of e2fsck.</para>
+ <screen>root# {stop lustre services for this device, if running}
+root# script /tmp/e2fsck.sda
+Script started, file is /tmp/e2fsck.sda
+root# mount -t ldiskfs /dev/sda /mnt/ost
+root# umount /mnt/ost
+root# e2fsck -fn /dev/sda # don't fix file system, just check for corrupt\
ion
-<anchor xml:id="dbdoclet.50438225_pgfId-1291268" xreflabel=""/>:
-<anchor xml:id="dbdoclet.50438225_pgfId-1291393" xreflabel=""/>[e2fsck output]
-<anchor xml:id="dbdoclet.50438225_pgfId-1291270" xreflabel=""/>:
-<anchor xml:id="dbdoclet.50438225_pgfId-1291399" xreflabel=""/>root# e2fsck -fp /dev/sda # fix filesystem using "prudent" answers (usually\
+:
+[e2fsck output]
+:
+root# e2fsck -fp /dev/sda # fix filesystem using "prudent" answers (usually\
'y')
</screen>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1291234" xreflabel=""/>In addition, the e2fsprogs package contains the lfsck tool, which does distributed coherency checking for the Lustre file system after e2fsck has been run. Running lfsck is NOT required in a large majority of cases, at a small risk of having some leaked space in the file system. To avoid a lengthy downtime, it can be run (with care) after Lustre is started.</para>
+ <para>In addition, the e2fsprogs package contains the lfsck tool, which does distributed coherency checking for the Lustre file system after e2fsck has been run. Running lfsck is NOT required in a large majority of cases, at a small risk of having some leaked space in the file system. To avoid a lengthy downtime, it can be run (with care) after Lustre is started.</para>
</section>
<section xml:id="dbdoclet.50438225_37365">
<title>27.2 Recovering from <anchor xml:id="dbdoclet.50438225_marker-1292186" xreflabel=""/>Corruption in the Lustre File System</title>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1289918" xreflabel=""/>In cases where the MDS or an OST becomes corrupt, you can run a distributed check on the file system to determine what sort of problems exist. Use lfsck to correct any defects found.</para>
+ <para>In cases where the MDS or an OST becomes corrupt, you can run a distributed check on the file system to determine what sort of problems exist. Use lfsck to correct any defects found.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290095" xreflabel=""/>Stop the Lustre file system.</para>
+ <para>Stop the Lustre file system.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290099" xreflabel=""/>Run e2fsck -f on the individual MDS / OST that had problems to fix any local file system damage.</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290108" xreflabel=""/>We recommend running e2fsck under script, to create a log of changes made to the file system in case it is needed later. After e2fsck is run, bring up the file system, if necessary, to reduce the outage window.</para>
+ <para>Run e2fsck -f on the individual MDS / OST that had problems to fix any local file system damage.</para>
+ <para>We recommend running e2fsck under script, to create a log of changes made to the file system in case it is needed later. After e2fsck is run, bring up the file system, if necessary, to reduce the outage window.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290111" xreflabel=""/>Run a full e2fsck of the MDS to create a database for lfsck. You <emphasis>must</emphasis> use the -n option for a mounted file system, otherwise you will corrupt the file system.</para>
- <screen><anchor xml:id="dbdoclet.50438225_pgfId-1290120" xreflabel=""/>e2fsck -n -v --mdsdb /tmp/mdsdb /dev/{mdsdev}
+ <para>Run a full e2fsck of the MDS to create a database for lfsck. You <emphasis>must</emphasis> use the -n option for a mounted file system, otherwise you will corrupt the file system.</para>
+ <screen>e2fsck -n -v --mdsdb /tmp/mdsdb /dev/{mdsdev}
</screen>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290115" xreflabel=""/>The mdsdb file can grow fairly large, depending on the number of files in the file system (10 GB or more for millions of files, though the actual file size is larger because the file is sparse). It is quicker to write the file to a local file system due to seeking and small writes. Depending on the number of files, this step can take several hours to complete.</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290147" xreflabel=""/><emphasis role="bold">Example</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438225_pgfId-1290148" xreflabel=""/>e2fsck -n -v --mdsdb /tmp/mdsdb /dev/sdb
-<anchor xml:id="dbdoclet.50438225_pgfId-1290149" xreflabel=""/>e2fsck 1.39.cfs1 (29-May-2006)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290150" xreflabel=""/>Warning: skipping journal recovery because doing a read-only filesystem che\
+ <para>The mdsdb file can grow fairly large, depending on the number of files in the file system (10 GB or more for millions of files, though the actual file size is larger because the file is sparse). It is quicker to write the file to a local file system due to seeking and small writes. Depending on the number of files, this step can take several hours to complete.</para>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <screen>e2fsck -n -v --mdsdb /tmp/mdsdb /dev/sdb
+e2fsck 1.39.cfs1 (29-May-2006)
+Warning: skipping journal recovery because doing a read-only filesystem che\
ck.
-<anchor xml:id="dbdoclet.50438225_pgfId-1290151" xreflabel=""/>lustre-MDT0000 contains a file system with errors, check forced.
-<anchor xml:id="dbdoclet.50438225_pgfId-1290152" xreflabel=""/>Pass 1: Checking inodes, blocks, and sizes
-<anchor xml:id="dbdoclet.50438225_pgfId-1290153" xreflabel=""/>MDS: ost_idx 0 max_id 288
-<anchor xml:id="dbdoclet.50438225_pgfId-1290154" xreflabel=""/>MDS: got 8 bytes = 1 entries in lov_objids
-<anchor xml:id="dbdoclet.50438225_pgfId-1290155" xreflabel=""/>MDS: max_files = 13
-<anchor xml:id="dbdoclet.50438225_pgfId-1290156" xreflabel=""/>MDS: num_osts = 1
-<anchor xml:id="dbdoclet.50438225_pgfId-1290157" xreflabel=""/>mds info db file written
-<anchor xml:id="dbdoclet.50438225_pgfId-1290158" xreflabel=""/>Pass 2: Checking directory structure
-<anchor xml:id="dbdoclet.50438225_pgfId-1290159" xreflabel=""/>Pass 3: Checking directory connectivity
-<anchor xml:id="dbdoclet.50438225_pgfId-1290160" xreflabel=""/>Pass 4: Checking reference counts
-<anchor xml:id="dbdoclet.50438225_pgfId-1290161" xreflabel=""/>Pass 5: Checking group summary information
-<anchor xml:id="dbdoclet.50438225_pgfId-1290162" xreflabel=""/>Free blocks count wrong (656160, counted=656058).
-<anchor xml:id="dbdoclet.50438225_pgfId-1290163" xreflabel=""/>Fix? no
-<anchor xml:id="dbdoclet.50438225_pgfId-1290164" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438225_pgfId-1290165" xreflabel=""/>Free inodes count wrong (786419, counted=786036).
-<anchor xml:id="dbdoclet.50438225_pgfId-1290166" xreflabel=""/>Fix? no
-<anchor xml:id="dbdoclet.50438225_pgfId-1290167" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438225_pgfId-1290168" xreflabel=""/>Pass 6: Acquiring information for lfsck
-<anchor xml:id="dbdoclet.50438225_pgfId-1290169" xreflabel=""/>MDS: max_files = 13
-<anchor xml:id="dbdoclet.50438225_pgfId-1290170" xreflabel=""/>MDS: num_osts = 1
-<anchor xml:id="dbdoclet.50438225_pgfId-1290171" xreflabel=""/>MDS: 'lustre-MDT0000_UUID' mdt idx 0: compat 0x4 rocomp 0x1 incomp 0x4
-<anchor xml:id="dbdoclet.50438225_pgfId-1290172" xreflabel=""/>lustre-MDT0000: ******* WARNING: Filesystem still has errors *******
-<anchor xml:id="dbdoclet.50438225_pgfId-1290173" xreflabel=""/> 13 inodes used (0%)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290174" xreflabel=""/> 2 non-contiguous inodes (15.4%)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290175" xreflabel=""/> # of inodes with ind/dind/tind blocks: 0/0/0
-<anchor xml:id="dbdoclet.50438225_pgfId-1290176" xreflabel=""/>130272 blocks used (16%)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290177" xreflabel=""/> 0 bad blocks
-<anchor xml:id="dbdoclet.50438225_pgfId-1290178" xreflabel=""/> 1 large file
-<anchor xml:id="dbdoclet.50438225_pgfId-1290179" xreflabel=""/> 296 regular files
-<anchor xml:id="dbdoclet.50438225_pgfId-1290180" xreflabel=""/> 91 directories
-<anchor xml:id="dbdoclet.50438225_pgfId-1290181" xreflabel=""/> 0 character device files
-<anchor xml:id="dbdoclet.50438225_pgfId-1290182" xreflabel=""/> 0 block device files
-<anchor xml:id="dbdoclet.50438225_pgfId-1290183" xreflabel=""/> 0 fifos
-<anchor xml:id="dbdoclet.50438225_pgfId-1290184" xreflabel=""/> 0 links
-<anchor xml:id="dbdoclet.50438225_pgfId-1290185" xreflabel=""/> 0 symbolic links (0 fast symbolic links)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290186" xreflabel=""/> 0 sockets
-<anchor xml:id="dbdoclet.50438225_pgfId-1290187" xreflabel=""/> --------
-<anchor xml:id="dbdoclet.50438225_pgfId-1290188" xreflabel=""/> 387 files
+lustre-MDT0000 contains a file system with errors, check forced.
+Pass 1: Checking inodes, blocks, and sizes
+MDS: ost_idx 0 max_id 288
+MDS: got 8 bytes = 1 entries in lov_objids
+MDS: max_files = 13
+MDS: num_osts = 1
+mds info db file written
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+Free blocks count wrong (656160, counted=656058).
+Fix? no
+
+Free inodes count wrong (786419, counted=786036).
+Fix? no
+
+Pass 6: Acquiring information for lfsck
+MDS: max_files = 13
+MDS: num_osts = 1
+MDS: 'lustre-MDT0000_UUID' mdt idx 0: compat 0x4 rocomp 0x1 incomp 0x4
+lustre-MDT0000: ******* WARNING: Filesystem still has errors *******
+ 13 inodes used (0%)
+ 2 non-contiguous inodes (15.4%)
+ # of inodes with ind/dind/tind blocks: 0/0/0
+130272 blocks used (16%)
+ 0 bad blocks
+ 1 large file
+ 296 regular files
+ 91 directories
+ 0 character device files
+ 0 block device files
+ 0 fifos
+ 0 links
+ 0 symbolic links (0 fast symbolic links)
+ 0 sockets
+ --------
+ 387 files
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290139" xreflabel=""/>Make this file accessible on all OSTs, either by using a shared file system or copying the file to the OSTs. The pdcp command is useful here.</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1289927" xreflabel=""/>The pdcp command (installed with pdsh), can be used to copy files to groups of hosts. Pdcp is available here:</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290242" xreflabel=""/><link xl:href="http://sourceforge.net/projects/pdsh">http://sourceforge.net/projects/pdsh</link></para>
+ <para>Make this file accessible on all OSTs, either by using a shared file system or copying the file to the OSTs. The pdcp command is useful here.</para>
+ <para>The pdcp command (installed with pdsh), can be used to copy files to groups of hosts. Pdcp is available here:</para>
+ <para><link xl:href="http://sourceforge.net/projects/pdsh">http://sourceforge.net/projects/pdsh</link></para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1289928" xreflabel=""/>Run a similar e2fsck step on the OSTs. The e2fsck --ostdb command can be run in parallel on all OSTs.</para>
- <screen><anchor xml:id="dbdoclet.50438225_pgfId-1290293" xreflabel=""/>e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ostNdb} \/dev/{ostNdev}
+ <para>Run a similar e2fsck step on the OSTs. The e2fsck --ostdb command can be run in parallel on all OSTs.</para>
+ <screen>e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ostNdb} \/dev/{ostNdev}
</screen>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290266" xreflabel=""/>The mdsdb file is read-only in this step; a single copy can be shared by all OSTs.</para>
+ <para>The mdsdb file is read-only in this step; a single copy can be shared by all OSTs.</para>
<note><para>If the OSTs do not have shared file system access to the MDS, a stub mdsdb file, {mdsdb}.mdshdr, is generated. This can be used instead of the full mdsdb file.</para></note>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1292837" xreflabel=""/><emphasis role="bold">Example:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438225_pgfId-1290319" xreflabel=""/>[root@oss161 ~]# e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb \ /tmp/ostdb /dev/\
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>[root@oss161 ~]# e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb \ /tmp/ostdb /dev/\
sda
-<anchor xml:id="dbdoclet.50438225_pgfId-1290349" xreflabel=""/>e2fsck 1.39.cfs1 (29-May-2006)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290321" xreflabel=""/>Warning: skipping journal recovery because doing a read-only filesystem che\
+e2fsck 1.39.cfs1 (29-May-2006)
+Warning: skipping journal recovery because doing a read-only filesystem che\
ck.
-<anchor xml:id="dbdoclet.50438225_pgfId-1290322" xreflabel=""/>lustre-OST0000 contains a file system with errors, check forced.
-<anchor xml:id="dbdoclet.50438225_pgfId-1290323" xreflabel=""/>Pass 1: Checking inodes, blocks, and sizes
-<anchor xml:id="dbdoclet.50438225_pgfId-1290324" xreflabel=""/>Pass 2: Checking directory structure
-<anchor xml:id="dbdoclet.50438225_pgfId-1290325" xreflabel=""/>Pass 3: Checking directory connectivity
-<anchor xml:id="dbdoclet.50438225_pgfId-1290326" xreflabel=""/>Pass 4: Checking reference counts
-<anchor xml:id="dbdoclet.50438225_pgfId-1290327" xreflabel=""/>Pass 5: Checking group summary information
-<anchor xml:id="dbdoclet.50438225_pgfId-1290328" xreflabel=""/>Free blocks count wrong (989015, counted=817968).
-<anchor xml:id="dbdoclet.50438225_pgfId-1290329" xreflabel=""/>Fix? no
-<anchor xml:id="dbdoclet.50438225_pgfId-1290330" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438225_pgfId-1290331" xreflabel=""/>Free inodes count wrong (262088, counted=261767).
-<anchor xml:id="dbdoclet.50438225_pgfId-1290332" xreflabel=""/>Fix? no
-<anchor xml:id="dbdoclet.50438225_pgfId-1290333" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438225_pgfId-1290334" xreflabel=""/>Pass 6: Acquiring information for lfsck
-<anchor xml:id="dbdoclet.50438225_pgfId-1290335" xreflabel=""/>OST: 'lustre-OST0000_UUID' ost idx 0: compat 0x2 rocomp 0 incomp 0x2
-<anchor xml:id="dbdoclet.50438225_pgfId-1290336" xreflabel=""/>OST: num files = 321
-<anchor xml:id="dbdoclet.50438225_pgfId-1290337" xreflabel=""/>OST: last_id = 321
-<anchor xml:id="dbdoclet.50438225_pgfId-1290338" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438225_pgfId-1290358" xreflabel=""/>lustre-OST0000: ******* WARNING: Filesystem still has errors *******
-<anchor xml:id="dbdoclet.50438225_pgfId-1290359" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438225_pgfId-1290360" xreflabel=""/> 56 inodes used (0%)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290361" xreflabel=""/> 27 non-contiguous inodes (48.2%)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290362" xreflabel=""/> # of inodes with ind/dind/tind blocks: 13/0/0
-<anchor xml:id="dbdoclet.50438225_pgfId-1290363" xreflabel=""/>59561 blocks used (5%)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290364" xreflabel=""/> 0 bad blocks
-<anchor xml:id="dbdoclet.50438225_pgfId-1290365" xreflabel=""/> 1 large file
-<anchor xml:id="dbdoclet.50438225_pgfId-1290366" xreflabel=""/> 329 regular files
-<anchor xml:id="dbdoclet.50438225_pgfId-1290367" xreflabel=""/> 39 directories
-<anchor xml:id="dbdoclet.50438225_pgfId-1290368" xreflabel=""/> 0 character device files
-<anchor xml:id="dbdoclet.50438225_pgfId-1290369" xreflabel=""/> 0 block device files
-<anchor xml:id="dbdoclet.50438225_pgfId-1290370" xreflabel=""/> 0 fifos
-<anchor xml:id="dbdoclet.50438225_pgfId-1290371" xreflabel=""/> 0 links
-<anchor xml:id="dbdoclet.50438225_pgfId-1290372" xreflabel=""/> 0 symbolic links (0 fast symbolic links)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290373" xreflabel=""/> 0 sockets
-<anchor xml:id="dbdoclet.50438225_pgfId-1290374" xreflabel=""/> --------
-<anchor xml:id="dbdoclet.50438225_pgfId-1290375" xreflabel=""/> 368 files
-<anchor xml:id="dbdoclet.50438225_pgfId-1290356" xreflabel=""/>
+lustre-OST0000 contains a file system with errors, check forced.
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+Free blocks count wrong (989015, counted=817968).
+Fix? no
+
+Free inodes count wrong (262088, counted=261767).
+Fix? no
+
+Pass 6: Acquiring information for lfsck
+OST: 'lustre-OST0000_UUID' ost idx 0: compat 0x2 rocomp 0 incomp 0x2
+OST: num files = 321
+OST: last_id = 321
+
+lustre-OST0000: ******* WARNING: Filesystem still has errors *******
+
+ 56 inodes used (0%)
+ 27 non-contiguous inodes (48.2%)
+ # of inodes with ind/dind/tind blocks: 13/0/0
+59561 blocks used (5%)
+ 0 bad blocks
+ 1 large file
+ 329 regular files
+ 39 directories
+ 0 character device files
+ 0 block device files
+ 0 fifos
+ 0 links
+ 0 symbolic links (0 fast symbolic links)
+ 0 sockets
+ --------
+ 368 files
+
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290316" xreflabel=""/>Make the mdsdb file and all ostdb files available on a mounted client and run lfsck to examine the file system. Optionally, correct the defects found by lfsck.</para>
- <screen><anchor xml:id="dbdoclet.50438225_pgfId-1289934" xreflabel=""/>script /root/lfsck.lustre.log
-<anchor xml:id="dbdoclet.50438225_pgfId-1290406" xreflabel=""/>lfsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ost1db} /tmp/{ost2db} ... /lus\
+ <para>Make the mdsdb file and all ostdb files available on a mounted client and run lfsck to examine the file system. Optionally, correct the defects found by lfsck.</para>
+ <screen>script /root/lfsck.lustre.log
+lfsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ost1db} /tmp/{ost2db} ... /lus\
tre/mount/point
</screen>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290428" xreflabel=""/><emphasis role="bold">Example:</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438225_pgfId-1290429" xreflabel=""/>script /root/lfsck.lustre.log
-<anchor xml:id="dbdoclet.50438225_pgfId-1290430" xreflabel=""/>lfsck -n -v --mdsdb /home/mdsdb --ostdb /home/{ost1db} \/mnt/lustre/client/
-<anchor xml:id="dbdoclet.50438225_pgfId-1290431" xreflabel=""/>MDSDB: /home/mdsdb
-<anchor xml:id="dbdoclet.50438225_pgfId-1290432" xreflabel=""/>OSTDB[0]: /home/ostdb
-<anchor xml:id="dbdoclet.50438225_pgfId-1290433" xreflabel=""/>MOUNTPOINT: /mnt/lustre/client/
-<anchor xml:id="dbdoclet.50438225_pgfId-1290434" xreflabel=""/>MDS: max_id 288 OST: max_id 321
-<anchor xml:id="dbdoclet.50438225_pgfId-1290435" xreflabel=""/>lfsck: ost_idx 0: pass1: check for duplicate objects
-<anchor xml:id="dbdoclet.50438225_pgfId-1290436" xreflabel=""/>lfsck: ost_idx 0: pass1 OK (287 files total)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290437" xreflabel=""/>lfsck: ost_idx 0: pass2: check for missing inode objects
-<anchor xml:id="dbdoclet.50438225_pgfId-1290438" xreflabel=""/>lfsck: ost_idx 0: pass2 OK (287 objects)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290439" xreflabel=""/>lfsck: ost_idx 0: pass3: check for orphan objects
-<anchor xml:id="dbdoclet.50438225_pgfId-1290440" xreflabel=""/>[0] uuid lustre-OST0000_UUID
-<anchor xml:id="dbdoclet.50438225_pgfId-1290441" xreflabel=""/>[0] last_id 288
-<anchor xml:id="dbdoclet.50438225_pgfId-1290442" xreflabel=""/>[0] zero-length orphan objid 1
-<anchor xml:id="dbdoclet.50438225_pgfId-1290443" xreflabel=""/>lfsck: ost_idx 0: pass3 OK (321 files total)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290444" xreflabel=""/>lfsck: pass4: check for duplicate object references
-<anchor xml:id="dbdoclet.50438225_pgfId-1290445" xreflabel=""/>lfsck: pass4 OK (no duplicates)
-<anchor xml:id="dbdoclet.50438225_pgfId-1290446" xreflabel=""/>lfsck: fixed 0 errors
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>script /root/lfsck.lustre.log
+lfsck -n -v --mdsdb /home/mdsdb --ostdb /home/{ost1db} \/mnt/lustre/client/
+MDSDB: /home/mdsdb
+OSTDB[0]: /home/ostdb
+MOUNTPOINT: /mnt/lustre/client/
+MDS: max_id 288 OST: max_id 321
+lfsck: ost_idx 0: pass1: check for duplicate objects
+lfsck: ost_idx 0: pass1 OK (287 files total)
+lfsck: ost_idx 0: pass2: check for missing inode objects
+lfsck: ost_idx 0: pass2 OK (287 objects)
+lfsck: ost_idx 0: pass3: check for orphan objects
+[0] uuid lustre-OST0000_UUID
+[0] last_id 288
+[0] zero-length orphan objid 1
+lfsck: ost_idx 0: pass3 OK (321 files total)
+lfsck: pass4: check for duplicate object references
+lfsck: pass4 OK (no duplicates)
+lfsck: fixed 0 errors
</screen>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290426" xreflabel=""/>By default, lfsck reports errors, but it does not repair any inconsistencies found. lfsck checks for three kinds of inconsistencies:</para>
+ <para>By default, lfsck reports errors, but it does not repair any inconsistencies found. lfsck checks for three kinds of inconsistencies:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290034" xreflabel=""/> Inode exists but has missing objects (dangling inode). This normally happens if there was a problem with an OST.</para>
+ <para> Inode exists but has missing objects (dangling inode). This normally happens if there was a problem with an OST.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290418" xreflabel=""/> Inode is missing but OST has unreferenced objects (orphan object). Normally, this happens if there was a problem with the MDS.</para>
+ <para> Inode is missing but OST has unreferenced objects (orphan object). Normally, this happens if there was a problem with the MDS.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290421" xreflabel=""/> Multiple inodes reference the same objects. This can happen if the MDS is corrupted or if the MDS storage is cached and loses some, but not all, writes.</para>
+ <para> Multiple inodes reference the same objects. This can happen if the MDS is corrupted or if the MDS storage is cached and loses some, but not all, writes.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1289943" xreflabel=""/>If the file system is in use and being modified while the --mdsdb and --ostdb steps are running, lfsck may report inconsistencies where none exist due to files and objects being created/removed after the database files were collected. Examine the lfsck results closely. You may want to re-run the test.</para>
+ <para>If the file system is in use and being modified while the --mdsdb and --ostdb steps are running, lfsck may report inconsistencies where none exist due to files and objects being created/removed after the database files were collected. Examine the lfsck results closely. You may want to re-run the test.</para>
</listitem></orderedlist>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438225_pgfId-1290574" xreflabel=""/>27.2.1 <anchor xml:id="dbdoclet.50438225_13916" xreflabel=""/>Working with Orphaned <anchor xml:id="dbdoclet.50438225_marker-1292187" xreflabel=""/>Objects</title>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1289945" xreflabel=""/>The easiest problem to resolve is that of orphaned objects. When the -l option for lfsck is used, these objects are linked to new files and put into lost+found in the Lustre file system, where they can be examined and saved or deleted as necessary. If you are certain the objects are not useful, run lfsck with the -d option to delete orphaned objects and free up any space they are using.</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1290628" xreflabel=""/>To fix dangling inodes, use lfsck with the -c option to create new, zero-length objects on the OSTs. These files read back with binary zeros for stripes that had objects re-created. Even without lfsck repair, these files can be read by entering:</para>
- <screen><anchor xml:id="dbdoclet.50438225_pgfId-1289949" xreflabel=""/>dd if=/lustre/bad/file of=/new/file bs=4k conv=sync,noerror
+ <title>27.2.1 <anchor xml:id="dbdoclet.50438225_13916" xreflabel=""/>Working with Orphaned <anchor xml:id="dbdoclet.50438225_marker-1292187" xreflabel=""/>Objects</title>
+ <para>The easiest problem to resolve is that of orphaned objects. When the -l option for lfsck is used, these objects are linked to new files and put into lost+found in the Lustre file system, where they can be examined and saved or deleted as necessary. If you are certain the objects are not useful, run lfsck with the -d option to delete orphaned objects and free up any space they are using.</para>
+ <para>To fix dangling inodes, use lfsck with the -c option to create new, zero-length objects on the OSTs. These files read back with binary zeros for stripes that had objects re-created. Even without lfsck repair, these files can be read by entering:</para>
+ <screen>dd if=/lustre/bad/file of=/new/file bs=4k conv=sync,noerror
</screen>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1289951" xreflabel=""/>Because it is rarely useful to have files with large holes in them, most users delete these files after reading them (if useful) and/or restoring them from backup.</para>
+ <para>Because it is rarely useful to have files with large holes in them, most users delete these files after reading them (if useful) and/or restoring them from backup.</para>
<note><para>You cannot write to the holes of such files without having lfsck re-create the objects. Generally, it is easier to delete these files and restore them from backup.</para></note>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1289916" xreflabel=""/>To fix inodes with duplicate objects, use lfsck with the -c option to copy the duplicate object to a new object and assign it to a file. One file will be okay and the duplicate will likely contain garbage. By itself, lfsck cannot tell which file is the usable one.</para>
+ <para>To fix inodes with duplicate objects, use lfsck with the -c option to copy the duplicate object to a new object and assign it to a file. One file will be okay and the duplicate will likely contain garbage. By itself, lfsck cannot tell which file is the usable one.</para>
</section>
</section>
<section xml:id="dbdoclet.50438225_12316">
<title>27.3 Recovering from an <anchor xml:id="dbdoclet.50438225_marker-1292768" xreflabel=""/>Unavailable OST</title>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1292771" xreflabel=""/>One of the most common problems encountered in a Lustre environment is when an OST becomes unavailable, because of a network partition, OSS node crash, etc. When this happens, the OST's clients pause and wait for the OST to become available again, either on the primary OSS or a failover OSS. When the OST comes back online, Lustre starts a recovery process to enable clients to reconnect to the OST. Lustre servers put a limit on the time they will wait in recovery for clients to reconnect. The timeout length is determined by the obd_timeout parameter.</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1292775" xreflabel=""/>During recovery, clients reconnect and replay their requests serially, in the same order they were done originally. Until a client receives a confirmation that a given transaction has been written to stable storage, the client holds on to the transaction, in case it needs to be replayed. Periodically, a progress message prints to the log, stating how_many/expected clients have reconnected. If the recovery is aborted, this log shows how many clients managed to reconnect. When all clients have completed recovery, or if the recovery timeout is reached, the recovery period ends and the OST resumes normal request processing.</para>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1292779" xreflabel=""/>If some clients fail to replay their requests during the recovery period, this will not stop the recovery from completing. You may have a situation where the OST recovers, but some clients are not able to participate in recovery (e.g. network problems or client failure), so they are evicted and their requests are not replayed. This would result in any operations on the evicted clients failing, including in-progress writes, which would cause cached writes to be lost. This is a normal outcome; the recovery cannot wait indefinitely, or the file system would be hung any time a client failed. The lost transactions are an unfortunate result of the recovery process.</para>
+ <para>One of the most common problems encountered in a Lustre environment is when an OST becomes unavailable, because of a network partition, OSS node crash, etc. When this happens, the OST's clients pause and wait for the OST to become available again, either on the primary OSS or a failover OSS. When the OST comes back online, Lustre starts a recovery process to enable clients to reconnect to the OST. Lustre servers put a limit on the time they will wait in recovery for clients to reconnect. The timeout length is determined by the obd_timeout parameter.</para>
+ <para>During recovery, clients reconnect and replay their requests serially, in the same order they were done originally. Until a client receives a confirmation that a given transaction has been written to stable storage, the client holds on to the transaction, in case it needs to be replayed. Periodically, a progress message prints to the log, stating how_many/expected clients have reconnected. If the recovery is aborted, this log shows how many clients managed to reconnect. When all clients have completed recovery, or if the recovery timeout is reached, the recovery period ends and the OST resumes normal request processing.</para>
+ <para>If some clients fail to replay their requests during the recovery period, this will not stop the recovery from completing. You may have a situation where the OST recovers, but some clients are not able to participate in recovery (e.g. network problems or client failure), so they are evicted and their requests are not replayed. This would result in any operations on the evicted clients failing, including in-progress writes, which would cause cached writes to be lost. This is a normal outcome; the recovery cannot wait indefinitely, or the file system would be hung any time a client failed. The lost transactions are an unfortunate result of the recovery process.</para>
<note><para>The version-based recovery (VBR) feature enables a failed client to be ''skipped'', so remaining clients can replay their requests, resulting in a more successful recovery from a downed OST. For more information about the VBR feature, see <xref linkend='lustrerecovery'/>(Version-based Recovery).</para></note>
</section>
</chapter>
<info>
<title xml:id="understandingfailover.title">Understanding Failover in Lustre</title>
</info>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304328" xreflabel=""/>This chapter describes failover in a Lustre system. It includes:</para>
+ <para>This chapter describes failover in a Lustre system. It includes:</para>
<itemizedlist>
<listitem>
<para>
</itemizedlist>
<section xml:id="dbdoclet.50540653_59957">
<title>3.1 What is Failover?</title>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304332" xreflabel=""/>A computer system is ''highly available'' when the services it provides are available with minimal downtime. In a highly-available system, if a failure condition occurs, such as the loss of a server or a network or software fault, the system's services continue without interruption. Generally, we measure availability by the percentage of time the system is required to be available.</para>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304333" xreflabel=""/>Availability is accomplished by replicating hardware and/or software so that when a primary server fails or is unavailable, a standby server can be switched into its place to run applications and associated resources. This process, called <emphasis role="italic">failover</emphasis>, should be automatic and, in most cases, completely application-transparent.</para>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304334" xreflabel=""/>A failover hardware setup requires a pair of servers with a shared resource (typically a physical storage device, which may be based on SAN, NAS, hardware RAID, SCSI or FC technology). The method of sharing storage should be essentially transparent at the device level; the same physical logical unit number (LUN) should be visible from both servers. To ensure high availability at the physical storage level, we encourage the use of RAID arrays to protect against drive-level failures.</para>
+ <para>A computer system is ''highly available'' when the services it provides are available with minimal downtime. In a highly-available system, if a failure condition occurs, such as the loss of a server or a network or software fault, the system's services continue without interruption. Generally, we measure availability by the percentage of time the system is required to be available.</para>
+ <para>Availability is accomplished by replicating hardware and/or software so that when a primary server fails or is unavailable, a standby server can be switched into its place to run applications and associated resources. This process, called <emphasis role="italic">failover</emphasis>, should be automatic and, in most cases, completely application-transparent.</para>
+ <para>A failover hardware setup requires a pair of servers with a shared resource (typically a physical storage device, which may be based on SAN, NAS, hardware RAID, SCSI or FC technology). The method of sharing storage should be essentially transparent at the device level; the same physical logical unit number (LUN) should be visible from both servers. To ensure high availability at the physical storage level, we encourage the use of RAID arrays to protect against drive-level failures.</para>
<note>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304335" xreflabel=""/>Lustre does not provide redundancy for data; it depends exclusively on redundancy of backing storage devices. The backing OST storage should be RAID 5 or, preferably, RAID 6 storage. MDT storage should be RAID 1 or RAID 0+1.</para>
+ <para>Lustre does not provide redundancy for data; it depends exclusively on redundancy of backing storage devices. The backing OST storage should be RAID 5 or, preferably, RAID 6 storage. MDT storage should be RAID 1 or RAID 0+1.</para>
</note>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50540653_pgfId-1304337" xreflabel=""/>3.1.1 Failover <anchor xml:id="dbdoclet.50540653_marker-1304336" xreflabel=""/>Capabilities</title>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304338" xreflabel=""/>To establish a highly-available Lustre file system, power management software or hardware and high availability (HA) software are used to provide the following failover capabilities:</para>
+ <title>3.1.1 Failover <anchor xml:id="dbdoclet.50540653_marker-1304336" xreflabel=""/>Capabilities</title>
+ <para>To establish a highly-available Lustre file system, power management software or hardware and high availability (HA) software are used to provide the following failover capabilities:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304339" xreflabel=""/><emphasis role="bold">Resource fencing</emphasis> - Protects physical storage from simultaneous access by two nodes.</para>
+ <para><emphasis role="bold">Resource fencing</emphasis> - Protects physical storage from simultaneous access by two nodes.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304340" xreflabel=""/><emphasis role="bold">Resource management</emphasis> - Starts and stops the Lustre resources as a part of failover, maintains the cluster state, and carries out other resource management tasks.</para>
+ <para><emphasis role="bold">Resource management</emphasis> - Starts and stops the Lustre resources as a part of failover, maintains the cluster state, and carries out other resource management tasks.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304341" xreflabel=""/><emphasis role="bold">Health monitoring</emphasis> - Verifies the availability of hardware and network resources and responds to health indications provided by Lustre.</para>
+ <para><emphasis role="bold">Health monitoring</emphasis> - Verifies the availability of hardware and network resources and responds to health indications provided by Lustre.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1305091" xreflabel=""/>These capabilities can be provided by a variety of software and/or hardware solutions. For more information about using power management software or hardware and high availability (HA) software with Lustre, see <xref linkend="configuringfailover"/>.</para>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1305272" xreflabel=""/>HA software is responsible for detecting failure of the primary Lustre server node and controlling the failover. Lustre works with any HA software that includes resource (I/O) fencing. For proper resource fencing, the HA software must be able to completely power off the failed server or disconnect it from the shared storage device. If two active nodes have access to the same storage device, data may be severely corrupted.</para>
+ <para>These capabilities can be provided by a variety of software and/or hardware solutions. For more information about using power management software or hardware and high availability (HA) software with Lustre, see <xref linkend="configuringfailover"/>.</para>
+ <para>HA software is responsible for detecting failure of the primary Lustre server node and controlling the failover. Lustre works with any HA software that includes resource (I/O) fencing. For proper resource fencing, the HA software must be able to completely power off the failed server or disconnect it from the shared storage device. If two active nodes have access to the same storage device, data may be severely corrupted.</para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50540653_pgfId-1304347" xreflabel=""/>3.1.2 Types of Failover<anchor xml:id="dbdoclet.50540653_marker-1304346" xreflabel=""/> Configurations</title>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304348" xreflabel=""/>Nodes in a cluster can be configured for failover in several ways. They are often configured in pairs (for example, two OSTs attached to a shared storage device), but other failover configurations are also possible. Failover configurations include:</para>
+ <title>3.1.2 Types of Failover<anchor xml:id="dbdoclet.50540653_marker-1304346" xreflabel=""/> Configurations</title>
+ <para>Nodes in a cluster can be configured for failover in several ways. They are often configured in pairs (for example, two OSTs attached to a shared storage device), but other failover configurations are also possible. Failover configurations include:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304349" xreflabel=""/><emphasis role="bold">Active/passive</emphasis> pair - In this configuration, the active node provides resources and serves data, while the passive node is usually standing by idle. If the active node fails, the passive node takes over and becomes active.</para>
+ <para><emphasis role="bold">Active/passive</emphasis> pair - In this configuration, the active node provides resources and serves data, while the passive node is usually standing by idle. If the active node fails, the passive node takes over and becomes active.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304350" xreflabel=""/><emphasis role="bold">Active/active</emphasis> pair - In this configuration, both nodes are active, each providing a subset of resources. In case of a failure, the second node takes over resources from the failed node.</para>
+ <para><emphasis role="bold">Active/active</emphasis> pair - In this configuration, both nodes are active, each providing a subset of resources. In case of a failure, the second node takes over resources from the failed node.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304351" xreflabel=""/>Typically, Lustre MDSs are configured as an active/passive pair, while OSSs are deployed in an active/active configuration that provides redundancy without extra overhead. Often the standby MDS is the active MDS for another Lustre file system or the MGS, so no nodes are idle in the cluster.</para>
+ <para>Typically, Lustre MDSs are configured as an active/passive pair, while OSSs are deployed in an active/active configuration that provides redundancy without extra overhead. Often the standby MDS is the active MDS for another Lustre file system or the MGS, so no nodes are idle in the cluster.</para>
</section>
</section>
<section xml:id="dbdoclet.50540653_97944">
<title>3.2 Failover Functionality in Lustre</title>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304355" xreflabel=""/>The failover functionality provided in Lustre can be used for the following failover scenario. When a client attempts to do I/O to a failed Lustre target, it continues to try until it receives an answer from any of the configured failover nodes for the Lustre target. A user-space application does not detect anything unusual, except that the I/O may take longer to complete.</para>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304356" xreflabel=""/>Lustre failover requires two nodes configured as a failover pair, which must share one or more storage devices. Lustre can be configured to provide MDT or OST failover.</para>
+ <para>The failover functionality provided in Lustre can be used for the following failover scenario. When a client attempts to do I/O to a failed Lustre target, it continues to try until it receives an answer from any of the configured failover nodes for the Lustre target. A user-space application does not detect anything unusual, except that the I/O may take longer to complete.</para>
+ <para>Lustre failover requires two nodes configured as a failover pair, which must share one or more storage devices. Lustre can be configured to provide MDT or OST failover.</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304357" xreflabel=""/>For MDT failover, two MDSs are configured to serve the same MDT. Only one MDS node can serve an MDT at a time.</para>
+ <para>For MDT failover, two MDSs are configured to serve the same MDT. Only one MDS node can serve an MDT at a time.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304358" xreflabel=""/>For OST failover, multiple OSS nodes are configured to be able to serve the same OST. However, only one OSS node can serve the OST at a time. An OST can be moved between OSS nodes that have access to the same storage device using <literal>umount/mount</literal> commands.</para>
+ <para>For OST failover, multiple OSS nodes are configured to be able to serve the same OST. However, only one OSS node can serve the OST at a time. An OST can be moved between OSS nodes that have access to the same storage device using <literal>umount/mount</literal> commands.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304359" xreflabel=""/>To add a failover partner to a Lustre configuration, the <literal>--failnode</literal> option is used. This can be done at creation time (using <literal>mkfs.lustre</literal>) or later when the Lustre system is active (using <literal>tunefs.lustre</literal>). For explanations of these utilities, see <xref linkend='dbdoclet.50438219_75432'/> and <xref linkend='dbdoclet.50438219_39574'/>.</para>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304366" xreflabel=""/>Lustre failover capability can be used to upgrade the Lustre software between successive minor versions without cluster downtime. For more information, see <xref linkend="upgradinglustre"/>.</para>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304370" xreflabel=""/>For information about configuring failover, see <xref linkend='configuringfailover'/>.</para>
+ <para>To add a failover partner to a Lustre configuration, the <literal>--failnode</literal> option is used. This can be done at creation time (using <literal>mkfs.lustre</literal>) or later when the Lustre system is active (using <literal>tunefs.lustre</literal>). For explanations of these utilities, see <xref linkend='dbdoclet.50438219_75432'/> and <xref linkend='dbdoclet.50438219_39574'/>.</para>
+ <para>Lustre failover capability can be used to upgrade the Lustre software between successive minor versions without cluster downtime. For more information, see <xref linkend="upgradinglustre"/>.</para>
+ <para>For information about configuring failover, see <xref linkend='configuringfailover'/>.</para>
<note>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304371" xreflabel=""/>Failover functionality in Lustre is provided only at the file system level. In a complete failover solution, failover functionality for system-level components, such as node failure detection or power control, must be provided by a third-party tool.</para>
+ <para>Failover functionality in Lustre is provided only at the file system level. In a complete failover solution, failover functionality for system-level components, such as node failure detection or power control, must be provided by a third-party tool.</para>
</note>
<caution>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304372" xreflabel=""/>OST failover functionality does not protect against corruption caused by a disk failure. If the storage media (i.e., physical disk) used for an OST fails, Lustre cannot recover it. We strongly recommend that some form of RAID be used for OSTs. Lustre functionality assumes that the storage is reliable, so it adds no extra reliability features.</para>
+ <para>OST failover functionality does not protect against corruption caused by a disk failure. If the storage media (i.e., physical disk) used for an OST fails, Lustre cannot recover it. We strongly recommend that some form of RAID be used for OSTs. Lustre functionality assumes that the storage is reliable, so it adds no extra reliability features.</para>
</caution>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50540653_pgfId-1304374" xreflabel=""/>3.2.1 MDT Failover Configuration <anchor xml:id="dbdoclet.50540653_marker-1304373" xreflabel=""/>(Active/Passive)</title>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304375" xreflabel=""/>Two MDSs are typically configured as an active/passive failover pair as shown in <xref linkend="understandingfailover.fig.configmdt"/>. Note that both nodes must have access to shared storage for the MDT(s) and the MGS. The primary (active) MDS manages the Lustre system metadata resources. If the primary MDS fails, the secondary (passive) MDS takes over these resources and serves the MDTs and the MGS.</para>
+ <title>3.2.1 MDT Failover Configuration <anchor xml:id="dbdoclet.50540653_marker-1304373" xreflabel=""/>(Active/Passive)</title>
+ <para>Two MDSs are typically configured as an active/passive failover pair as shown in <xref linkend="understandingfailover.fig.configmdt"/>. Note that both nodes must have access to shared storage for the MDT(s) and the MGS. The primary (active) MDS manages the Lustre system metadata resources. If the primary MDS fails, the secondary (passive) MDS takes over these resources and serves the MDTs and the MGS.</para>
<note>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304376" xreflabel=""/>In an environment with multiple file systems, the MDSs can be configured in a quasi active/active configuration, with each MDS managing metadata for a subset of the Lustre file system.</para>
+ <para>In an environment with multiple file systems, the MDSs can be configured in a quasi active/active configuration, with each MDS managing metadata for a subset of the Lustre file system.</para>
</note>
<figure>
<title xml:id="understandingfailover.fig.configmdt"> Lustre failover configuration for an MDT </title>
</figure>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50540653_pgfId-1304383" xreflabel=""/>3.2.2 OST Failover Configuration <anchor xml:id="dbdoclet.50540653_marker-1304382" xreflabel=""/>(Active/Active)</title>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304384" xreflabel=""/>OSTs are usually configured in a load-balanced, active/active failover configuration. A failover cluster is built from two OSSs as shown in <xref linkend="understandingfailover.fig.configost"/>.</para>
+ <title>3.2.2 OST Failover Configuration <anchor xml:id="dbdoclet.50540653_marker-1304382" xreflabel=""/>(Active/Active)</title>
+ <para>OSTs are usually configured in a load-balanced, active/active failover configuration. A failover cluster is built from two OSSs as shown in <xref linkend="understandingfailover.fig.configost"/>.</para>
<note>
<para>OSSs configured as a failover pair must have shared disks/RAID.</para>
</note>
</textobject>
</mediaobject>
</figure>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304719" xreflabel=""/>In an active configuration, 50% of the available OSTs are assigned to one OSS and the remaining OSTs are assigned to the other OSS. Each OSS serves as the primary node for half the OSTs and as a failover node for the remaining OSTs.</para>
- <para><anchor xml:id="dbdoclet.50540653_pgfId-1304392" xreflabel=""/>In this mode, if one OSS fails, the other OSS takes over all of the failed OSTs. The clients attempt to connect to each OSS serving the OST, until one of them responds. Data on the OST is written synchronously, and the clients replay transactions that were in progress and uncommitted to disk before the OST failure.</para>
+ <para>In an active configuration, 50% of the available OSTs are assigned to one OSS and the remaining OSTs are assigned to the other OSS. Each OSS serves as the primary node for half the OSTs and as a failover node for the remaining OSTs.</para>
+ <para>In this mode, if one OSS fails, the other OSS takes over all of the failed OSTs. The clients attempt to connect to each OSS serving the OST, until one of them responds. Data on the OST is written synchronously, and the clients replay transactions that were in progress and uncommitted to disk before the OST failure.</para>
</section>
</section>
</chapter>
<info>
<title xml:id="understandinglustre.title">Understanding Lustre</title>
</info>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294042" xreflabel=""/>This chapter describes the Lustre architecture and features of Lustre. It includes the following sections:</para>
+ <para>This chapter describes the Lustre architecture and features of Lustre. It includes the following sections:</para>
<itemizedlist>
<listitem>
<para>
</itemizedlist>
<section xml:id="understandinglustre.whatislustre">
<title>1.1 <anchor xml:id="dbdoclet.50438250_92658" xreflabel=""/>What Lustre Is (and What It Isn't)</title>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294027" xreflabel=""/>Lustre is a storage architecture for clusters. The central component of the Lustre architecture is the Lustre file system, which is supported on the Linux operating system and provides a POSIX-compliant UNIX file system interface.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294028" xreflabel=""/>The Lustre storage architecture is used for many different kinds of clusters. It is best known for powering seven of the ten largest high-performance computing (HPC) clusters worldwide, with tens of thousands of client systems, petabytes (PB) of storage and hundreds of gigabytes per second (GB/sec) of I/O throughput. Many HPC sites use Lustre as a site-wide global file system, serving dozens of clusters.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1297491" xreflabel=""/>The ability of a Lustre file system to scale capacity and performance for any need reduces the need to deploy many separate file systems, such as one for each compute cluster. Storage management is simplified by avoiding the need to copy data between compute clusters. In addition to aggregating storage capacity of many servers, the I/O throughput is also aggregated and scales with additional servers. Moreover, throughput and/or capacity can be easily increased by adding servers dynamically.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1297493" xreflabel=""/>While Lustre can function in many work environments, it is not necessarily the best choice for all applications. It is best suited for uses that exceed the capacity that a single server can provide, though in some use cases Lustre can perform better with a single server than other filesystems due to its strong locking and data coherency.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1297495" xreflabel=""/>Lustre is currently not particularly well suited for "peer-to-peer" usage models where there are clients and servers running on the same node, each sharing a small amount of storage, due to the lack of Lustre-level data replication. In such uses, if one client/server fails, then the data stored on that node will not be accessible until the node is restarted.</para>
+ <para>Lustre is a storage architecture for clusters. The central component of the Lustre architecture is the Lustre file system, which is supported on the Linux operating system and provides a POSIX-compliant UNIX file system interface.</para>
+ <para>The Lustre storage architecture is used for many different kinds of clusters. It is best known for powering seven of the ten largest high-performance computing (HPC) clusters worldwide, with tens of thousands of client systems, petabytes (PB) of storage and hundreds of gigabytes per second (GB/sec) of I/O throughput. Many HPC sites use Lustre as a site-wide global file system, serving dozens of clusters.</para>
+ <para>The ability of a Lustre file system to scale capacity and performance for any need reduces the need to deploy many separate file systems, such as one for each compute cluster. Storage management is simplified by avoiding the need to copy data between compute clusters. In addition to aggregating storage capacity of many servers, the I/O throughput is also aggregated and scales with additional servers. Moreover, throughput and/or capacity can be easily increased by adding servers dynamically.</para>
+ <para>While Lustre can function in many work environments, it is not necessarily the best choice for all applications. It is best suited for uses that exceed the capacity that a single server can provide, though in some use cases Lustre can perform better with a single server than other filesystems due to its strong locking and data coherency.</para>
+ <para>Lustre is currently not particularly well suited for "peer-to-peer" usage models where there are clients and servers running on the same node, each sharing a small amount of storage, due to the lack of Lustre-level data replication. In such uses, if one client/server fails, then the data stored on that node will not be accessible until the node is restarted.</para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438250_pgfId-1293793" xreflabel=""/>1.1.1 Lustre <anchor xml:id="dbdoclet.50438250_marker-1293792" xreflabel=""/>Features</title>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294716" xreflabel=""/>Lustre runs on a variety of vendor's kernels. For more details, see <link xl:href="http://wiki.whamcloud.com/">Lustre Release Information</link> on the Whamcloud wiki.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294223" xreflabel=""/>A Lustre installation can be scaled up or down with respect to the number of client nodes, disk storage and bandwidth. Scalability and performance are dependent on available disk and network bandwith and the processing power of the servers in the system. Lustre can be deployed in a wide variety of configurations that can be scaled well beyond the size and performance observed in production systems to date.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294224" xreflabel=""/><xref linkend="understandinglustre.tab1"/> shows the practical range of scalability and performance characteristics of the Lustre file system and some test results in production systems.</para>
+ <title>1.1.1 Lustre <anchor xml:id="dbdoclet.50438250_marker-1293792" xreflabel=""/>Features</title>
+ <para>Lustre runs on a variety of vendor's kernels. For more details, see <link xl:href="http://wiki.whamcloud.com/">Lustre Release Information</link> on the Whamcloud wiki.</para>
+ <para>A Lustre installation can be scaled up or down with respect to the number of client nodes, disk storage and bandwidth. Scalability and performance are dependent on available disk and network bandwith and the processing power of the servers in the system. Lustre can be deployed in a wide variety of configurations that can be scaled well beyond the size and performance observed in production systems to date.</para>
+ <para><xref linkend="understandinglustre.tab1"/> shows the practical range of scalability and performance characteristics of the Lustre file system and some test results in production systems.</para>
<table frame="all">
<title xml:id="understandinglustre.tab1">Lustre Scalability and Performance</title>
<tgroup cols="4">
<row>
<entry>
<para><emphasis role="bold">
- <anchor xml:id="dbdoclet.50438250_pgfId-1294236" xreflabel=""/>
+
</emphasis></para>
</entry>
<entry>
- <para><emphasis role="bold"><anchor xml:id="dbdoclet.50438250_pgfId-1294238" xreflabel=""/>Feature</emphasis></para>
+ <para><emphasis role="bold">Feature</emphasis></para>
</entry>
<entry>
- <para><emphasis role="bold"><anchor xml:id="dbdoclet.50438250_pgfId-1294240" xreflabel=""/>Practical Range</emphasis></para>
+ <para><emphasis role="bold">Practical Range</emphasis></para>
</entry>
<entry>
- <para><emphasis role="bold"><anchor xml:id="dbdoclet.50438250_pgfId-1294242" xreflabel=""/>Tested in Production</emphasis></para>
+ <para><emphasis role="bold">Tested in Production</emphasis></para>
</entry>
</row>
</thead>
<tbody>
<row>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294244" xreflabel=""/><emphasis role="bold">Clients</emphasis></para>
+ <para> <emphasis role="bold">Clients</emphasis></para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294246" xreflabel=""/>Scalabity</para>
+ <para> Scalabity</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294248" xreflabel=""/>100-100,000</para>
+ <para> 100-100,000</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294250" xreflabel=""/>50,000+ clients</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294251" xreflabel=""/>Many installations in 10,000 to 20,000 range</para>
+ <para> 50,000+ clients</para>
+ <para>Many installations in 10,000 to 20,000 range</para>
</entry>
</row>
<row>
<entry/>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294255" xreflabel=""/>Performance</para>
+ <para> Performance</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294257" xreflabel=""/><emphasis>Single client: </emphasis> 2 GB/sec I/O,</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294258" xreflabel=""/>1,000 metadata ops/sec</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294259" xreflabel=""/><emphasis>File system:</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294260" xreflabel=""/>2.5 TB/sec</para>
+ <para> <emphasis>Single client: </emphasis> 2 GB/sec I/O,</para>
+ <para>1,000 metadata ops/sec</para>
+ <para><emphasis>File system:</emphasis></para>
+ <para>2.5 TB/sec</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294262" xreflabel=""/><emphasis>Single client: </emphasis> 2 GB/sec</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294263" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294264" xreflabel=""/><emphasis>File system:</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294265" xreflabel=""/>240 GB/sec I/O</para>
+ <para> <emphasis>Single client: </emphasis> 2 GB/sec</para>
+ <para> </para>
+ <para><emphasis>File system:</emphasis></para>
+ <para>240 GB/sec I/O</para>
</entry>
</row>
<row>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294267" xreflabel=""/><emphasis role="bold">OSSs</emphasis></para>
+ <para> <emphasis role="bold">OSSs</emphasis></para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294269" xreflabel=""/>Scalabity</para>
+ <para> Scalabity</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294271" xreflabel=""/><emphasis>OSSs:</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294272" xreflabel=""/>4-500 with up to 4000 OSTs</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294273" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294274" xreflabel=""/><emphasis>File system:</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294275" xreflabel=""/>64 PB, file size 320 TB</para>
+ <para> <emphasis>OSSs:</emphasis></para>
+ <para>4-500 with up to 4000 OSTs</para>
+ <para> </para>
+ <para><emphasis>File system:</emphasis></para>
+ <para>64 PB, file size 320 TB</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294277" xreflabel=""/><emphasis>OSSs:</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294278" xreflabel=""/>450 OSSs with 1,000 OSTs</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294279" xreflabel=""/>192 OSSs with 1344 OSTs</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294280" xreflabel=""/><emphasis>File system:</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1294281" xreflabel=""/>10 PB, file size multi-TB</para>
+ <para> <emphasis>OSSs:</emphasis></para>
+ <para>450 OSSs with 1,000 OSTs</para>
+ <para>192 OSSs with 1344 OSTs</para>
+ <para><emphasis>File system:</emphasis></para>
+ <para>10 PB, file size multi-TB</para>
</entry>
</row>
<row>
<entry/>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294285" xreflabel=""/>Performance</para>
+ <para> Performance</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294287" xreflabel=""/>Up to 5 GB/sec</para>
+ <para> Up to 5 GB/sec</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294289" xreflabel=""/>OSS throughput at 2.0+ GB/sec</para>
+ <para> OSS throughput at 2.0+ GB/sec</para>
</entry>
</row>
<row>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294291" xreflabel=""/><emphasis role="bold">MDSs</emphasis></para>
+ <para> <emphasis role="bold">MDSs</emphasis></para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294293" xreflabel=""/>Scalabity</para>
+ <para> Scalabity</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294295" xreflabel=""/>1 + 1 (failover with one backup)</para>
+ <para> 1 + 1 (failover with one backup)</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294297" xreflabel=""/> </para>
+ <para>  </para>
</entry>
</row>
<row>
<entry/>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294301" xreflabel=""/>Performance</para>
+ <para> Performance</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294303" xreflabel=""/>Up to 35,000/s create, 100,000/s stat metadata operations</para>
+ <para> Up to 35,000/s create, 100,000/s stat metadata operations</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1294305" xreflabel=""/>15,000/s create, 25,000/s stat metadata operations</para>
+ <para> 15,000/s create, 25,000/s stat metadata operations</para>
</entry>
</row>
</tbody>
</tgroup>
</table>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293868" xreflabel=""/>Other Lustre features are:</para>
+ <para>Other Lustre features are:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1295598" xreflabel=""/><emphasis role="bold">Performance-enhanced ext4 file system:</emphasis> Lustre uses a modified version of the ext4 journaling file system to store data and metadata. This version, called <emphasis role="italic">ldiskfs</emphasis>, has been enhanced to improve performance and provide additional functionality needed by Lustre.</para>
+ <para><emphasis role="bold">Performance-enhanced ext4 file system:</emphasis> Lustre uses a modified version of the ext4 journaling file system to store data and metadata. This version, called <emphasis role="italic">ldiskfs</emphasis>, has been enhanced to improve performance and provide additional functionality needed by Lustre.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1295599" xreflabel=""/><emphasis role="bold">POSIX compliance</emphasis> : The full POSIX test suite passes with limited exceptions on Lustre clients. In a cluster, most operations are atomic so that clients never see stale data or metadata. Lustre supports mmap() file I/O.</para>
+ <para><emphasis role="bold">POSIX compliance</emphasis> : The full POSIX test suite passes with limited exceptions on Lustre clients. In a cluster, most operations are atomic so that clients never see stale data or metadata. Lustre supports mmap() file I/O.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293870" xreflabel=""/><emphasis role="bold">High-performance heterogeneous networking:</emphasis> Lustre supports a variety of high performance, low latency networks and permits Remote Direct Memory Access (RDMA) for Infiniband (OFED). This enables multiple, bridging RDMA networks to useLustre routing for maximum performance. Lustre also provides integrated network diagnostics.</para>
+ <para><emphasis role="bold">High-performance heterogeneous networking:</emphasis> Lustre supports a variety of high performance, low latency networks and permits Remote Direct Memory Access (RDMA) for Infiniband (OFED). This enables multiple, bridging RDMA networks to useLustre routing for maximum performance. Lustre also provides integrated network diagnostics.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293871" xreflabel=""/><emphasis role="bold">High-availability:</emphasis> Lustre offers active/active failover using shared storage partitions for OSS targets (OSTs) and active/passive failover using a shared storage partition for the MDS target (MDT). This allows application transparent recovery. Lustre can work with a variety of high availability (HA) managers to allow automated failover and has no single point of failure (NSPF). Multiple mount protection (MMP) provides integrated protection from errors in highly-available systems that would otherwise cause file system corruption.</para>
+ <para><emphasis role="bold">High-availability:</emphasis> Lustre offers active/active failover using shared storage partitions for OSS targets (OSTs) and active/passive failover using a shared storage partition for the MDS target (MDT). This allows application transparent recovery. Lustre can work with a variety of high availability (HA) managers to allow automated failover and has no single point of failure (NSPF). Multiple mount protection (MMP) provides integrated protection from errors in highly-available systems that would otherwise cause file system corruption.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293872" xreflabel=""/><emphasis role="bold">Security:</emphasis> In Lustre, an option is available to have TCP connections only from privileged ports. Group membership handling is server-based.</para>
+ <para><emphasis role="bold">Security:</emphasis> In Lustre, an option is available to have TCP connections only from privileged ports. Group membership handling is server-based.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293873" xreflabel=""/><emphasis role="bold">Access control list (ACL), exended attributes</emphasis> : Currently, the Lustre security model follows that of a UNIX file system, enhanced with POSIX ACLs. Noteworthy additional features include root squash and connecting only from privileged ports.</para>
+ <para><emphasis role="bold">Access control list (ACL), exended attributes</emphasis> : Currently, the Lustre security model follows that of a UNIX file system, enhanced with POSIX ACLs. Noteworthy additional features include root squash and connecting only from privileged ports.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293874" xreflabel=""/><emphasis role="bold">Interoperability:</emphasis> Lustre runs on a variety of CPU architectures and mixed-endian clusters and is interoperable between adjacent Lustre software releases.</para>
+ <para><emphasis role="bold">Interoperability:</emphasis> Lustre runs on a variety of CPU architectures and mixed-endian clusters and is interoperable between adjacent Lustre software releases.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293875" xreflabel=""/><emphasis role="bold">Object-based architecture:</emphasis> Clients are isolated from the on-disk file structure enabling upgrading of the storage architecture without affecting the client.</para>
+ <para><emphasis role="bold">Object-based architecture:</emphasis> Clients are isolated from the on-disk file structure enabling upgrading of the storage architecture without affecting the client.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293876" xreflabel=""/><emphasis role="bold">Byte-granular file and fine-grained metadata locking:</emphasis> Any clients can operate on the same file and directory concurrently. A Lustre distributed lock manager (DLM) ensures that files are coherent between all the clients in a file system and the servers. Multiple clients can access the same files concurrently, and the DLM ensures that all the clients see consistent data at all times. The DLM on each MDT and each OST manages the locking for objects stored in that file system. The MDT manages locks on inodes permissions and path names. OST manages locks for each stripe of a file and the data within each object</para>
+ <para><emphasis role="bold">Byte-granular file and fine-grained metadata locking:</emphasis> Any clients can operate on the same file and directory concurrently. A Lustre distributed lock manager (DLM) ensures that files are coherent between all the clients in a file system and the servers. Multiple clients can access the same files concurrently, and the DLM ensures that all the clients see consistent data at all times. The DLM on each MDT and each OST manages the locking for objects stored in that file system. The MDT manages locks on inodes permissions and path names. OST manages locks for each stripe of a file and the data within each object</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293877" xreflabel=""/><emphasis role="bold">Quotas</emphasis> : User and group quotas are available for Lustre.</para>
+ <para><emphasis role="bold">Quotas</emphasis> : User and group quotas are available for Lustre.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293878" xreflabel=""/><emphasis role="bold">OSS addition</emphasis> : The capacity of a Lustre file system and aggregate cluster bandwidth can be increased without interrupting any operations by adding a new OSS with OSTs to the cluster.</para>
+ <para><emphasis role="bold">OSS addition</emphasis> : The capacity of a Lustre file system and aggregate cluster bandwidth can be increased without interrupting any operations by adding a new OSS with OSTs to the cluster.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293879" xreflabel=""/><emphasis role="bold">Controlled striping</emphasis> : The distribution of files across OSTs can be configured on a per file, per directory, or per file system basis. This allows file I/O to be tuned to specific application requirements. Lustre uses RAID-0 striping and balances space usage across OSTs.</para>
+ <para><emphasis role="bold">Controlled striping</emphasis> : The distribution of files across OSTs can be configured on a per file, per directory, or per file system basis. This allows file I/O to be tuned to specific application requirements. Lustre uses RAID-0 striping and balances space usage across OSTs.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293880" xreflabel=""/><emphasis role="bold">Network data integrity protection:</emphasis> A checksum of all data sent from the client to the OSS protects against corruption during data transfer.</para>
+ <para><emphasis role="bold">Network data integrity protection:</emphasis> A checksum of all data sent from the client to the OSS protects against corruption during data transfer.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293881" xreflabel=""/><emphasis role="bold">MPI I/O:</emphasis> Lustre has a dedicated MPI ADIO layer that optimizes parallel I/O to match the underlying file system architecture.</para>
+ <para><emphasis role="bold">MPI I/O:</emphasis> Lustre has a dedicated MPI ADIO layer that optimizes parallel I/O to match the underlying file system architecture.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293882" xreflabel=""/><emphasis role="bold">NFS and CIFS export:</emphasis> Lustre files can be re-exported using NFS or CIFS (via Samba) enabling them to be shared with a non-Linux client.</para>
+ <para><emphasis role="bold">NFS and CIFS export:</emphasis> Lustre files can be re-exported using NFS or CIFS (via Samba) enabling them to be shared with a non-Linux client.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293883" xreflabel=""/><emphasis role="bold">Disaster recovery tool:</emphasis> Lustre provides a distributed file system check (lfsck) that can restore consistency between storage components in case of a major file system error. Lustre can operate even in the presence of file system inconsistencies, so lfsck is not required before returning the file system to production.</para>
+ <para><emphasis role="bold">Disaster recovery tool:</emphasis> Lustre provides a distributed file system check (lfsck) that can restore consistency between storage components in case of a major file system error. Lustre can operate even in the presence of file system inconsistencies, so lfsck is not required before returning the file system to production.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293884" xreflabel=""/><emphasis role="bold">Internal monitoring and instrumentation interfaces:</emphasis> Lustre offers a variety of mechanisms to examine performance and tuning.</para>
+ <para><emphasis role="bold">Internal monitoring and instrumentation interfaces:</emphasis> Lustre offers a variety of mechanisms to examine performance and tuning.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293885" xreflabel=""/><emphasis role="bold">Open source</emphasis> : Lustre is licensed under the GPL 2.0 license for use with Linux.</para>
+ <para><emphasis role="bold">Open source</emphasis> : Lustre is licensed under the GPL 2.0 license for use with Linux.</para>
</listitem>
</itemizedlist>
</section>
</section>
<section xml:id="understandinglustre.components">
<title>1.2 <anchor xml:id="dbdoclet.50438250_17402" xreflabel=""/>Lustre Components</title>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293889" xreflabel=""/>An installation of the Lustre software includes a management server (MGS) and one or more Lustre file systems interconnected with Lustre networking (LNET).</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1296520" xreflabel=""/>A basic configuration of Lustre components is shown in <xref linkend="understandinglustre.fig.cluster"/>.</para>
+ <para>An installation of the Lustre software includes a management server (MGS) and one or more Lustre file systems interconnected with Lustre networking (LNET).</para>
+ <para>A basic configuration of Lustre components is shown in <xref linkend="understandinglustre.fig.cluster"/>.</para>
<figure>
<title xml:id="understandinglustre.fig.cluster">Lustre components in a basic cluster </title>
<mediaobject>
</mediaobject>
</figure>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438250_pgfId-1296512" xreflabel=""/>1.2.1 Management Server (MGS)</title>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1296493" xreflabel=""/>The MGS stores configuration information for all the Lustre file systems in a cluster and provides this information to other Lustre components. Each Lustre target contacts the MGS to provide information, and Lustre clients contact the MGS to retrieve information.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293901" xreflabel=""/>It is preferable that the MGS have its own storage space so that it can be managed independently. However, the MGS can be co-located and share storage space with an MDS as shown in <xref linkend="understandinglustre.fig.cluster"/>.</para>
+ <title>1.2.1 Management Server (MGS)</title>
+ <para>The MGS stores configuration information for all the Lustre file systems in a cluster and provides this information to other Lustre components. Each Lustre target contacts the MGS to provide information, and Lustre clients contact the MGS to retrieve information.</para>
+ <para>It is preferable that the MGS have its own storage space so that it can be managed independently. However, the MGS can be co-located and share storage space with an MDS as shown in <xref linkend="understandinglustre.fig.cluster"/>.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438250_pgfId-1293902" xreflabel=""/>1.2.2 Lustre File System Components</title>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293903" xreflabel=""/>Each Lustre file system consists of the following components:</para>
+ <title>1.2.2 Lustre File System Components</title>
+ <para>Each Lustre file system consists of the following components:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293904" xreflabel=""/><emphasis role="bold">Metadata Server (MDS)</emphasis> - The MDS makes metadata stored in one or more MDTs available to Lustre clients. Each MDS manages the names and directories in the Lustre file system(s) and provides network request handling for one or more local MDTs.</para>
+ <para><emphasis role="bold">Metadata Server (MDS)</emphasis> - The MDS makes metadata stored in one or more MDTs available to Lustre clients. Each MDS manages the names and directories in the Lustre file system(s) and provides network request handling for one or more local MDTs.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293905" xreflabel=""/><emphasis role="bold">Metadata Target (MDT</emphasis> ) - The MDT stores metadata (such as filenames, directories, permissions and file layout) on storage attached to an MDS. Each file system has one MDT. An MDT on a shared storage target can be available to multiple MDSs, although only one can access it at a time. If an active MDS fails, a standby MDS can serve the MDT and make it available to clients. This is referred to as MDS failover.</para>
+ <para><emphasis role="bold">Metadata Target (MDT</emphasis> ) - The MDT stores metadata (such as filenames, directories, permissions and file layout) on storage attached to an MDS. Each file system has one MDT. An MDT on a shared storage target can be available to multiple MDSs, although only one can access it at a time. If an active MDS fails, a standby MDS can serve the MDT and make it available to clients. This is referred to as MDS failover.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293906" xreflabel=""/><emphasis role="bold">Object Storage Servers (OSS)</emphasis> : The OSS provides file I/O service, and network request handling for one or more local OSTs. Typically, an OSS serves between 2 and 8 OSTs, up to 16 TB each. A typical configuration is an MDT on a dedicated node, two or more OSTs on each OSS node, and a client on each of a large number of compute nodes.</para>
+ <para><emphasis role="bold">Object Storage Servers (OSS)</emphasis> : The OSS provides file I/O service, and network request handling for one or more local OSTs. Typically, an OSS serves between 2 and 8 OSTs, up to 16 TB each. A typical configuration is an MDT on a dedicated node, two or more OSTs on each OSS node, and a client on each of a large number of compute nodes.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293907" xreflabel=""/><emphasis role="bold">Object Storage Target (OST)</emphasis> : User file data is stored in one or more objects, each object on a separate OST in a Lustre file system. The number of objects per file is configurable by the user and can be tuned to optimize performance for a given workload.</para>
+ <para><emphasis role="bold">Object Storage Target (OST)</emphasis> : User file data is stored in one or more objects, each object on a separate OST in a Lustre file system. The number of objects per file is configurable by the user and can be tuned to optimize performance for a given workload.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293908" xreflabel=""/><emphasis role="bold">Lustre clients</emphasis> : Lustre clients are computational, visualization or desktop nodes that are running Lustre client software, allowing them to mount the Lustre file system.</para>
+ <para><emphasis role="bold">Lustre clients</emphasis> : Lustre clients are computational, visualization or desktop nodes that are running Lustre client software, allowing them to mount the Lustre file system.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293909" xreflabel=""/>The Lustre client software provides an interface between the Linux virtual file system and the Lustre servers. The client software includes a Management Client (MGC), a Metadata Client (MDC), and multiple Object Storage Clients (OSCs), one corresponding to each OST in the file system.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293910" xreflabel=""/>A logical object volume (LOV) aggregates the OSCs to provide transparent access across all the OSTs. Thus, a client with the Lustre file system mounted sees a single, coherent, synchronized namespace. Several clients can write to different parts of the same file simultaneously, while, at the same time, other clients can read from the file.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293911" xreflabel=""/><xref linkend="understandinglustre.tab.storagerequire"/> provides the requirements for attached storage for each Lustre file system component and describes desirable characterics of the hardware used.</para>
+ <para>The Lustre client software provides an interface between the Linux virtual file system and the Lustre servers. The client software includes a Management Client (MGC), a Metadata Client (MDC), and multiple Object Storage Clients (OSCs), one corresponding to each OST in the file system.</para>
+ <para>A logical object volume (LOV) aggregates the OSCs to provide transparent access across all the OSTs. Thus, a client with the Lustre file system mounted sees a single, coherent, synchronized namespace. Several clients can write to different parts of the same file simultaneously, while, at the same time, other clients can read from the file.</para>
+ <para><xref linkend="understandinglustre.tab.storagerequire"/> provides the requirements for attached storage for each Lustre file system component and describes desirable characterics of the hardware used.</para>
<table frame="all">
<title xml:id="understandinglustre.tab.storagerequire">Storage and hardware requirements for Lustre components</title>
<tgroup cols="3">
<row>
<entry>
<para><emphasis role="bold">
- <anchor xml:id="dbdoclet.50438250_pgfId-1293914" xreflabel=""/>
+
</emphasis></para>
</entry>
<entry>
- <para><emphasis role="bold"><anchor xml:id="dbdoclet.50438250_pgfId-1293916" xreflabel=""/>Required attached storage</emphasis></para>
+ <para><emphasis role="bold">Required attached storage</emphasis></para>
</entry>
<entry>
- <para><emphasis role="bold"><anchor xml:id="dbdoclet.50438250_pgfId-1293918" xreflabel=""/>Desirable hardware characteristics</emphasis></para>
+ <para><emphasis role="bold">Desirable hardware characteristics</emphasis></para>
</entry>
</row>
</thead>
<tbody>
<row>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1293920" xreflabel=""/><emphasis role="bold">MDSs</emphasis></para>
+ <para> <emphasis role="bold">MDSs</emphasis></para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1293922" xreflabel=""/>1-2% of file system capacity</para>
+ <para> 1-2% of file system capacity</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1293924" xreflabel=""/>Adequate CPU power, plenty of memory, fast disk storage.</para>
+ <para> Adequate CPU power, plenty of memory, fast disk storage.</para>
</entry>
</row>
<row>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1293926" xreflabel=""/><emphasis role="bold">OSSs</emphasis></para>
+ <para> <emphasis role="bold">OSSs</emphasis></para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1293928" xreflabel=""/>1-16 TB per OST, 1-8 OSTs per OSS</para>
+ <para> 1-16 TB per OST, 1-8 OSTs per OSS</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1293930" xreflabel=""/>Good bus bandwidth. Recommended that storage be balanced evenly across OSSs.</para>
+ <para> Good bus bandwidth. Recommended that storage be balanced evenly across OSSs.</para>
</entry>
</row>
<row>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1293932" xreflabel=""/><emphasis role="bold">Clients</emphasis></para>
+ <para> <emphasis role="bold">Clients</emphasis></para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1293934" xreflabel=""/>None</para>
+ <para> None</para>
</entry>
<entry>
- <para> <anchor xml:id="dbdoclet.50438250_pgfId-1293936" xreflabel=""/>Low latency, high bandwith network.</para>
+ <para> Low latency, high bandwith network.</para>
</entry>
</row>
</tbody>
</tgroup>
</table>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293938" xreflabel=""/>For additional hardware requirements and considerations, see <xref linkend="settinguplustresystem"/>.</para>
+ <para>For additional hardware requirements and considerations, see <xref linkend="settinguplustresystem"/>.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438250_pgfId-1295546" xreflabel=""/>1.2.3 Lustre Networking (LNET)</title>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293939" xreflabel=""/>Lustre Networking (LNET) is a custom networking API that provides the communication infrastructure that handles metadata and file I/O data for the Lustre file system servers and clients. For more information about LNET, see something <xref linkend="understandinglustrenetworking"/>.</para>
+ <title>1.2.3 Lustre Networking (LNET)</title>
+ <para>Lustre Networking (LNET) is a custom networking API that provides the communication infrastructure that handles metadata and file I/O data for the Lustre file system servers and clients. For more information about LNET, see something <xref linkend="understandinglustrenetworking"/>.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438250_pgfId-1293940" xreflabel=""/>1.2.4 Lustre Cluster</title>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293941" xreflabel=""/>At scale, the Lustre cluster can include hundreds of OSSs and thousands of clients (see <xref linkend="understandinglustre.fig.lustrescale"/>). More than one type of network can be used in a Lustre cluster. Shared storage between OSSs enables failover capability. For more details about OSS failover, see <xref linkend="understandingfailover"/>.</para>
+ <title>1.2.4 Lustre Cluster</title>
+ <para>At scale, the Lustre cluster can include hundreds of OSSs and thousands of clients (see <xref linkend="understandinglustre.fig.lustrescale"/>). More than one type of network can be used in a Lustre cluster. Shared storage between OSSs enables failover capability. For more details about OSS failover, see <xref linkend="understandingfailover"/>.</para>
<figure>
<title xml:id="understandinglustre.fig.lustrescale">Lustre cluster at scale</title>
<mediaobject>
</section>
<section xml:id="understandinglustre.storageio">
<title>1.3 <anchor xml:id="dbdoclet.50438250_38077" xreflabel=""/>Lustre Storage and I/O</title>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1296581" xreflabel=""/>In a Lustre file system, a file stored on the MDT points to one or more objects associated with a data file, as shown in <xref linkend="understandinglustre.fig.mdtost"/>. Each object contains data and is stored on an OST. If the MDT file points to one object, all the file data is stored in that object. If the file points to more than one object, the file data is 'striped' across the objects (using RAID 0) and each object is stored on a different OST. (For more information about how striping is implemented in Lustre, see <xref linkend="dbdoclet.50438250_89922"/>)</para>
+ <para>In a Lustre file system, a file stored on the MDT points to one or more objects associated with a data file, as shown in <xref linkend="understandinglustre.fig.mdtost"/>. Each object contains data and is stored on an OST. If the MDT file points to one object, all the file data is stored in that object. If the file points to more than one object, the file data is 'striped' across the objects (using RAID 0) and each object is stored on a different OST. (For more information about how striping is implemented in Lustre, see <xref linkend="dbdoclet.50438250_89922"/>)</para>
<para>In <xref linkend="understandinglustre.fig.mdtost"/>, each filename points to an inode. The inode contains all of the file attributes, such as owner, access permissions, Lustre striping layout, access time, and access control. Multiple filenames may point to the same inode.</para>
<figure>
<title xml:id="understandinglustre.fig.mdtost">MDT file points to objects on OSTs containing file data</title>
</textobject>
</mediaobject>
</figure>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1296662" xreflabel=""/>When a client opens a file, the <literal>fileopen</literal> operation transfers the file layout from the MDS to the client. The client then uses this information to perform I/O on the file, directly interacting with the OSS nodes where the objects are stored. This process is illustrated in <xref linkend="understandinglustre.fig.fileio"/>.</para>
+ <para>When a client opens a file, the <literal>fileopen</literal> operation transfers the file layout from the MDS to the client. The client then uses this information to perform I/O on the file, directly interacting with the OSS nodes where the objects are stored. This process is illustrated in <xref linkend="understandinglustre.fig.fileio"/>.</para>
<figure>
<title xml:id="understandinglustre.fig.fileio">File open and file I/O in Lustre</title>
<mediaobject>
</textobject>
</mediaobject>
</figure>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1296842" xreflabel=""/>Each file on the MDT contains the layout of the associated data file, including the OST number and object identifier. Clients request the file layout from the MDS and then perform file I/O operations by communicating directly with the OSSs that manage that file data.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1297084" xreflabel=""/>The available bandwidth of a Lustre file system is determined as follows:</para>
+ <para>Each file on the MDT contains the layout of the associated data file, including the OST number and object identifier. Clients request the file layout from the MDS and then perform file I/O operations by communicating directly with the OSSs that manage that file data.</para>
+ <para>The available bandwidth of a Lustre file system is determined as follows:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293969" xreflabel=""/> The <emphasis>network bandwidth</emphasis> equals the aggregated bandwidth of the OSSs to the targets.</para>
+ <para> The <emphasis>network bandwidth</emphasis> equals the aggregated bandwidth of the OSSs to the targets.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293970" xreflabel=""/> The <emphasis>disk bandwidth</emphasis> equals the sum of the disk bandwidths of the storage targets (OSTs) up to the limit of the network bandwidth.</para>
+ <para> The <emphasis>disk bandwidth</emphasis> equals the sum of the disk bandwidths of the storage targets (OSTs) up to the limit of the network bandwidth.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293971" xreflabel=""/> The <emphasis>aggregate bandwidth</emphasis> equals the minimium of the disk bandwidth and the network bandwidth.</para>
+ <para> The <emphasis>aggregate bandwidth</emphasis> equals the minimium of the disk bandwidth and the network bandwidth.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293972" xreflabel=""/> The <emphasis>available file system space</emphasis> equals the sum of the available space of all the OSTs.</para>
+ <para> The <emphasis>available file system space</emphasis> equals the sum of the available space of all the OSTs.</para>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438250_89922">
<title>1.3.1 Lustre File System and Striping</title>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1297909" xreflabel=""/>One of the main factors leading to the high performance of Lustre file systems is the ability to stripe data across multiple OSTs in a round-robin fashion. Users can optionally configure for each file the number of stripes, stripe size, and OSTs that are used.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1297885" xreflabel=""/>Striping can be used to improve performance when the aggregate bandwidth to a single file exeeds the bandwidth of a single OST. The ability to stripe is also useful when a single OST does not have anough free space to hold an entire file. For more information about benefits and drawbacks of file striping, see <xref linkend="dbdoclet.50438209_48033"/>.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1295480" xreflabel=""/>Striping allows segments or 'chunks' of data in a file to be stored on different OSTs, as shown in <xref linkend="understandinglustre.fig.filestripe"/>. In the Lustre file system, a RAID 0 pattern is used in which data is "striped" across a certain number of objects. The number of objects in a single file is called the <literal>stripe_count</literal>.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293978" xreflabel=""/>Each object contains a chunk of data from the file. When the chunk of data being written to a particular object exceeds the <literal>stripe_size</literal>, the next chunk of data in the file is stored on the next object.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293979" xreflabel=""/>Default values for <literal>stripe_count</literal> and <literal>stripe_size</literal> are set for the file system. The default value for <literal>stripe_count</literal> is 1 stripe for file and the default value for <literal>stripe_size</literal> is 1MB. The user may change these values on a per directory or per file basis. For more details, see <xref linkend="dbdoclet.50438209_78664"/>.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1293983" xreflabel=""/><xref linkend="understandinglustre.fig.filestripe"/>, the <literal>stripe_size</literal> for File C is larger than the <literal>stripe_size</literal> for File A, allowing more data to be stored in a single stripe for File C. The <literal>stripe_count</literal> for File A is 3, resulting in data striped across three objects, while the <literal>stripe_count</literal> for File B and File C is 1.</para>
- <para><anchor xml:id="dbdoclet.50438250_pgfId-1297121" xreflabel=""/>No space is reserved on the OST for unwritten data. File A in <xref linkend="understandinglustre.fig.filestripe"/>.</para>
+ <para>One of the main factors leading to the high performance of Lustre file systems is the ability to stripe data across multiple OSTs in a round-robin fashion. Users can optionally configure for each file the number of stripes, stripe size, and OSTs that are used.</para>
+ <para>Striping can be used to improve performance when the aggregate bandwidth to a single file exeeds the bandwidth of a single OST. The ability to stripe is also useful when a single OST does not have anough free space to hold an entire file. For more information about benefits and drawbacks of file striping, see <xref linkend="dbdoclet.50438209_48033"/>.</para>
+ <para>Striping allows segments or 'chunks' of data in a file to be stored on different OSTs, as shown in <xref linkend="understandinglustre.fig.filestripe"/>. In the Lustre file system, a RAID 0 pattern is used in which data is "striped" across a certain number of objects. The number of objects in a single file is called the <literal>stripe_count</literal>.</para>
+ <para>Each object contains a chunk of data from the file. When the chunk of data being written to a particular object exceeds the <literal>stripe_size</literal>, the next chunk of data in the file is stored on the next object.</para>
+ <para>Default values for <literal>stripe_count</literal> and <literal>stripe_size</literal> are set for the file system. The default value for <literal>stripe_count</literal> is 1 stripe for file and the default value for <literal>stripe_size</literal> is 1MB. The user may change these values on a per directory or per file basis. For more details, see <xref linkend="dbdoclet.50438209_78664"/>.</para>
+ <para><xref linkend="understandinglustre.fig.filestripe"/>, the <literal>stripe_size</literal> for File C is larger than the <literal>stripe_size</literal> for File A, allowing more data to be stored in a single stripe for File C. The <literal>stripe_count</literal> for File A is 3, resulting in data striped across three objects, while the <literal>stripe_count</literal> for File B and File C is 1.</para>
+ <para>No space is reserved on the OST for unwritten data. File A in <xref linkend="understandinglustre.fig.filestripe"/>.</para>
<figure>
<title xml:id="understandinglustre.fig.filestripe">File striping on Lustre</title>
<mediaobject>
<info>
<title xml:id="understandinglustrenetworking.title">Understanding Lustre Networking (LNET)</title>
</info>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289855" xreflabel=""/>This chapter introduces Lustre Networking (LNET) and includes the following sections:</para>
+ <para>This chapter introduces Lustre Networking (LNET) and includes the following sections:</para>
<itemizedlist>
<listitem>
<para>
</itemizedlist>
<section xml:id="dbdoclet.50438191_22878">
<title>2.1 Introducing LNET</title>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289866" xreflabel=""/>In a cluster with a Lustre file system, the system network connecting the servers and the clients is implemented using Lustre Networking (LNET), which provides the communication infrastructure required by the Lustre file system.</para>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289867" xreflabel=""/>LNET supports many commonly-used network types, such as InfiniBand and IP networks, and allows simultaneous availability across multiple network types with routing between them. Remote Direct Memory Access (RDMA) is permitted when supported by underlying networks using the appropriate Lustre network driver (LND). High availability and recovery features enable transparent recovery in conjunction with failover servers.</para>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289868" xreflabel=""/>An LND is a pluggable driver that provides support for a particular network type. LNDs are loaded into the driver stack, with one LND for each network type in use.</para>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1290033" xreflabel=""/>For information about configuring LNET, see <xref linkend="configuringlnet"/>.</para>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1290034" xreflabel=""/>For information about administering LNET, see <xref linkend="adminlustrepart3"/>.</para>
+ <para>In a cluster with a Lustre file system, the system network connecting the servers and the clients is implemented using Lustre Networking (LNET), which provides the communication infrastructure required by the Lustre file system.</para>
+ <para>LNET supports many commonly-used network types, such as InfiniBand and IP networks, and allows simultaneous availability across multiple network types with routing between them. Remote Direct Memory Access (RDMA) is permitted when supported by underlying networks using the appropriate Lustre network driver (LND). High availability and recovery features enable transparent recovery in conjunction with failover servers.</para>
+ <para>An LND is a pluggable driver that provides support for a particular network type. LNDs are loaded into the driver stack, with one LND for each network type in use.</para>
+ <para>For information about configuring LNET, see <xref linkend="configuringlnet"/>.</para>
+ <para>For information about administering LNET, see <xref linkend="adminlustrepart3"/>.</para>
</section>
<section xml:id="dbdoclet.50438191_19625">
<title>2.2 Key Features of LNET</title>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289918" xreflabel=""/>Key features of LNET include:</para>
+ <para>Key features of LNET include:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289871" xreflabel=""/><anchor xml:id="dbdoclet.50438191_57931" xreflabel=""/>RDMA, when supported by underlying networks such as InfiniBand or Myrinet MX</para>
+ <para><anchor xml:id="dbdoclet.50438191_57931" xreflabel=""/>RDMA, when supported by underlying networks such as InfiniBand or Myrinet MX</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289872" xreflabel=""/>Support for many commonly-used network types such as InfiniBand and TCP/IP</para>
+ <para>Support for many commonly-used network types such as InfiniBand and TCP/IP</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289873" xreflabel=""/>High availability and recovery features enabling transparent recovery in conjunction with failover servers</para>
+ <para>High availability and recovery features enabling transparent recovery in conjunction with failover servers</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289874" xreflabel=""/>Simultaneous availability of multiple network types with routing between them.</para>
+ <para>Simultaneous availability of multiple network types with routing between them.</para>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1290430" xreflabel=""/>LNET provides end-to-end throughput over Gigabit Ethernet (GigE) networks in excess of 100 MB/s, throughput up to 1.5 GB/s over InfiniBand double data rate (DDR) links, and throughput over 1 GB/s across 10GigE interfaces.</para>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1290434" xreflabel=""/>Lustre can use bonded networks, such as bonded Ethernet networks, when the underlying network technology supports bonding. For more information, see <xref linkend="settingupbonding"/>.</para>
+ <para>LNET provides end-to-end throughput over Gigabit Ethernet (GigE) networks in excess of 100 MB/s, throughput up to 1.5 GB/s over InfiniBand double data rate (DDR) links, and throughput over 1 GB/s across 10GigE interfaces.</para>
+ <para>Lustre can use bonded networks, such as bonded Ethernet networks, when the underlying network technology supports bonding. For more information, see <xref linkend="settingupbonding"/>.</para>
</section>
<section xml:id="dbdoclet.50438191_20721">
<title>2.3 Supported Network Types</title>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289882" xreflabel=""/>LNET includes LNDs to support many network types including:</para>
+ <para>LNET includes LNDs to support many network types including:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289883" xreflabel=""/> InfiniBand: OpenFabrics OFED (o2ib)</para>
+ <para> InfiniBand: OpenFabrics OFED (o2ib)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289884" xreflabel=""/> TCP (any network carrying TCP traffic, including GigE, 10GigE, and IPoIB)</para>
+ <para> TCP (any network carrying TCP traffic, including GigE, 10GigE, and IPoIB)</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289885" xreflabel=""/> Cray: Seastar</para>
+ <para> Cray: Seastar</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289886" xreflabel=""/> Myrinet: MX</para>
+ <para> Myrinet: MX</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289888" xreflabel=""/><anchor xml:id="dbdoclet.50438191_marker-1289887" xreflabel=""/> RapidArray: ra</para>
+ <para><anchor xml:id="dbdoclet.50438191_marker-1289887" xreflabel=""/> RapidArray: ra</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_pgfId-1289890" xreflabel=""/><anchor xml:id="dbdoclet.50438191_marker-1289889" xreflabel=""/> Quadrics: Elan</para>
+ <para><anchor xml:id="dbdoclet.50438191_marker-1289889" xreflabel=""/> Quadrics: Elan</para>
</listitem>
</itemizedlist>
</section>
<info>
<title xml:id='upgradinglustre.title'>Upgrading Lustre</title>
</info>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1289198" xreflabel=""/>This chapter describes Lustre interoperability and how to upgrade to Lustre 2.0, and includes the following sections:</para>
+ <para>This chapter describes Lustre interoperability and how to upgrade to Lustre 2.0, and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438205_82079"/>Lustre Interoperability</para>
<section xml:id="dbdoclet.50438205_82079">
<title>16.1 Lustre <anchor xml:id="dbdoclet.50438205_marker-1293321" xreflabel=""/>Interoperability</title>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1294233" xreflabel=""/>Lustre 2.0 is built on a new architectural code base, which is different than the one used with Lustre 1.8. These architectural changes require existing Lustre 1.8.x users to follow a slightly different procedure to upgrade to Lustre 2.0 - requiring clients to be unmounted and the file system be shut down. Once the servers are upgraded and restarted, then the clients can be remounted. After the upgrade, Lustre 2.0 servers can interoperate with compatible 1.8 clients and servers. Lustre 2.0 does <emphasis>not</emphasis> support 2.0 clients interoperating with 1.8 servers.</para>
+ <para>Lustre 2.0 is built on a new architectural code base, which is different than the one used with Lustre 1.8. These architectural changes require existing Lustre 1.8.x users to follow a slightly different procedure to upgrade to Lustre 2.0 - requiring clients to be unmounted and the file system be shut down. Once the servers are upgraded and restarted, then the clients can be remounted. After the upgrade, Lustre 2.0 servers can interoperate with compatible 1.8 clients and servers. Lustre 2.0 does <emphasis>not</emphasis> support 2.0 clients interoperating with 1.8 servers.</para>
<note><para>Lustre 1.8 clients support a mix of 1.8 and 2.0 OSTs, not all OSSs need to be upgraded at the same time.</para></note>
<note><para>Lustre 2.0 is compatible with version 1.8.4 and above. If you are planning a heterogeneous environment (mixed 1.8 and 2.0 servers), make sure that version 1.8.4 is installed on the client and server nodes that are not upgraded to 2.0.</para></note>
</section>
<section xml:id="dbdoclet.50438205_51369">
<title>16.2 Upgrading <anchor xml:id="dbdoclet.50438205_marker-1294652" xreflabel=""/>Lustre 1.8.x to 2.0</title>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1294654" xreflabel=""/>Upgrading to Lustre 2.0 involves shutting down the file system and upgrading servers, and optionally clients, all at the same time. Lustre 2.0 does not support a rolling upgrade in which the file system operates continuously while individual servers (or their failover partners) and clients are upgraded one at a time.</para>
+ <para>Upgrading to Lustre 2.0 involves shutting down the file system and upgrading servers, and optionally clients, all at the same time. Lustre 2.0 does not support a rolling upgrade in which the file system operates continuously while individual servers (or their failover partners) and clients are upgraded one at a time.</para>
<note><para>Although the Lustre 1.8 to 2.0 upgrade path has been tested, for best results we recommend performing a fresh Lustre 2.0 installation, rather than upgrading from 1.8 to 2.0.</para></note>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438205_pgfId-1291181" xreflabel=""/>16.2.1 <anchor xml:id="dbdoclet.50438205_80840" xreflabel=""/>Performing a <anchor xml:id="dbdoclet.50438205_marker-1293327" xreflabel=""/>File System Upgrade</title>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1291188" xreflabel=""/>This procedure describes a file system upgrade in which Lustre 2.0 packages are installed on multiple 1.8.x servers and, optionally, clients, requiring a file system shut down. You can choose to upgrade the entire Lustre file system to 2.0 or just upgrade the Lustre servers to 2.0 and leave the clients at 1.8.x. Lustre 2.0 servers can interoperate with compatible 1.8 clients and servers.</para>
+ <title>16.2.1 <anchor xml:id="dbdoclet.50438205_80840" xreflabel=""/>Performing a <anchor xml:id="dbdoclet.50438205_marker-1293327" xreflabel=""/>File System Upgrade</title>
+ <para>This procedure describes a file system upgrade in which Lustre 2.0 packages are installed on multiple 1.8.x servers and, optionally, clients, requiring a file system shut down. You can choose to upgrade the entire Lustre file system to 2.0 or just upgrade the Lustre servers to 2.0 and leave the clients at 1.8.x. Lustre 2.0 servers can interoperate with compatible 1.8 clients and servers.</para>
<tip><para>In a Lustre upgrade, the package install and file system unmount steps are reversible; you can do either step first. To minimize downtime, this procedure first performs the 2.0 package installation, and then unmounts the file system.</para></tip>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1291279" xreflabel=""/>Make a complete, restorable file system backup before upgrading Lustre.</para>
+ <para>Make a complete, restorable file system backup before upgrading Lustre.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1294770" xreflabel=""/>If any Lustre nodes will not be upgraded to 2.0, make sure that these client and server nodes are at version 1.8.4.</para>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1294786" xreflabel=""/>Lustre 2.0 is compatible with version 1.8.4 and above. If you are planning a heterogeneous environment (mixed 1.8 and 2.0 clients and servers), make sure that version 1.8.4 is installed on nodes that are not upgraded to 2.0.</para>
+ <para>If any Lustre nodes will not be upgraded to 2.0, make sure that these client and server nodes are at version 1.8.4.</para>
+ <para>Lustre 2.0 is compatible with version 1.8.4 and above. If you are planning a heterogeneous environment (mixed 1.8 and 2.0 clients and servers), make sure that version 1.8.4 is installed on nodes that are not upgraded to 2.0.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1290500" xreflabel=""/>Install the 2.0 packages on the Lustre servers and, optionally, the clients.</para>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1291180" xreflabel=""/>Some or all servers can be upgraded. Some or all clients can be upgraded.</para>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1290700" xreflabel=""/>For help determining where to install a specific package, see <link xl:href="InstallingLustre.html#50438261_21654">TABLE 8-1</link> (Lustre packages, descriptions and installation guidance).</para>
+ <para>Install the 2.0 packages on the Lustre servers and, optionally, the clients.</para>
+ <para>Some or all servers can be upgraded. Some or all clients can be upgraded.</para>
+ <para>For help determining where to install a specific package, see <link xl:href="InstallingLustre.html#50438261_21654">TABLE 8-1</link> (Lustre packages, descriptions and installation guidance).</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1290965" xreflabel=""/>Install the kernel, modules and ldiskfs packages. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438205_pgfId-1290527" xreflabel=""/>$ rpm -ivh
-<anchor xml:id="dbdoclet.50438205_pgfId-1292146" xreflabel=""/>kernel-lustre-smp-<ver> \
-<anchor xml:id="dbdoclet.50438205_pgfId-1290528" xreflabel=""/>kernel-ib-<ver> \
-<anchor xml:id="dbdoclet.50438205_pgfId-1290529" xreflabel=""/>lustre-modules-<ver> \
-<anchor xml:id="dbdoclet.50438205_pgfId-1290530" xreflabel=""/>lustre-ldiskfs-<ver>
+ <para>Install the kernel, modules and ldiskfs packages. For example:</para>
+ <screen>$ rpm -ivh
+kernel-lustre-smp-<ver> \
+kernel-ib-<ver> \
+lustre-modules-<ver> \
+lustre-ldiskfs-<ver>
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1290595" xreflabel=""/>Upgrade the utilities/userspace packages. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438205_pgfId-1290597" xreflabel=""/>$ rpm -Uvh lustre-<ver>
+ <para>Upgrade the utilities/userspace packages. For example:</para>
+ <screen>$ rpm -Uvh lustre-<ver>
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1291701" xreflabel=""/>If a new e2fsprogs package is available, upgrade it. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438205_pgfId-1291702" xreflabel=""/>$ rpm -Uvh e2fsprogs-<ver>
+ <para>If a new e2fsprogs package is available, upgrade it. For example:</para>
+ <screen>$ rpm -Uvh e2fsprogs-<ver>
</screen>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1294595" xreflabel=""/>Use e2fsprogs-1.41-10 or later, available at:</para>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1294597" xreflabel=""/><link xl:href="http://downloads.lustre.org/public/tools/e2fsprogs/">http://downloads.lustre.org/public/tools/e2fsprogs/</link></para>
+ <para>Use e2fsprogs-1.41-10 or later, available at:</para>
+ <para><link xl:href="http://downloads.lustre.org/public/tools/e2fsprogs/">http://downloads.lustre.org/public/tools/e2fsprogs/</link></para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1291712" xreflabel=""/>(Optional) If you want to add optional packages to your Lustre system, install them now.</para>
+ <para>(Optional) If you want to add optional packages to your Lustre system, install them now.</para>
</listitem></orderedlist>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1290496" xreflabel=""/>Shut down the file system.</para>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1290712" xreflabel=""/>Shut down the components in this order: clients, then the MDT, then OSTs. Unmounting a block device causes Lustre to be shut down on that node.</para>
+ <para>Shut down the file system.</para>
+ <para>Shut down the components in this order: clients, then the MDT, then OSTs. Unmounting a block device causes Lustre to be shut down on that node.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1290276" xreflabel=""/>Unmount the clients. On each client node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438205_pgfId-1290492" xreflabel=""/>umount <mount point>
+ <para>Unmount the clients. On each client node, run:</para>
+ <screen>umount <mount point>
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1291085" xreflabel=""/>Unmount the MDT. On the MDS node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438205_pgfId-1291086" xreflabel=""/>umount <mount point>
+ <para>Unmount the MDT. On the MDS node, run:</para>
+ <screen>umount <mount point>
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1291083" xreflabel=""/>Unmount the OSTs (be sure to unmount all OSTs). On each OSS node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438205_pgfId-1290318" xreflabel=""/>umount <mount point>
+ <para>Unmount the OSTs (be sure to unmount all OSTs). On each OSS node, run:</para>
+ <screen>umount <mount point>
</screen>
</listitem></orderedlist>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1292164" xreflabel=""/>Unload the old Lustre modules by rebooting the node or manually removing the Lustre modules.</para>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1294825" xreflabel=""/>Run lustre_rmmod several times and use lsmod to check the currently loaded modules.</para>
+ <para>Unload the old Lustre modules by rebooting the node or manually removing the Lustre modules.</para>
+ <para>Run lustre_rmmod several times and use lsmod to check the currently loaded modules.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1292162" xreflabel=""/>Start the upgraded file system.</para>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1290713" xreflabel=""/>Start the components in this order: OSTs, then the MDT, then clients.</para>
+ <para>Start the upgraded file system.</para>
+ <para>Start the components in this order: OSTs, then the MDT, then clients.</para>
<orderedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1292183" xreflabel=""/>Mount the OSTs (be sure to mount all OSTs). On each OSS node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438205_pgfId-1290430" xreflabel=""/>mount -t lustre <block device name> <mount point>
+ <para>Mount the OSTs (be sure to mount all OSTs). On each OSS node, run:</para>
+ <screen>mount -t lustre <block device name> <mount point>
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1292179" xreflabel=""/>Mount the MDT. On the MDS node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438205_pgfId-1290431" xreflabel=""/>mount -t lustre <block device name> <mount point>
+ <para>Mount the MDT. On the MDS node, run:</para>
+ <screen>mount -t lustre <block device name> <mount point>
</screen>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1290404" xreflabel=""/>Mount the file system on the clients. On each client node, run:</para>
- <screen><anchor xml:id="dbdoclet.50438205_pgfId-1290467" xreflabel=""/>mount -t lustre <MGS node>:/<fsname> <mount point>
+ <para>Mount the file system on the clients. On each client node, run:</para>
+ <screen>mount -t lustre <MGS node>:/<fsname> <mount point>
</screen>
</listitem></orderedlist></listitem></orderedlist>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1294471" xreflabel=""/>If you have a problem upgrading Lustre, contact us via the <link xl:href="https://jira.whamcloud.com">Whamcloud Jira</link> bug tracker.</para>
- <para><anchor xml:id="dbdoclet.50438205_pgfId-1292981" xreflabel=""/> </para>
+ <para>If you have a problem upgrading Lustre, contact us via the <link xl:href="https://jira.whamcloud.com">Whamcloud Jira</link> bug tracker.</para>
+ <para> </para>
</section>
</section>
</chapter>
<info>
<title xml:id='userutilities.title'>User Utilities</title>
</info>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305183" xreflabel=""/>This chapter describes user utilities and includes the following sections:</para>
+ <para>This chapter describes user utilities and includes the following sections:</para>
<itemizedlist><listitem>
<para><xref linkend="dbdoclet.50438206_94597"/></para>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438206_94597" >
<title>32.1 <anchor xreflabel=""/>l<anchor xml:id="dbdoclet.50438206_marker-1305209" xreflabel=""/>fs</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305211" xreflabel=""/>The lfs utility can be used for user configuration routines and monitoring.</para>
+ <para>The lfs utility can be used for user configuration routines and monitoring.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305212" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1305213" xreflabel=""/>lfs
-<anchor xml:id="dbdoclet.50438206_pgfId-1305214" xreflabel=""/>lfs changelog [--follow] <mdtname> [startrec [endrec]]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305215" xreflabel=""/>lfs changelog_clear <mdtname> <id> <endrec>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305216" xreflabel=""/>lfs check <mds|osts|servers>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305217" xreflabel=""/>lfs df [-i] [-h] [--pool]-p <fsname>[.<pool>] [path]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305218" xreflabel=""/>lfs find [[!] --atime|-A [-+]N] [[!] --mtime|-M [-+]N]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305219" xreflabel=""/> [[!] --ctime|-C [-+]N] [--maxdepth|-D N] [--name|-n <pattern>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305220" xreflabel=""/> [--print|-p] [--print0|-P] [[!] --obd|-O <uuid[s]>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305221" xreflabel=""/> [[!] --size|-S [+-]N[kMGTPE]] --type |-t {bcdflpsD}]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305222" xreflabel=""/> [[!] --gid|-g|--group|-G <gname>|<gid>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305223" xreflabel=""/> [[!] --uid|-u|--user|-U <uname>|<uid>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305224" xreflabel=""/> <dirname|filename>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305225" xreflabel=""/>lfs osts [path]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305226" xreflabel=""/>lfs getstripe [--obd|-O <uuid>] [--quiet|-q] [--verbose|-v]
-<anchor xml:id="dbdoclet.50438206_pgfId-1306270" xreflabel=""/> [--count|-c] [--index|-i | --offset|-o]
-<anchor xml:id="dbdoclet.50438206_pgfId-1306271" xreflabel=""/> [--size|-s] [--pool|-p] [--directory|-d]
-<anchor xml:id="dbdoclet.50438206_pgfId-1306272" xreflabel=""/> [--recursive|-r] <dirname|filename> ...
-<anchor xml:id="dbdoclet.50438206_pgfId-1305230" xreflabel=""/>lfs setstripe [--size|-s stripe_size] [--count|-c stripe_cnt]
-<anchor xml:id="dbdoclet.50438206_pgfId-1306280" xreflabel=""/> [--index|-i|--offset|-o start_ost_index]
-<anchor xml:id="dbdoclet.50438206_pgfId-1306281" xreflabel=""/> [--pool|-p <pool>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1306282" xreflabel=""/> <dirname|filename>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305233" xreflabel=""/>lfs setstripe -d <dir>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305234" xreflabel=""/>lfs poollist <filesystem>[.<pool>]|<pathname>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305235" xreflabel=""/>lfs quota [-q] [-v] [-o obd_uuid|-I ost_idx|-i mdt_idx] [-u <uname>| -u <uid>|-\
+ <title>Synopsis</title>
+ <screen>lfs
+lfs changelog [--follow] <mdtname> [startrec [endrec]]
+lfs changelog_clear <mdtname> <id> <endrec>
+lfs check <mds|osts|servers>
+lfs df [-i] [-h] [--pool]-p <fsname>[.<pool>] [path]
+lfs find [[!] --atime|-A [-+]N] [[!] --mtime|-M [-+]N]
+ [[!] --ctime|-C [-+]N] [--maxdepth|-D N] [--name|-n <pattern>]
+ [--print|-p] [--print0|-P] [[!] --obd|-O <uuid[s]>]
+ [[!] --size|-S [+-]N[kMGTPE]] --type |-t {bcdflpsD}]
+ [[!] --gid|-g|--group|-G <gname>|<gid>]
+ [[!] --uid|-u|--user|-U <uname>|<uid>]
+ <dirname|filename>
+lfs osts [path]
+lfs getstripe [--obd|-O <uuid>] [--quiet|-q] [--verbose|-v]
+ [--count|-c] [--index|-i | --offset|-o]
+ [--size|-s] [--pool|-p] [--directory|-d]
+ [--recursive|-r] <dirname|filename> ...
+lfs setstripe [--size|-s stripe_size] [--count|-c stripe_cnt]
+ [--index|-i|--offset|-o start_ost_index]
+ [--pool|-p <pool>]
+ <dirname|filename>
+lfs setstripe -d <dir>
+lfs poollist <filesystem>[.<pool>]|<pathname>
+lfs quota [-q] [-v] [-o obd_uuid|-I ost_idx|-i mdt_idx] [-u <uname>| -u <uid>|-\
g <gname>| -g <gid>] <filesystem>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305236" xreflabel=""/>lfs quota -t <-u|-g> <filesystem>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305237" xreflabel=""/>lfs quotacheck [-ug] <filesystem>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305238" xreflabel=""/>lfs quotachown [-i] <filesystem>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305239" xreflabel=""/>lfs quotaon [-ugf] <filesystem>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305240" xreflabel=""/>lfs quotaoff [-ug] <filesystem>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305241" xreflabel=""/>lfs quotainv [-ug] [-f] <filesystem>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305242" xreflabel=""/>lfs setquota <-u|--user|-g|--group> <uname|uid|gname|gid>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305243" xreflabel=""/> [--block-softlimit <block-softlimit>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305244" xreflabel=""/> [--block-hardlimit <block-hardlimit>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305245" xreflabel=""/> [--inode-softlimit <inode-softlimit>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305246" xreflabel=""/> [--inode-hardlimit <inode-hardlimit>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305247" xreflabel=""/> <filesystem>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305248" xreflabel=""/>lfs setquota <-u|--user|-g|--group> <uname|uid|gname|gid>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305249" xreflabel=""/> [-b <block-softlimit>] [-B <block-hardlimit>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305250" xreflabel=""/> [-i <inode-softlimit>] [-I <inode-hardlimit>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305251" xreflabel=""/> <filesystem>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305252" xreflabel=""/>lfs setquota -t <-u|-g>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305253" xreflabel=""/> [--block-grace <block-grace>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305254" xreflabel=""/> [--inode-grace <inode-grace>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305255" xreflabel=""/> <filesystem>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305256" xreflabel=""/>lfs setquota -t <-u|-g>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305257" xreflabel=""/> [-b <block-grace>] [-i <inode-grace>]
-<anchor xml:id="dbdoclet.50438206_pgfId-1305258" xreflabel=""/> <filesystem>
-<anchor xml:id="dbdoclet.50438206_pgfId-1305259" xreflabel=""/>lfs help
+lfs quota -t <-u|-g> <filesystem>
+lfs quotacheck [-ug] <filesystem>
+lfs quotachown [-i] <filesystem>
+lfs quotaon [-ugf] <filesystem>
+lfs quotaoff [-ug] <filesystem>
+lfs quotainv [-ug] [-f] <filesystem>
+lfs setquota <-u|--user|-g|--group> <uname|uid|gname|gid>
+ [--block-softlimit <block-softlimit>]
+ [--block-hardlimit <block-hardlimit>]
+ [--inode-softlimit <inode-softlimit>]
+ [--inode-hardlimit <inode-hardlimit>]
+ <filesystem>
+lfs setquota <-u|--user|-g|--group> <uname|uid|gname|gid>
+ [-b <block-softlimit>] [-B <block-hardlimit>]
+ [-i <inode-softlimit>] [-I <inode-hardlimit>]
+ <filesystem>
+lfs setquota -t <-u|-g>
+ [--block-grace <block-grace>]
+ [--inode-grace <inode-grace>]
+ <filesystem>
+lfs setquota -t <-u|-g>
+ [-b <block-grace>] [-i <inode-grace>]
+ <filesystem>
+lfs help
</screen>
<note><para>In the above example, the <filesystem> parameter refers to the mount point of the Lustre file system. The default mount point is /mnt/lustre</para></note>
<note><para>The old lfs quota output was very detailed and contained cluster-wide quota statistics (including cluster-wide limits for a user/group and cluster-wide usage for a user/group), as well as statistics for each MDS/OST. Now, lfs quota has been updated to provide only cluster-wide statistics, by default. To obtain the full report of cluster-wide limits, usage and statistics, use the -v option with lfs quota.</para></note>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305262" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305263" xreflabel=""/>The lfs utility is used to create a new file with a specific striping pattern, determine the default striping pattern, gather the extended attributes (object numbers and location) for a specific file, find files with specific attributes, list OST information or set quota limits. It can be invoked interactively without any arguments or in a non-interactive mode with one of the supported arguments.</para>
+ <title>Description</title>
+ <para>The lfs utility is used to create a new file with a specific striping pattern, determine the default striping pattern, gather the extended attributes (object numbers and location) for a specific file, find files with specific attributes, list OST information or set quota limits. It can be invoked interactively without any arguments or in a non-interactive mode with one of the supported arguments.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305264" xreflabel=""/>Options</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305265" xreflabel=""/>The various lfs options are listed and described below. For a complete list of available options, type help at the lfs prompt.</para>
+ <title>Options</title>
+ <para>The various lfs options are listed and described below. For a complete list of available options, type help at the lfs prompt.</para>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
<colspec colname="c3" colwidth="33*"/>
<thead>
<row>
- <entry nameend="c2" namest="c1"><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438206_pgfId-1305268" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438206_pgfId-1305272" xreflabel=""/>Description</emphasis></para></entry>
+ <entry nameend="c2" namest="c1"><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305276" xreflabel=""/><emphasis role="bold">changelog</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305280" xreflabel=""/>Shows the metadata changes on an MDT. Start and end points are optional. The <emphasis role="bold">--follow</emphasis> option blocks on new changes; this option is only valid when run directly on the MDT node.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">changelog</emphasis></para></entry>
+ <entry><para> Shows the metadata changes on an MDT. Start and end points are optional. The <emphasis role="bold">--follow</emphasis> option blocks on new changes; this option is only valid when run directly on the MDT node.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305284" xreflabel=""/><emphasis role="bold">changelog_clear</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305288" xreflabel=""/>Indicates that changelog records previous to <endrec> are no longer of interest to a particular consumer <id>, potentially allowing the MDT to free up disk space. An <endrec> of 0 indicates the current last record. Changelog consumers must be registered on the MDT node using lctl.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">changelog_clear</emphasis></para></entry>
+ <entry><para> Indicates that changelog records previous to <endrec> are no longer of interest to a particular consumer <id>, potentially allowing the MDT to free up disk space. An <endrec> of 0 indicates the current last record. Changelog consumers must be registered on the MDT node using lctl.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305292" xreflabel=""/><emphasis role="bold">check</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305296" xreflabel=""/>Displays the status of MDS or OSTs (as specified in the command) or all servers (MDS and OSTs).</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">check</emphasis></para></entry>
+ <entry><para> Displays the status of MDS or OSTs (as specified in the command) or all servers (MDS and OSTs).</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305300" xreflabel=""/><emphasis role="bold">df [-i] [-h] [--pool|-p <fsname>[.<pool>] [path]</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305304" xreflabel=""/>Report file system disk space usage or inode usage (with <emphasis role="bold">-i</emphasis>) of each MDT/OST or a subset of OSTs if a pool is specified with <emphasis role="bold">-p</emphasis>. By default, prints the usage of all mounted Lustre file systems. Otherwise, if <emphasis role="bold">path</emphasis> is specified, prints only the usage of that file system. If <emphasis role="bold">-h</emphasis> is given, the output is printed in human-readable format, using SI base-2 suffixes for <emphasis role="bold">M</emphasis>ega-, <emphasis role="bold">G</emphasis>iga-, <emphasis role="bold">T</emphasis>era-, <emphasis role="bold">P</emphasis>eta-, or <emphasis role="bold">E</emphasis>xabytes.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">df [-i] [-h] [--pool|-p <fsname>[.<pool>] [path]</emphasis></para></entry>
+ <entry><para> Report file system disk space usage or inode usage (with <emphasis role="bold">-i</emphasis>) of each MDT/OST or a subset of OSTs if a pool is specified with <emphasis role="bold">-p</emphasis>. By default, prints the usage of all mounted Lustre file systems. Otherwise, if <emphasis role="bold">path</emphasis> is specified, prints only the usage of that file system. If <emphasis role="bold">-h</emphasis> is given, the output is printed in human-readable format, using SI base-2 suffixes for <emphasis role="bold">M</emphasis>ega-, <emphasis role="bold">G</emphasis>iga-, <emphasis role="bold">T</emphasis>era-, <emphasis role="bold">P</emphasis>eta-, or <emphasis role="bold">E</emphasis>xabytes.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305308" xreflabel=""/><emphasis role="bold">find</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305312" xreflabel=""/> Searches the directory tree rooted at the given directory/filename for files that match the given parameters.</para><para><anchor xml:id="dbdoclet.50438206_pgfId-1305313" xreflabel=""/> </para><para><anchor xml:id="dbdoclet.50438206_pgfId-1305314" xreflabel=""/>Using ! before an option negates its meaning (files NOT matching the parameter). Using + before a numeric value means files with the parameter OR MORE. Using - before a numeric value means files with the parameter OR LESS.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">find</emphasis></para></entry>
+ <entry><para> Searches the directory tree rooted at the given directory/filename for files that match the given parameters.</para><para> </para><para>Using ! before an option negates its meaning (files NOT matching the parameter). Using + before a numeric value means files with the parameter OR MORE. Using - before a numeric value means files with the parameter OR LESS.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305318" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305320" xreflabel=""/><emphasis role="bold">--atime</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305322" xreflabel=""/>File was last accessed N*24 hours ago. (There is no guarantee that <emphasis role="bold">atime</emphasis> is kept coherent across the cluster.)</para><para><anchor xml:id="dbdoclet.50438206_pgfId-1305323" xreflabel=""/> </para><para><anchor xml:id="dbdoclet.50438206_pgfId-1305324" xreflabel=""/>OSTs store a transient atime that is updated when clients do read requests. Permanent <emphasis role="bold">atime</emphasis> is written to the MDS when the file is closed. However, on-disk atime is only updated if it is more than 60 seconds old (/proc/fs/lustre/mds/*/max_atime_diff). Lustre considers the latest <emphasis role="bold">atime</emphasis> from all OSTs. If a setattr is set by user, then it is updated on both the MDS and OST, allowing the <emphasis role="bold">atime</emphasis> to go backward.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--atime</emphasis></para></entry>
+ <entry><para> File was last accessed N*24 hours ago. (There is no guarantee that <emphasis role="bold">atime</emphasis> is kept coherent across the cluster.)</para><para> </para><para>OSTs store a transient atime that is updated when clients do read requests. Permanent <emphasis role="bold">atime</emphasis> is written to the MDS when the file is closed. However, on-disk atime is only updated if it is more than 60 seconds old (/proc/fs/lustre/mds/*/max_atime_diff). Lustre considers the latest <emphasis role="bold">atime</emphasis> from all OSTs. If a setattr is set by user, then it is updated on both the MDS and OST, allowing the <emphasis role="bold">atime</emphasis> to go backward.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305328" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305330" xreflabel=""/><emphasis role="bold">--ctime</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305332" xreflabel=""/>File status was last changed N*24 hours ago.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--ctime</emphasis></para></entry>
+ <entry><para> File status was last changed N*24 hours ago.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305336" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305338" xreflabel=""/><emphasis role="bold">--mtime</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305340" xreflabel=""/>File data was last modified N*24 hours ago.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--mtime</emphasis></para></entry>
+ <entry><para> File data was last modified N*24 hours ago.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305344" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305346" xreflabel=""/><emphasis role="bold">--obd</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305348" xreflabel=""/>File has an object on a specific OST(s).</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--obd</emphasis></para></entry>
+ <entry><para> File has an object on a specific OST(s).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305352" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305354" xreflabel=""/><emphasis role="bold">--size</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305356" xreflabel=""/>File has a size in bytes, or kilo-, Mega-, Giga-, Tera-, Peta- or Exabytes if a suffix is given.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--size</emphasis></para></entry>
+ <entry><para> File has a size in bytes, or kilo-, Mega-, Giga-, Tera-, Peta- or Exabytes if a suffix is given.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305360" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305362" xreflabel=""/><emphasis role="bold">--type</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305364" xreflabel=""/>File has the type (block, character, directory, pipe, file, symlink, socket or Door [Solaris]).</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--type</emphasis></para></entry>
+ <entry><para> File has the type (block, character, directory, pipe, file, symlink, socket or Door [Solaris]).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305368" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305370" xreflabel=""/><emphasis role="bold">--uid</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305372" xreflabel=""/>File has a specific numeric user ID.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--uid</emphasis></para></entry>
+ <entry><para> File has a specific numeric user ID.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305376" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305378" xreflabel=""/><emphasis role="bold">--user</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305380" xreflabel=""/>File owned by a specific user (numeric user ID allowed).</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--user</emphasis></para></entry>
+ <entry><para> File owned by a specific user (numeric user ID allowed).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305384" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305386" xreflabel=""/><emphasis role="bold">--gid</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305388" xreflabel=""/>File has a specific group ID.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--gid</emphasis></para></entry>
+ <entry><para> File has a specific group ID.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305392" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305394" xreflabel=""/><emphasis role="bold">--group</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305396" xreflabel=""/>File belongs to a specific group (numeric group ID allowed).</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--group</emphasis></para></entry>
+ <entry><para> File belongs to a specific group (numeric group ID allowed).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305400" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305402" xreflabel=""/><emphasis role="bold">--maxdepth</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305404" xreflabel=""/>Limits <emphasis role="bold">find</emphasis> to descend at most N levels of the directory tree.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--maxdepth</emphasis></para></entry>
+ <entry><para> Limits <emphasis role="bold">find</emphasis> to descend at most N levels of the directory tree.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305408" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305410" xreflabel=""/><emphasis role="bold">--print / --print0</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305412" xreflabel=""/>Prints the full filename, followed by a new line or NULL character correspondingly.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--print / --print0</emphasis></para></entry>
+ <entry><para> Prints the full filename, followed by a new line or NULL character correspondingly.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305416" xreflabel=""/><emphasis role="bold">osts [path]</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306293" xreflabel=""/>Lists all OSTs for the file system. If a path located on a Lustre-mounted file system is specified, then only OSTs belonging to this file system are displayed.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">osts [path]</emphasis></para></entry>
+ <entry><para> Lists all OSTs for the file system. If a path located on a Lustre-mounted file system is specified, then only OSTs belonging to this file system are displayed.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305424" xreflabel=""/><emphasis role="bold">getstripe</emphasis></para></entry>
- <entry><para>
<anchor xml:id="dbdoclet.50438206_pgfId-1305428" xreflabel=""/>Lists striping information for a given filename or directory. By default, the stripe count, stripe size and offset are returned.</para><para> If you only want specific striping information, then the options of --count,--size,--index or --offset plus various combinations of these options can be used to retrieve specific information.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">getstripe</emphasis></para></entry>
+ <entry><para> Lists striping information for a given filename or directory. By default, the stripe count, stripe size and offset are returned.</para><para> If you only want specific striping information, then the options of --count,--size,--index or --offset plus various combinations of these options can be used to retrieve specific information.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305432" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305434" xreflabel=""/><emphasis role="bold">--obd <uuid></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305436" xreflabel=""/>Lists files that have an object on a specific OST.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--obd <uuid></emphasis></para></entry>
+ <entry><para> Lists files that have an object on a specific OST.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305440" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305442" xreflabel=""/><emphasis role="bold">--quiet</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305444" xreflabel=""/>Lists details about the file's object ID information.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--quiet</emphasis></para></entry>
+ <entry><para> Lists details about the file's object ID information.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305448" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305450" xreflabel=""/><emphasis role="bold">--verbose</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305452" xreflabel=""/>Prints additional striping information.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--verbose</emphasis></para></entry>
+ <entry><para> Prints additional striping information.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305456" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305458" xreflabel=""/><emphasis role="bold">--count</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305460" xreflabel=""/>Lists the stripe count (how many OSTs to use).</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--count</emphasis></para></entry>
+ <entry><para> Lists the stripe count (how many OSTs to use).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306303" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306305" xreflabel=""/><emphasis role="bold">--index</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306307" xreflabel=""/>Lists the index for each OST in the file system.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--index</emphasis></para></entry>
+ <entry><para> Lists the index for each OST in the file system.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306311" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306313" xreflabel=""/><emphasis role="bold">--offset</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306315" xreflabel=""/>Lists the OST index on which file striping starts.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--offset</emphasis></para></entry>
+ <entry><para> Lists the OST index on which file striping starts.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306319" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306321" xreflabel=""/><emphasis role="bold">--pool</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306323" xreflabel=""/>Lists the pools to which a file belongs.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--pool</emphasis></para></entry>
+ <entry><para> Lists the pools to which a file belongs.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305464" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305466" xreflabel=""/><emphasis role="bold">--size</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305468" xreflabel=""/>Lists the stripe size (how much data to write to one OST before moving to the next OST).</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--size</emphasis></para></entry>
+ <entry><para> Lists the stripe size (how much data to write to one OST before moving to the next OST).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305496" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305498" xreflabel=""/><emphasis role="bold">--directory</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305500" xreflabel=""/>Lists entries about a specified directory instead of its contents (in the same manner as ls -d).</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--directory</emphasis></para></entry>
+ <entry><para> Lists entries about a specified directory instead of its contents (in the same manner as ls -d).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305504" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305506" xreflabel=""/><emphasis role="bold">--recursive</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305508" xreflabel=""/>Recurses into all sub-directories.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--recursive</emphasis></para></entry>
+ <entry><para> Recurses into all sub-directories.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305512" xreflabel=""/><emphasis role="bold">setstripe</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305516" xreflabel=""/>Create new files with a specific file layout (stripe pattern) configuration.<footnote><para><anchor xml:id="dbdoclet.50438206_pgfId-1305519" xreflabel=""/>The file cannot exist prior to using setstripe. A directory must exist prior to using setstripe.</para></footnote></para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">setstripe</emphasis></para></entry>
+ <entry><para> Create new files with a specific file layout (stripe pattern) configuration.<footnote><para>The file cannot exist prior to using setstripe. A directory must exist prior to using setstripe.</para></footnote></para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306331" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306333" xreflabel=""/><emphasis role="bold">--count stripe_cnt</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306335" xreflabel=""/>Number of OSTs over which to stripe a file. A <emphasis role="bold">stripe_cnt</emphasis> of 0 uses the file system-wide default stripe count (default is 1). A <emphasis role="bold">stripe_cnt</emphasis> of -1 stripes over all available OSTs, and normally results in a file with 80 stripes.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--count stripe_cnt</emphasis></para></entry>
+ <entry><para> Number of OSTs over which to stripe a file. A <emphasis role="bold">stripe_cnt</emphasis> of 0 uses the file system-wide default stripe count (default is 1). A <emphasis role="bold">stripe_cnt</emphasis> of -1 stripes over all available OSTs, and normally results in a file with 80 stripes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305523" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305528" xreflabel=""/><emphasis role="bold">--size stripe_size</emphasis><footnote><para><anchor xml:id="dbdoclet.50438206_pgfId-1305527" xreflabel=""/>The default stripe-size is 0. The default start-ost is -1. Do NOT confuse them! If you set start-ost to 0, all new file creations occur on OST 0 (seldom a good idea).</para></footnote> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305531" xreflabel=""/> Number of bytes to store on an OST before moving to the next OST. A <emphasis role="bold">stripe_size</emphasis> of 0 uses the file system's default stripe size, (default is 1 MB). Can be specified with <emphasis role="bold">k</emphasis> (KB), <emphasis role="bold">m</emphasis> (MB), or <emphasis role="bold">g</emphasis> (GB), respectively.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--size stripe_size</emphasis><footnote><para>The default stripe-size is 0. The default start-ost is -1. Do NOT confuse them! If you set start-ost to 0, all new file creations occur on OST 0 (seldom a good idea).</para></footnote> </para></entry>
+ <entry><para> Number of bytes to store on an OST before moving to the next OST. A <emphasis role="bold">stripe_size</emphasis> of 0 uses the file system's default stripe size, (default is 1 MB). Can be specified with <emphasis role="bold">k</emphasis> (KB), <emphasis role="bold">m</emphasis> (MB), or <emphasis role="bold">g</emphasis> (GB), respectively.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305543" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305545" xreflabel=""/><emphasis role="bold">--index --offset start_ost_index</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305547" xreflabel=""/> The OST index (base 10, starting at 0) on which to start striping for this file. A start_ost_index value of -1 allows the MDS to choose the starting index. This is the default value, and it means that the MDS selects the starting OST as it wants. We strongly recommend selecting this default, as it allows space and load balancing to be done by the MDS as needed. The <emphasis role="bold">start_ost_index</emphasis> value has no relevance on whether the MDS will use round-robin or QoS weighted allocation for the remaining stripes in the file.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--index --offset start_ost_index</emphasis></para></entry>
+ <entry><para> The OST index (base 10, starting at 0) on which to start striping for this file. A start_ost_index value of -1 allows the MDS to choose the starting index. This is the default value, and it means that the MDS selects the starting OST as it wants. We strongly recommend selecting this default, as it allows space and load balancing to be done by the MDS as needed. The <emphasis role="bold">start_ost_index</emphasis> value has no relevance on whether the MDS will use round-robin or QoS weighted allocation for the remaining stripes in the file.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305551" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305553" xreflabel=""/><emphasis role="bold">--pool <pool></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305558" xreflabel=""/>Name of the pre-defined pool of OSTs (see <link xl:href="SystemConfigurationUtilities.html#50438219_38274">lctl</link>) that will be used for striping. The <emphasis role="bold">stripe_cnt</emphasis>, <emphasis role="bold">stripe_size</emphasis> and <emphasis role="bold">start_ost</emphasis> values are used as well. The <emphasis role="bold">start-ost</emphasis> value must be part of the pool or an error is returned.</para></entry>
+ <entry><para> </para></entry>
+ <entry><para> <emphasis role="bold">--pool <pool></emphasis></para></entry>
+ <entry><para> Name of the pre-defined pool of OSTs (see <link xl:href="SystemConfigurationUtilities.html#50438219_38274">lctl</link>) that will be used for striping. The <emphasis role="bold">stripe_cnt</emphasis>, <emphasis role="bold">stripe_size</emphasis> and <emphasis role="bold">start_ost</emphasis> values are used as well. The <emphasis role="bold">start-ost</emphasis> value must be part of the pool or an error is returned.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305570" xreflabel=""/><emphasis role="bold">setstripe -d</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305574" xreflabel=""/>Deletes default striping on the specified directory.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">setstripe -d</emphasis></para></entry>
+ <entry><para> Deletes default striping on the specified directory.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306343" xreflabel=""/><emphasis role="bold">poollist {filesystem} [.poolname]|{pathname}</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305582" xreflabel=""/>Lists pools in the file system or pathname, or OSTs in the file system's pool.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">poollist {filesystem} [.poolname]|{pathname}</emphasis></para></entry>
+ <entry><para> Lists pools in the file system or pathname, or OSTs in the file system's pool.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305586" xreflabel=""/><emphasis role="bold">quota [-q] [-v] [-o obd_uuid|-i mdt_idx|-I ost_idx] [-u|-g <uname>|<uid>|<gname>|<gid>] <filesystem></emphasis></para><para><anchor xml:id="dbdoclet.50438206_pgfId-1305587" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305591" xreflabel=""/>Displays disk usage and limits, either for the full file system or for objects on a specific OBD. A user or group name or an ID can be specified. If both user and group are omitted, quotas for the current UID/GID are shown. The <emphasis role="bold">-q</emphasis> option disables printing of additional descriptions (including column titles). It fills in blank spaces in the ''grace'' column with zeros (when there is no grace period set), to ensure that the number of columns is consistent. The <emphasis role="bold">-v</emphasis> option provides more verbose (per-OBD statistics) output.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">quota [-q] [-v] [-o obd_uuid|-i mdt_idx|-I ost_idx] [-u|-g <uname>|<uid>|<gname>|<gid>] <filesystem></emphasis></para><para> </para></entry>
+ <entry><para> Displays disk usage and limits, either for the full file system or for objects on a specific OBD. A user or group name or an ID can be specified. If both user and group are omitted, quotas for the current UID/GID are shown. The <emphasis role="bold">-q</emphasis> option disables printing of additional descriptions (including column titles). It fills in blank spaces in the ''grace'' column with zeros (when there is no grace period set), to ensure that the number of columns is consistent. The <emphasis role="bold">-v</emphasis> option provides more verbose (per-OBD statistics) output.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305595" xreflabel=""/><emphasis role="bold">quota -t <-u|-g> <filesystem></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305599" xreflabel=""/>Displays block and inode grace times for user (-u) or group (-g) quotas.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">quota -t <-u|-g> <filesystem></emphasis></para></entry>
+ <entry><para> Displays block and inode grace times for user (-u) or group (-g) quotas.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305603" xreflabel=""/><emphasis role="bold">quotachown</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305607" xreflabel=""/>Changes the file's owner and group on OSTs of the specified file system.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">quotachown</emphasis></para></entry>
+ <entry><para> Changes the file's owner and group on OSTs of the specified file system.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305611" xreflabel=""/><emphasis role="bold">quotacheck [-ugf] <filesystem></emphasis></para><para><anchor xml:id="dbdoclet.50438206_pgfId-1305612" xreflabel=""/> </para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305616" xreflabel=""/>Scans the specified file system for disk usage, and creates or updates quota files. Options specify quota for users (-u), groups (-g), and force (-f).</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">quotacheck [-ugf] <filesystem></emphasis></para><para> </para></entry>
+ <entry><para> Scans the specified file system for disk usage, and creates or updates quota files. Options specify quota for users (-u), groups (-g), and force (-f).</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305620" xreflabel=""/><emphasis role="bold">quotaon [-ugf] <filesystem></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305624" xreflabel=""/>Turns on file system quotas. Options specify quota for users (-u), groups (-g), and force (-f).</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">quotaon [-ugf] <filesystem></emphasis></para></entry>
+ <entry><para> Turns on file system quotas. Options specify quota for users (-u), groups (-g), and force (-f).</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305628" xreflabel=""/><emphasis role="bold">quotaoff [-ugf] <filesystem></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305632" xreflabel=""/>Turns off file system quotas. Options specify quota for users (-u), groups (-g), and force (-f).</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">quotaoff [-ugf] <filesystem></emphasis></para></entry>
+ <entry><para> Turns off file system quotas. Options specify quota for users (-u), groups (-g), and force (-f).</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305636" xreflabel=""/><emphasis role="bold">quotainv [-ug] [-f] <filesystem></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305640" xreflabel=""/>Clears quota files (administrative quota files if used without <emphasis role="bold">-f</emphasis>, operational quota files otherwise), all of their quota entries for users (-u) or groups (-g). After running <emphasis role="bold">quotainv</emphasis>, you must run <emphasis role="bold">quotacheck</emphasis> before using quotas.</para><para><anchor xml:id="dbdoclet.50438206_pgfId-1305641" xreflabel=""/><emphasis role="bold">CAUTION</emphasis>: Use extreme caution when using this command; its results cannot be undone.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">quotainv [-ug] [-f] <filesystem></emphasis></para></entry>
+ <entry><para> Clears quota files (administrative quota files if used without <emphasis role="bold">-f</emphasis>, operational quota files otherwise), all of their quota entries for users (-u) or groups (-g). After running <emphasis role="bold">quotainv</emphasis>, you must run <emphasis role="bold">quotacheck</emphasis> before using quotas.</para><para><emphasis role="bold">CAUTION</emphasis>: Use extreme caution when using this command; its results cannot be undone.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305645" xreflabel=""/><emphasis role="bold">setquota <-u|-g> <uname>|<uid>|<gname>|<gid> [--block-softlimit <block-softlimit>] [--block-hardlimit <block-hardlimit>] [--inode-softlimit <inode-softlimit>] [--inode-hardlimit <inode-hardlimit>] <filesystem></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305649" xreflabel=""/>Sets file system quotas for users or groups. Limits can be specified with --{block|inode}-{softlimit|hardlimit} or their short equivalents -b, -B, -i, -I. Users can set 1, 2, 3 or 4 limits.<footnote><para><anchor xml:id="dbdoclet.50438206_pgfId-1305652" xreflabel=""/>The old setquota interface is supported, but it may be removed in a future Lustre release.</para></footnote> Also, limits can be specified with special suffixes, -b, -k, -m, -g, -t, and -p to indicate units of 1, 2^10, 2^20, 2^30, 2^40 and 2^50, respectively. By default, the block limits unit is 1 kilobyte (1,024), and block limits are always kilobyte-grained (even if specified in bytes). See <xref linkend="dbdoclet.50438206_11903"/>.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">setquota <-u|-g> <uname>|<uid>|<gname>|<gid> [--block-softlimit <block-softlimit>] [--block-hardlimit <block-hardlimit>] [--inode-softlimit <inode-softlimit>] [--inode-hardlimit <inode-hardlimit>] <filesystem></emphasis></para></entry>
+ <entry><para> Sets file system quotas for users or groups. Limits can be specified with --{block|inode}-{softlimit|hardlimit} or their short equivalents -b, -B, -i, -I. Users can set 1, 2, 3 or 4 limits.<footnote><para>The old setquota interface is supported, but it may be removed in a future Lustre release.</para></footnote> Also, limits can be specified with special suffixes, -b, -k, -m, -g, -t, and -p to indicate units of 1, 2^10, 2^20, 2^30, 2^40 and 2^50, respectively. By default, the block limits unit is 1 kilobyte (1,024), and block limits are always kilobyte-grained (even if specified in bytes). See <xref linkend="dbdoclet.50438206_11903"/>.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305659" xreflabel=""/><emphasis role="bold">setquota -t <-u|-g></emphasis></para><para><anchor xml:id="dbdoclet.50438206_pgfId-1305660" xreflabel=""/><emphasis role="bold">[--block-grace <block-grace>]</emphasis></para><para><anchor xml:id="dbdoclet.50438206_pgfId-1305661" xreflabel=""/><emphasis role="bold">[--inode-grace <inode-grace>] <filesystem></emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305665" xreflabel=""/>Sets the file system quota grace times for users or groups. Grace time is specified in 'XXwXXdXXhXXmXXs†format or as an integer seconds value. See <xref linkend="dbdoclet.50438206_11903"/>.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">setquota -t <-u|-g></emphasis></para><para><emphasis role="bold">[--block-grace <block-grace>]</emphasis></para><para><emphasis role="bold">[--inode-grace <inode-grace>] <filesystem></emphasis></para></entry>
+ <entry><para> Sets the file system quota grace times for users or groups. Grace time is specified in 'XXwXXdXXhXXmXXs†format or as an integer seconds value. See <xref linkend="dbdoclet.50438206_11903"/>.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305672" xreflabel=""/><emphasis role="bold">help</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305676" xreflabel=""/>Provides brief help on various lfs arguments.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">help</emphasis></para></entry>
+ <entry><para> Provides brief help on various lfs arguments.</para></entry>
</row>
<row>
- <entry nameend="c2" namest="c1"><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305680" xreflabel=""/><emphasis role="bold">exit/quit</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305684" xreflabel=""/>Quits the interactive lfs session.</para></entry>
+ <entry nameend="c2" namest="c1"><para> <emphasis role="bold">exit/quit</emphasis></para></entry>
+ <entry><para> Quits the interactive lfs session.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_
pgfId-1306365" xreflabel=""/><anchor xml:id="dbdoclet.50438206_11903" xreflabel=""/><anchor xml:id="dbdoclet.50438206_74082" xreflabel=""/>Examples</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306366" xreflabel=""/>Creates a file striped on two OSTs with 128 KB on each stripe.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306367" xreflabel=""/>$ lfs setstripe -s 128k -c 2 /mnt/lustre/file1
+ <title><anchor xml:id="dbdoclet.50438206_11903" xreflabel=""/><anchor xml:id="dbdoclet.50438206_74082" xreflabel=""/>Examples</title>
+ <para>Creates a file striped on two OSTs with 128 KB on each stripe.</para>
+ <screen>$ lfs setstripe -s 128k -c 2 /mnt/lustre/file1
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306369" xreflabel=""/>Deletes a default stripe pattern on a given directory. New files use the default striping pattern.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306370" xreflabel=""/>$ lfs setstripe -d /mnt/lustre/dir
+ <para>Deletes a default stripe pattern on a given directory. New files use the default striping pattern.</para>
+ <screen>$ lfs setstripe -d /mnt/lustre/dir
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306372" xreflabel=""/>Lists the detailed object allocation of a given file.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306373" xreflabel=""/>$ lfs getstripe -v /mnt/lustre/file1
+ <para>Lists the detailed object allocation of a given file.</para>
+ <screen>$ lfs getstripe -v /mnt/lustre/file1
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306375" xreflabel=""/>Efficiently lists all files in a given directory and its subdirectories.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306376" xreflabel=""/>$ lfs find /mnt/lustre
+ <para>Efficiently lists all files in a given directory and its subdirectories.</para>
+ <screen>$ lfs find /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306378" xreflabel=""/>Recursively lists all regular files in a given directory more than 30 days old.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306379" xreflabel=""/>$ lfs find /mnt/lustre -mtime +30 -type f -print
+ <para>Recursively lists all regular files in a given directory more than 30 days old.</para>
+ <screen>$ lfs find /mnt/lustre -mtime +30 -type f -print
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306381" xreflabel=""/>Recursively lists all files in a given directory that have objects on OST2-UUID. The lfs check servers command checks the status of all servers (MDT and OSTs).</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306382" xreflabel=""/>$ lfs find --obd OST2-UUID /mnt/lustre/
+ <para>Recursively lists all files in a given directory that have objects on OST2-UUID. The lfs check servers command checks the status of all servers (MDT and OSTs).</para>
+ <screen>$ lfs find --obd OST2-UUID /mnt/lustre/
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306384" xreflabel=""/>Lists all OSTs in the file system.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306385" xreflabel=""/>$ lfs osts
+ <para>Lists all OSTs in the file system.</para>
+ <screen>$ lfs osts
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306387" xreflabel=""/>Lists space usage per OST and MDT in human-readable format.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306388" xreflabel=""/>$ lfs df -h
+ <para>Lists space usage per OST and MDT in human-readable format.</para>
+ <screen>$ lfs df -h
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306390" xreflabel=""/>Lists inode usage per OST and MDT.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306391" xreflabel=""/>$ lfs df -i
+ <para>Lists inode usage per OST and MDT.</para>
+ <screen>$ lfs df -i
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306393" xreflabel=""/>List space or inode usage for a specific OST pool.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306394" xreflabel=""/>$ lfs df --pool <filesystem>[.<pool>] | <pathname>
+ <para>List space or inode usage for a specific OST pool.</para>
+ <screen>$ lfs df --pool <filesystem>[.<pool>] | <pathname>
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306396" xreflabel=""/>List quotas of user 'bob'.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306397" xreflabel=""/>$ lfs quota -u bob /mnt/lustre
+ <para>List quotas of user 'bob'.</para>
+ <screen>$ lfs quota -u bob /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306399" xreflabel=""/>Show grace times for user quotas on /mnt/lustre.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306400" xreflabel=""/>$ lfs quota -t -u /mnt/lustre
+ <para>Show grace times for user quotas on /mnt/lustre.</para>
+ <screen>$ lfs quota -t -u /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306402" xreflabel=""/>Changes file owner and group.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306403" xreflabel=""/>$ lfs quotachown -i /mnt/lustre
+ <para>Changes file owner and group.</para>
+ <screen>$ lfs quotachown -i /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306405" xreflabel=""/>Checks quotas for user and group. Turns on quotas after making the check.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306406" xreflabel=""/>$ lfs quotacheck -ug /mnt/lustre
+ <para>Checks quotas for user and group. Turns on quotas after making the check.</para>
+ <screen>$ lfs quotacheck -ug /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306408" xreflabel=""/>Turns on quotas of user and group.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306409" xreflabel=""/>$ lfs quotaon -ug /mnt/lustre
+ <para>Turns on quotas of user and group.</para>
+ <screen>$ lfs quotaon -ug /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306411" xreflabel=""/>Turns off quotas of user and group.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306412" xreflabel=""/>$ lfs quotaoff -ug /mnt/lustre
+ <para>Turns off quotas of user and group.</para>
+ <screen>$ lfs quotaoff -ug /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306414" xreflabel=""/>Sets quotas of user 'bob', with a 1 GB block quota hardlimit and a 2 GB block quota softlimit.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306415" xreflabel=""/>$ lfs setquota -u bob --block-softlimit 2000000 --block-hardlimit 1000000 /\
+ <para>Sets quotas of user 'bob', with a 1 GB block quota hardlimit and a 2 GB block quota softlimit.</para>
+ <screen>$ lfs setquota -u bob --block-softlimit 2000000 --block-hardlimit 1000000 /\
mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306417" xreflabel=""/>Sets grace times for user quotas: 1000 seconds for block quotas, 1 week and 4 days for inode quotas.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306418" xreflabel=""/>$ lfs setquota -t -u --block-grace 1000 --inode-grace 1w4d /mnt/lustre
+ <para>Sets grace times for user quotas: 1000 seconds for block quotas, 1 week and 4 days for inode quotas.</para>
+ <screen>$ lfs setquota -t -u --block-grace 1000 --inode-grace 1w4d /mnt/lustre
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306420" xreflabel=""/>Checks the status of all servers (MDT, OST)</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306421" xreflabel=""/>$ lfs check servers
+ <para>Checks the status of all servers (MDT, OST)</para>
+ <screen>$ lfs check servers
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306423" xreflabel=""/>Creates a file striped on two OSTs from the pool my_pool</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306424" xreflabel=""/>$ lfs setstripe --pool my_pool -c 2 /mnt/lustre/file
+ <para>Creates a file striped on two OSTs from the pool my_pool</para>
+ <screen>$ lfs setstripe --pool my_pool -c 2 /mnt/lustre/file
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306426" xreflabel=""/>Lists the pools defined for the mounted Lustre file system /mnt/lustre</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306427" xreflabel=""/>$ lfs poollist /mnt/lustre/
+ <para>Lists the pools defined for the mounted Lustre file system /mnt/lustre</para>
+ <screen>$ lfs poollist /mnt/lustre/
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306429" xreflabel=""/>Lists the OSTs which are members of the pool my_pool in file system my_fs</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306430" xreflabel=""/>$ lfs poollist my_fs.my_pool
+ <para>Lists the OSTs which are members of the pool my_pool in file system my_fs</para>
+ <screen>$ lfs poollist my_fs.my_pool
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306432" xreflabel=""/>Finds all directories/files associated with poolA.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306433" xreflabel=""/>$ lfs find /mnt/lustre --pool poolA
+ <para>Finds all directories/files associated with poolA.</para>
+ <screen>$ lfs find /mnt/lustre --pool poolA
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306435" xreflabel=""/>Finds all directories/files not associated with a pool.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306436" xreflabel=""/>$ lfs find /mnt//lustre --pool ""
+ <para>Finds all directories/files not associated with a pool.</para>
+ <screen>$ lfs find /mnt//lustre --pool ""
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306438" xreflabel=""/>Finds all directories/files associated with pool.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306439" xreflabel=""/>$ lfs find /mnt/lustre ! --pool ""
+ <para>Finds all directories/files associated with pool.</para>
+ <screen>$ lfs find /mnt/lustre ! --pool ""
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306441" xreflabel=""/>Associates a directory with the pool my_pool, so all new files and directories are created in the pool.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306442" xreflabel=""/>$ lfs setstripe --pool my_pool /mnt/lustre/dir
+ <para>Associates a directory with the pool my_pool, so all new files and directories are created in the pool.</para>
+ <screen>$ lfs setstripe --pool my_pool /mnt/lustre/dir
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305765" xreflabel=""/>See Also</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305769" xreflabel=""/><link xl:href="SystemConfigurationUtilities.html#50438219_38274">lctl</link></para>
+ <title>See Also</title>
+ <para><link xl:href="SystemConfigurationUtilities.html#50438219_38274">lctl</link></para>
</section>
</section>
<section xml:id="dbdoclet.50438206_42260">
<title>32.2 lfs_migrate</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305772" xreflabel=""/>The lfs_migrate utility is a simple tool to migrate files between Lustre OSTs.</para>
+ <para>The lfs_migrate utility is a simple tool to migrate files between Lustre OSTs.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305773" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1305774" xreflabel=""/>lfs_migrate [-c|-s] [-h] [-l] [-n] [-y] [file|directory ...]
+ <title>Synopsis</title>
+ <screen>lfs_migrate [-c|-s] [-h] [-l] [-n] [-y] [file|directory ...]
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305775" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305776" xreflabel=""/>The lfs_migrate utility is a simple tool to assist migration of files between Lustre OSTs. The utility copies each specified file to a new file, verifies the file contents have not changed, and then renames the new file to the original filename. This allows balanced space usage between OSTs, moving files of OSTs that are starting to show hardware problems (though are still functional) or OSTs that will be discontinued.</para>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305777" xreflabel=""/>Because lfs_migrate is not closely integrated with the MDS, it cannot determine whether a file is currently open and/or in-use by other applications or nodes. This makes it UNSAFE for use on files that might be modified by other applications, since the migrated file is only a copy of the current file. This results in the old file becoming an open-unlinked file and any modifications to that file are lost.</para>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305778" xreflabel=""/>Files to be migrated can be specified as command-line arguments. If a directory is specified on the command-line then all files within the directory are migrated. If no files are specified on the command-line, then a list of files is read from the standard input, making lfs_migrate suitable for use with lfs find to locate files on specific OSTs and/or matching other file attributes.</para>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305779" xreflabel=""/>The current file allocation policies on the MDS dictate where the new files are placed, taking into account whether specific OSTs have been disabled on the MDS via lctl (preventing new files from being allocated there), whether some OSTs are overly full (reducing the number of files placed on those OSTs), or if there is a specific default file striping for the target directory (potentially changing the stripe count, stripe size, OST pool, or OST index of a new file).</para>
+ <title>Description</title>
+ <para>The lfs_migrate utility is a simple tool to assist migration of files between Lustre OSTs. The utility copies each specified file to a new file, verifies the file contents have not changed, and then renames the new file to the original filename. This allows balanced space usage between OSTs, moving files of OSTs that are starting to show hardware problems (though are still functional) or OSTs that will be discontinued.</para>
+ <para>Because lfs_migrate is not closely integrated with the MDS, it cannot determine whether a file is currently open and/or in-use by other applications or nodes. This makes it UNSAFE for use on files that might be modified by other applications, since the migrated file is only a copy of the current file. This results in the old file becoming an open-unlinked file and any modifications to that file are lost.</para>
+ <para>Files to be migrated can be specified as command-line arguments. If a directory is specified on the command-line then all files within the directory are migrated. If no files are specified on the command-line, then a list of files is read from the standard input, making lfs_migrate suitable for use with lfs find to locate files on specific OSTs and/or matching other file attributes.</para>
+ <para>The current file allocation policies on the MDS dictate where the new files are placed, taking into account whether specific OSTs have been disabled on the MDS via lctl (preventing new files from being allocated there), whether some OSTs are overly full (reducing the number of files placed on those OSTs), or if there is a specific default file striping for the target directory (potentially changing the stripe count, stripe size, OST pool, or OST index of a new file).</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305780" xreflabel=""/>Options</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305830" xreflabel=""/>Options supporting lfs_migrate are described below.</para>
+ <title>Options</title>
+ <para>Options supporting lfs_migrate are described below.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438206_pgfId-1305783" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438206_pgfId-1305785" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305789" xreflabel=""/><emphasis role="bold">-c</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305791" xreflabel=""/>Compares file data after migrate (default value, use <emphasis role="bold">-s</emphasis> to disable).</para></entry>
+ <entry><para> <emphasis role="bold">-c</emphasis></para></entry>
+ <entry><para> Compares file data after migrate (default value, use <emphasis role="bold">-s</emphasis> to disable).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305795" xreflabel=""/><emphasis role="bold">-s</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305797" xreflabel=""/>Skips file data comparison after migrate (use <emphasis role="bold">-c</emphasis> to enable).</para></entry>
+ <entry><para> <emphasis role="bold">-s</emphasis></para></entry>
+ <entry><para> Skips file data comparison after migrate (use <emphasis role="bold">-c</emphasis> to enable).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305801" xreflabel=""/><emphasis role="bold">-h</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305803" xreflabel=""/>Displays help information.</para></entry>
+ <entry><para> <emphasis role="bold">-h</emphasis></para></entry>
+ <entry><para> Displays help information.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305807" xreflabel=""/><emphasis role="bold">-l</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305809" xreflabel=""/>Migrates files with hard links (skips, by default). Files with multiple hard links are split into multiple separate files by lfs_migrate, so they are skipped, by default, to avoid breaking the hard links.</para></entry>
+ <entry><para> <emphasis role="bold">-l</emphasis></para></entry>
+ <entry><para> Migrates files with hard links (skips, by default). Files with multiple hard links are split into multiple separate files by lfs_migrate, so they are skipped, by default, to avoid breaking the hard links.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305813" xreflabel=""/><emphasis role="bold">-n</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305815" xreflabel=""/>Only prints the names of files to be migrated.</para></entry>
+ <entry><para> <emphasis role="bold">-n</emphasis></para></entry>
+ <entry><para> Only prints the names of files to be migrated.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305819" xreflabel=""/><emphasis role="bold">-q</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305821" xreflabel=""/>Runs quietly (does not print filenames or status).</para></entry>
+ <entry><para> <emphasis role="bold">-q</emphasis></para></entry>
+ <entry><para> Runs quietly (does not print filenames or status).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305825" xreflabel=""/><emphasis role="bold">--y</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305827" xreflabel=""/>Answers 'y' to usage warning without prompting (for scripts).</para></entry>
+ <entry><para> <emphasis role="bold">--y</emphasis></para></entry>
+ <entry><para> Answers 'y' to usage warning without prompting (for scripts).</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305831" xreflabel=""/>Examples</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305833" xreflabel=""/>Rebalances all files in /mnt/lustre/dir.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1305832" xreflabel=""/>$ lfs_migrate /mnt/lustre/file
+ <title>Examples</title>
+ <para>Rebalances all files in /mnt/lustre/dir.</para>
+ <screen>$ lfs_migrate /mnt/lustre/file
</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305834" xreflabel=""/> </para>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305836" xreflabel=""/>Migrates files in /test filesystem on OST004 larger than 4 GB in size.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1305835" xreflabel=""/>$ lfs find /test -obd test-OST004 -size +4G | lfs_migrate -y
+ <para> </para>
+ <para>Migrates files in /test filesystem on OST004 larger than 4 GB in size.</para>
+ <screen>$ lfs find /test -obd test-OST004 -size +4G | lfs_migrate -y
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305837" xreflabel=""/>See Also</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305841" xreflabel=""/><xref linkend="dbdoclet.50438206_94597"/></para>
+ <title>See Also</title>
+ <para><xref linkend="dbdoclet.50438206_94597"/></para>
</section>
</section>
<section xml:id="dbdoclet.50438206_91700">
<title>32.3 lf<anchor xml:id="dbdoclet.50438206_marker-1305843" xreflabel=""/>sck</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305845" xreflabel=""/>Lfsck ensures that objects are not referenced by multiple MDS files, that there are no orphan objects on the OSTs (objects that do not have any file on the MDS which references them), and that all of the objects referenced by the MDS exist. Under normal circumstances, Lustre maintains such coherency by distributed logging mechanisms, but under exceptional circumstances that may fail (e.g. disk failure, file system corruption leading to e2fsck repair). To avoid lengthy downtime, you can also run lfsck once Lustre is already started.</para>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305846" xreflabel=""/>The e2fsck utility is run on each of the local MDS and OST device file systems and verifies that the underlying ldiskfs is consistent. After e2fsck is run, lfsck does distributed coherency checking for the Lustre file system. In most cases, e2fsck is sufficient to repair any file system issues and lfsck is not required.</para>
+ <para>Lfsck ensures that objects are not referenced by multiple MDS files, that there are no orphan objects on the OSTs (objects that do not have any file on the MDS which references them), and that all of the objects referenced by the MDS exist. Under normal circumstances, Lustre maintains such coherency by distributed logging mechanisms, but under exceptional circumstances that may fail (e.g. disk failure, file system corruption leading to e2fsck repair). To avoid lengthy downtime, you can also run lfsck once Lustre is already started.</para>
+ <para>The e2fsck utility is run on each of the local MDS and OST device file systems and verifies that the underlying ldiskfs is consistent. After e2fsck is run, lfsck does distributed coherency checking for the Lustre file system. In most cases, e2fsck is sufficient to repair any file system issues and lfsck is not required.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305847" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1305848" xreflabel=""/>lfsck [-c|--create] [-d|--delete] [-f|--force] [-h|--help] [-l|--lostfound]\
+ <title>Synopsis</title>
+ <screen>lfsck [-c|--create] [-d|--delete] [-f|--force] [-h|--help] [-l|--lostfound]\
[-n|--nofix] [-v|--verbose] --mdsdb mds_database_file --ostdb ost1_databas\
e_file [ost2_database_file...] <filesystem>
</screen>
<note><para>For lfsck, database filenames must be provided as absolute pathnames. Relative paths do not work, the databases cannot be properly opened.</para></note>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305851" xreflabel=""/>Options</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305909" xreflabel=""/>The options and descriptions for the lfsck command are listed below.</para>
+ <title>Options</title>
+ <para>The options and descriptions for the lfsck command are listed below.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438206_pgfId-1305854" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438206_pgfId-1305858" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305860" xreflabel=""/><emphasis role="bold">-c</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305864" xreflabel=""/>Creates (empty) missing OST objects referenced by MDS inodes.</para></entry>
+ <entry><para> <emphasis role="bold">-c</emphasis></para></entry>
+ <entry><para> Creates (empty) missing OST objects referenced by MDS inodes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305866" xreflabel=""/><emphasis role="bold">-d</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305870" xreflabel=""/>Deletes orphaned objects from the file system. Since objects on the OST are often only one of several stripes of a file, it can be difficult to compile multiple objects together in a single, usable file.</para></entry>
+ <entry><para> <emphasis role="bold">-d</emphasis></para></entry>
+ <entry><para> Deletes orphaned objects from the file system. Since objects on the OST are often only one of several stripes of a file, it can be difficult to compile multiple objects together in a single, usable file.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305872" xreflabel=""/><emphasis role="bold">-h</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305876" xreflabel=""/>Prints a brief help message.</para></entry>
+ <entry><para> <emphasis role="bold">-h</emphasis></para></entry>
+ <entry><para> Prints a brief help message.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305878" xreflabel=""/><emphasis role="bold">-l</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305882" xreflabel=""/> Puts orphaned objects into a lost+found directory in the root of the file system.</para></entry>
+ <entry><para> <emphasis role="bold">-l</emphasis></para></entry>
+ <entry><para> Puts orphaned objects into a lost+found directory in the root of the file system.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305884" xreflabel=""/><emphasis role="bold">-n</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305888" xreflabel=""/>Does not repair the file system, just performs a read-only check (default).</para></entry>
+ <entry><para> <emphasis role="bold">-n</emphasis></para></entry>
+ <entry><para> Does not repair the file system, just performs a read-only check (default).</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305890" xreflabel=""/><emphasis role="bold">-v</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305894" xreflabel=""/>Verbose operation - more verbosity by specifying the option multiple times.</para></entry>
+ <entry><para> <emphasis role="bold">-v</emphasis></para></entry>
+ <entry><para> Verbose operation - more verbosity by specifying the option multiple times.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305896" xreflabel=""/><emphasis role="bold">--mdsdb</emphasis></para><para><anchor xml:id="dbdoclet.50438206_pgfId-1305897" xreflabel=""/><emphasis role="bold">mds_database_file</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305901" xreflabel=""/> MDS database file created by running e2fsck --mdsdb mds_database_file <device> on the MDS backing device. This is required.</para></entry>
+ <entry><para> <emphasis role="bold">--mdsdb</emphasis></para><para><emphasis role="bold">mds_database_file</emphasis></para></entry>
+ <entry><para> MDS database file created by running e2fsck --mdsdb mds_database_file <device> on the MDS backing device. This is required.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305903" xreflabel=""/><emphasis role="bold">--ostdb ost1_database_file</emphasis></para><para><anchor xml:id="dbdoclet.50438206_pgfId-1305904" xreflabel=""/><emphasis role="bold">[ost2_database_file...]</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305908" xreflabel=""/> OST database files created by running e2fsck --ostdb ost_database_file <device> on each of the OST backing devices. These are required unless an OST is unavailable, in which case all objects thereon are considered missing.</para></entry>
+ <entry><para> <emphasis role="bold">--ostdb ost1_database_file</emphasis></para><para><emphasis role="bold">[ost2_database_file...]</emphasis></para></entry>
+ <entry><para> OST database files created by running e2fsck --ostdb ost_database_file <device> on each of the OST backing devices. These are required unless an OST is unavailable, in which case all objects thereon are considered missing.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305910" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305911" xreflabel=""/>The lfsck utility is used to check and repair the distributed coherency of a Lustre file system. If an MDS or an OST becomes corrupt, run a distributed check on the file system to determine what sort of problems exist. Use lfsck to correct any defects found.</para>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305914" xreflabel=""/>For more information on using e2fsck and lfsck, including examples, see <xref linkend="dbdoclet.50438268_83826"/> (Commit on Share). For information on resolving orphaned objects, see <xref linkend="dbdoclet.50438225_13916"/> (Working with Orphaned Objects).</para>
+ <title>Description</title>
+ <para>The lfsck utility is used to check and repair the distributed coherency of a Lustre file system. If an MDS or an OST becomes corrupt, run a distributed check on the file system to determine what sort of problems exist. Use lfsck to correct any defects found.</para>
+ <para>For more information on using e2fsck and lfsck, including examples, see <xref linkend="dbdoclet.50438268_83826"/> (Commit on Share). For information on resolving orphaned objects, see <xref linkend="dbdoclet.50438225_13916"/> (Working with Orphaned Objects).</para>
</section>
</section>
<section xml:id="dbdoclet.50438206_75125">
<title>32.4 File<anchor xml:id="dbdoclet.50438206_marker-1305920" xreflabel=""/>frag</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305922" xreflabel=""/>The e2fsprogs package contains the filefrag tool which reports the extent of file fragmentation.</para>
+ <para>The e2fsprogs package contains the filefrag tool which reports the extent of file fragmentation.</para>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305923" xreflabel=""/>Synopsis</title>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1305924" xreflabel=""/>filefrag [ -belsv ] [ files... ]
+ <title>Synopsis</title>
+ <screen>filefrag [ -belsv ] [ files... ]
</screen>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305925" xreflabel=""/>Description</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305926" xreflabel=""/>The filefrag utility reports the extent of fragmentation in a given file. Initially, filefrag attempts to obtain extent information using FIEMAP ioctl, which is efficient and fast. If FIEMAP is not supported, then filefrag uses FIBMAP.</para>
+ <title>Description</title>
+ <para>The filefrag utility reports the extent of fragmentation in a given file. Initially, filefrag attempts to obtain extent information using FIEMAP ioctl, which is efficient and fast. If FIEMAP is not supported, then filefrag uses FIBMAP.</para>
<note><para>Lustre only supports FIEMAP ioctl. FIBMAP ioctl is not supported.</para></note>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305931" xreflabel=""/>In default mode <footnote><para><anchor xml:id="dbdoclet.50438206_pgfId-1305930" xreflabel=""/>The default mode is faster than the verbose/extent mode.</para></footnote>, filefrag returns the number of physically discontiguous extents in the file. In extent or verbose mode, each extent is printed with details. For Lustre, the extents are printed in device offset order, not logical offset order.</para>
+ <para>In default mode <footnote><para>The default mode is faster than the verbose/extent mode.</para></footnote>, filefrag returns the number of physically discontiguous extents in the file. In extent or verbose mode, each extent is printed with details. For Lustre, the extents are printed in device offset order, not logical offset order.</para>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305932" xreflabel=""/>Options</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305971" xreflabel=""/>The options and descriptions for the filefrag utility are listed below.</para>
+ <title>Options</title>
+ <para>The options and descriptions for the filefrag utility are listed below.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438206_pgfId-1305935" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438206_pgfId-1305937" xreflabel=""/> <anchor xml:id="dbdoclet.50438206_pgfId-1305938" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Option</emphasis></para></entry>
+ <entry><para><emphasis role="bold"> Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305942" xreflabel=""/><emphasis role="bold">-b</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305944" xreflabel=""/>Uses the 1024-byte blocksize for the output. By default, this blocksize is used by Lustre, since OSTs may use different block sizes.</para></entry>
+ <entry><para> <emphasis role="bold">-b</emphasis></para></entry>
+ <entry><para> Uses the 1024-byte blocksize for the output. By default, this blocksize is used by Lustre, since OSTs may use different block sizes.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305948" xreflabel=""/><emphasis role="bold">-e</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305950" xreflabel=""/> Uses the extent mode when printing the output.</para></entry>
+ <entry><para> <emphasis role="bold">-e</emphasis></para></entry>
+ <entry><para> Uses the extent mode when printing the output.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305954" xreflabel=""/><emphasis role="bold">-l</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305956" xreflabel=""/>Displays extents in LUN offset order.</para></entry>
+ <entry><para> <emphasis role="bold">-l</emphasis></para></entry>
+ <entry><para> Displays extents in LUN offset order.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305960" xreflabel=""/><emphasis role="bold">-s</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305962" xreflabel=""/>Synchronizes the file before requesting the mapping.</para></entry>
+ <entry><para> <emphasis role="bold">-s</emphasis></para></entry>
+ <entry><para> Synchronizes the file before requesting the mapping.</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305966" xreflabel=""/><emphasis role="bold">-v</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1305968" xreflabel=""/>Uses the verbose mode when checking file fragmentation.</para></entry>
+ <entry><para> <emphasis role="bold">-v</emphasis></para></entry>
+ <entry><para> Uses the verbose mode when checking file fragmentation.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section remap="h5">
- <title><anchor xml:id="dbdoclet.50438206_pgfId-1305972" xreflabel=""/>Examples</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305973" xreflabel=""/>Lists default output.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1305974" xreflabel=""/>$ filefrag /mnt/lustre/foo
-<anchor xml:id="dbdoclet.50438206_pgfId-1305975" xreflabel=""/>/mnt/lustre/foo: 6 extents found
-</screen>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305976" xreflabel=""/>Lists verbose output in extent format.</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1305977" xreflabel=""/>$ filefrag -ve /mnt/lustre/foo
-<anchor xml:id="dbdoclet.50438206_pgfId-1305978" xreflabel=""/>Checking /mnt/lustre/foo
-<anchor xml:id="dbdoclet.50438206_pgfId-1305979" xreflabel=""/>Filesystem type is: bd00bd0
-<anchor xml:id="dbdoclet.50438206_pgfId-1305980" xreflabel=""/>Filesystem cylinder groups is approximately 5
-<anchor xml:id="dbdoclet.50438206_pgfId-1305981" xreflabel=""/>File size of /mnt/lustre/foo is 157286400 (153600 blocks)
-<anchor xml:id="dbdoclet.50438206_pgfId-1305982" xreflabel=""/>ext: device_logical: start..end physical: \
+ <title>Examples</title>
+ <para>Lists default output.</para>
+ <screen>$ filefrag /mnt/lustre/foo
+/mnt/lustre/foo: 6 extents found
+</screen>
+ <para>Lists verbose output in extent format.</para>
+ <screen>$ filefrag -ve /mnt/lustre/foo
+Checking /mnt/lustre/foo
+Filesystem type is: bd00bd0
+Filesystem cylinder groups is approximately 5
+File size of /mnt/lustre/foo is 157286400 (153600 blocks)
+ext: device_logical: start..end physical: \
start..end: length: device: flags:
-<anchor xml:id="dbdoclet.50438206_pgfId-1305983" xreflabel=""/>0: 0.. 49151: 212992.. \
+0: 0.. 49151: 212992.. \
262144: 49152: 0: remote
-<anchor xml:id="dbdoclet.50438206_pgfId-1305984" xreflabel=""/>1: 49152.. 73727: 270336.. \
+1: 49152.. 73727: 270336.. \
294912: 24576: 0: remote
-<anchor xml:id="dbdoclet.50438206_pgfId-1305985" xreflabel=""/>2: 73728.. 76799: 24576.. \
+2: 73728.. 76799: 24576.. \
27648: 3072: 0: remote
-<anchor xml:id="dbdoclet.50438206_pgfId-1305986" xreflabel=""/>3: 0.. 57343: 196608.. \
+3: 0.. 57343: 196608.. \
253952: 57344: 1: remote
-<anchor xml:id="dbdoclet.50438206_pgfId-1305987" xreflabel=""/>4: 57344.. 65535: 139264.. \
+4: 57344.. 65535: 139264.. \
147456: 8192: 1: remote
-<anchor xml:id="dbdoclet.50438206_pgfId-1305988" xreflabel=""/>5: 65536.. 76799: 163840.. \
+5: 65536.. 76799: 163840.. \
175104: 11264: 1: remote
-<anchor xml:id="dbdoclet.50438206_pgfId-1305989" xreflabel=""/>/mnt/lustre/foo: 6 extents found
+/mnt/lustre/foo: 6 extents found
</screen>
</section>
</section>
<section xml:id="dbdoclet.50438206_86244">
<title>32.5 Mou<anchor xml:id="dbdoclet.50438206_marker-1305992" xreflabel=""/>nt</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1305994" xreflabel=""/>Lustre uses the standard mount(8) Linux command. When mounting a Lustre file system, mount(8) executes the /sbin/mount.lustre command to complete the mount. The mount command supports these Lustre-specific options:</para>
+ <para>Lustre uses the standard mount(8) Linux command. When mounting a Lustre file system, mount(8) executes the /sbin/mount.lustre command to complete the mount. The mount command supports these Lustre-specific options:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438206_pgfId-1305997" xreflabel=""/>Server options</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438206_pgfId-1305999" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Server options</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306001" xreflabel=""/>abort_recov</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306003" xreflabel=""/>Aborts recovery when starting a target</para></entry>
+ <entry><para> abort_recov</para></entry>
+ <entry><para> Aborts recovery when starting a target</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306005" xreflabel=""/>nosvc</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306007" xreflabel=""/>Starts only MGS/MGC servers</para></entry>
+ <entry><para> nosvc</para></entry>
+ <entry><para> Starts only MGS/MGC servers</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306009" xreflabel=""/>exclude</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306011" xreflabel=""/>Starts with a dead OST</para></entry>
+ <entry><para> exclude</para></entry>
+ <entry><para> Starts with a dead OST</para></entry>
</row>
</tbody>
</tgroup>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438206_pgfId-1306014" xreflabel=""/>Client options</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438206_pgfId-1306016" xreflabel=""/>Description</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Client options</emphasis></para></entry>
+ <entry><para><emphasis role="bold">Description</emphasis></para></entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306018" xreflabel=""/>flock</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306020" xreflabel=""/>Enables/disables flock support</para></entry>
+ <entry><para> flock</para></entry>
+ <entry><para> Enables/disables flock support</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306022" xreflabel=""/>user_xattr/nouser_xattr</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306024" xreflabel=""/>Enables/disables user-extended attributes</para></entry>
+ <entry><para> user_xattr/nouser_xattr</para></entry>
+ <entry><para> Enables/disables user-extended attributes</para></entry>
</row>
<row>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306026" xreflabel=""/>retry=</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438206_pgfId-1306028" xreflabel=""/>Number of times a client will retry to mount the file system</para></entry>
+ <entry><para> retry=</para></entry>
+ <entry><para> Number of times a client will retry to mount the file system</para></entry>
</row>
</tbody>
</tgroup>
</section>
<section xml:id="dbdoclet.50438206_56217">
<title>32.6 Handling <anchor xml:id="dbdoclet.50438206_marker-1306030" xreflabel=""/>Timeouts</title>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306032" xreflabel=""/>Timeouts are the most common cause of hung applications. After a timeout involving an MDS or failover OST, applications attempting to access the disconnected resource wait until the connection gets established.</para>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306033" xreflabel=""/>When a client performs any remote operation, it gives the server a reasonable amount of time to respond. If a server does not reply either due to a down network, hung server, or any other reason, a timeout occurs which requires a recovery.</para>
- <para><anchor xml:id="dbdoclet.50438206_pgfId-1306034" xreflabel=""/>If a timeout occurs, a message (similar to this one), appears on the console of the client, and in /var/log/messages:</para>
- <screen><anchor xml:id="dbdoclet.50438206_pgfId-1306035" xreflabel=""/>LustreError: 26597:(client.c:810:ptlrpc_expire_one_request()) @@@ timeout
-<anchor xml:id="dbdoclet.50438206_pgfId-1306036" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438206_pgfId-1306037" xreflabel=""/>req@a2d45200 x5886/t0 o38->mds_svc_UUID@NID_mds_UUID:12 lens 168/64 ref 1 fl
-<anchor xml:id="dbdoclet.50438206_pgfId-1306038" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438206_pgfId-1306039" xreflabel=""/>RPC:/0/0 rc 0
-<anchor xml:id="dbdoclet.50438206_pgfId-1293091" xreflabel=""/>
+ <para>Timeouts are the most common cause of hung applications. After a timeout involving an MDS or failover OST, applications attempting to access the disconnected resource wait until the connection gets established.</para>
+ <para>When a client performs any remote operation, it gives the server a reasonable amount of time to respond. If a server does not reply either due to a down network, hung server, or any other reason, a timeout occurs which requires a recovery.</para>
+ <para>If a timeout occurs, a message (similar to this one), appears on the console of the client, and in /var/log/messages:</para>
+ <screen>LustreError: 26597:(client.c:810:ptlrpc_expire_one_request()) @@@ timeout
+
+req@a2d45200 x5886/t0 o38->mds_svc_UUID@NID_mds_UUID:12 lens 168/64 ref 1 fl
+
+RPC:/0/0 rc 0
+
</screen>
</section>
</chapter>
<holder>Oracle and/or its affiliates.</holder>
<holder>Whamcloud, Inc.</holder>
<info>
- <para><anchor xml:id="dbdoclet.50438245_pgfId-999159" xreflabel=""/>This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.</para>
- <para><anchor xml:id="dbdoclet.50438245_pgfId-999160" xreflabel=""/>The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.</para>
- <para><anchor xml:id="dbdoclet.50438245_pgfId-999161" xreflabel=""/>If this is software or related software documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable:</para>
- <para><anchor xml:id="dbdoclet.50438245_pgfId-999162" xreflabel=""/>U.S. GOVERNMENT RIGHTS. Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065.</para>
- <para><anchor xml:id="dbdoclet.50438245_pgfId-999163" xreflabel=""/>This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications which may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.</para>
- <para><anchor xml:id="dbdoclet.50438245_pgfId-999164" xreflabel=""/>Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.</para>
- <para><anchor xml:id="dbdoclet.50438245_pgfId-999165" xreflabel=""/>AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open Company, Ltd.</para>
- <para><anchor xml:id="dbdoclet.50438245_pgfId-999166" xreflabel=""/>This software or hardware and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services.</para>
+ <para>This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.</para>
+ <para>The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.</para>
+ <para>If this is software or related software documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable:</para>
+ <para>U.S. GOVERNMENT RIGHTS. Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065.</para>
+ <para>This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications which may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.</para>
+ <para>Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.</para>
+ <para>AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open Company, Ltd.</para>
+ <para>This software or hardware and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services.</para>
<para>Ce logiciel ou matériel et la documentation qui l’accompagne peuvent fournir des informations ou des liens donnant accès à des contenus, des produits et des services émanant de tiers. Oracle Corporation et ses affiliés déclinent toute responsabilité ou garantie expresse quant aux contenus, produits ou services émanant de tiers. En aucun cas, Oracle Corporation et ses affiliés ne sauraient être tenus pour responsables des pertes subies, des coûts occasionnés ou des dommages causés par l’accès à des contenus, produits ou services tiers, ou à leur utilisation.</para>
- <para><anchor xml:id="dbdoclet.50438245_pgfId-998576" xreflabel=""/>This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License. To view a copy of this license and obtain more information about Creative Commons licensing, visit <link href="http://creativecommons.org/licenses/by-sa/3.0/us">Creative Commons Attribution-Share Alike 3.0 United States</link> or send a letter to Creative Commons, 171 2nd Street, Suite 300, San Francisco, California 94105, USA.</para>
+ <para>This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License. To view a copy of this license and obtain more information about Creative Commons licensing, visit <link href="http://creativecommons.org/licenses/by-sa/3.0/us">Creative Commons Attribution-Share Alike 3.0 United States</link> or send a letter to Creative Commons, 171 2nd Street, Suite 300, San Francisco, California 94105, USA.</para>
</info>
</copyright>
<book version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink">
<info>
- <title><anchor xml:id="dbdoclet.50438145_pgfId-108319" xreflabel=""/>Lustre 2.x Filesystem</title>
+ <title>Lustre 2.x Filesystem</title>
<subtitle>Operations Manual</subtitle>
<xi:include href="copyright.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
</tgroup>
</informaltable>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-24675" xreflabel=""/>A<anchor xml:id="dbdoclet.50438146_Symbols" xreflabel=""/><anchor xml:id="dbdoclet.50438146_Numerics" xreflabel=""/><anchor xml:id="dbdoclet.50438146_A" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24679" xreflabel=""/></para>
+ <title>A<anchor xml:id="dbdoclet.50438146_Symbols" xreflabel=""/><anchor xml:id="dbdoclet.50438146_Numerics" xreflabel=""/><anchor xml:id="dbdoclet.50438146_A" xreflabel=""/></title>
+ <para></para>
<para>access control list (ACL), <link xl:href="ManagingSecurity.html#50438221_marker-1292306">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24681" xreflabel=""/></para>
+ <para></para>
<para>ACL, using, <link xl:href="ManagingSecurity.html#50438221_marker-1292306">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24683" xreflabel=""/>ACLs</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24684" xreflabel=""/></para>
+ <para>ACLs</para>
+ <para></para>
<para>examples, <link xl:href="ManagingSecurity.html#50438221_marker-1292325">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24686" xreflabel=""/></para>
+ <para></para>
<para>Lustre support, <link xl:href="ManagingSecurity.html#50438221_marker-1292314">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24688" xreflabel=""/>adaptive timeouts</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24689" xreflabel=""/></para>
+ <para>adaptive timeouts</para>
+ <para></para>
<para>configuring, <link xl:href="LustreProc.html#50438271_marker-1293381">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24691" xreflabel=""/></para>
+ <para></para>
<para>interpreting, <link xl:href="LustreProc.html#50438271_marker-1293383">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24693" xreflabel=""/></para>
+ <para></para>
<para>introduction, <link xl:href="LustreProc.html#50438271_marker-1293380">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24695" xreflabel=""/>adding</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24696" xreflabel=""/></para>
+ <para>adding</para>
+ <para></para>
<para>clients, <link xl:href="ConfiguringLustre.html#50438267_marker-1292440">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24698" xreflabel=""/></para>
+ <para></para>
<para>OSTs, <link xl:href="ConfiguringLustre.html#50438267_marker-1292440">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24700" xreflabel=""/></para>
+ <para></para>
<para>allocating quotas, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290226">1</link></para>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-24702" xreflabel=""/>B<anchor xml:id="dbdoclet.50438146_B" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24704" xreflabel=""/></para>
+ <title>B<anchor xml:id="dbdoclet.50438146_B" xreflabel=""/></title>
+ <para></para>
<para>bonding, <link xl:href="SettingUpBonding.html#50438258_marker-1301678">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24706" xreflabel=""/></para>
+ <para></para>
<para>configuring Lustre, <link xl:href="SettingUpBonding.html#50438258_marker-1301911">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24708" xreflabel=""/></para>
+ <para></para>
<para>module parameters, <link xl:href="SettingUpBonding.html#50438258_marker-1301747">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24710" xreflabel=""/></para>
+ <para></para>
<para>references, <link xl:href="SettingUpBonding.html#50438258_marker-1301915">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24712" xreflabel=""/></para>
+ <para></para>
<para>requirements, <link xl:href="SettingUpBonding.html#50438258_marker-1301684">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24714" xreflabel=""/></para>
+ <para></para>
<para>setting up, <link xl:href="SettingUpBonding.html#50438258_marker-1301757">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-24716" xreflabel=""/>C<anchor xml:id="dbdoclet.50438146_C" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24718" xreflabel=""/>calculating</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24719" xreflabel=""/></para>
+ <title>C<anchor xml:id="dbdoclet.50438146_C" xreflabel=""/></title>
+ <para>calculating</para>
+ <para></para>
<para>OSS memory requirements, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1292245">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24721" xreflabel=""/></para>
+ <para></para>
<para>changelogs, <link xl:href="LustreMonitoring.html#50438273_marker-1297383">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24723" xreflabel=""/></para>
+ <para></para>
<para>checksums, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1291974">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24725" xreflabel=""/>client</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24726" xreflabel=""/></para>
+ <para>client</para>
+ <para></para>
<para>eviction, <link xl:href="LustreRecovery.html#50438268_marker-1292164">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24728" xreflabel=""/>client read/write</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24729" xreflabel=""/></para>
+ <para>client read/write</para>
+ <para></para>
<para>extents survey, <link xl:href="LustreProc.html#50438271_marker-1290612">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24731" xreflabel=""/></para>
+ <para></para>
<para>offset survey, <link xl:href="LustreProc.html#50438271_marker-1290564">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24733" xreflabel=""/>clients</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24734" xreflabel=""/></para>
+ <para>clients</para>
+ <para></para>
<para>adding, <link xl:href="ConfiguringLustre.html#50438267_marker-1292440">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24736" xreflabel=""/></para>
+ <para></para>
<para>CollectL, <link xl:href="LustreMonitoring.html#50438273_marker-1297391">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24738" xreflabel=""/>command</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24739" xreflabel=""/></para>
+ <para>command</para>
+ <para></para>
<para>filefrag, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305920">1</link>, <link xl:href="LustreProgrammingInterfaces.html#50438291_marker-1294565">2</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24742" xreflabel=""/></para>
+ <para></para>
<para>lfs, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305209">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24744" xreflabel=""/></para>
+ <para></para>
<para>lfsck, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305843">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24746" xreflabel=""/></para>
+ <para></para>
<para>mount, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305992">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24748" xreflabel=""/></para>
+ <para></para>
<para>Commit on Share (COS), <link xl:href="LustreRecovery.html#50438268_marker-1292182">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24750" xreflabel=""/></para>
+ <para></para>
<para>components, Lustre, <link xl:href="UnderstandingLustre.html#50438250_marker-1293887">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24752" xreflabel=""/></para>
+ <para></para>
<para>configuration example, Lustre, <link xl:href="ConfiguringLustre.html#50438267_marker-1290955">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24754" xreflabel=""/></para>
+ <para></para>
<para>configuration, logs, regenerating, <link xl:href="LustreMaintenance.html#50438199_marker-1305736">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24756" xreflabel=""/>configuration, more complex</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24757" xreflabel=""/></para>
+ <para>configuration, more complex</para>
+ <para></para>
<para>failover, <link xl:href="LustreOperations.html#50438194_marker-1306313">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24759" xreflabel=""/>configuring</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24760" xreflabel=""/></para>
+ <para>configuring</para>
+ <para></para>
<para>adaptive timeouts, <link xl:href="LustreProc.html#50438271_marker-1293381">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24762" xreflabel=""/></para>
+ <para></para>
<para>root squash, <link xl:href="ManagingSecurity.html#50438221_marker-1294610">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-24764" xreflabel=""/>D<anchor xml:id="dbdoclet.50438146_D" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24766" xreflabel=""/></para>
+ <title>D<anchor xml:id="dbdoclet.50438146_D" xreflabel=""/></title>
+ <para></para>
<para>debug_mb, <link xl:href="LustreDebugging.html#50438274_marker-1295885">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24768" xreflabel=""/>debugging</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24769" xreflabel=""/></para>
+ <para>debugging</para>
+ <para></para>
<para>adding debugging to source code, <link xl:href="LustreDebugging.html#50438274_marker-1296026">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24771" xreflabel=""/></para>
+ <para></para>
<para>buffer, <link xl:href="LustreDebugging.html#50438274_marker-1295885">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24773" xreflabel=""/></para>
+ <para></para>
<para>controlling the kernel debug log, <link xl:href="LustreDebugging.html#50438274_marker-1295955">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24775" xreflabel=""/></para>
+ <para></para>
<para>finding Lustre UUID of an OST, <link xl:href="LustreDebugging.html#50438274_marker-1296007">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24777" xreflabel=""/></para>
+ <para></para>
<para>finding memory leaks, <link xl:href="LustreDebugging.html#50438274_marker-1296153">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24779" xreflabel=""/></para>
+ <para></para>
<para>looking at disk content, <link xl:href="LustreDebugging.html#50438274_marker-1295982">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24781" xreflabel=""/></para>
+ <para></para>
<para>messages, <link xl:href="LustreDebugging.html#50438274_marker-1295746">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24783" xreflabel=""/></para>
+ <para></para>
<para>Ptlrpc request history, <link xl:href="LustreDebugging.html#50438274_marker-1296099">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24785" xreflabel=""/></para>
+ <para></para>
<para>sample lctl run, <link xl:href="LustreDebugging.html#50438274_marker-1295914">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24787" xreflabel=""/></para>
+ <para></para>
<para>tracing lock traffic, <link xl:href="LustreDebugging.html#50438274_marker-1296017">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24789" xreflabel=""/></para>
+ <para></para>
<para>troubleshooting with strace, <link xl:href="LustreDebugging.html#50438274_marker-1295969">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24791" xreflabel=""/></para>
+ <para></para>
<para>directory statahead, using, <link xl:href="LustreProc.html#50438271_marker-1294292">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24793" xreflabel=""/></para>
+ <para></para>
<para>downed routers, <link xl:href="ManagingLNET.html#50438203_marker-1288828">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-24795" xreflabel=""/>E<anchor xml:id="dbdoclet.50438146_E" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24797" xreflabel=""/></para>
+ <title>E<anchor xml:id="dbdoclet.50438146_E" xreflabel=""/></title>
+ <para></para>
<para>e2fsprogs, <link xl:href="InstallingLustre.html#50438261_marker-1292893">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24799" xreflabel=""/></para>
+ <para></para>
<para>e2scan, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317398">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24801" xreflabel=""/></para>
+ <para></para>
<para>Elan (Quadrics Elan), <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289889">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24803" xreflabel=""/></para>
+ <para></para>
<para>environmental requirements, <link xl:href="InstallingLustre.html#50438261_marker-1292914">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24805" xreflabel=""/></para>
+ <para></para>
<para>error messages, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291323">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24807" xreflabel=""/></para>
+ <para></para>
<para>error numbers, <link xl:href="LustreTroubleshooting.html#50438198_marker-1296744">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24809" xreflabel=""/></para>
+ <para></para>
<para>external journal, creating, <link xl:href="ConfiguringStorage.html#50438208_marker-1289927">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-24811" xreflabel=""/>F<anchor xml:id="dbdoclet.50438146_F" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24813" xreflabel=""/></para>
+ <title>F<anchor xml:id="dbdoclet.50438146_F" xreflabel=""/></title>
+ <para></para>
<para>failover, <link xl:href="UnderstandingFailover.html#50438253_marker-1304330">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24815" xreflabel=""/></para>
+ <para></para>
<para>capabilities, <link xl:href="UnderstandingFailover.html#50438253_marker-1304336">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24817" xreflabel=""/></para>
+ <para></para>
<para>configuration types, <link xl:href="UnderstandingFailover.html#50438253_marker-1304346">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24819" xreflabel=""/></para>
+ <para></para>
<para>configuring, <link xl:href="LustreOperations.html#50438194_marker-1306313">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24821" xreflabel=""/></para>
+ <para></para>
<para>failover and MMP, <link xl:href="ManagingFailover.html#50438213_marker-1301522">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24823" xreflabel=""/></para>
+ <para></para>
<para>MDT (active/passive), <link xl:href="UnderstandingFailover.html#50438253_marker-1304373">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24825" xreflabel=""/></para>
+ <para></para>
<para>OST (active/active), <link xl:href="UnderstandingFailover.html#50438253_marker-1304382">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24827" xreflabel=""/></para>
+ <para></para>
<para>file formats, quotas, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290326">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24829" xreflabel=""/></para>
+ <para></para>
<para>file readahead, using, <link xl:href="LustreProc.html#50438271_marker-1294292">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24831" xreflabel=""/></para>
+ <para></para>
<para>file size, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290761">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24833" xreflabel=""/></para>
+ <para></para>
<para>file striping, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1291832">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24835" xreflabel=""/></para>
+ <para></para>
<para>file system size, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290753">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24837" xreflabel=""/></para>
+ <para></para>
<para>filefrag command, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305920">1</link>, <link xl:href="LustreProgrammingInterfaces.html#50438291_marker-1294565">2</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24840" xreflabel=""/></para>
+ <para></para>
<para>filename length, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290791">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24842" xreflabel=""/></para>
+ <para></para>
<para>flock utility, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1295910">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24844" xreflabel=""/>free space management</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24845" xreflabel=""/></para>
+ <para>free space management</para>
+ <para></para>
<para>adjusting weighting between free space and location, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1294033">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24847" xreflabel=""/></para>
+ <para></para>
<para>round-robin allocator, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1293988">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24849" xreflabel=""/></para>
+ <para></para>
<para>weighted allocator, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1294020">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24851" xreflabel=""/></para>
+ <para></para>
<para>free space, managing, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1295520">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-24853" xreflabel=""/>H<anchor xml:id="dbdoclet.50438146_G" xreflabel=""/><anchor xml:id="dbdoclet.50438146_H" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24856" xreflabel=""/></para>
+ <title>H<anchor xml:id="dbdoclet.50438146_G" xreflabel=""/><anchor xml:id="dbdoclet.50438146_H" xreflabel=""/></title>
+ <para></para>
<para>HA software, <link xl:href="InstallingLustre.html#50438261_marker-1292902">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24858" xreflabel=""/></para>
+ <para></para>
<para>handling full OSTs, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1295529">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24860" xreflabel=""/></para>
+ <para></para>
<para>handling timeouts, <link xl:href="UserUtilities_HTML.html#50438206_marker-1306030">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-24862" xreflabel=""/>I<anchor xml:id="dbdoclet.50438146_I" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24864" xreflabel=""/>I/O options</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24865" xreflabel=""/></para>
+ <title>I<anchor xml:id="dbdoclet.50438146_I" xreflabel=""/></title>
+ <para>I/O options</para>
+ <para></para>
<para>checksums, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1291974">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24867" xreflabel=""/></para>
+ <para></para>
<para>I/O tunables, <link xl:href="LustreProc.html#50438271_marker-1290508">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24869" xreflabel=""/></para>
+ <para></para>
<para>I/O, direct, performing, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1291962">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24871" xreflabel=""/></para>
+ <para></para>
<para>inode number, OST, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1293235">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24873" xreflabel=""/></para>
+ <para></para>
<para>inode size, MDT, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1293231">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24875" xreflabel=""/></para>
+ <para></para>
<para>installing Lustre, environmental requirements, <link xl:href="InstallingLustre.html#50438261_marker-1292914">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24877" xreflabel=""/></para>
+ <para></para>
<para>installing Lustre, HA software, <link xl:href="InstallingLustre.html#50438261_marker-1292902">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24879" xreflabel=""/></para>
+ <para></para>
<para>installing Lustre, memory requirements, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1292212">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24881" xreflabel=""/></para>
+ <para></para>
<para>installing Lustre, required tools / utilities, <link xl:href="InstallingLustre.html#50438261_marker-1292890">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24883" xreflabel=""/></para>
+ <para></para>
<para>interoperability, <link xl:href="UpgradingLustre.html#50438205_marker-1293321">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24885" xreflabel=""/>interpreting</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24886" xreflabel=""/></para>
+ <para>interpreting</para>
+ <para></para>
<para>adaptive timeouts, <link xl:href="LustreProc.html#50438271_marker-1293383">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-24888" xreflabel=""/>K<anchor xml:id="dbdoclet.50438146_J" xreflabel=""/><anchor xml:id="dbdoclet.50438146_K" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24891" xreflabel=""/></para>
+ <title>K<anchor xml:id="dbdoclet.50438146_J" xreflabel=""/><anchor xml:id="dbdoclet.50438146_K" xreflabel=""/></title>
+ <para></para>
<para>key features, <link xl:href="UnderstandingLustre.html#50438250_marker-1293792">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-24893" xreflabel=""/>L<anchor xml:id="dbdoclet.50438146_L" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24895" xreflabel=""/></para>
+ <title>L<anchor xml:id="dbdoclet.50438146_L" xreflabel=""/></title>
+ <para></para>
<para>l_getidentity, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1318227">1</link>, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317750">2</link>, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317690">3</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24899" xreflabel=""/></para>
+ <para></para>
<para>lctl, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1302539">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24901" xreflabel=""/></para>
+ <para></para>
<para>setting parameters, <link xl:href="LustreOperations.html#50438194_marker-1305718">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24903" xreflabel=""/></para>
+ <para></para>
<para>lfs command, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305209">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24905" xreflabel=""/></para>
+ <para></para>
<para>lfsck command, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305843">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24907" xreflabel=""/></para>
+ <para></para>
<para>ll_recover_lost_found_objs, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317865">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24909" xreflabel=""/></para>
+ <para></para>
<para>llapi, <link xl:href="SettingLustreProperties_HTML.html#50438215_marker-1297700">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24911" xreflabel=""/></para>
+ <para></para>
<para>llstat, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1318052">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24913" xreflabel=""/>LNET</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24914" xreflabel=""/></para>
+ <para>LNET</para>
+ <para></para>
<para>starting, <link xl:href="ManagingLNET.html#50438203_marker-1287400">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24916" xreflabel=""/></para>
+ <para></para>
<para>stopping, <link xl:href="ManagingLNET.html#50438203_marker-1288543">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24918" xreflabel=""/>LNET self-test</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24919" xreflabel=""/></para>
+ <para>LNET self-test</para>
+ <para></para>
<para>commands, <link xl:href="LNETSelfTest.html#50438223_marker-1298562">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24921" xreflabel=""/>Load balancing with InfiniBand</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24922" xreflabel=""/></para>
+ <para>Load balancing with InfiniBand</para>
+ <para></para>
<para>modprobe.conf, <link xl:href="ManagingLNET.html#50438203_marker-1290316">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24924" xreflabel=""/></para>
+ <para></para>
<para>locking proc entries, <link xl:href="LustreProc.html#50438271_marker-1290874">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24926" xreflabel=""/></para>
+ <para></para>
<para>lst, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1318121">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24928" xreflabel=""/>Lustre</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24929" xreflabel=""/></para>
+ <para>Lustre</para>
+ <para></para>
<para>administration, aborting recovery, <link xl:href="LustreMaintenance.html#50438199_marker-1299023">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24931" xreflabel=""/></para>
+ <para></para>
<para>administration, changing a server NID, <link xl:href="LustreMaintenance.html#50438199_marker-1305737">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24933" xreflabel=""/></para>
+ <para></para>
<para>administration, determining which machine is serving an OST, <link xl:href="LustreMaintenance.html#50438199_marker-1305741">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24935" xreflabel=""/></para>
+ <para></para>
<para>administration, failout / failover mode for OSTs, <link xl:href="LustreOperations.html#50438194_marker-1298926">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24937" xreflabel=""/></para>
+ <para></para>
<para>administration, finding nodes in the file system, <link xl:href="LustreMaintenance.html#50438199_marker-1298897">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24939" xreflabel=""/></para>
+ <para></para>
<para>administration, mounting a server, <link xl:href="LustreOperations.html#50438194_marker-1298863">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24941" xreflabel=""/></para>
+ <para></para>
<para>administration, mounting a server without Lustre service, <link xl:href="LustreMaintenance.html#50438199_marker-1298918">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24943" xreflabel=""/></para>
+ <para></para>
<para>administration, regenerating Lustre configuration logs, <link xl:href="LustreMaintenance.html#50438199_marker-1305736">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24945" xreflabel=""/></para>
+ <para></para>
<para>administration, removing and restoring OSTs, <link xl:href="LustreMaintenance.html#50438199_marker-1298976">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24947" xreflabel=""/></para>
+ <para></para>
<para>administration, running multiple Lustre file systems, <link xl:href="LustreOperations.html#50438194_marker-1298939">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24949" xreflabel=""/></para>
+ <para></para>
<para>administration, setting and retrieving Lustre parameters, <link xl:href="LustreOperations.html#50438194_marker-1302467">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24951" xreflabel=""/></para>
+ <para></para>
<para>administration, starting Lustre, <link xl:href="LustreOperations.html#50438194_marker-1305696">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24953" xreflabel=""/></para>
+ <para></para>
<para>administration, working with inactive OSTs, <link xl:href="LustreMaintenance.html#50438199_marker-1298888">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24955" xreflabel=""/></para>
+ <para></para>
<para>adminstration, unmounting a server, <link xl:href="LustreOperations.html#50438194_marker-1298879">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24957" xreflabel=""/></para>
+ <para></para>
<para>components, <link xl:href="UnderstandingLustre.html#50438250_marker-1293887">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24959" xreflabel=""/></para>
+ <para></para>
<para>configuration example, <link xl:href="ConfiguringLustre.html#50438267_marker-1290955">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24961" xreflabel=""/></para>
+ <para></para>
<para>COS, <link xl:href="LustreRecovery.html#50438268_marker-1292182">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24963" xreflabel=""/></para>
+ <para></para>
<para>failover, <link xl:href="UnderstandingFailover.html#50438253_marker-1304330">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24965" xreflabel=""/></para>
+ <para></para>
<para>installing, environmental requirements, <link xl:href="InstallingLustre.html#50438261_marker-1292914">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24967" xreflabel=""/></para>
+ <para></para>
<para>installing, HA software, <link xl:href="InstallingLustre.html#50438261_marker-1292902">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24969" xreflabel=""/></para>
+ <para></para>
<para>installing, memory requirements, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1292212">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24971" xreflabel=""/></para>
+ <para></para>
<para>installing, required tools / utilities, <link xl:href="InstallingLustre.html#50438261_marker-1292890">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24973" xreflabel=""/></para>
+ <para></para>
<para>interoperability, <link xl:href="UpgradingLustre.html#50438205_marker-1293321">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24975" xreflabel=""/></para>
+ <para></para>
<para>key features, <link xl:href="UnderstandingLustre.html#50438250_marker-1293792">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24977" xreflabel=""/></para>
+ <para></para>
<para>metadata replay, <link xl:href="LustreRecovery.html#50438268_marker-1292175">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24979" xreflabel=""/></para>
+ <para></para>
<para>orphaned objects, <link xl:href="TroubleShootingRecovery.html#50438225_marker-1292187">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24981" xreflabel=""/></para>
+ <para></para>
<para>parameters, reporting current, <link xl:href="LustreOperations.html#50438194_marker-1302474">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24983" xreflabel=""/></para>
+ <para></para>
<para>parameters, setting and retrieving, <link xl:href="LustreOperations.html#50438194_marker-1302467">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24985" xreflabel=""/></para>
+ <para></para>
<para>parameters, setting with lctl, <link xl:href="LustreOperations.html#50438194_marker-1305718">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24987" xreflabel=""/></para>
+ <para></para>
<para>parameters, setting with mkfs.lustre, <link xl:href="LustreOperations.html#50438194_marker-1305722">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24989" xreflabel=""/></para>
+ <para></para>
<para>parameters, setting with tunefs.lustre, <link xl:href="LustreOperations.html#50438194_marker-1305720">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24991" xreflabel=""/></para>
+ <para></para>
<para>recovering from corruption in Lustre file system, <link xl:href="TroubleShootingRecovery.html#50438225_marker-1292186">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24993" xreflabel=""/></para>
+ <para></para>
<para>recovering from errors on backing file system, <link xl:href="TroubleShootingRecovery.html#50438225_marker-1292184">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24995" xreflabel=""/></para>
+ <para></para>
<para>recovery, client eviction, <link xl:href="LustreRecovery.html#50438268_marker-1292164">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24997" xreflabel=""/></para>
+ <para></para>
<para>reply reconstruction, <link xl:href="LustreRecovery.html#50438268_marker-1292176">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-24999" xreflabel=""/></para>
+ <para></para>
<para>scaling, <link xl:href="ConfiguringLustre.html#50438267_marker-1292440">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25001" xreflabel=""/></para>
+ <para></para>
<para>starting, <link xl:href="LustreOperations.html#50438194_marker-1305696">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25003" xreflabel=""/></para>
+ <para></para>
<para>upgrading, 1.6.x to 1.8.x, <link xl:href="UpgradingLustre.html#50438205_marker-1294652">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25005" xreflabel=""/></para>
+ <para></para>
<para>VBR, introduction, <link xl:href="LustreRecovery.html#50438268_marker-1288580">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25007" xreflabel=""/></para>
+ <para></para>
<para>VBR, tips, <link xl:href="LustreRecovery.html#50438268_marker-1288584">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25009" xreflabel=""/></para>
+ <para></para>
<para>VBR, working with, <link xl:href="LustreRecovery.html#50438268_marker-1288583">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25011" xreflabel=""/>Lustre I/O kit</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25012" xreflabel=""/></para>
+ <para>Lustre I/O kit</para>
+ <para></para>
<para>downloading, <link xl:href="BenchmarkingTests.html#50438212_marker-1289913">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25014" xreflabel=""/></para>
+ <para></para>
<para>obdfilter_survey tool, <link xl:href="BenchmarkingTests.html#50438212_marker-1289957">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25016" xreflabel=""/></para>
+ <para></para>
<para>ost_survey tool, <link xl:href="BenchmarkingTests.html#50438212_marker-1290067">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25018" xreflabel=""/></para>
+ <para></para>
<para>sgpdd_survey tool, <link xl:href="BenchmarkingTests.html#50438212_marker-1302844">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25020" xreflabel=""/></para>
+ <para></para>
<para>Lustre Monitoring Tool (LMT), <link xl:href="LustreMonitoring.html#50438273_marker-1297386">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25022" xreflabel=""/></para>
+ <para></para>
<para>lustre_rmmod.sh, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317379">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25024" xreflabel=""/></para>
+ <para></para>
<para>lustre_rsync, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317224">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-25026" xreflabel=""/>M<anchor xml:id="dbdoclet.50438146_M" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25028" xreflabel=""/>man1</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25029" xreflabel=""/></para>
+ <title>M<anchor xml:id="dbdoclet.50438146_M" xreflabel=""/></title>
+ <para>man1</para>
+ <para></para>
<para>filefrag, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305920">1</link>, <link xl:href="LustreProgrammingInterfaces.html#50438291_marker-1294565">2</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25032" xreflabel=""/></para>
+ <para></para>
<para>lfs, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305209">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25034" xreflabel=""/></para>
+ <para></para>
<para>lfsck, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305843">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25036" xreflabel=""/></para>
+ <para></para>
<para>mount, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305992">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25038" xreflabel=""/>man2</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25039" xreflabel=""/></para>
+ <para>man2</para>
+ <para></para>
<para>user/group cache upcall, <link xl:href="LustreProgrammingInterfaces.html#50438291_marker-1293215">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25041" xreflabel=""/>man5</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25042" xreflabel=""/></para>
+ <para>man5</para>
+ <para></para>
<para>LNET options, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1293320">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25044" xreflabel=""/></para>
+ <para></para>
<para>module options, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1293311">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25046" xreflabel=""/></para>
+ <para></para>
<para>MX LND, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1295997">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25048" xreflabel=""/></para>
+ <para></para>
<para>Portals LND (Linux), <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1293719">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25050" xreflabel=""/></para>
+ <para></para>
<para>SOCKLND kernel TCP/IP LND, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1293448">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25052" xreflabel=""/>man8</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25053" xreflabel=""/></para>
+ <para>man8</para>
+ <para></para>
<para>application profiling utilties, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1313051">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25055" xreflabel=""/></para>
+ <para></para>
<para>l_getidentity, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1318227">1</link>, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317750">2</link>, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317690">3</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25059" xreflabel=""/></para>
+ <para></para>
<para>lctl, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1302539">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25061" xreflabel=""/></para>
+ <para></para>
<para>ll_recover_lost_found_objs, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317865">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25063" xreflabel=""/></para>
+ <para></para>
<para>llstat, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1318052">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25065" xreflabel=""/></para>
+ <para></para>
<para>lst, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1318121">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25067" xreflabel=""/></para>
+ <para></para>
<para>lustre_rsync, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317224">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25069" xreflabel=""/></para>
+ <para></para>
<para>mkfs.lustre, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1316695">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25071" xreflabel=""/></para>
+ <para></para>
<para>mount.lustre, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1310637">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25073" xreflabel=""/></para>
+ <para></para>
<para>plot-llstat, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317989">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25075" xreflabel=""/></para>
+ <para></para>
<para>proc statistics, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1313052">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25077" xreflabel=""/></para>
+ <para></para>
<para>routerstat, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317921">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25079" xreflabel=""/></para>
+ <para></para>
<para>system configuration utilities, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1313048">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25081" xreflabel=""/></para>
+ <para></para>
<para>test/debug utilities, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1313053">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25083" xreflabel=""/></para>
+ <para></para>
<para>tunefs.lustre, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317076">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25085" xreflabel=""/></para>
+ <para></para>
<para>managing free space, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1295520">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25087" xreflabel=""/>maximum</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25088" xreflabel=""/></para>
+ <para>maximum</para>
+ <para></para>
<para>file size, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290761">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25090" xreflabel=""/></para>
+ <para></para>
<para>file system size, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290753">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25092" xreflabel=""/></para>
+ <para></para>
<para>filename/pathname length, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290791">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25094" xreflabel=""/></para>
+ <para></para>
<para>number of clients, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290746">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25096" xreflabel=""/></para>
+ <para></para>
<para>number of files in a directory, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290770">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25098" xreflabel=""/></para>
+ <para></para>
<para>number of open files, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290800">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25100" xreflabel=""/></para>
+ <para></para>
<para>number of OSTs and MDTs, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290738">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25102" xreflabel=""/></para>
+ <para></para>
<para>stripe count, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290714">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25104" xreflabel=""/></para>
+ <para></para>
<para>stripe size, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290722">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25106" xreflabel=""/>mballoc</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25107" xreflabel=""/></para>
+ <para>mballoc</para>
+ <para></para>
<para>history, <link xl:href="LustreProc.html#50438271_marker-1297045">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25109" xreflabel=""/>mballoc3</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25110" xreflabel=""/></para>
+ <para>mballoc3</para>
+ <para></para>
<para>tunables, <link xl:href="LustreProc.html#50438271_marker-1290795">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25112" xreflabel=""/>MDS</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25113" xreflabel=""/></para>
+ <para>MDS</para>
+ <para></para>
<para>service thread count, <link xl:href="LustreTuning.html#50438272_marker-1293755">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25115" xreflabel=""/>MDT</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25116" xreflabel=""/></para>
+ <para>MDT</para>
+ <para></para>
<para>failover, <link xl:href="UnderstandingFailover.html#50438253_marker-1304373">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25118" xreflabel=""/></para>
+ <para></para>
<para>inode size, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1293231">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25120" xreflabel=""/></para>
+ <para></para>
<para>memory requirements, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1292212">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25122" xreflabel=""/></para>
+ <para></para>
<para>metadata replay, <link xl:href="LustreRecovery.html#50438268_marker-1292175">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25124" xreflabel=""/>minimum</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25125" xreflabel=""/></para>
+ <para>minimum</para>
+ <para></para>
<para>stripe size, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290730">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25127" xreflabel=""/></para>
+ <para></para>
<para>mkfs.lustre, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1316695">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25129" xreflabel=""/></para>
+ <para></para>
<para>setting parameters, <link xl:href="LustreOperations.html#50438194_marker-1305722">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25131" xreflabel=""/>MMP</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25132" xreflabel=""/></para>
+ <para>MMP</para>
+ <para></para>
<para>MMP and failover, <link xl:href="ManagingFailover.html#50438213_marker-1301522">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25134" xreflabel=""/></para>
+ <para></para>
<para>modprobe.conf, <link xl:href="ManagingLNET.html#50438203_marker-1290316">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25136" xreflabel=""/>monitoring</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25137" xreflabel=""/></para>
+ <para>monitoring</para>
+ <para></para>
<para>changelogs, <link xl:href="LustreMonitoring.html#50438273_marker-1297383">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25139" xreflabel=""/></para>
+ <para></para>
<para>CollectL, <link xl:href="LustreMonitoring.html#50438273_marker-1297391">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25141" xreflabel=""/></para>
+ <para></para>
<para>Lustre Monitoring Tool, <link xl:href="LustreMonitoring.html#50438273_marker-1297386">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25143" xreflabel=""/></para>
+ <para></para>
<para>mount command, <link xl:href="UserUtilities_HTML.html#50438206_marker-1305992">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25145" xreflabel=""/></para>
+ <para></para>
<para>mount.lustre, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1310637">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25147" xreflabel=""/></para>
+ <para></para>
<para>MX LND, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1295997">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-25149" xreflabel=""/>N<anchor xml:id="dbdoclet.50438146_N" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25151" xreflabel=""/>network</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25152" xreflabel=""/></para>
+ <title>N<anchor xml:id="dbdoclet.50438146_N" xreflabel=""/></title>
+ <para>network</para>
+ <para></para>
<para>bonding, <link xl:href="SettingUpBonding.html#50438258_marker-1301678">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25154" xreflabel=""/>networks, supported</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25155" xreflabel=""/></para>
+ <para>networks, supported</para>
+ <para></para>
<para>Elan (Quadrics Elan), <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289889">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25157" xreflabel=""/></para>
+ <para></para>
<para>ra (RapidArray), <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289887">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25159" xreflabel=""/></para>
+ <para></para>
<para>NID, server, changing, <link xl:href="LustreMaintenance.html#50438199_marker-1305737">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25161" xreflabel=""/></para>
+ <para></para>
<para>number of clients, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290746">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25163" xreflabel=""/></para>
+ <para></para>
<para>number of files in a directory, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290770">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25165" xreflabel=""/></para>
+ <para></para>
<para>number of open files, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290800">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-25167" xreflabel=""/>O<anchor xml:id="dbdoclet.50438146_O" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25169" xreflabel=""/></para>
+ <title>O<anchor xml:id="dbdoclet.50438146_O" xreflabel=""/></title>
+ <para></para>
<para>obdfilter_survey tool, <link xl:href="BenchmarkingTests.html#50438212_marker-1289957">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25171" xreflabel=""/></para>
+ <para></para>
<para>orphaned objects, working with, <link xl:href="TroubleShootingRecovery.html#50438225_marker-1292187">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25173" xreflabel=""/>OSS</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25174" xreflabel=""/></para>
+ <para>OSS</para>
+ <para></para>
<para>memory, determining, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1292245">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25176" xreflabel=""/></para>
+ <para></para>
<para>service thread count, <link xl:href="LustreTuning.html#50438272_marker-1294858">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25178" xreflabel=""/></para>
+ <para></para>
<para>OSS read cache, <link xl:href="LustreProc.html#50438271_marker-1296183">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25180" xreflabel=""/>OST</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25181" xreflabel=""/></para>
+ <para>OST</para>
+ <para></para>
<para>failover, <link xl:href="UnderstandingFailover.html#50438253_marker-1304382">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25183" xreflabel=""/></para>
+ <para></para>
<para>number of inodes, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1293235">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25185" xreflabel=""/></para>
+ <para></para>
<para>removing and restoring, <link xl:href="LustreMaintenance.html#50438199_marker-1298976">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25187" xreflabel=""/></para>
+ <para></para>
<para>OST block I/O stream, watching, <link xl:href="LustreProc.html#50438271_marker-1290631">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25189" xreflabel=""/></para>
+ <para></para>
<para>OST pools, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1295531">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25191" xreflabel=""/></para>
+ <para></para>
<para>OST, adding, <link xl:href="LustreMaintenance.html#50438199_marker-1306353">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25193" xreflabel=""/></para>
+ <para></para>
<para>OST, determining which machine is serving, <link xl:href="LustreMaintenance.html#50438199_marker-1305741">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25195" xreflabel=""/></para>
+ <para></para>
<para>ost_survey tool, <link xl:href="BenchmarkingTests.html#50438212_marker-1290067">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25197" xreflabel=""/>OSTs</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25198" xreflabel=""/></para>
+ <para>OSTs</para>
+ <para></para>
<para>adding, <link xl:href="ConfiguringLustre.html#50438267_marker-1292440">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25200" xreflabel=""/></para>
+ <para></para>
<para>OSTs and MDTs, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290738">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25202" xreflabel=""/></para>
+ <para></para>
<para>OSTs, full, handling, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1295529">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-25204" xreflabel=""/>P<anchor xml:id="dbdoclet.50438146_P" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25206" xreflabel=""/></para>
+ <title>P<anchor xml:id="dbdoclet.50438146_P" xreflabel=""/></title>
+ <para></para>
<para>parameters, setting with lctl, <link xl:href="LustreOperations.html#50438194_marker-1305718">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25208" xreflabel=""/></para>
+ <para></para>
<para>parameters, setting with mkfs.lustre, <link xl:href="LustreOperations.html#50438194_marker-1305722">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25210" xreflabel=""/></para>
+ <para></para>
<para>parameters, setting with tunefs.lustre, <link xl:href="LustreOperations.html#50438194_marker-1305720">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25212" xreflabel=""/></para>
+ <para></para>
<para>pathname length, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290791">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25214" xreflabel=""/></para>
+ <para></para>
<para>performing direct I/O, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1291962">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25216" xreflabel=""/></para>
+ <para></para>
<para>Perl, <link xl:href="InstallingLustre.html#50438261_marker-1292899">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25218" xreflabel=""/></para>
+ <para></para>
<para>plot-llstat, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317989">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25220" xreflabel=""/></para>
+ <para></para>
<para>pools, OST, <link xl:href="ManagingFileSystemIO.html#50438211_marker-1295531">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25222" xreflabel=""/>Portals LND</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25223" xreflabel=""/></para>
+ <para>Portals LND</para>
+ <para></para>
<para>Linux, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1293719">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25225" xreflabel=""/>proc entries</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25226" xreflabel=""/></para>
+ <para>proc entries</para>
+ <para></para>
<para>free space distribution, <link xl:href="LustreProc.html#50438271_marker-1296165">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25228" xreflabel=""/></para>
+ <para></para>
<para>LNET information, <link xl:href="LustreProc.html#50438271_marker-1296164">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25230" xreflabel=""/></para>
+ <para></para>
<para>locating file systems and servers, <link xl:href="LustreProc.html#50438271_marker-1296151">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25232" xreflabel=""/></para>
+ <para></para>
<para>locking, <link xl:href="LustreProc.html#50438271_marker-1290874">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25234" xreflabel=""/></para>
+ <para></para>
<para>timeouts, <link xl:href="LustreProc.html#50438271_marker-1296153">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-25236" xreflabel=""/>Q<anchor xml:id="dbdoclet.50438146_Q" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25238" xreflabel=""/></para>
+ <title>Q<anchor xml:id="dbdoclet.50438146_Q" xreflabel=""/></title>
+ <para></para>
<para>Quadrics Elan, <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289889">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25240" xreflabel=""/></para>
+ <para></para>
<para>quota limits, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290282">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25242" xreflabel=""/></para>
+ <para></para>
<para>quota statistics, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290343">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25244" xreflabel=""/>quotas</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25245" xreflabel=""/></para>
+ <para>quotas</para>
+ <para></para>
<para>administering, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290170">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25247" xreflabel=""/></para>
+ <para></para>
<para>allocating, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290226">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25249" xreflabel=""/></para>
+ <para></para>
<para>creating files, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290170">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25251" xreflabel=""/></para>
+ <para></para>
<para>enabling, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290128">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25253" xreflabel=""/></para>
+ <para></para>
<para>file formats, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290326">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25255" xreflabel=""/></para>
+ <para></para>
<para>granted cache, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290272">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25257" xreflabel=""/></para>
+ <para></para>
<para>known issues, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290269">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25259" xreflabel=""/></para>
+ <para></para>
<para>limits, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290282">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25261" xreflabel=""/></para>
+ <para></para>
<para>statistics, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290343">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25263" xreflabel=""/></para>
+ <para></para>
<para>working with, <link xl:href="ConfiguringQuotas.html#50438217_marker-1290118">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-25265" xreflabel=""/>R<anchor xml:id="dbdoclet.50438146_R" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25267" xreflabel=""/></para>
+ <title>R<anchor xml:id="dbdoclet.50438146_R" xreflabel=""/></title>
+ <para></para>
<para>ra (RapidArray), <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289887">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25269" xreflabel=""/>RAID</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25270" xreflabel=""/></para>
+ <para>RAID</para>
+ <para></para>
<para>creating an external journal, <link xl:href="ConfiguringStorage.html#50438208_marker-1289927">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25272" xreflabel=""/></para>
+ <para></para>
<para>formatting options, <link xl:href="ConfiguringStorage.html#50438208_marker-1291599">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25274" xreflabel=""/></para>
+ <para></para>
<para>handling degraded arrays, <link xl:href="LustreOperations.html#50438194_marker-1307136">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25276" xreflabel=""/></para>
+ <para></para>
<para>performance tradeoffs, <link xl:href="ConfiguringStorage.html#50438208_marker-1291593">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25278" xreflabel=""/></para>
+ <para></para>
<para>reliability best practices, <link xl:href="ConfiguringStorage.html#50438208_marker-1291581">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25280" xreflabel=""/></para>
+ <para></para>
<para>selecting storage for MDS or OSTs, <link xl:href="ConfiguringStorage.html#50438208_marker-1291567">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25282" xreflabel=""/></para>
+ <para></para>
<para>RapidArray, <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289887">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25284" xreflabel=""/></para>
+ <para></para>
<para>readahead, tuning, <link xl:href="LustreProc.html#50438271_marker-1295183">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25286" xreflabel=""/>recovery mode, failure types</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25287" xreflabel=""/></para>
+ <para>recovery mode, failure types</para>
+ <para></para>
<para>client failure, <link xl:href="LustreRecovery.html#50438268_marker-1287394">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25289" xreflabel=""/></para>
+ <para></para>
<para>MDS failure/failover, <link xl:href="LustreRecovery.html#50438268_marker-1287397">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25291" xreflabel=""/></para>
+ <para></para>
<para>network partition, <link xl:href="LustreRecovery.html#50438268_marker-1289388">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25293" xreflabel=""/></para>
+ <para></para>
<para>OST failure, <link xl:href="LustreRecovery.html#50438268_marker-1289240">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25295" xreflabel=""/></para>
+ <para></para>
<para>recovery, aborting, <link xl:href="LustreMaintenance.html#50438199_marker-1299023">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25297" xreflabel=""/></para>
+ <para></para>
<para>regenerating configuration logs, <link xl:href="LustreMaintenance.html#50438199_marker-1305736">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25299" xreflabel=""/></para>
+ <para></para>
<para>reply reconstruction, <link xl:href="LustreRecovery.html#50438268_marker-1292176">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25301" xreflabel=""/></para>
+ <para></para>
<para>reporting current Lustre parameters, <link xl:href="LustreOperations.html#50438194_marker-1302474">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25303" xreflabel=""/></para>
+ <para></para>
<para>required tools / utilities, <link xl:href="InstallingLustre.html#50438261_marker-1292890">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25305" xreflabel=""/>root squash</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25306" xreflabel=""/></para>
+ <para>root squash</para>
+ <para></para>
<para>configuring, <link xl:href="ManagingSecurity.html#50438221_marker-1294610">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25308" xreflabel=""/></para>
+ <para></para>
<para>tips, <link xl:href="ManagingSecurity.html#50438221_marker-1296366">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25310" xreflabel=""/></para>
+ <para></para>
<para>tuning, <link xl:href="ManagingSecurity.html#50438221_marker-1294611">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25312" xreflabel=""/></para>
+ <para></para>
<para>root squash, using, <link xl:href="ManagingSecurity.html#50438221_marker-1294644">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25314" xreflabel=""/></para>
+ <para></para>
<para>round-robin allocator, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1293988">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25316" xreflabel=""/></para>
+ <para></para>
<para>routers, downed, <link xl:href="ManagingLNET.html#50438203_marker-1288828">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25318" xreflabel=""/></para>
+ <para></para>
<para>routerstat, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317921">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25320" xreflabel=""/></para>
+ <para></para>
<para>RPC stream tunables, <link xl:href="LustreProc.html#50438271_marker-1290514">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25322" xreflabel=""/></para>
+ <para></para>
<para>RPC stream, watching, <link xl:href="LustreProc.html#50438271_marker-1290535">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-25324" xreflabel=""/>S<anchor xml:id="dbdoclet.50438146_S" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25326" xreflabel=""/></para>
+ <title>S<anchor xml:id="dbdoclet.50438146_S" xreflabel=""/></title>
+ <para></para>
<para>scaling Lustre, <link xl:href="ConfiguringLustre.html#50438267_marker-1292440">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25328" xreflabel=""/>server</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25329" xreflabel=""/></para>
+ <para>server</para>
+ <para></para>
<para>mounting, <link xl:href="LustreOperations.html#50438194_marker-1298863">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25331" xreflabel=""/></para>
+ <para></para>
<para>unmounting, <link xl:href="LustreOperations.html#50438194_marker-1298879">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25333" xreflabel=""/></para>
+ <para></para>
<para>server NID, changing, <link xl:href="LustreMaintenance.html#50438199_marker-1305737">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25335" xreflabel=""/>service threads</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25336" xreflabel=""/></para>
+ <para>service threads</para>
+ <para></para>
<para>MDS, <link xl:href="LustreTuning.html#50438272_marker-1293755">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25338" xreflabel=""/></para>
+ <para></para>
<para>OSS, <link xl:href="LustreTuning.html#50438272_marker-1294858">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25340" xreflabel=""/>setting</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25341" xreflabel=""/></para>
+ <para>setting</para>
+ <para></para>
<para>SCSI I/O sizes, <link xl:href="LustreTroubleshooting.html#50438198_marker-1294800">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25343" xreflabel=""/></para>
+ <para></para>
<para>setting and retrieving Lustre parameters, <link xl:href="LustreOperations.html#50438194_marker-1302467">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25345" xreflabel=""/></para>
+ <para></para>
<para>sgpdd_survey tool, <link xl:href="BenchmarkingTests.html#50438212_marker-1302844">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25347" xreflabel=""/></para>
+ <para></para>
<para>SOCKLND kernel TCP/IP LND, <link xl:href="ConfigurationFilesModuleParameters.html#50438293_marker-1293448">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25349" xreflabel=""/>starting</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25350" xreflabel=""/></para>
+ <para>starting</para>
+ <para></para>
<para>LNET, <link xl:href="ManagingLNET.html#50438203_marker-1287400">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25352" xreflabel=""/></para>
+ <para></para>
<para>statahead, tuning, <link xl:href="LustreProc.html#50438271_marker-1295184">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25354" xreflabel=""/>stopping</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25355" xreflabel=""/></para>
+ <para>stopping</para>
+ <para></para>
<para>LNET, <link xl:href="ManagingLNET.html#50438203_marker-1288543">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25357" xreflabel=""/></para>
+ <para></para>
<para>strace, <link xl:href="LustreDebugging.html#50438274_marker-1295969">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25359" xreflabel=""/></para>
+ <para></para>
<para>stripe count, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290714">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25361" xreflabel=""/></para>
+ <para></para>
<para>stripe size, maximum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290722">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25363" xreflabel=""/></para>
+ <para></para>
<para>stripe size, minimum, <link xl:href="SettingUpLustreSystem.html#50438256_marker-1290730">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25365" xreflabel=""/>striping</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25366" xreflabel=""/></para>
+ <para>striping</para>
+ <para></para>
<para>managing free space, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1295520">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25368" xreflabel=""/></para>
+ <para></para>
<para>size, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1291859">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25370" xreflabel=""/></para>
+ <para></para>
<para>striping using llapi, <link xl:href="SettingLustreProperties_HTML.html#50438215_marker-1297700">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25372" xreflabel=""/>supported networks</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25373" xreflabel=""/></para>
+ <para>supported networks</para>
+ <para></para>
<para>Elan (Quadrics Elan), <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289889">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25375" xreflabel=""/></para>
+ <para></para>
<para>ra (RapidArray), <link xl:href="UnderstandingLustreNetworking.html#50438191_marker-1289887">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-25377" xreflabel=""/>T<anchor xml:id="dbdoclet.50438146_T" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25379" xreflabel=""/></para>
+ <title>T<anchor xml:id="dbdoclet.50438146_T" xreflabel=""/></title>
+ <para></para>
<para>timeouts, handling, <link xl:href="UserUtilities_HTML.html#50438206_marker-1306030">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25381" xreflabel=""/>troubleshooting</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25382" xreflabel=""/></para>
+ <para>troubleshooting</para>
+ <para></para>
<para>drawbacks in doing multi-client O_APPEND writes, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291520">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25384" xreflabel=""/></para>
+ <para></para>
<para>erasing a file system, <link xl:href="LustreOperations.html#50438194_marker-1307237">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25386" xreflabel=""/></para>
+ <para></para>
<para>error messages, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291323">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25388" xreflabel=""/></para>
+ <para></para>
<para>error numbers, <link xl:href="LustreTroubleshooting.html#50438198_marker-1296744">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25390" xreflabel=""/></para>
+ <para></para>
<para>handling timeouts on initial Lustre setup, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291503">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25392" xreflabel=""/>handling/debugging "bind</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25393" xreflabel=""/></para>
+ <para>handling/debugging "bind</para>
+ <para></para>
<para>address already in use" error, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291446">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25395" xreflabel=""/>handling/debugging "Lustre Error</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25396" xreflabel=""/></para>
+ <para>handling/debugging "Lustre Error</para>
+ <para></para>
<para>xxx went back in time", <link xl:href="LustreTroubleshooting.html#50438198_marker-1291509">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25398" xreflabel=""/></para>
+ <para></para>
<para>handling/debugging error "28", <link xl:href="LustreTroubleshooting.html#50438198_marker-1291470">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25400" xreflabel=""/></para>
+ <para></para>
<para>identifying a missing OST, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291366">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25402" xreflabel=""/></para>
+ <para></para>
<para>improving Lustre performance when working with small files, <link xl:href="LustreTuning.html#50438272_marker-1295851">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25404" xreflabel=""/></para>
+ <para></para>
<para>log message 'out of memory' on OST, <link xl:href="LustreTroubleshooting.html#50438198_marker-1292113">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25406" xreflabel=""/>Lustre Error</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25407" xreflabel=""/></para>
+ <para>Lustre Error</para>
+ <para></para>
<para>"slow start_page_write", <link xl:href="LustreTroubleshooting.html#50438198_marker-1291517">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25409" xreflabel=""/></para>
+ <para></para>
<para>OST object missing or damaged, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291349">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25411" xreflabel=""/></para>
+ <para></para>
<para>OSTs become read-only, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291361">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25413" xreflabel=""/></para>
+ <para></para>
<para>reclaiming reserved disk space, <link xl:href="LustreOperations.html#50438194_marker-1307251">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25415" xreflabel=""/></para>
+ <para></para>
<para>recovering from an unavailable OST, <link xl:href="TroubleShootingRecovery.html#50438225_marker-1292768">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25417" xreflabel=""/></para>
+ <para></para>
<para>replacing an existing OST or MDS, <link xl:href="LustreOperations.html#50438194_marker-1307278">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25419" xreflabel=""/></para>
+ <para></para>
<para>reporting a Lustre bug, <link xl:href="LustreTroubleshooting.html#50438198_marker-1296753">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25421" xreflabel=""/></para>
+ <para></para>
<para>setting SCSI I/O sizes, <link xl:href="LustreTroubleshooting.html#50438198_marker-1294800">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25423" xreflabel=""/></para>
+ <para></para>
<para>slowdown occurs during Lustre startup, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291921">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25425" xreflabel=""/></para>
+ <para></para>
<para>triggering watchdog for PID NNN, <link xl:href="LustreTroubleshooting.html#50438198_marker-1291480">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25427" xreflabel=""/>tunables</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25428" xreflabel=""/></para>
+ <para>tunables</para>
+ <para></para>
<para>RPC stream, <link xl:href="LustreProc.html#50438271_marker-1290514">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25430" xreflabel=""/></para>
+ <para></para>
<para>tunables, lockless, <link xl:href="LustreTuning.html#50438272_marker-1291703">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25432" xreflabel=""/></para>
+ <para></para>
<para>tunefs.lustre, <link xl:href="SystemConfigurationUtilities_HTML.html#50438219_marker-1317076">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25434" xreflabel=""/></para>
+ <para></para>
<para>setting parameters, <link xl:href="LustreOperations.html#50438194_marker-1305720">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25436" xreflabel=""/>tuning</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25437" xreflabel=""/></para>
+ <para>tuning</para>
+ <para></para>
<para>directory statahead, <link xl:href="LustreProc.html#50438271_marker-1295184">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25439" xreflabel=""/></para>
+ <para></para>
<para>file readahead, <link xl:href="LustreProc.html#50438271_marker-1295183">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25441" xreflabel=""/></para>
+ <para></para>
<para>lockless tunables, <link xl:href="LustreTuning.html#50438272_marker-1291703">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25443" xreflabel=""/></para>
+ <para></para>
<para>MDS threads, <link xl:href="LustreTuning.html#50438272_marker-1293755">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25445" xreflabel=""/></para>
+ <para></para>
<para>OSS threads, <link xl:href="LustreTuning.html#50438272_marker-1294858">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25447" xreflabel=""/></para>
+ <para></para>
<para>root squash, <link xl:href="ManagingSecurity.html#50438221_marker-1294611">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-25449" xreflabel=""/>U<anchor xml:id="dbdoclet.50438146_U" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25451" xreflabel=""/>upgrade</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25452" xreflabel=""/></para>
+ <title>U<anchor xml:id="dbdoclet.50438146_U" xreflabel=""/></title>
+ <para>upgrade</para>
+ <para></para>
<para>1.6.x to 1.8.x, <link xl:href="UpgradingLustre.html#50438205_marker-1294652">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25454" xreflabel=""/></para>
+ <para></para>
<para>complete file system, <link xl:href="UpgradingLustre.html#50438205_marker-1293327">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25456" xreflabel=""/>utilities, third-party</para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25457" xreflabel=""/></para>
+ <para>utilities, third-party</para>
+ <para></para>
<para>e2fsprogs, <link xl:href="InstallingLustre.html#50438261_marker-1292893">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25459" xreflabel=""/></para>
+ <para></para>
<para>Perl, <link xl:href="InstallingLustre.html#50438261_marker-1292899">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-25461" xreflabel=""/>V<anchor xml:id="dbdoclet.50438146_V" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25463" xreflabel=""/></para>
+ <title>V<anchor xml:id="dbdoclet.50438146_V" xreflabel=""/></title>
+ <para></para>
<para>VBR, introduction, <link xl:href="LustreRecovery.html#50438268_marker-1288580">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25465" xreflabel=""/></para>
+ <para></para>
<para>VBR, tips, <link xl:href="LustreRecovery.html#50438268_marker-1288584">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25467" xreflabel=""/></para>
+ <para></para>
<para>VBR, working with, <link xl:href="LustreRecovery.html#50438268_marker-1288583">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25469" xreflabel=""/></para>
+ <para></para>
<para>Version-based recovery (VBR), <link xl:href="LustreRecovery.html#50438268_marker-1288580">1</link></para>
</section>
<section remap="h3">
- <title>
<anchor xml:id="dbdoclet.50438146_pgfId-25471" xreflabel=""/>W<anchor xml:id="dbdoclet.50438146_W" xreflabel=""/></title>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25473" xreflabel=""/></para>
+ <title>W<anchor xml:id="dbdoclet.50438146_W" xreflabel=""/></title>
+ <para></para>
<para>weighted allocator, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1294020">1</link></para>
- <para><anchor xml:id="dbdoclet.50438146_pgfId-25475" xreflabel=""/></para>
+ <para></para>
<para>weighting, adjusting between free space and location, <link xl:href="ManagingStripingFreeSpace.html#50438209_marker-1294033">1</link></para>
<!--
Begin SiteCatalyst code version: G.5.
git://git.whamcloud.com / doc / manual.git / commitdiff