-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='backupandrestore'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="backupandrestore">
<info>
- <title xml:id='backupandrestore.title'>Backing Up and Restoring a File System</title>
+ <title xml:id="backupandrestore.title">Backing Up and Restoring a File System</title>
</info>
<para>Lustre provides backups at the file system-level, device-level and file-level. This chapter describes how to backup and restore on Lustre, and includes the following sections:</para>
- <itemizedlist><listitem>
- <para><xref linkend="dbdoclet.50438207_56395"/></para>
- </listitem>
- <listitem>
- <para><xref linkend="dbdoclet.50438207_71633"/></para>
- </listitem>
- <listitem>
- <para><xref linkend="dbdoclet.50438207_21638"/></para>
- </listitem>
- <listitem>
- <para><xref linkend="dbdoclet.50438207_22325"/></para>
- </listitem>
- <listitem>
- <para><xref linkend="dbdoclet.50438207_31553"/></para>
- </listitem>
-</itemizedlist>
-
- <section xml:id="dbdoclet.50438207_56395">
- <title>17.1 Backing up a File System</title>
- <para>Backing up a complete file system gives you full control over the files to back up, and allows restoration of individual files as needed. File system-level backups are also the easiest to integrate into existing backup solutions.</para>
- <para>File system backups are performed from a Lustre client (or many clients working parallel in different directories) rather than on individual server nodes; this is no different than backing up any other file system.</para>
- <para>However, due to the large size of most Lustre file systems, it is not always possible to get a complete backup. We recommend that you back up subsets of a file system. This includes subdirectories of the entire file system, filesets for a single user, files incremented by date, and so on.</para>
-
- <note><para>In order to allow Lustre to scale the filesystem namespace for future applications, Lustre 2.x internally uses a 128-bit file identifier for all files. To interface with user applications, Lustre presents 64-bit inode numbers for the stat(), fstat(), and readdir() system calls on 64-bit applications, and 32-bit inode numbers to 32-bit applications.</para><para> Some 32-bit applications accessing Lustre filesystems (on both 32-bit and 64-bit CPUs) may experience problems with the stat(), fstat() or readdir() system calls under certain circumstances, though the Lustre client should return 32-bit inode numbers to these applications.</para><para> In particular, if the Lustre filesystem is exported from a 64-bit client via NFS to a 32-bit client, the Linux NFS server will export 64-bit inode numbers to applications running on the NFS client. If the 32-bit applications are not compiled with Large File Support (LFS), then they return EOVERFLOW errors when accessing the Lustre files. To avoid this problem, Linux NFS clients can use the kernel command-line option "nfs.enable_ino64=0" in order to force the NFS client to export 32-bit inode numbers to the client.</para><para><emphasis role="bold">Workaround</emphasis>: We very strongly recommend that backups using tar(1) and other utilities that depend on the inode number to uniquely identify an inode to be run on 64-bit clients. The 128-bit Lustre file identifiers cannot be uniquely mapped to a 32-bit inode number, and as a result these utilities may operate incorrectly on 32-bit clients.</para></note>
-
- <section remap="h3">
- <title>17.1.1 Lustre_rsync</title>
- <para>The lustre_rsync feature keeps the entire file system in sync on a backup by replicating the file system's changes to a second file system (the second file system need not be a Lustre file system, but it must be sufficiently large). Lustre_rsync uses Lustre changelogs to efficiently synchronize the file systems without having to scan (directory walk) the Lustre file system. This efficiency is critically important for large file systems, and distinguishes the Lustre lustre_rsync feature from other replication/backup solutions.</para>
- <section remap="h4">
- <title>17.1.1.1 Using Lustre_rsync</title>
- <para>The lustre_rsync feature works by periodically running lustre_rsync, a userspace program used to synchronize changes in the Lustre file system onto the target file system. The lustre_rsync utility keeps a status file, which enables it to be safely interrupted and restarted without losing synchronization between the file systems.</para>
- <para>The first time that lustre_rsync is run, the user must specify a set of parameters for the program to use. These parameters are described in the following table and in <link xl:href="SystemConfigurationUtilities.html#50438219_63667">lustre_rsync</link>. On subsequent runs, these parameters are stored in the the status file, and only the name of the status file needs to be passed to lustre_rsync.</para>
- <para>Before using lustre_rsync:</para>
- <itemizedlist><listitem>
- <para> Register the changelog user. For details, see the <xref linkend='systemconfigurationutilities'/> (changelog_register) parameter in the <xref linkend='systemconfigurationutilities'/> (lctl).</para>
- </listitem>
-</itemizedlist>
- <para>- AND -</para>
- <itemizedlist><listitem>
- <para> Verify that the Lustre file system (source) and the replica file system (target) are identical <emphasis>before</emphasis> registering the changelog user. If the file systems are discrepant, use a utility, e.g. regular rsync (not lustre_rsync), to make them identical.</para>
- </listitem>
-</itemizedlist>
- <para>The lustre_rsync utility uses the following parameters:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="3*"/>
- <colspec colname="c2" colwidth="10*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> --source=<src></para></entry>
- <entry><para> The path to the root of the Lustre file system (source) which will be synchronized. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
- </row>
- <row>
- <entry><para> --target=<tgt></para></entry>
- <entry><para> The path to the root where the source file system will be synchronized (target). This is a mandatory option if the status log created during a previous synchronization operation (--statuslog) is not specified. This option can be repeated if multiple synchronization targets are desired.</para></entry>
- </row>
- <row>
- <entry><para> --mdt=<mdt></para></entry>
- <entry><para> The metadata device to be synchronized. A changelog user must be registered for this device. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
- </row>
- <row>
- <entry><para> --user=<user id></para></entry>
- <entry><para> The changelog user ID for the specified MDT. To use lustre_rsync, the changelog user must be registered. For details, see the changelog_register parameter in <xref linkend='systemconfigurationutilities'/> (lctl). This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
- </row>
- <row>
- <entry><para> --statuslog=<log></para></entry>
- <entry><para> A log file to which synchronization status is saved. When the lustre_rsync utility starts, if the status log from a previous synchronization operation is specified, then the state is read from the log and otherwise mandatory --source, --target and --mdt options can be skipped. Specifying the --source, --target and/or --mdt options, in addition to the --statuslog option, causes the specified parameters in the status log to be overriden. Command line options take precedence over options in the status log.</para></entry>
- </row>
- <row>
- <entry><para> --xattr <yes|no></para></entry>
- <entry><para> Specifies whether extended attributes (xattrs) are synchronized or not. The default is to synchronize extended attributes.</para><para><note><para>Disabling xattrs causes Lustre striping information not to be synchronized.</para></note></para></entry>
- </row>
- <row>
- <entry><para> --verbose</para></entry>
- <entry><para> Produces verbose output.</para></entry>
- </row>
- <row>
- <entry><para> --dry-run</para></entry>
- <entry><para> Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para></entry>
- </row>
- <row>
- <entry><para> --abort-on-err</para></entry>
- <entry><para> Stops processing the lustre_rsync operation if an error occurs. The default is to continue the operation.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h4">
- <title>17.1.1.2 lustre_rsync Examples</title>
- <para>Sample lustre_rsync commands are listed below.</para>
- <para>Register a changelog user for an MDT (e.g. lustre-MDT0000).</para>
- <screen># lctl --device lustre-MDT0000 changelog_register lustre-MDT0000 Registered\
- changelog userid 'cl1'
-</screen>
- <para>Synchronize a Lustre file system (/mnt/lustre) to a target file system (/mnt/target).</para>
- <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target --mdt=lustre-MDT00\
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438207_56395"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438207_71633"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438207_21638"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438207_22325"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438207_31553"/></para>
+ </listitem>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438207_56395">
+ <title>17.1 Backing up a File System</title>
+ <para>Backing up a complete file system gives you full control over the files to back up, and allows restoration of individual files as needed. File system-level backups are also the easiest to integrate into existing backup solutions.</para>
+ <para>File system backups are performed from a Lustre client (or many clients working parallel in different directories) rather than on individual server nodes; this is no different than backing up any other file system.</para>
+ <para>However, due to the large size of most Lustre file systems, it is not always possible to get a complete backup. We recommend that you back up subsets of a file system. This includes subdirectories of the entire file system, filesets for a single user, files incremented by date, and so on.</para>
+ <note>
+ <para>In order to allow Lustre to scale the filesystem namespace for future applications, Lustre 2.x internally uses a 128-bit file identifier for all files. To interface with user applications, Lustre presents 64-bit inode numbers for the <literal>stat()</literal>, <literal>fstat()</literal>, and <literal>readdir()</literal> system calls on 64-bit applications, and 32-bit inode numbers to 32-bit applications.</para>
+ <para> Some 32-bit applications accessing Lustre filesystems (on both 32-bit and 64-bit CPUs) may experience problems with the <literal>stat()</literal>, <literal>fstat()</literal> or<literal> readdir()</literal> system calls under certain circumstances, though the Lustre client should return 32-bit inode numbers to these applications.</para>
+ <para> In particular, if the Lustre filesystem is exported from a 64-bit client via NFS to a 32-bit client, the Linux NFS server will export 64-bit inode numbers to applications running on the NFS client. If the 32-bit applications are not compiled with Large File Support (LFS), then they return <literal>EOVERFLOW</literal> errors when accessing the Lustre files. To avoid this problem, Linux NFS clients can use the kernel command-line option "<literal>nfs.enable_ino64=0</literal>" in order to force the NFS client to export 32-bit inode numbers to the client.</para>
+ <para><emphasis role="bold">Workaround</emphasis>: We very strongly recommend that backups using <literal>tar(1)</literal> and other utilities that depend on the inode number to uniquely identify an inode to be run on 64-bit clients. The 128-bit Lustre file identifiers cannot be uniquely mapped to a 32-bit inode number, and as a result these utilities may operate incorrectly on 32-bit clients.</para>
+ </note>
+ <section remap="h3">
+ <title>17.1.1 Lustre_rsync</title>
+ <para>The <literal>lustre_rsync </literal>feature keeps the entire file system in sync on a backup by replicating the file system's changes to a second file system (the second file system need not be a Lustre file system, but it must be sufficiently large). <literal>lustre_rsync </literal>uses Lustre changelogs to efficiently synchronize the file systems without having to scan (directory walk) the Lustre file system. This efficiency is critically important for large file systems, and distinguishes the Lustre <literal>lustre_rsync</literal> feature from other replication/backup solutions.</para>
+ <section remap="h4">
+ <title>17.1.1.1 Using Lustre_rsync</title>
+ <para>The <literal>lustre_rsync</literal> feature works by periodically running <literal>lustre_rsync</literal>, a userspace program used to synchronize changes in the Lustre file system onto the target file system. The <literal>lustre_rsync</literal> utility keeps a status file, which enables it to be safely interrupted and restarted without losing synchronization between the file systems.</para>
+ <para>The first time that <literal>lustre_rsync</literal> is run, the user must specify a set of parameters for the program to use. These parameters are described in the following table and in <xref linkend="dbdoclet.50438219_63667"/>. On subsequent runs, these parameters are stored in the the status file, and only the name of the status file needs to be passed to <literal>lustre_rsync</literal>.</para>
+ <para>Before using <literal>lustre_rsync</literal>:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Register the changelog user. For details, see the <xref linkend="systemconfigurationutilities"/> (<literal>changelog_register</literal>) parameter in the <xref linkend="systemconfigurationutilities"/> (<literal>lctl</literal>).</para>
+ </listitem>
+ </itemizedlist>
+ <para>- AND -</para>
+ <itemizedlist>
+ <listitem>
+ <para>Verify that the Lustre file system (source) and the replica file system (target) are identical <emphasis>before</emphasis> registering the changelog user. If the file systems are discrepant, use a utility, e.g. regular <literal>rsync</literal> (not <literal>lustre_rsync</literal>), to make them identical.</para>
+ </listitem>
+ </itemizedlist>
+ <para>The <literal>lustre_rsync</literal> utility uses the following parameters:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="3*"/>
+ <colspec colname="c2" colwidth="10*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>--source=<src></literal></para>
+ </entry>
+ <entry>
+ <para>The path to the root of the Lustre file system (source) which will be synchronized. This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statuslog</literal>) is not specified.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--target=<tgt></literal></para>
+ </entry>
+ <entry>
+ <para>The path to the root where the source file system will be synchronized (target). This is a mandatory option if the status log created during a previous synchronization operation (<literal>--statuslog</literal>) is not specified. This option can be repeated if multiple synchronization targets are desired.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--mdt=<mdt></literal></para>
+ </entry>
+ <entry>
+ <para>The metadata device to be synchronized. A changelog user must be registered for this device. This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statuslog</literal>) is not specified.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--user=<user id></literal></para>
+ </entry>
+ <entry>
+ <para> The changelog user ID for the specified MDT. To use <literal>lustre_rsync</literal>, the changelog user must be registered. For details, see the <literal>changelog_register</literal> parameter in <xref linkend="systemconfigurationutilities"/> (<literal>lctl</literal>). This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statusl</literal>) is not specified.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--statuslog=<log></literal></para>
+ </entry>
+ <entry>
+ <para>A log file to which synchronization status is saved. When the <literal>lustre_rsync</literal> utility starts, if the status log from a previous synchronization operation is specified, then the state is read from the log and otherwise mandatory <literal>--source</literal>, <literal>--target</literal> and <literal>--mdt</literal> options can be skipped. Specifying the <literal>--source</literal>, <literal>--target</literal> and/or <literal>--mdt</literal> options, in addition to the <literal>--statuslog</literal> option, causes the specified parameters in the status log to be overriden. Command line options take precedence over options in the status log.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--xattr <yes|no></para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Specifies whether extended attributes (<literal>xattrs</literal>) are synchronized or not. The default is to synchronize extended attributes.</para>
+ <para><note>
+ <para>Disabling xattrs causes Lustre striping information not to be synchronized.</para>
+ </note></para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--verbose</literal></para>
+ </entry>
+ <entry>
+ <para> Produces verbose output.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--dry-run</literal></para>
+ </entry>
+ <entry>
+ <para> Shows the output of <literal>lustre_rsync</literal> commands (<literal>copy</literal>, <literal>mkdir</literal>, etc.) on the target file system without actually executing them.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--abort-on-err</literal></para>
+ </entry>
+ <entry>
+ <para> Stops processing the <literal>lustre_rsync</literal> operation if an error occurs. The default is to continue the operation.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h4">
+ <title>17.1.1.2 lustre_rsync Examples</title>
+ <para>Sample <literal>lustre_rsync</literal> commands are listed below.</para>
+ <para>Register a changelog user for an MDT (e.g. <literal>lustre-MDT0000</literal>).</para>
+ <screen># lctl --device lustre-MDT0000 changelog_register lustre-MDT0000 Registered\
+ changelog userid 'cl1'</screen>
+ <para>Synchronize a Lustre file system (<literal>/mnt/lustre</literal>) to a target file system (<literal>/mnt/target</literal>).</para>
+ <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target --mdt=lustre-MDT00\
00 --user=cl1 --statuslog sync.log --verbose
Lustre filesystem: lustre
MDT device: lustre-MDT0000
Starting changelog record: 0
Errors: 0
lustre_rsync took 1 seconds
-Changelog records consumed: 22
-</screen>
- <para>After the file system undergoes changes, synchronize the changes onto the target file system. Only the statuslog name needs to be specified, as it has all the parameters passed earlier.</para>
- <screen>$ lustre_rsync --statuslog sync.log --verbose
+Changelog records consumed: 22</screen>
+ <para>After the file system undergoes changes, synchronize the changes onto the target file system. Only the <literal>statuslog</literal> name needs to be specified, as it has all the parameters passed earlier.</para>
+ <screen>$ lustre_rsync --statuslog sync.log --verbose
Replicating Lustre filesystem: lustre
MDT device: lustre-MDT0000
Source: /mnt/lustre
Starting changelog record: 22
Errors: 0
lustre_rsync took 2 seconds
-Changelog records consumed: 42
-</screen>
- <para>To synchronize a Lustre file system (/mnt/lustre) to two target file systems (/mnt/target1 and /mnt/target2).</para>
- <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target1 --target=/mnt/tar\
+Changelog records consumed: 42</screen>
+ <para>To synchronize a Lustre file system (<literal>/mnt/lustre</literal>) to two target file systems (<literal>/mnt/target1</literal> and <literal>/mnt/target2</literal>).</para>
+ <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target1 --target=/mnt/tar\
get2 \
--mdt=lustre-MDT0000 --user=cl1
- --statuslog sync.log
-</screen>
- </section>
+ --statuslog sync.log</screen>
</section>
</section>
- <section xml:id="dbdoclet.50438207_71633">
- <title>17.2 Backing Up and Restoring an MDS or OST (Device Level)</title>
- <para>In some cases, it is useful to do a full device-level backup of an individual device (MDT or OST), before replacing hardware, performing maintenance, etc. Doing full device-level backups ensures that all of the data and configuration files is preserved in the original state and is the easiest method of doing a backup. For the MDT file system, it may also be the fastest way to perform the backup and restore, since it can do large streaming read and write operations at the maximum bandwidth of the underlying devices.</para>
-
- <note><para>Keeping an updated full backup of the MDT is especially important because a permanent failure of the MDT file system renders the much larger amount of data in all the OSTs largely inaccessible and unusable.</para></note>
-
- <note><para>In Lustre 2.0 and 2.1 the only correct way to perform an MDT backup and restore is to do a device-level backup as is described in this section. The ability to do MDT file-level backups is not functional in these releases because of the inability to restore the Object Index (OI) file correctly (see bug 22741 for details).</para></note>
-
- <para>If hardware replacement is the reason for the backup or if a spare storage device is available, it is possible to do a raw copy of the MDT or OST from one block device to the other, as long as the new device is at least as large as the original device. To do this, run:</para>
- <screen>dd if=/dev/{original} of=/dev/{new} bs=1M
-</screen>
- <para>If hardware errors cause read problems on the original device, use the command below to allow as much data as possible to be read from the original device while skipping sections of the disk with errors:</para>
- <screen>dd if=/dev/{original} of=/dev/{new} bs=4k conv=sync,noerror count={original\
- size in 4kB blocks}
-</screen>
- <para>Even in the face of hardware errors, the ldiskfs file system is very robust and it may be possible to recover the file system data after running e2fsck -f on the new device.</para>
- </section>
- <section xml:id="dbdoclet.50438207_21638">
- <title>17.3 Making a File-Level Backup of an OST File System</title>
- <para>This procedure provides another way to backup or migrate the data of an OST at the file level, so that the unused space of the OST does not need to be backed up. Backing up a single OST device is not necessarily the best way to perform backups of the Lustre file system, since the files stored in the backup are not usable without metadata stored on the MDT. However, it is the preferred method for migration of OST devices, especially when it is desirable to reformat the underlying file system with different configuration options or to reduce fragmentation.</para>
-
- <note><para>In Lustre 2.0 and 2.1 the only correct way to perform an MDT backup and restore is to do a device-level backup as is described in this section. The ability to do MDT file-level backups is not functional in these releases because of the inability to restore the Object Index (OI) file correctly (see bug 22741 for details).</para></note>
- <orderedlist><listitem>
- <para>Make a mountpoint for the file system.</para>
- <screen>[oss]# mkdir -p /mnt/ost
-</screen>
-</listitem><listitem>
- <para>Mount the file system.</para>
- <screen>[oss]# mount -t ldiskfs /<emphasis>dev</emphasis>/{ostdev} /mnt/ost
-</screen>
-</listitem><listitem>
- <para>Change to the mountpoint being backed up.</para>
- <screen>[oss]# cd /mnt/ost
-</screen>
-</listitem><listitem>
- <para>Back up the extended attributes.</para>
- <screen>[oss]# getfattr -R -d -m '.*' -e hex -P . > ea-$(date +%Y%m%d).bak
-</screen>
- <note><para>If the tar(1) command supports the --xattr option, the getfattr step may be unnecessary as long as it does a backup of the "trusted" attributes. However, completing this step is not harmful and can serve as an added safety measure.</para></note>
- <note><para>In most distributions, the getfattr command is part of the "attr" package. If the getfattr command returns errors like Operation not supported, then the kernel does not correctly support EAs. Stop and use a different backup method.</para></note>
-
-</listitem><listitem>
- <para>Verify that the ea-$date.bak file has properly backed up the EA data on the OST.</para>
- <para>Without this attribute data, the restore process may be missing extra data that can be very useful in case of later file system corruption. Look at this file with more or a text editor. Each object file should hae a corresponding item similar to this:</para>
- <screen>[oss]# file: O/0/d0/100992
+ </section>
+ <section xml:id="dbdoclet.50438207_71633">
+ <title>17.2 Backing Up and Restoring an MDS or OST (Device Level)</title>
+ <para>In some cases, it is useful to do a full device-level backup of an individual device (MDT or OST), before replacing hardware, performing maintenance, etc. Doing full device-level backups ensures that all of the data and configuration files is preserved in the original state and is the easiest method of doing a backup. For the MDT file system, it may also be the fastest way to perform the backup and restore, since it can do large streaming read and write operations at the maximum bandwidth of the underlying devices.</para>
+ <note>
+ <para>Keeping an updated full backup of the MDT is especially important because a permanent failure of the MDT file system renders the much larger amount of data in all the OSTs largely inaccessible and unusable.</para>
+ </note>
+ <note>
+ <para>In Lustre 2.0 and 2.1 the only correct way to perform an MDT backup and restore is to do a device-level backup as is described in this section. The ability to do MDT file-level backups is not functional in these releases because of the inability to restore the Object Index (OI) file correctly (see bug 22741 for details).</para>
+ </note>
+ <para>If hardware replacement is the reason for the backup or if a spare storage device is available, it is possible to do a raw copy of the MDT or OST from one block device to the other, as long as the new device is at least as large as the original device. To do this, run:</para>
+ <screen>dd if=/dev/{original} of=/dev/{new} bs=1M</screen>
+ <para>If hardware errors cause read problems on the original device, use the command below to allow as much data as possible to be read from the original device while skipping sections of the disk with errors:</para>
+ <screen>dd if=/dev/{original} of=/dev/{new} bs=4k conv=sync,noerror count={original\
+ size in 4kB blocks}</screen>
+ <para>Even in the face of hardware errors, the <literal>ldiskfs</literal> file system is very robust and it may be possible to recover the file system data after running <literal>e2fsck -f</literal> on the new device.</para>
+ </section>
+ <section xml:id="dbdoclet.50438207_21638">
+ <title>17.3 Making a File-Level Backup of an OST File System</title>
+ <para>This procedure provides another way to backup or migrate the data of an OST at the file level, so that the unused space of the OST does not need to be backed up. Backing up a single OST device is not necessarily the best way to perform backups of the Lustre file system, since the files stored in the backup are not usable without metadata stored on the MDT. However, it is the preferred method for migration of OST devices, especially when it is desirable to reformat the underlying file system with different configuration options or to reduce fragmentation.</para>
+ <note>
+ <para>In Lustre 2.0 and 2.1 the only correct way to perform an MDT backup and restore is to do a device-level backup as is described in this section. The ability to do MDT file-level backups is not functional in these releases because of the inability to restore the Object Index (OI) file correctly (see bug 22741 for details).</para>
+ </note>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Make a mountpoint for the file system.</emphasis></para>
+ <screen>[oss]# mkdir -p /mnt/ost</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Mount the file system.</emphasis></para>
+ <screen>[oss]# mount -t ldiskfs /<emphasis>dev</emphasis>/{ostdev} /mnt/ost</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Change to the mountpoint being backed up.</emphasis></para>
+ <screen>[oss]# cd /mnt/ost</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Back up the extended attributes.</emphasis></para>
+ <screen>[oss]# getfattr -R -d -m '.*' -e hex -P . > ea-$(date +%Y%m%d).bak</screen>
+ <note>
+ <para>If the <literal>tar(1)</literal> command supports the <literal>--xattr</literal> option, the <literal>getfattr</literal> step may be unnecessary as long as it does a backup of the "trusted" attributes. However, completing this step is not harmful and can serve as an added safety measure.</para>
+ </note>
+ <note>
+ <para>In most distributions, the <literal>getfattr</literal> command is part of the "<literal>attr</literal>" package. If the <literal>getfattr</literal> command returns errors like <literal>Operation not supported</literal>, then the kernel does not correctly support EAs. Stop and use a different backup method.</para>
+ </note>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Verify that the <literal>ea-$date.bak</literal> file has properly backed up the EA data on the OST.</emphasis></para>
+ <para>Without this attribute data, the restore process may be missing extra data that can be very useful in case of later file system corruption. Look at this file with more or a text editor. Each object file should have a corresponding item similar to this:</para>
+ <screen>[oss]# file: O/0/d0/100992
trusted.fid= \
-0x0d822200000000004a8a73e500000000808a0100000000000000000000000000
-</screen>
-</listitem><listitem>
- <para>Back up all file system data.</para>
- <screen>[oss]# tar czvf {backup file}.tgz --sparse .
-</screen>
- <note><para>In Lustre 1.6.7 and later, the --sparse option reduces the size of the backup file. Be sure to use it so the tar command does not mistakenly create an archive full of zeros.</para></note>
-
-</listitem><listitem>
- <para>Change directory out of the file system.</para>
- <screen>[oss]# cd -
-</screen>
-</listitem><listitem>
- <para>Unmount the file system.</para>
- <screen>[oss]# umount /mnt/ost
-</screen>
- <note><para>When restoring an OST backup on a different node as part of an OST migration, you also have to change server NIDs and use the --writeconf command to re-generate the configuration logs. See <xref linkend='lustremaintenance'/> (Changing a Server NID).</para></note>
-
-
-
- </listitem></orderedlist>
- </section>
- <section xml:id="dbdoclet.50438207_22325">
- <title>17.4 Restoring a File-Level Backup</title>
-
<para>To restore data from a file-level backup, you need to format the device, restore the file data and then restore the EA data.</para>
-
- <orderedlist><listitem>
- <para>Format the new device.</para>
- <screen>[oss]# mkfs.lustre --ost --index {<emphasis>OST index</emphasis>} {<emphasis>other options</emphasis>} newdev}
-</screen>
-</listitem><listitem>
- <para>Mount the file system.</para>
- <screen>[oss]# mount -t ldiskfs {<emphasis>newdev</emphasis>} /mnt/ost
-</screen>
-</listitem><listitem>
- <para>Change to the new file system mount point.</para>
- <screen>[oss]# cd /mnt/ost
-</screen>
-</listitem><listitem>
- <para>Restore the file system backup.</para>
- <screen>[oss]# tar xzvpf {<emphasis>backup file</emphasis>} --sparse
-</screen>
-</listitem><listitem>
- <para>Restore the file system extended attributes.</para>
- <screen>[oss]# setfattr --restore=ea-${date}.bak
-</screen>
-</listitem><listitem>
- <para>Verify that the extended attributes were restored.</para>
- <screen>[oss]# getfattr -d -m ".*" -e hex O/0/d0/100992 trusted.fid= \
-0x0d822200000000004a8a73e500000000808a0100000000000000000000000000
-</screen>
-</listitem><listitem>
- <para>Change directory out of the file system.</para>
- <screen>[oss]# cd -
-</screen>
-</listitem><listitem>
- <para>Unmount the new file system.</para>
- <screen>[oss]# umount /mnt/ost
-
-</screen>
-</listitem></orderedlist>
- <para>If the file system was used between the time the backup was made and when it was restored, then the lfsck tool (part of Lustre e2fsprogs) can optionally be run to ensure the file system is coherent. If all of the device file systems were backed up at the same time after the entire Lustre file system was stopped, this is not necessary. In either case, the file system should be immediately usable even if lfsck is not run, though there may be I/O errors reading from files that are present on the MDT but not the OSTs, and files that were created after the MDT backup will not be accessible/visible.</para>
- </section>
- <section xml:id="dbdoclet.50438207_31553">
- <title>17.5 Using LVM Snapshots with Lustre</title>
- <para>If you want to perform disk-based backups (because, for example, access to the backup system needs to be as fast as to the primary Lustre file system), you can use the Linux LVM snapshot tool to maintain multiple, incremental file system backups.</para>
- <para>Because LVM snapshots cost CPU cycles as new files are written, taking snapshots of the main Lustre file system will probably result in unacceptable performance losses. You should create a new, backup Lustre file system and periodically (e.g., nightly) back up new/changed files to it. Periodic snapshots can be taken of this backup file system to create a series of "full" backups.</para>
-
- <note><para>Creating an LVM snapshot is not as reliable as making a separate backup, because the LVM snapshot shares the same disks as the primary MDT device, and depends on the primary MDT device for much of its data. If the primary MDT device becomes corrupted, this may result in the snapshot being corrupted.</para></note>
-
- <section remap="h3">
- <title>17.5.1 Creating an LVM-based Backup File System</title>
- <para>Use this procedure to create a backup Lustre file system for use with the LVM snapshot mechanism.</para>
- <orderedlist><listitem>
- <para>Create LVM volumes for the MDT and OSTs.</para>
- <para>Create LVM devices for your MDT and OST targets. Make sure not to use the entire disk for the targets; save some room for the snapshots. The snapshots start out as 0 size, but grow as you make changes to the current file system. If you expect to change 20% of the file system between backups, the most recent snapshot will be 20% of the target size, the next older one will be 40%, etc. Here is an example:</para>
- <screen>cfs21:~# pvcreate /dev/sda1
+0x0d822200000000004a8a73e500000000808a0100000000000000000000000000</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Back up all file system data.</emphasis></para>
+ <screen>[oss]# tar czvf {backup file}.tgz --sparse .</screen>
+ <note>
+ <para>In Lustre 1.6.7 and later, the <literal>--sparse</literal> option reduces the size of the backup file. Be sure to use it so the tar command does not mistakenly create an archive full of zeros.</para>
+ </note>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Change directory out of the file system.</emphasis></para>
+ <screen>[oss]# cd -</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Unmount the file system.</emphasis></para>
+ <screen>[oss]# umount /mnt/ost</screen>
+ <note>
+ <para>When restoring an OST backup on a different node as part of an OST migration, you also have to change server NIDs and use the --writeconf command to re-generate the configuration logs. See <xref linkend="lustremaintenance"/> (Changing a Server NID).</para>
+ </note>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section xml:id="dbdoclet.50438207_22325">
+ <title>17.4 Restoring a File-Level Backup</title>
+ <para>To restore data from a file-level backup, you need to format the device, restore the file data and then restore the EA data.</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Format the new device.</emphasis></para>
+ <screen>[oss]# mkfs.lustre --ost --index {<emphasis>OST index</emphasis>} {<emphasis>other options</emphasis>} newdev}</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Mount the file system.</emphasis></para>
+ <screen>[oss]# mount -t ldiskfs {<emphasis>newdev</emphasis>} /mnt/ost</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Change to the new file system mount point.</emphasis></para>
+ <screen>[oss]# cd /mnt/ost</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Restore the file system backup.</emphasis></para>
+ <screen>[oss]# tar xzvpf {<emphasis>backup file</emphasis>} --sparse</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Restore the file system extended attributes.</emphasis></para>
+ <screen>[oss]# setfattr --restore=ea-${date}.bak</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Verify that the extended attributes were restored.</emphasis></para>
+ <screen>[oss]# getfattr -d -m ".*" -e hex O/0/d0/100992 trusted.fid= \
+0x0d822200000000004a8a73e500000000808a0100000000000000000000000000</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Change directory out of the file system.</emphasis></para>
+ <screen>[oss]# cd -</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Unmount the new file system.</emphasis></para>
+ <screen>[oss]# umount /mnt/ost</screen>
+ </listitem>
+ </orderedlist>
+ <para>If the file system was used between the time the backup was made and when it was restored, then the <literal>lfsck</literal> tool (part of Lustre <literal>e2fsprogs</literal>) can optionally be run to ensure the file system is coherent. If all of the device file systems were backed up at the same time after the entire Lustre file system was stopped, this is not necessary. In either case, the file system should be immediately usable even if <literal>lfsck</literal> is not run, though there may be I/O errors reading from files that are present on the MDT but not the OSTs, and files that were created after the MDT backup will not be accessible/visible.</para>
+ </section>
+ <section xml:id="dbdoclet.50438207_31553">
+ <title>17.5 Using LVM Snapshots with Lustre</title>
+ <para>If you want to perform disk-based backups (because, for example, access to the backup system needs to be as fast as to the primary Lustre file system), you can use the Linux LVM snapshot tool to maintain multiple, incremental file system backups.</para>
+ <para>Because LVM snapshots cost CPU cycles as new files are written, taking snapshots of the main Lustre file system will probably result in unacceptable performance losses. You should create a new, backup Lustre file system and periodically (e.g., nightly) back up new/changed files to it. Periodic snapshots can be taken of this backup file system to create a series of "full" backups.</para>
+ <note>
+ <para>Creating an LVM snapshot is not as reliable as making a separate backup, because the LVM snapshot shares the same disks as the primary MDT device, and depends on the primary MDT device for much of its data. If the primary MDT device becomes corrupted, this may result in the snapshot being corrupted.</para>
+ </note>
+ <section remap="h3">
+ <title>17.5.1 Creating an LVM-based Backup File System</title>
+ <para>Use this procedure to create a backup Lustre file system for use with the LVM snapshot mechanism.</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Create LVM volumes for the MDT and OSTs.</emphasis></para>
+
<para>Create LVM devices for your MDT and OST targets. Make sure not to use the entire disk for the targets; save some room for the snapshots. The snapshots start out as 0 size, but grow as you make changes to the current file system. If you expect to change 20% of the file system between backups, the most recent snapshot will be 20% of the target size, the next older one will be 40%, etc. Here is an example:</para>
+ <screen>cfs21:~# pvcreate /dev/sda1
Physical volume "/dev/sda1" successfully created
cfs21:~# vgcreate volgroup /dev/sda1
Volume group "volgroup" successfully created
Logical volume "OST0" created
cfs21:~# lvscan
ACTIVE '/dev/volgroup/MDT' [200.00 MB] inherit
- ACTIVE '/dev/volgroup/OST0' [200.00 MB] inherit
-</screen>
-</listitem><listitem>
- <para>Format the LVM volumes as Lustre targets.</para>
- <para>In this example, the backup file system is called 'main' and designates the current, most up-to-date backup.</para>
- <screen>cfs21:~# mkfs.lustre --mdt --fsname=main /dev/volgroup/MDT
+ ACTIVE '/dev/volgroup/OST0' [200.00 MB] inherit</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Format the LVM volumes as Lustre targets.</emphasis></para>
+ <para>In this example, the backup file system is called 'main' and designates the current, most up-to-date backup.</para>
+ <screen>cfs21:~# mkfs.lustre --mdt --fsname=main /dev/volgroup/MDT
No management node specified, adding MGS to this MDT.
Permanent disk data:
Target: main-MDTffff
Writing CONFIGS/mountdata
cfs21:~# mount -t lustre /dev/volgroup/MDT /mnt/mdt
cfs21:~# mount -t lustre /dev/volgroup/OST0 /mnt/ost
-cfs21:~# mount -t lustre cfs21:/main /mnt/main
-</screen>
-</listitem></orderedlist>
- </section>
- <section remap="h3">
- <title>17.5.2 Backing up New/Changed Files to the Backup File System</title>
-
<para>At periodic intervals e.g., nightly, back up new and changed files to the LVM-based backup file system.</para>
- <screen>cfs21:~# cp /etc/passwd /mnt/main
+cfs21:~# mount -t lustre cfs21:/main /mnt/main</screen>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section remap="h3">
+ <title>17.5.2 Backing up New/Changed Files to the Backup File System</title>
+ <para>At periodic intervals e.g., nightly, back up new and changed files to the LVM-based backup file system.</para>
+ <screen>cfs21:~# cp /etc/passwd /mnt/main
cfs21:~# cp /etc/fstab /mnt/main
cfs21:~# ls /mnt/main
-fstab passwd
-</screen>
- </section>
- <section remap="h3">
- <title>17.5.3 Creating Snapshot Volumes</title>
- <para>Whenever you want to make a "checkpoint" of the main Lustre file system, create LVM snapshots of all target MDT and OSTs in the LVM-based backup file system. You must decide the maximum size of a snapshot ahead of time, although you can dynamically change this later. The size of a daily snapshot is dependent on the amount of data changed daily in the main Lustre file system. It is likely that a two-day old snapshot will be twice as big as a one-day old snapshot.</para>
- <para>You can create as many snapshots as you have room for in the volume group. If necessary, you can dynamically add disks to the volume group.</para>
- <para>The snapshots of the target MDT and OSTs should be taken at the same point in time. Make sure that the cronjob updating the backup file system is not running, since that is the only thing writing to the disks. Here is an example:</para>
- <screen>cfs21:~# modprobe dm-snapshot
+fstab passwd</screen>
+ </section>
+ <section remap="h3">
+ <title>17.5.3 Creating Snapshot Volumes</title>
+ <para>Whenever you want to make a "checkpoint" of the main Lustre file system, create LVM snapshots of all target MDT and OSTs in the LVM-based backup file system. You must decide the maximum size of a snapshot ahead of time, although you can dynamically change this later. The size of a daily snapshot is dependent on the amount of data changed daily in the main Lustre file system. It is likely that a two-day old snapshot will be twice as big as a one-day old snapshot.</para>
+ <para>You can create as many snapshots as you have room for in the volume group. If necessary, you can dynamically add disks to the volume group.</para>
+ <para>The snapshots of the target MDT and OSTs should be taken at the same point in time. Make sure that the cronjob updating the backup file system is not running, since that is the only thing writing to the disks. Here is an example:</para>
+ <screen>cfs21:~# modprobe dm-snapshot
cfs21:~# lvcreate -L50M -s -n MDTb1 /dev/volgroup/MDT
Rounding up size to full physical extent 52.00 MB
Logical volume "MDTb1" created
cfs21:~# lvcreate -L50M -s -n OSTb1 /dev/volgroup/OST0
Rounding up size to full physical extent 52.00 MB
- Logical volume "OSTb1" created
-</screen>
- <para>After the snapshots are taken, you can continue to back up new/changed files to "main". The snapshots will not contain the new files.</para>
- <screen>cfs21:~# cp /etc/termcap /mnt/main
+ Logical volume "OSTb1" created</screen>
+ <para>After the snapshots are taken, you can continue to back up new/changed files to "main". The snapshots will not contain the new files.</para>
+ <screen>cfs21:~# cp /etc/termcap /mnt/main
cfs21:~# ls /mnt/main
-fstab passwd termcap
-</screen>
- </section>
- <section remap="h3">
- <title>17.5.4 Restoring the File System From a Snapshot</title>
- <para>Use this procedure to restore the file system from an LVM snapshot.</para>
- <orderedlist><listitem>
- <para>Rename the LVM snapshot.</para>
- <para>Rename the file system snapshot from "main" to "back" so you can mount it without unmounting "main". This is recommended, but not required. Use the --reformat flag to tunefs.lustre to force the name change. For example:</para>
- <screen>cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/volgroup/M\
+fstab passwd termcap</screen>
+ </section>
+ <section remap="h3">
+ <title>17.5.4 Restoring the File System From a Snapshot</title>
+ <para>Use this procedure to restore the file system from an LVM snapshot.</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Rename the LVM snapshot.</emphasis></para>
+ <para>Rename the file system snapshot from "main" to "back" so you can mount it without unmounting "main". This is recommended, but not required. Use the <literal>--reformat</literal> flag to <literal>tunefs.lustre</literal> to force the name change. For example:</para>
+ <screen>cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/volgroup/M\
DTb1
checking for existing Lustre data
found Lustre data
cfs21:~# umount /mnt/mdtback
cfs21:~# mount -t ldiskfs /dev/volgroup/OSTb1 /mnt/ostback
cfs21:~# rm /mnt/ostback/last_rcvd
-cfs21:~# umount /mnt/ostback
-</screen>
-</listitem><listitem>
- <para>Mount the file system from the LVM snapshot.</para>
- <para>For example:</para>
- <screen>cfs21:~# mount -t lustre /dev/volgroup/MDTb1 /mnt/mdtback \
+cfs21:~# umount /mnt/ostback</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Mount the file system from the LVM snapshot.</emphasis></para>
+
<para>For example:</para>
+ <screen>cfs21:~# mount -t lustre /dev/volgroup/MDTb1 /mnt/mdtback \
cfs21:~# mount -t lustre /dev/volgroup/OSTb1 /mnt/ostback
-cfs21:~# mount -t lustre cfs21:/back /mnt/back
-</screen>
-</listitem><listitem>
- <para> 3. Note the old directory contents, as of the snapshot time.</para>
- <para>For example:</para>
- <screen>cfs21:~/cfs/b1_5/lustre/utils# ls /mnt/back
-fstab passwds
-</screen>
-</listitem></orderedlist>
-
- </section>
- <section remap="h3">
- <title>17.5.5 Deleting Old Snapshots</title>
- <para>To reclaim disk space, you can erase old snapshots as your backup policy dictates. Run:</para>
- <screen>lvremove /dev/volgroup/MDTb1
-</screen>
- </section>
- <section remap="h3">
- <title>17.5.6 Changing Snapshot Volume Size</title>
- <para>You can also extend or shrink snapshot volumes if you find your daily deltas are smaller or larger than expected. Run:</para>
- <screen>lvextend -L10G /dev/volgroup/MDTb1
-</screen>
- <note><para>Extending snapshots seems to be broken in older LVM. It is working in LVM v2.02.01.</para></note>
- </section>
+cfs21:~# mount -t lustre cfs21:/back /mnt/back</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Note the old directory contents, as of the snapshot time.</emphasis></para>
+ <para>For example:</para>
+ <screen>cfs21:~/cfs/b1_5/lustre/utils# ls /mnt/back
+fstab passwds</screen>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section remap="h3">
+ <title>17.5.5 Deleting Old Snapshots</title>
+ <para>To reclaim disk space, you can erase old snapshots as your backup policy dictates. Run:</para>
+ <screen>lvremove /dev/volgroup/MDTb1</screen>
+ </section>
+ <section remap="h3">
+ <title>17.5.6 Changing Snapshot Volume Size</title>
+ <para>You can also extend or shrink snapshot volumes if you find your daily deltas are smaller or larger than expected. Run:</para>
+ <screen>lvextend -L10G /dev/volgroup/MDTb1</screen>
+ <note>
+ <para>Extending snapshots seems to be broken in older LVM. It is working in LVM v2.02.01.</para>
+ </note>
+ </section>
</section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='benchmarkingtests'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="benchmarkingtests">
<info>
- <title xml:id='benchmarkingtests.title'>Benchmarking Lustre Performance (Lustre I/O Kit)</title>
+ <title xml:id="benchmarkingtests.title">Benchmarking Lustre Performance (Lustre I/O Kit)</title>
</info>
-
<para>This chapter describes the Lustre I/O kit, a collection of I/O benchmarking tools for a Lustre cluster, and PIOS, a parallel I/O simulator for Linux and Solaris. It includes:</para>
- <itemizedlist><listitem>
- <para><xref linkend="dbdoclet.50438212_44437"/></para>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438212_44437"/></para>
</listitem>
-
-<listitem>
- <para><xref linkend="dbdoclet.50438212_51053"/></para>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438212_51053"/></para>
</listitem>
-
-<listitem>
- <para><xref linkend="dbdoclet.50438212_26516"/></para>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438212_26516"/></para>
</listitem>
-
-<listitem>
- <para><xref linkend="dbdoclet.50438212_85136"/></para>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438212_85136"/></para>
</listitem>
-
-<listitem>
- <para><xref linkend="dbdoclet.50438212_58201"/></para>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438212_58201"/></para>
</listitem>
-
-</itemizedlist>
-
- <section xml:id="dbdoclet.50438212_44437">
- <title>24.1 Using Lustre I/O Kit Tools</title>
- <para>The tools in the Lustre I/O Kit are used to benchmark Lustre hardware and validate that it is working as expected before you install the Lustre software. It can also be used to to validate the performance of the various hardware and software layers in the cluster and also to find and troubleshoot I/O issues.</para>
- <para>Typically, performance is measured starting with single raw devices and then proceeding to groups of devices. Once raw performance has been established, other software layers are then added incrementally and tested.</para>
- <section remap="h3">
- <title>24.1.1 Contents of the Lustre I/O Kit</title>
- <para>The I/O kit contains three tests, each of which tests a progressively higher layer in the Lustre stack:</para>
- <itemizedlist><listitem>
- <para>sgpdd_survey - Measure basic 'bare metal' performance of devices while bypassing the kernel block device layers, buffer cache, and file system.</para>
- </listitem>
-
-<listitem>
- <para>obdfilter_survey - Measure the performance of one or more OSTs directly on the OSS node or alternately over the network from a Lustre client.</para>
- </listitem>
-
-<listitem>
- <para>ost_survey - Performs I/O against OSTs individually to allow performance comparisons to detect if an OST is performing suboptimally due to hardware issues.</para>
- </listitem>
-
-</itemizedlist>
- <para>Typically with these tests, Lustre should deliver 85-90% of the raw device performance.</para>
- <para>A utility stats-collect is also provided to collect application profiling information from Lustre clients and servers. See <link xl:href="BenchmarkingTests.html#50438212_58201">Collecting Application Profiling Information (stats-collect)</link> for more information.</para>
- </section>
- <section remap="h3">
- <title>24.1.2 Preparing to Use the Lustre <anchor xml:id="dbdoclet.50438212_marker-1289913" xreflabel=""/>I/O Kit</title>
- <para>The following prerequisites must be met to use the tests in the Lustre I/O kit:</para>
- <itemizedlist><listitem>
- <para> Password-free remote access to nodes in the system (provided by ssh or rsh).</para>
- </listitem>
-
-<listitem>
- <para> LNET self-test completed to test that Lustre Networking has been properly installed and configured. See <xref linkend='lnetselftest'/>.</para>
- </listitem>
-
-<listitem>
- <para> Lustre file system software installed.</para>
- </listitem>
-
-<listitem>
- <para>sg3_utils package providing the sgp_dd tool (sg3_utils is a separate RPM package available online using YUM).</para>
- </listitem>
-
-</itemizedlist>
- <para>Download the Lustre I/O kit (lustre-iokit)from:</para>
- <para><link xl:href="http://downloads.lustre.org/public/tools/lustre-iokit/">http://downloads.lustre.org/public/tools/lustre-iokit/</link></para>
- </section>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438212_44437">
+ <title>24.1 Using Lustre I/O Kit Tools</title>
+ <para>The tools in the Lustre I/O Kit are used to benchmark Lustre hardware and validate that it is working as expected before you install the Lustre software. It can also be used to to validate the performance of the various hardware and software layers in the cluster and also to find and troubleshoot I/O issues.</para>
+ <para>Typically, performance is measured starting with single raw devices and then proceeding to groups of devices. Once raw performance has been established, other software layers are then added incrementally and tested.</para>
+ <section remap="h3">
+ <title>24.1.1 Contents of the Lustre I/O Kit</title>
+ <para>The I/O kit contains three tests, each of which tests a progressively higher layer in the Lustre stack:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>sgpdd_survey</literal> - Measure basic 'bare metal' performance of devices while bypassing the kernel block device layers, buffer cache, and file system.</para>
+ </listitem>
+ <listitem>
+ <para><literal>obdfilter_survey</literal> - Measure the performance of one or more OSTs directly on the OSS node or alternately over the network from a Lustre client.</para>
+ </listitem>
+ <listitem>
+ <para><literal>ost_survey</literal> - Performs I/O against OSTs individually to allow performance comparisons to detect if an OST is performing suboptimally due to hardware issues.</para>
+ </listitem>
+ </itemizedlist>
+ <para>Typically with these tests, Lustre should deliver 85-90% of the raw device performance.</para>
+ <para>A utility <literal>stats-collect</literal> is also provided to collect application profiling information from Lustre clients and servers. See <xref linkend="dbdoclet.50438212_58201">Collecting Application Profiling Information (stats-collect)</xref> for more information.</para>
</section>
- <section xml:id="dbdoclet.50438212_51053">
- <title>24.2 Testing I/O Performance of Raw Hardware (sgpdd_survey<anchor xml:id="dbdoclet.50438212_marker-1302844" xreflabel=""/>)</title>
- <para>The sgpdd_survey tool is used to test bare metal I/O performance of the raw hardware, while bypassing as much of the kernel as possible. This survey may be used to characterize the performance of a SCSI device by simulating an OST serving multiple stripe files. The data gathered by this survey can help set expectations for the performance of a Lustre OST using this device.</para>
- <para>The script uses sgp_dd to carry out raw sequential disk I/O. It runs with variable numbers of sgp_dd threads to show how performance varies with different request queue depths.</para>
- <para>The script spawns variable numbers of sgp_dd instances, each reading or writing a separate area of the disk to demonstrate performance variance within a number of concurrent stripe files.</para>
- <para>Several tips and insights for disk performance measurement are described below. Some of this information is specific to RAID arrays and/or the Linux RAID implementation.</para>
- <itemizedlist><listitem>
- <para><emphasis>Performance is limited by the slowest disk.</emphasis></para>
- </listitem>
-
-</itemizedlist>
- <para>Before creating a RAID array, benchmark all disks individually. We have frequently encountered situations where drive performance was not consistent for all devices in the array. Replace any disks that are significantly slower than the rest.</para>
- <itemizedlist><listitem>
- <para><emphasis>Disks and arrays are very sensitive to request size.</emphasis></para>
- </listitem>
-
-</itemizedlist>
- <para>To identify the optimal request size for a given disk, benchmark the disk with different record sizes ranging from 4 KB to 1 to 2 MB.</para>
-
- <caution><para>The sgpdd_survey script overwrites the device being tested, which results in the <emphasis>LOSS OF ALL DATA</emphasis> on that device. Exercise caution when selecting the device to be tested.</para></caution>
-
- <note><para>Array performance with all LUNs loaded does not always match the performance of a single LUN when tested in isolation.</para></note>
-
- <para></para>
- <para><emphasis role="bold">Prequisites:</emphasis></para>
- <itemizedlist><listitem>
- <para>sgp_dd tool in the sg3_utils package</para>
- </listitem>
-
-<listitem>
- <para> Lustre software is <emphasis>NOT</emphasis> required</para>
- </listitem>
-
-</itemizedlist>
- <para>The device(s) being tested must meet one of these two requirements:</para>
- <itemizedlist><listitem>
- <para> If the device is a SCSI device, it must appear in the output of sg_map (make sure the kernel module sg is loaded).</para>
- </listitem>
-
-<listitem>
- <para> If the device is a raw device, it must appear in the output of raw -qa.</para>
- </listitem>
-
-</itemizedlist>
- <para>Raw and SCSI devices cannot be mixed in the test specification.</para>
-
- <note><para>If you need to create raw devices to use the sgpdd_survey tool, note that raw device 0 cannot be used due to a bug in certain versions of the "raw" utility (including that shipped with RHEL4U4.)</para></note>
-
- <section remap="h3">
- <title>24.2.1 Tuning Linux Storage Devices</title>
- <para>To get large I/O transfers (1 MB) to disk, it may be necessary to tune several kernel parameters as specified:</para>
- <screen>/sys/block/sdN/queue/max_sectors_kb = 4096
+ <section remap="h3">
+ <title>24.1.2 Preparing to Use the Lustre I/O Kit</title>
+ <para>The following prerequisites must be met to use the tests in the Lustre I/O kit:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Password-free remote access to nodes in the system (provided by <literal>ssh</literal> or <literal>rsh</literal>).</para>
+ </listitem>
+ <listitem>
+ <para>LNET self-test completed to test that Lustre Networking has been properly installed and configured. See <xref linkend="lnetselftest"/>.</para>
+ </listitem>
+ <listitem>
+ <para>Lustre file system software installed.</para>
+ </listitem>
+ <listitem>
+ <para><literal>sg3_utils</literal> package providing the <literal>sgp_dd</literal> tool (<literal>sg3_utils</literal> is a separate RPM package available online using YUM).</para>
+ </listitem>
+ </itemizedlist>
+ <para>Download the Lustre I/O kit (<literal>lustre-iokit</literal>)from:</para>
+ <para><ulink xl:href="http://downloads.whamcloud.com/">http://downloads.whamcloud.com/</ulink></para>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438212_51053">
+ <title>24.2 Testing I/O Performance of Raw Hardware (<literal>sgpdd_survey</literal>)</title>
+ <para>The <literal>sgpdd_survey</literal> tool is used to test bare metal I/O performance of the raw hardware, while bypassing as much of the kernel as possible. This survey may be used to characterize the performance of a SCSI device by simulating an OST serving multiple stripe files. The data gathered by this survey can help set expectations for the performance of a Lustre OST using this device.</para>
+ <para>The script uses <literal>sgp_dd</literal> to carry out raw sequential disk I/O. It runs with variable numbers of <literal>sgp_dd</literal> threads to show how performance varies with different request queue depths.</para>
+ <para>The script spawns variable numbers of <literal>sgp_dd</literal> instances, each reading or writing a separate area of the disk to demonstrate performance variance within a number of concurrent stripe files.</para>
+ <para>Several tips and insights for disk performance measurement are described below. Some of this information is specific to RAID arrays and/or the Linux RAID implementation.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis>Performance is limited by the slowest disk.</emphasis></para>
+ <para>Before creating a RAID array, benchmark all disks individually. We have frequently encountered situations where drive performance was not consistent for all devices in the array. Replace any disks that are significantly slower than the rest.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis>Disks and arrays are very sensitive to request size.</emphasis></para>
+ <para>To identify the optimal request size for a given disk, benchmark the disk with different record sizes ranging from 4 KB to 1 to 2 MB.</para>
+ </listitem>
+ </itemizedlist>
+ <caution>
+ <para>The <literal>sgpdd_survey</literal> script overwrites the device being tested, which results in the <emphasis>
+ <emphasis role="bold">LOSS OF ALL DATA</emphasis>
+ </emphasis> on that device. Exercise caution when selecting the device to be tested.</para>
+ </caution>
+ <note>
+ <para>Array performance with all LUNs loaded does not always match the performance of a single LUN when tested in isolation.</para>
+ </note>
+ <para><emphasis role="bold">Prequisites:</emphasis></para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>sgp_dd</literal> tool in the <literal>sg3_utils</literal> package</para>
+ </listitem>
+ <listitem>
+ <para>Lustre software is <emphasis>NOT</emphasis> required</para>
+ </listitem>
+ </itemizedlist>
+ <para>The device(s) being tested must meet one of these two requirements:</para>
+ <itemizedlist>
+ <listitem>
+ <para>If the device is a SCSI device, it must appear in the output of <literal>sg_map</literal> (make sure the kernel module <literal>sg</literal> is loaded).</para>
+ </listitem>
+ <listitem>
+ <para>If the device is a raw device, it must appear in the output of <literal>raw -qa</literal>.</para>
+ </listitem>
+ </itemizedlist>
+ <para>Raw and SCSI devices cannot be mixed in the test specification.</para>
+ <note>
+ <para>If you need to create raw devices to use the <literal>sgpdd_survey</literal> tool, note that raw device 0 cannot be used due to a bug in certain versions of the "raw" utility (including that shipped with RHEL4U4.)</para>
+ </note>
+ <section remap="h3">
+ <title>24.2.1 Tuning Linux Storage Devices</title>
+ <para>To get large I/O transfers (1 MB) to disk, it may be necessary to tune several kernel parameters as specified:</para>
+ <screen>/sys/block/sdN/queue/max_sectors_kb = 4096
/sys/block/sdN/queue/max_phys_segments = 256
/proc/scsi/sg/allow_dio = 1
/sys/module/ib_srp/parameters/srp_sg_tablesize = 255
/sys/block/sdN/queue/scheduler</screen>
- </section>
- <section remap="h3">
- <title>24.2.2 Running sgpdd_survey</title>
- <para>The sgpdd_survey script must be customized for the particular device being tested and for the location where the script saves its working and result files (by specifying the ${rslt} variable). Customization variables are described at the beginning of the script.</para>
- <para>When the sgpdd_survey script runs, it creates a number of working files and a pair of result files. The names of all the files created start with the prefixdefined in the variable ${rslt}. (The default value is /tmp.) The files include:</para>
- <itemizedlist><listitem>
- <para> File containing standard output data (same as stdout)</para>
- </listitem>
-
-</itemizedlist>
- <screen>${rslt}_<emphasis><date/time></emphasis>.summary<emphasis/></screen>
- <itemizedlist><listitem>
- <para> Temporary (tmp) files</para>
- </listitem>
-
-</itemizedlist>
- <screen>${rslt}_<emphasis><date/time></emphasis>_*
+ </section>
+ <section remap="h3">
+ <title>24.2.2 Running sgpdd_survey</title>
+ <para>The <literal>sgpdd_survey</literal> script must be customized for the particular device being tested and for the location where the script saves its working and result files (by specifying the <literal>${rslt}</literal> variable). Customization variables are described at the beginning of the script.</para>
+ <para>When the <literal>sgpdd_survey</literal> script runs, it creates a number of working files and a pair of result files. The names of all the files created start with the prefixdefined in the variable <literal>${rslt}</literal>. (The default value is <literal>/tmp</literal>.) The files include:</para>
+ <itemizedlist>
+ <listitem>
+ <para>File containing standard output data (same as <literal>stdout</literal>)</para>
+ <screen>${rslt}_<emphasis><date/time></emphasis>.summary</screen>
+ </listitem>
+ <listitem>
+ <para> Temporary (tmp) files</para>
+ <screen>${rslt}_<emphasis><date/time></emphasis>_*
</screen>
- <itemizedlist><listitem>
- <para> Collected tmp files for post-mortem</para>
- </listitem>
-
-</itemizedlist>
- <screen>${rslt}_<emphasis><date/time></emphasis>.detail
+ </listitem>
+ <listitem>
+ <para> Collected tmp files for post-mortem</para>
+ <screen>${rslt}_<emphasis><date/time></emphasis>.detail
</screen>
- <para>The stdout and the .summary file will contain lines like this:</para>
- <screen>total_size 8388608K rsz 1024 thr 1 crg 1 180.45 MB/s 1 x 180.50 \=/ 180.50 \
+ </listitem>
+ </itemizedlist>
+ <para>The <literal>stdout</literal> and the <literal>.summary</literal> file will contain lines like this:</para>
+ <screen>total_size 8388608K rsz 1024 thr 1 crg 1 180.45 MB/s 1 x 180.50 \=/ 180.50 \
MB/s
</screen>
- <para>Each line corresponds to a run of the test. Each test run will have a different number of threads, record size, or number of regions.</para>
- <itemizedlist><listitem>
- <para>total_size - Size of file being tested in KBs (8 GB in above example).</para>
- </listitem>
-
-<listitem>
- <para>rsz - Record size in KBs (1 MB in above example).</para>
- </listitem>
-
-<listitem>
- <para>thr - Number of threads generating I/O (1 thread in above example).</para>
- </listitem>
-
-<listitem>
- <para> crg - Current regions, the number of disjount areas on the disk to which I/O is being sent (1 region in above example, indicating that no seeking is done).</para>
- </listitem>
-
-<listitem>
- <para>MB/s - Aggregate bandwidth measured by dividing the total amount of data by the elapsed time (180.45 MB/s in the above example).</para>
- </listitem>
-
-<listitem>
- <para>MB/s - The remaining numbers show the number of regions X performance of the slowest disk as a sanity check on the aggregate bandwidth.</para>
- </listitem>
-
-</itemizedlist>
- <para>If there are so many threads that the sgp_dd script is unlikely to be able to allocate I/O buffers, then ENOMEM is printed in place of the aggregate bandwidth result.</para>
- <para>If one or more sgp_dd instances do not successfully report a bandwidth number, then FAILED is printed in place of the aggregate bandwidth result.</para>
- </section>
+ <para>Each line corresponds to a run of the test. Each test run will have a different number of threads, record size, or number of regions.</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>total_size</literal> - Size of file being tested in KBs (8 GB in above example).</para>
+ </listitem>
+ <listitem>
+ <para><literal>rsz</literal> - Record size in KBs (1 MB in above example).</para>
+ </listitem>
+ <listitem>
+ <para><literal>thr</literal> - Number of threads generating I/O (1 thread in above example).</para>
+ </listitem>
+ <listitem>
+ <para><literal>crg</literal> - Current regions, the number of disjount areas on the disk to which I/O is being sent (1 region in above example, indicating that no seeking is done).</para>
+ </listitem>
+ <listitem>
+ <para><literal>MB/s</literal> - Aggregate bandwidth measured by dividing the total amount of data by the elapsed time (180.45 MB/s in the above example).</para>
+ </listitem>
+ <listitem>
+ <para><literal>MB/s</literal> - The remaining numbers show the number of regions X performance of the slowest disk as a sanity check on the aggregate bandwidth.</para>
+ </listitem>
+ </itemizedlist>
+ <para>If there are so many threads that the <literal>sgp_dd</literal> script is unlikely to be able to allocate I/O buffers, then <literal>ENOMEM</literal> is printed in place of the aggregate bandwidth result.</para>
+ <para>If one or more <literal>sgp_dd</literal> instances do not successfully report a bandwidth number, then <literal>FAILED</literal> is printed in place of the aggregate bandwidth result.</para>
</section>
- <section xml:id="dbdoclet.50438212_26516">
- <title>24.3 <anchor xml:id="dbdoclet.50438212_40624" xreflabel=""/>Testing OST Performance (obdfilter_survey<anchor xml:id="dbdoclet.50438212_marker-1289957" xreflabel=""/>)</title>
- <para>The obdfilter_survey script generates sequential I/O from varying numbers of threads and objects (files) to simulate the I/O patterns of a Lustre client.</para>
- <para>The obdfilter_survey script can be run directly on the OSS node to measure the OST storage performance without any intervening network, or it can be run remotely on a Lustre client to measure the OST performance including network overhead.</para>
- <para>The obdfilter_survey is used to characterize the performance of the following:</para>
- <itemizedlist><listitem>
- <para><emphasis role="bold">Local file system</emphasis> - In this mode, the obdfilter_survey script exercises one or more instances of the obdfilter directly. The script may run on one or more OSS nodes, for example, when the OSSs are all attached to the same multi-ported disk subsystem.</para>
- </listitem>
-
-</itemizedlist>
- <para>Run the script using the case=disk parameter to run the test against all the local OSTs. The script automatically detects all local OSTs and includes them in the survey.</para>
- <para>To run the test against only specific OSTs, run the script using the target= parameter to list the OSTs to be tested explicitly. If some OSTs are on remote nodes, specify their hostnames in addition to the OST name (for example, oss2:lustre-OST0004).</para>
- <para>All obdfilter instances are driven directly. The script automatically loads the obdecho module (if required) and creates one instance of echo_client for each obdfilter instance in order to generate I/O requests directly to the OST.</para>
- <para>For more details, see <link xl:href="BenchmarkingTests.html#50438212_59319">Testing Local Disk Performance</link>.</para>
- <itemizedlist><listitem>
- <para><emphasis role="bold">Network</emphasis> - In this mode, the Lustre client generates I/O requests over the network but these requests are not sent to the OST file system. The OSS node runs the obdecho server to receive the requests but discards them before they are sent to the disk.</para>
- </listitem>
-
-</itemizedlist>
- <para>Pass the parameters case=network and target=<emphasis><hostname</emphasis>|<emphasis>IP_of_server></emphasis> to the script. For each network case, the script does the required setup.</para>
- <para>For more details, see <link xl:href="BenchmarkingTests.html#50438212_36037">Testing Network Performance</link></para>
- <itemizedlist><listitem>
- <para><emphasis role="bold">Remote file system over the network</emphasis> - In this mode the obdfilter_survey script generates I/O from a Lustre client to a remote OSS to write the data to the file system.</para>
- </listitem>
-
-</itemizedlist>
+ </section>
+ <section xml:id="dbdoclet.50438212_26516">
+ <title>24.3 Testing OST Performance (<literal>obdfilter_survey</literal>)</title>
+ <para>The <literal>obdfilter_survey</literal> script generates sequential I/O from varying numbers of threads and objects (files) to simulate the I/O patterns of a Lustre client.</para>
+ <para>The <literal>obdfilter_survey</literal> script can be run directly on the OSS node to measure the OST storage performance without any intervening network, or it can be run remotely on a Lustre client to measure the OST performance including network overhead.</para>
+ <para>The <literal>obdfilter_survey</literal> is used to characterize the performance of the following:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">Local file system</emphasis> - In this mode, the <literal>obdfilter_survey</literal> script exercises one or more instances of the obdfilter directly. The script may run on one or more OSS nodes, for example, when the OSSs are all attached to the same multi-ported disk subsystem.</para>
+ <para>Run the script using the <literal>case=disk</literal> parameter to run the test against all the local OSTs. The script automatically detects all local OSTs and includes them in the survey.</para>
+ <para>To run the test against only specific OSTs, run the script using the <literal>target=parameter</literal> to list the OSTs to be tested explicitly. If some OSTs are on remote nodes, specify their hostnames in addition to the OST name (for example, <literal>oss2:lustre-OST0004</literal>).</para>
+ <para>All <literal>obdfilter</literal> instances are driven directly. The script automatically loads the <literal>obdecho</literal> module (if required) and creates one instance of <literal>echo_client</literal> for each <literal>obdfilter</literal> instance in order to generate I/O requests directly to the OST.</para>
+ <para>For more details, see <xref linkend="dbdoclet.50438212_59319">Testing Local Disk Performance</xref>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Network</emphasis> - In this mode, the Lustre client generates I/O requests over the network but these requests are not sent to the OST file system. The OSS node runs the obdecho server to receive the requests but discards them before they are sent to the disk.</para>
+ <para>Pass the parameters <literal>case=network</literal> and <literal>target=<emphasis><hostname</emphasis>|<emphasis>IP_of_server></emphasis></literal> to the script. For each network case, the script does the required setup.</para>
+ <para>For more details, see <xref linkend="dbdoclet.50438212_36037">Testing Network Performance</xref></para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Remote file system over the network</emphasis> - In this mode the <literal>obdfilter_survey</literal> script generates I/O from a Lustre client to a remote OSS to write the data to the file system.</para>
+ </listitem>
<para>To run the test against all the local OSCs, pass the parameter case=netdisk to the script. Alternately you can pass the target= parameter with one or more OSC devices (e.g., lustre-OST0000-osc-ffff88007754bc00) against which the tests are to be run.</para>
- <para>For more details, see <link xl:href="BenchmarkingTests.html#50438212_62662">Testing Remote Disk Performance</link>.</para>
- <caution><para>The obdfilter_survey script is destructive and should not be run on devices that containing existing data that needs to be preserved. Thus, tests using obdfilter_survey should be run before the Lustre file system is placed in production.</para></caution>
-
- <note><para>If the obdfilter_survey test is terminated before it completes, some small amount of space is leaked. you can either ignore it or reformat the file system.</para></note>
- <note><para>The obdfilter_survey script is <emphasis>NOT</emphasis> scalable beyond tens of OSTs since it is only intended to measure the I/O performance of individual storage subsystems, not the scalability of the entire system.</para></note>
-
- <note><para>The obdfilter_survey script must be customized, depending on the components under test and where the script's working files should be kept. Customization variables are described at the beginning of the obdfilter_survey script. In particular, pay attention to the listed maximum values listed for each parameter in the script.</para></note>
-
- <section remap="h3">
- <title>24.3.1 <anchor xml:id="dbdoclet.50438212_59319" xreflabel=""/>Testing Local Disk Performance</title>
- <para>The obdfilter_survey script can be run automatically or manually against a local disk. This script profiles the overall throughput of storage hardware, including the file system and RAID layers managing the storage, by sending workloads to the OSTs that vary in thread count, object count, and I/O size.</para>
- <para>When the obdfilter_survey script is run, it provides information about the performance abilities of the storage hardware and shows the saturation points.</para>
- <para>The plot-obdfilter script generates from the output of the obdfilter_survey a CSV file and parameters for importing into a spreadsheet or gnuplot to visualize the data.</para>
- <para>To run the obdfilter_survey script, create a standard Lustre configuration; no special setup is needed.</para>
- <para><emphasis role="bold">To perform an automatic run:</emphasis></para>
- <orderedlist><listitem>
- <para> 1. Start the Lustre OSTs.</para>
- <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
- </listitem><listitem>
- <para> 2. Verify that the obdecho module is loaded. Run:</para>
- <screen>modprobe obdecho</screen>
- </listitem><listitem>
- <para> 3. Run the obdfilter_survey script with the parameter case=disk.</para>
- <para>For example, to run a local test with up to two objects (nobjhi), up to two threads (thrhi), and 1024 MB transfer size (size):</para>
- <screen>$ nobjhi=2 thrhi=2 size=1024 case=disk sh obdfilter-survey
-
-</screen>
- </listitem></orderedlist>
- <para><emphasis role="bold">To perform a manual run:</emphasis></para>
- <orderedlist><listitem>
- <para> 1. Start the Lustre OSTs.</para>
- <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
- </listitem><listitem>
- <para> 2. Verify that the obdecho module is loaded. Run:</para>
- <para>modprobe obdecho</para>
- </listitem><listitem>
- <para> 3. Determine the OST names.</para>
- <para>On the OSS nodes to be tested, run the lctldl command. The OST device names are listed in the fourth column of the output. For example:</para>
- <screen>$ lctl dl |grep obdfilter
+ <para>For more details, see <xref linkend="dbdoclet.50438212_62662">Testing Remote Disk Performance</xref>.</para>
+ </itemizedlist>
+ <caution>
+ <para>The <literal>obdfilter_survey</literal> script is destructive and should not be run on devices that containing existing data that needs to be preserved. Thus, tests using <literal>obdfilter_survey</literal> should be run before the Lustre file system is placed in production.</para>
+ </caution>
+ <note>
+ <para>If the <literal>obdfilter_survey</literal> test is terminated before it completes, some small amount of space is leaked. you can either ignore it or reformat the file system.</para>
+ </note>
+ <note>
+ <para>The <literal>obdfilter_survey</literal> script is <emphasis>NOT</emphasis> scalable beyond tens of OSTs since it is only intended to measure the I/O performance of individual storage subsystems, not the scalability of the entire system.</para>
+ </note>
+ <note>
+ <para>The <literal>obdfilter_survey</literal> script must be customized, depending on the components under test and where the script's working files should be kept. Customization variables are described at the beginning of the <literal>obdfilter_survey</literal> script. In particular, pay attention to the listed maximum values listed for each parameter in the script.</para>
+ </note>
+ <section xml:id='dbdoclet.50438212_59319'>
+ <title>24.3.1 Testing Local Disk Performance</title>
+ <para>The <literal>obdfilter_survey</literal> script can be run automatically or manually against a local disk. This script profiles the overall throughput of storage hardware, including the file system and RAID layers managing the storage, by sending workloads to the OSTs that vary in thread count, object count, and I/O size.</para>
+ <para>When the <literal>obdfilter_survey</literal> script is run, it provides information about the performance abilities of the storage hardware and shows the saturation points.</para>
+ <para>The <literal>plot-obdfilter</literal> script generates from the output of the <literal>obdfilter_survey</literal> a CSV file and parameters for importing into a spreadsheet or gnuplot to visualize the data.</para>
+ <para>To run the <literal>obdfilter_survey</literal> script, create a standard Lustre configuration; no special setup is needed.</para>
+ <para><emphasis role="bold">To perform an automatic run:</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Start the Lustre OSTs.</emphasis></para>
+ <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Verify that the obdecho module is loaded. Run:</emphasis></para>
+ <screen>modprobe obdecho</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Run the <literal>obdfilter_survey</literal> script with the parameter <literal>case=disk</literal>.</emphasis></para>
+ <para>For example, to run a local test with up to two objects (nobjhi), up to two threads (thrhi), and 1024 MB transfer size (size):</para>
+ <screen>$ nobjhi=2 thrhi=2 size=1024 case=disk sh obdfilter-survey</screen>
+ </listitem>
+ </orderedlist>
+ <para><emphasis role="italic">To perform a manual run:</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Start the Lustre OSTs.</emphasis></para>
+ <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Verify that the <literal>obdecho</literal> module is loaded. Run:</emphasis></para>
+ <screen><para>modprobe obdecho</para></screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Determine the OST names.</emphasis></para>
+ <para>On the OSS nodes to be tested, run the lctldl command. The OST device names are listed in the fourth column of the output. For example:</para>
+ <screen>$ lctl dl |grep obdfilter
0 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 1159
2 UP obdfilter lustre-OST0002 lustre-OST0002_UUID 1159
-...
-</screen>
- </listitem><listitem>
- <para> 4. List all OSTs you want to test.</para>
- <para>Use the target= parameter to list the OSTs separated by spaces. List the individual OSTs by name using the format <emphasis><fsname>-<OSTnumber></emphasis> (for example, lustre-OST0001). You do not have to specify an MDS or LOV.</para>
- </listitem><listitem>
- <para> 5. Run the obdfilter_survey script with the target= parameter.</para>
- <para>For example, to run a local test with up to two objects (nobjhi), up to two threads (thrhi), and 1024 Mb (size) transfer size:</para>
- <screen>$ nobjhi=2 thrhi=2 size=1024 targets='lustre-OST0001 \
-lustre-OST0002' sh obdfilter-survey
-</screen>
- </listitem></orderedlist>
- </section>
- <section remap="h3">
- <title>24.3.2 <anchor xml:id="dbdoclet.50438212_36037" xreflabel=""/>Testing Network Performance</title>
- <para>The obdfilter_survey script can only be run automatically against a network; no manual test is provided.</para>
- <para>To run the network test, a specific Lustre setup is needed. Make sure that these configuration requirements have been met.</para>
- <para><emphasis role="bold">To perform an automatic run:</emphasis></para>
- <orderedlist><listitem>
- <para> 1. Start the Lustre OSTs.</para>
- <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
- </listitem><listitem>
- <para> 2. Verify that the obdecho module is loaded. Run:</para>
- <screen>modprobe obdecho</screen>
- </listitem><listitem>
- <para> 3. Start lctl and check the device list, which must be empty. Run:</para>
- <screen>lctl dl
-</screen>
- </listitem><listitem>
- <para> 4. Run the obdfilter_survey script with the parameters case=network and <emphasis role="bold">targets=</emphasis><emphasis><hostname</emphasis>|<emphasis>ip_of_server></emphasis>. For example:</para>
- <screen>$ nobjhi=2 thrhi=2 size=1024 targets='oss1 oss2' case=network sh obdfilte\
-r-survey
-</screen>
- </listitem><listitem>
- <para> 5. On the server side, view the statistics at:</para>
- <screen>/proc/fs/lustre/obdecho/<emphasis><echo_srv></emphasis>/stats
-</screen>
- <para>where <emphasis><echo_srv></emphasis> is the obdecho server created by the script.</para>
- </listitem></orderedlist>
- </section>
- <section remap="h3">
- <title>24.3.3 <anchor xml:id="dbdoclet.50438212_62662" xreflabel=""/>Testing Remote Disk Performance</title>
- <para>The obdfilter_survey script can be run automatically or manually against a network disk. To run the network disk test, start with a standard Lustre configuration. No special setup is needed.</para>
- <para><emphasis role="bold">To perform an automatic run:</emphasis></para>
- <orderedlist><listitem>
- <para> 1. Start the Lustre OSTs.</para>
- <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
- </listitem><listitem>
- <para> 2. Verify that the obdecho module is loaded. Run:</para>
- <screen>modprobe obdecho</screen>
- </listitem><listitem>
- <para> 3. Run the obdfilter_survey script with the parameter case=netdisk. For example:</para>
- <screen>$ nobjhi=2 thrhi=2 size=1024 case=netdisk sh obdfilter-survey
+...</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">List all OSTs you want to test.</emphasis></para>
+ <para>Use the <literal>target=parameter</literal> to list the OSTs separated by spaces. List the individual OSTs by name using the format <emphasis>
+ <literal><fsname>-<OSTnumber></literal>
+ </emphasis> (for example, lustre-OST0001). You do not have to specify an MDS or LOV.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Run the <literal>obdfilter_survey</literal> script with the <literal>target=parameter</literal>.</emphasis></para>
+ <para>For example, to run a local test with up to two objects (<literal>nobjhi</literal>), up to two threads (<literal>thrhi</literal>), and 1024 Mb (size) transfer size:</para>
+ <screen>$ nobjhi=2 thrhi=2 size=1024 targets='lustre-OST0001 \
+lustre-OST0002' sh obdfilter-survey</screen>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section xml:id='dbdoclet.50438212_36037'>
+ <title>24.3.2 Testing Network Performance</title>
+ <para>The <literal>obdfilter_survey</literal> script can only be run automatically against a network; no manual test is provided.</para>
+ <para>To run the network test, a specific Lustre setup is needed. Make sure that these configuration requirements have been met.</para>
+ <para><emphasis role="bold">To perform an automatic run:</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Start the Lustre OSTs.</emphasis></para>
+ <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Verify that the <literal>obdecho</literal> module is loaded. Run:</emphasis></para>
+ <screen>modprobe obdecho</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Start <literal>lctl</literal> and check the device list, which must be empty. Run</emphasis>:</para>
+ <screen>lctl dl</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Run the <literal>obdfilter_survey</literal> script with the parameters <literal>case=network</literal> and <literal><emphasis role="bold">targets=</emphasis><emphasis><hostname</emphasis>|<emphasis>ip_of_server></emphasis></literal>. For example:</emphasis></para>
+ <screen>$ nobjhi=2 thrhi=2 size=1024 targets='oss1 oss2' case=network sh obdfilte\
+sh odbfilter-survey</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">On the server side, view the statistics at:</emphasis></para>
+ <screen>/proc/fs/lustre/obdecho/<emphasis><echo_srv></emphasis>/stats</screen>
+ <para>where <emphasis>
+ <literal><echo_srv></literal>
+ </emphasis> is the <literal>obdecho</literal> server created by the script.</para>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section xml:id='dbdoclet.50438212_62662'>
+ <title>24.3.3 Testing Remote Disk Performance</title>
+ <para>The <literal>obdfilter_survey</literal> script can be run automatically or manually against a network disk. To run the network disk test, start with a standard Lustre configuration. No special setup is needed.</para>
+ <para><emphasis role="bold">To perform an automatic run:</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Start the Lustre OSTs.</emphasis></para>
+ <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Verify that the <literal>obdecho</literal> module is loaded. Run:</emphasis></para>
+ <screen>modprobe obdecho</screen>
+ </listitem>
+ <listitem>
+ <para>Run the <literal>obdfilter_survey</literal> script with the parameter <literal>case=netdisk</literal>. For example:</para>
+ <screen>$ nobjhi=2 thrhi=2 size=1024 case=netdisk sh obdfilter-survey
</screen>
- </listitem></orderedlist>
- <para><emphasis role="bold">To perform a manual run:</emphasis></para>
- <orderedlist><listitem>
- <para> 1. Start the Lustre OSTs.</para>
- <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
- </listitem><listitem>
- <para> 2. Verify that the obdecho module is loaded. Run:</para>
- <para>modprobe obdecho</para>
- </listitem><listitem>
- <para> 3. Determine the OSC names.</para>
- <para>On the OSS nodes to be tested, run the lctldl command. The OSC device names are listed in the fourth column of the output. For example:</para>
- <screen>$ lctl dl |grep obdfilter
-3 UP osc lustre-OST0000-osc-ffff88007754bc00 54b91eab-0ea9-1516-b571-5e6df3\
-49592e 5
-4 UP osc lustre-OST0001-osc-ffff88007754bc00 54b91eab-0ea9-1516-b571-5e6df3\
-49592e 5
+ </listitem>
+ </orderedlist>
+ <para><emphasis role="bold">To perform a manual run:</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Start the Lustre OSTs.</emphasis></para>
+ <para>The Lustre OSTs should be mounted on the OSS node(s) to be tested. The Lustre client is not required to be mounted at this time.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Verify that the <literal>obdecho</literal> module is loaded. Run:</emphasis></para>
+ <para>modprobe obdecho</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Determine the OSC names.</emphasis></para>
+ <para>On the OSS nodes to be tested, run the <literal>lctl dl</literal> command. The OSC device names are listed in the fourth column of the output. For example:</para>
+ <screen>$ lctl dl |grep obdfilter
+3 UP osc lustre-OST0000-osc-ffff88007754bc00 \
+ 54b91eab-0ea9-1516-b571-5e6df349592e 5
+4 UP osc lustre-OST0001-osc-ffff88007754bc00 \
+ 54b91eab-0ea9-1516-b571-5e6df349592e 5
...
</screen>
- </listitem><listitem>
- <para> 4. List all OSCs you want to test.</para>
- <para>Use the target= parameter to list the OSCs separated by spaces. List the individual OSCs by name seperated by spaces using the format <emphasis><fsname>-<OST_name></emphasis>-osc-<emphasis><OSC_number></emphasis> (for example, lustre-OST0000-osc-ffff88007754bc00). You <emphasis role="bold">do not have to specify an MDS or LOV.</emphasis></para>
- </listitem><listitem>
- <para> 5. Run the <emphasis role="bold">o</emphasis>bdfilter_survey script with the target= parameter and case=netdisk.</para>
- <para>An example of a local test run with up to two objects (nobjhi), up to two threads (thrhi), and 1024 Mb (size) transfer size is shown below:</para>
- <screen>$ nobjhi=2 thrhi=2 size=1024 \
-targets="lustre-OST0000-osc-ffff88007754bc00 \
-lustre-OST0001-osc-ffff88007754bc00" \
-sh obdfilter-survey
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">List all OSCs you want to test.</emphasis></para>
+ <para>Use the <literal>target=parameter</literal> to list the OSCs separated by spaces. List the individual OSCs by name seperated by spaces using the format <literal><emphasis><fsname>-<OST_name></emphasis>-osc-<emphasis><OSC_number></emphasis></literal> (for example, <literal>lustre-OST0000-osc-ffff88007754bc00</literal>). You <emphasis role="bold">do not have to specify an MDS or LOV.</emphasis></para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Run the <literal><emphasis role="bold">o</emphasis>bdfilter_survey</literal> script with the <literal>target=parameter</literal> and <literal>case=netdisk</literal>.</emphasis></para>
+ <para>An example of a local test run with up to two objects (<literal>nobjhi</literal>), up to two threads (<literal>thrhi</literal>), and 1024 Mb (size) transfer size is shown below:</para>
+ <screen>$ nobjhi=2 thrhi=2 size=1024 \
+ targets="lustre-OST0000-osc-ffff88007754bc00 \
+ lustre-OST0001-osc-ffff88007754bc00" \
+ sh obdfilter-survey
+</screen>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section remap="h3">
+ <title>24.3.4 Output Files</title>
+ <para>When the <literal>obdfilter_survey</literal> script runs, it creates a number of working files and a pair of result files. All files start with the prefix defined in the variable <literal>${rslt}</literal>.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">File</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>${rslt}.summary</literal></para>
+ </entry>
+ <entry>
+ <para> Same as stdout</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>${rslt}.script_*</literal></para>
+ </entry>
+ <entry>
+ <para> Per-host test script files</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>${rslt}.detail_tmp*</literal></para>
+ </entry>
+ <entry>
+ <para> Per-OST result files</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>${rslt}.detail</literal></para>
+ </entry>
+ <entry>
+ <para> Collected result files for post-mortem</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para>The <literal>obdfilter_survey</literal> script iterates over the given number of threads and objects performing the specified tests and checks that all test processes have completed successfully.</para>
+ <note>
+ <para>The <literal>obdfilter_survey</literal> script may not clean up properly if it is aborted or if it encounters an unrecoverable error. In this case, a manual cleanup may be required, possibly including killing any running instances of <literal>lctl</literal> (local or remote), removing <literal>echo_client</literal> instances created by the script and unloading <literal>obdecho</literal>.</para>
+ </note>
+ <section remap="h4">
+ <title>24.3.4.1 Script Output</title>
+ <para>The <literal>.summary</literal> file and <literal>stdout</literal> of the <literal>obdfilter_survey</literal> script contain lines like:</para>
+ <screen>ost 8 sz 67108864K rsz 1024 obj 8 thr 8 write 613.54 [ 64.00, 82.00]
</screen>
- </listitem></orderedlist>
- </section>
- <section remap="h3">
- <title>24.3.4 Output Files</title>
- <para>When the obdfilter_survey script runs, it creates a number of working files and a pair of result files. All files start with the prefix defined in the variable ${rslt}.</para>
+ <para>Where:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold">File</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
+ <entry>
+ <para><emphasis role="bold">Parameter and value</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> ${rslt}.summary</para></entry>
- <entry><para> Same as stdout</para></entry>
+ <entry>
+ <para> ost 8</para>
+ </entry>
+ <entry>
+ <para> Total number of OSTs being tested.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> sz 67108864K</para>
+ </entry>
+ <entry>
+ <para> Total amount of data read or written (in KB).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> rsz 1024</para>
+ </entry>
+ <entry>
+ <para> Record size (size of each echo_client I/O, in KB).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> obj 8</para>
+ </entry>
+ <entry>
+ <para> Total number of objects over all OSTs.</para>
+ </entry>
</row>
<row>
- <entry><para> ${rslt}.script_*</para></entry>
- <entry><para> Per-host test script files</para></entry>
+ <entry>
+ <para> thr 8</para>
+ </entry>
+ <entry>
+ <para> Total number of threads over all OSTs and objects.</para>
+ </entry>
</row>
<row>
- <entry><para> ${rslt}.detail_tmp*</para></entry>
- <entry><para> Per-OST result files</para></entry>
+ <entry>
+ <para> write</para>
+ </entry>
+ <entry>
+ <para> Test name. If more tests have been specified, they all appear on the same line.</para>
+ </entry>
</row>
<row>
- <entry><para> ${rslt}.detail</para></entry>
- <entry><para> Collected result files for post-mortem</para></entry>
+ <entry>
+ <para> 613.54</para>
+ </entry>
+ <entry>
+ <para> Aggregate bandwidth over all OSTs (measured by dividing the total number of MB by the elapsed time).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> [64, 82.00]</para>
+ </entry>
+ <entry>
+ <para> Minimum and maximum instantaneous bandwidths on an individual OST.</para>
+ </entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para>The obdfilter_survey script iterates over the given number of threads and objects performing the specified tests and checks that all test processes have completed successfully.</para>
-
- <note><para>The obdfilter_survey script may not clean up properly if it is aborted or if it encounters an unrecoverable error. In this case, a manual cleanup may be required, possibly including killing any running instances of lctl (local or remote), removing echo_client instances created by the script and unloading obdecho.</para></note>
-
- <section remap="h4">
- <title>24.3.4.1 Script Output</title>
- <para>The .summary file and stdout of the obdfilter_survey script contain lines like:</para>
- <screen>ost 8 sz 67108864K rsz 1024 obj 8 thr 8 write 613.54 [ 64.00, 82.00]
-</screen>
- <para>Where:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter and value</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> ost 8</para></entry>
- <entry><para> Total number of OSTs being tested.</para></entry>
- </row>
- <row>
- <entry><para> sz 67108864K</para></entry>
- <entry><para> Total amount of data read or written (in KB).</para></entry>
- </row>
- <row>
- <entry><para> rsz 1024</para></entry>
- <entry><para> Record size (size of each echo_client I/O, in KB).</para></entry>
- </row>
- <row>
- <entry><para> obj 8</para></entry>
- <entry><para> Total number of objects over all OSTs.</para></entry>
- </row>
- <row>
- <entry><para> thr 8</para></entry>
- <entry><para> Total number of threads over all OSTs and objects.</para></entry>
- </row>
- <row>
- <entry><para> write</para></entry>
- <entry><para> Test name. If more tests have been specified, they all appear on the same line.</para></entry>
- </row>
- <row>
- <entry><para> 613.54</para></entry>
- <entry><para> Aggregate bandwidth over all OSTs (measured by dividing the total number of MB by the elapsed time).</para></entry>
- </row>
- <row>
- <entry><para> [64, 82.00]</para></entry>
- <entry><para> Minimum and maximum instantaneous bandwidths on an individual OST.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <note><para>Although the numbers of threads and objects are specified per-OST in the customization section of the script, the reported results are aggregated over all OSTs.</para></note>
- </section>
- <section remap="h4">
- <title>24.3.4.2 Visualizing Results</title>
- <para>It is useful to import the obdfilter_survey script summary data (it is fixed width) into Excel (or any graphing package) and graph the bandwidth versus the number of threads for varying numbers of concurrent regions. This shows how the OSS performs for a given number of concurrently-accessed objects (files) with varying numbers of I/Os in flight.</para>
- <para>It is also useful to monitor and record average disk I/O sizes during each test using the 'disk io size' histogram in the file /proc/fs/lustre/obdfilter/ (see <link xl:href="LustreProc.html#50438271_55057">Watching the OST Block I/O Stream</link> for details). These numbers help identify problems in the system when full-sized I/Os are not submitted to the underlying disk. This may be caused by problems in the device driver or Linux block layer.</para>
- <para> */brw_stats</para>
- <para>The plot-obdfilter script included in the I/O toolkit is an example of processing output files to a .csv format and plotting a graph using gnuplot.</para>
- </section>
+ <note>
+ <para>Although the numbers of threads and objects are specified per-OST in the customization section of the script, the reported results are aggregated over all OSTs.</para>
+ </note>
+ </section>
+ <section remap="h4">
+ <title>24.3.4.2 Visualizing Results</title>
+ <para>It is useful to import the <literal>obdfilter_survey</literal> script summary data (it is fixed width) into Excel (or any graphing package) and graph the bandwidth versus the number of threads for varying numbers of concurrent regions. This shows how the OSS performs for a given number of concurrently-accessed objects (files) with varying numbers of I/Os in flight.</para>
+ <para>It is also useful to monitor and record average disk I/O sizes during each test using the 'disk io size' histogram in the file <literal>/proc/fs/lustre/obdfilter/</literal> (see <xref linkend="dbdoclet.50438271_55057">Watching the OST Block I/O Stream</xref> for details). These numbers help identify problems in the system when full-sized I/Os are not submitted to the underlying disk. This may be caused by problems in the device driver or Linux block layer.</para>
+ <screen> */brw_stats</screen>
+ <para>The <literal>plot-obdfilter</literal> script included in the I/O toolkit is an example of processing output files to a .csv format and plotting a graph using <literal>gnuplot</literal>.</para>
</section>
</section>
- <section xml:id="dbdoclet.50438212_85136">
- <title>24.4 Testing OST I/O Performance (ost_<anchor xml:id="dbdoclet.50438212_marker-1290067" xreflabel=""/>survey)</title>
- <para>The ost_survey tool is a shell script that uses lfs setstripe to perform I/O against a single OST. The script writes a file (currently using dd) to each OST in the Lustre file system, and compares read and write speeds. The ost_survey tool is used to detect anomalies between otherwise identical disk subsystems.</para>
- <note><para>We have frequently discovered wide performance variations across all LUNs in a cluster. This may be caused by faulty disks, RAID parity reconstruction during the test, or faulty network hardware.</para></note>
-
- <para>To run the ost_survey script, supply a file size (in KB) and the Lustre mount point. For example, run:</para>
- <screen>$ ./ost-survey.sh 10 /mnt/lustre
+ </section>
+ <section xml:id="dbdoclet.50438212_85136">
+ <title>24.4 Testing OST I/O Performance (<literal>ost_survey</literal>)</title>
+ <para>The <literal>ost_survey</literal> tool is a shell script that uses <literal>lfs setstripe</literal> to perform I/O against a single OST. The script writes a file (currently using <literal>dd</literal>) to each OST in the Lustre file system, and compares read and write speeds. The <literal>ost_survey</literal> tool is used to detect anomalies between otherwise identical disk subsystems.</para>
+ <note>
+ <para>We have frequently discovered wide performance variations across all LUNs in a cluster. This may be caused by faulty disks, RAID parity reconstruction during the test, or faulty network hardware.</para>
+ </note>
+ <para>To run the <literal>ost_survey</literal> script, supply a file size (in KB) and the Lustre mount point. For example, run:</para>
+ <screen>$ ./ost-survey.sh 10 /mnt/lustre
</screen>
-
<para>Typical output is:</para>
- <screen>Average read Speed: 6.73
+ <para>Typical output is:</para>
+ <screen>Average read Speed: 6.73
Average write Speed: 5.41
read - Worst OST indx 0 5.84 MB/s
write - Worst OST indx 0 3.77 MB/s
Ost index 2 Read time 0.14 Write time \
0.16
</screen>
- </section>
- <section xml:id="dbdoclet.50438212_58201">
- <title>24.5 Collecting Application Profiling Information (stats-collect)</title>
- <para>The stats-collect utility contains the following scripts used to collect application profiling information from Lustre clients and servers:</para>
- <itemizedlist><listitem>
- <para>lstat.sh - Script for a single node that is run on each profile node.</para>
- </listitem>
-
-<listitem>
- <para>gather_stats_everywhere.sh - Script that collect statistics.</para>
- </listitem>
-
-<listitem>
- <para>config.sh - Script that contains customized configuration descriptions.</para>
- </listitem>
-
-</itemizedlist>
- <para>The stats-collect utility requires:</para>
- <itemizedlist><listitem>
- <para> Lustre to be installed and set up on your cluster</para>
- </listitem>
-
-<listitem>
- <para> SSH and SCP access to these nodes without requiring a password</para>
- </listitem>
-
-</itemizedlist>
- <section remap="h3">
- <title>24.5.1 Using stats-collect</title>
- <para>The stats-collect utility is configured by including profiling configuration variables in the config.sh script. Each configuration variable takes the following form, where 0 indicates statistics are to be collected only when the script starts and stops and <emphasis>n</emphasis> indicates the interval in seconds at which statistics are to be collected:</para>
- <screen><emphasis><statistic></emphasis>_INTERVAL=<emphasis>[</emphasis>0<emphasis>|n]</emphasis></screen>
- <para>Statistics that can be collected include:</para>
- <itemizedlist><listitem>
- <para>VMSTAT - Memory and CPU usage and aggregate read/write operations</para>
- </listitem>
-
-<listitem>
- <para>SERVICE - Lustre OST and MDT RPC service statistics</para>
- </listitem>
-
-<listitem>
- <para>BRW - OST block read/write statistics (brw_stats)</para>
- </listitem>
-
-<listitem>
- <para>SDIO - SCSI disk IO statistics (sd_iostats)</para>
- </listitem>
-
-<listitem>
- <para>MBALLOC - ldiskfs block allocation statistics</para>
- </listitem>
-
-<listitem>
- <para>IO - Lustre target operations statistics</para>
- </listitem>
-
-<listitem>
- <para>JBD - ldisfs journal statistics</para>
- </listitem>
-
-<listitem>
- <para>CLIENT - Lustre OSC request statistics</para>
- </listitem>
-
-</itemizedlist>
- <para>To collect profile information:</para>
- <para> 1. Begin collecting statistics on each node specified in the config.sh script.</para>
- <para>Starting the collect profile daemon on each node by entering:</para>
- <screen>sh gather_stats_everywhere.sh config.sh start
+ </section>
+ <section xml:id="dbdoclet.50438212_58201">
+ <title>24.5 Collecting Application Profiling Information (<literal>stats-collect</literal>)</title>
+ <para>The <literal>stats-collect</literal> utility contains the following scripts used to collect application profiling information from Lustre clients and servers:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>lstat.sh</literal> - Script for a single node that is run on each profile node.</para>
+ </listitem>
+ <listitem>
+ <para><literal>gather_stats_everywhere.sh</literal> - Script that collect statistics.</para>
+ </listitem>
+ <listitem>
+ <para><literal>config.sh</literal> - Script that contains customized configuration descriptions.</para>
+ </listitem>
+ </itemizedlist>
+ <para>The <literal>stats-collect</literal> utility requires:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Lustre to be installed and set up on your cluster</para>
+ </listitem>
+ <listitem>
+ <para> SSH and SCP access to these nodes without requiring a password</para>
+ </listitem>
+ </itemizedlist>
+ <section remap="h3">
+ <title>24.5.1 Using <literal>stats-collect</literal></title>
+ <para>The stats-collect utility is configured by including profiling configuration variables in the config.sh script. Each configuration variable takes the following form, where 0 indicates statistics are to be collected only when the script starts and stops and <emphasis>n</emphasis> indicates the interval in seconds at which statistics are to be collected:</para>
+ <screen><emphasis><statistic></emphasis>_INTERVAL=<emphasis>[</emphasis>0<emphasis>|n]</emphasis></screen>
+ <para>Statistics that can be collected include:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>VMSTAT</literal> - Memory and CPU usage and aggregate read/write operations</para>
+ </listitem>
+ <listitem>
+ <para><literal>SERVICE</literal> - Lustre OST and MDT RPC service statistics</para>
+ </listitem>
+ <listitem>
+ <para><literal>BRW</literal> - OST block read/write statistics (brw_stats)</para>
+ </listitem>
+ <listitem>
+ <para><literal>SDIO</literal> - SCSI disk IO statistics (sd_iostats)</para>
+ </listitem>
+ <listitem>
+ <para><literal>MBALLOC</literal> - <literal>ldiskfs</literal> block allocation statistics</para>
+ </listitem>
+ <listitem>
+ <para><literal>IO</literal> - Lustre target operations statistics</para>
+ </listitem>
+ <listitem>
+ <para><literal>JBD</literal> - ldisfs journal statistics</para>
+ </listitem>
+ <listitem>
+ <para><literal>CLIENT</literal> - Lustre OSC request statistics</para>
+ </listitem>
+ </itemizedlist>
+ <para>To collect profile information:</para>
+ <para>Begin collecting statistics on each node specified in the config.sh script.</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Starting the collect profile daemon on each node by entering:</emphasis></para>
+ <screen>sh gather_stats_everywhere.sh config.sh start
</screen>
- <para> 2. Run the test.</para>
- <para> 3. Stop collecting statistics on each node, clean up the temporary file, and create a profiling tarball.</para>
- <para>Enter:</para>
- <screen>sh gather_stats_everywhere.sh config.sh stop <emphasis><log_name.tgz></emphasis></screen>
- <para>When <emphasis><log_name.tgz></emphasis> is specified, a profile tarball /tmp/<emphasis><log_name.tgz></emphasis> is created.</para>
- <para> 4. Analyze the collected statistics and create a csv tarball for the specified profiling data.</para>
- <screen>sh gather_stats_everywhere.sh config.sh analyse log_tarball.tgz csv
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Run the test.</emphasis></para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Stop collecting statistics on each node, clean up the temporary file, and create a profiling tarball.</emphasis></para>
+ <para>Enter:</para>
+ <screen>sh gather_stats_everywhere.sh config.sh stop <emphasis><log_name.tgz></emphasis></screen>
+ <para>When <emphasis>
+ <literal><log_name.tgz></literal>
+ </emphasis> is specified, a profile tarball <literal>/tmp/<emphasis><log_name.tgz></emphasis></literal> is created.</para>
+ </listitem>
+ <listitem>
+ <para>Analyze the collected statistics and create a csv tarball for the specified profiling data.</para>
+ <screen>sh gather_stats_everywhere.sh config.sh analyse log_tarball.tgz csv
</screen>
- <para> </para>
- </section>
+ </listitem>
+ </orderedlist>
+ </section>
</section>
</chapter>
<para><emphasis role="bold"><emphasis><emphasis role="italic">(Optional)</emphasis> Run benchmarking to</emphasis>ols to validate the performance of hardware and software layers in the cluster. Available tools include:</emphasis></para>
<itemizedlist>
<listitem>
- <para><literal>obdfilter_survey</literal> - Characterizes the storage performance of a Lustre file system. For details, see <xref linkend="dbdoclet.50438212_40624"/>.</para>
+ <para><literal>obdfilter_survey</literal> - Characterizes the storage performance of a Lustre file system. For details, see <xref linkend="dbdoclet.50438212_26516"/>.</para>
</listitem>
<listitem>
<para><literal>ost_survey</literal> - Performs I/O against OSTs to detect anomalies between otherwise identical disk subsystems. For details, see <xref linkend="dbdoclet.50438212_85136"/>.</para>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='configuringquotas'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringquotas">
<info>
- <title xml:id='configuringquotas.title'>Configuring and Managing Quotas</title>
+ <title xml:id="configuringquotas.title">Configuring and Managing Quotas</title>
</info>
-
<para>This chapter describes how to configure quotas and includes the following sections:</para>
-
- <itemizedlist><listitem>
+ <itemizedlist>
+ <listitem>
<para><xref linkend="dbdoclet.50438217_54945"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438217_31982"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438217_49939"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438217_15106"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438217_27895"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438217_20772"/></para>
</listitem>
-</itemizedlist>
-
- <section xml:id="dbdoclet.50438217_54945">
- <title>21.1 Working with <anchor xml:id="dbdoclet.50438217_marker-1290118" xreflabel=""/>Quotas</title>
- <para>Quotas allow a system administrator to limit the amount of disk space a user or group can use in a directory. Quotas are set by root, and can be specified for individual users and/or groups. Before a file is written to a partition where quotas are set, the quota of the creator's group is checked. If a quota exists, then the file size counts towards the group's quota. If no quota exists, then the owner's user quota is checked before the file is written. Similarly, inode usage for specific functions can be controlled if a user over-uses the allocated space.</para>
- <para>Lustre quota enforcement differs from standard Linux quota enforcement in several ways:</para>
- <itemizedlist><listitem>
- <para> Quotas are administered via the lfs command (post-mount).</para>
- </listitem>
-<listitem>
- <para> Quotas are distributed (as Lustre is a distributed file system), which has several ramifications.</para>
- </listitem>
-<listitem>
- <para> Quotas are allocated and consumed in a quantized fashion.</para>
- </listitem>
-
-<listitem>
- <para> Client does not set the usrquota or grpquota options to mount. When quota is enabled, it is enabled for all clients of the file system; started automatically using quota_type or started manually with lfs quotaon.</para>
- </listitem>
-
-</itemizedlist>
- <caution><para>Although quotas are available in Lustre, root quotas are NOT enforced.</para><para>lfs setquota -u root (limits are not enforced)</para><para>lfs quota -u root (usage includes internal Lustre data that is dynamic in size and does not accurately reflect mount point visible block and inode usage).</para></caution>
-
- </section>
- <section xml:id="dbdoclet.50438217_31982">
- <title>21.2 Enabling <anchor xml:id="dbdoclet.50438217_marker-1290128" xreflabel=""/>Disk Quotas</title>
- <para>Use this procedure to enable (configure) disk quotas in Lustre.</para>
- <orderedlist><listitem>
- <para> 1. If you have re-complied your Linux kernel, be sure that CONFIG_QUOTA and CONFIG_QUOTACTL are enabled. Also, verify that CONFIG_QFMT_V1 and/or CONFIG_QFMT_V2 are enabled.</para>
- <para>Quota is enabled in all Linux 2.6 kernels supplied for Lustre.</para>
-
- </listitem><listitem>
- <para> 2. Start the server.</para>
- </listitem><listitem>
- <para> 3. Mount the Lustre file system on the client and verify that the lquota module has loaded properly by using the lsmod command.</para>
- <screen>$ lsmod
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438217_54945">
+ <title>21.1 Working with Quotas</title>
+ <para>Quotas allow a system administrator to limit the amount of disk space a user or group can use in a directory. Quotas are set by root, and can be specified for individual users and/or groups. Before a file is written to a partition where quotas are set, the quota of the creator's group is checked. If a quota exists, then the file size counts towards the group's quota. If no quota exists, then the owner's user quota is checked before the file is written. Similarly, inode usage for specific functions can be controlled if a user over-uses the allocated space.</para>
+ <para>Lustre quota enforcement differs from standard Linux quota enforcement in several ways:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Quotas are administered via the <literal>lfs</literal> command (post-mount).</para>
+ </listitem>
+ <listitem>
+ <para>Quotas are distributed (as Lustre is a distributed file system), which has several ramifications.</para>
+ </listitem>
+ <listitem>
+ <para>Quotas are allocated and consumed in a quantized fashion.</para>
+ </listitem>
+ <listitem>
+ <para>Client does not set the <literal>usrquota</literal> or <literal>grpquota</literal> options to mount. When quota is enabled, it is enabled for all clients of the file system; started automatically using <literal>quota_type</literal> or started manually with <literal>lfs quotaon</literal>.</para>
+ </listitem>
+ </itemizedlist>
+ <caution>
+ <para>Although quotas are available in Lustre, root quotas are NOT enforced.</para>
+ <para><literal>lfs setquota -u root</literal> (limits are not enforced)</para>
+ <para><literal>lfs quota -u root</literal> (usage includes internal Lustre data that is dynamic in size and does not accurately reflect mount point visible block and inode usage).</para>
+ </caution>
+ </section>
+ <section xml:id="dbdoclet.50438217_31982">
+ <title>21.2 Enabling Disk Quotas</title>
+ <para>Use this procedure to enable (configure) disk quotas in Lustre.</para>
+ <orderedlist>
+ <listitem>
+ <emphasis role="bold">
+ <para>If you have re-complied your Linux kernel, be sure that <literal>CONFIG_QUOTA</literal> and <literal>CONFIG_QUOTACTL</literal> are enabled. Also, verify that <literal>CONFIG_QFMT_V1</literal> and/or <literal>CONFIG_QFMT_V2</literal> are enabled.</para>
+ </emphasis>
+ <para>Quota is enabled in all Linux 2.6 kernels supplied for Lustre.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Start the server.</emphasis></para>
+ </listitem>
+ <listitem>
+ <emphasis role="bold">
+ <para>Mount the Lustre file system on the client and verify that the <literal>lquota</literal> module has loaded properly by using the <literal>lsmod</literal> command.</para>
+ </emphasis>
+ <screen>$ lsmod
[root@oss161 ~]# lsmod
Module Size Used by
obdfilter 220532 1
lov 289064 1 lustre
lquota 107048 4 obdfilter
mdc 95016 1 lustre
-ksocklnd 111812 1
-</screen>
- </listitem></orderedlist>
-
- <para>The Lustre mount command no longer recognizes the usrquota and grpquota options. If they were previously specified, remove them from /etc/fstab.</para>
- <para>When quota is enabled, it is enabled for all file system clients (started automatically using quota_type or manually with lfs quotaon).</para>
-
- <note><para>Lustre with the Linux kernel 2.4 does <emphasis>not</emphasis> support quotas.</para></note>
-
- <para>To enable quotas automatically when the file system is started, you must set the mdt.quota_type and ost.quota_type parameters, respectively, on the MDT and OSTs. The parameters can be set to the string u (user), g (group) or ug for both users and groups.</para>
- <para>You can enable quotas at mkfs time (mkfs.lustre --param mdt.quota_type=ug) or with tunefs.lustre. As an example:</para>
- <screen>tunefs.lustre --param ost.quota_type=ug $ost_dev
-</screen>
- <caution><para>If you are using mkfs.lustre --param mdt.quota_type=ug or tunefs.lustre --param ost.quota_type=ug, be sure to run the command on all OSTs and the MDT. Otherwise, abnormal results may occur.</para></caution>
-
- <section remap="h4">
- <title>21.2.0.1 Administrative and Operational Quotas</title>
- <para>Lustre has two kinds of quota files:</para>
- <itemizedlist><listitem>
- <para> Administrative quotas (for the MDT), which contain limits for users/groups for the entire cluster.</para>
- </listitem>
-
-<listitem>
- <para> Operational quotas (for the MDT and OSTs), which contain quota information dedicated to a cluster node.</para>
- </listitem>
-
-</itemizedlist>
- <para>Lustre 1.6.5 introduced the v2 file format for administrative quota files, with continued support for the old file format (v1). The mdt.quota_type parameter also handles '1' and '2' options, to specify the Lustre quota versions that will be used. For example:</para>
- <screen>--param mdt.quota_type=ug1
---param mdt.quota_type=u2
-</screen>
- <para>Lustre 1.6.6 introduced the v2 file format for operational quotas, with continued support for the old file format (v1). The ost.quota_type parameter handles '1' and '2' options, to specify the Lustre quota versions that will be used. For example:</para>
- <screen>--param ost.quota_type=ug2
---param ost.quota_type=u1
-</screen>
- <para>For more information about the v1 and v2 formats, see <link xl:href="ConfiguringQuotas.html#50438217_66360">Quota File Formats</link>.</para>
- </section>
- </section>
- <section xml:id="dbdoclet.50438217_49939">
- <title>21.3 Creating Quota <anchor xml:id="dbdoclet.50438217_marker-1290170" xreflabel=""/>Files and Quota Administration</title>
- <para>Once each quota-enabled file system is remounted, it is capable of working with disk quotas. However, the file system is not yet ready to support quotas. If umount has been done regularly, run the lfs command with the quotaon option. If umount has not been done, perform these steps:</para>
- <orderedlist><listitem>
- <para> 1. Take Lustre ''offline''.</para>
- <para>That is, verify that no write operations (append, write, truncate, create or delete) are being performed (preparing to run lfs quotacheck). Operations that do not change Lustre files (such as read or mount) are okay to run.</para>
-
- <caution><para>When lfsquotacheck is run, Lustre must NOT be performing any write operations. Failure to follow this caution may cause the statistic information of quota to be inaccurate. For example, the number of blocks used by OSTs for users or groups will be inaccurate, which can cause unexpected quota problems.</para></caution>
-
- </listitem><listitem>
- <para> 2. Run the <emphasis role="bold">lfs</emphasis> command with the <emphasis role="bold">quotacheck</emphasis> option:</para>
- <screen># lfs quotacheck -ug /mnt/lustre
-</screen>
- <para>By default, quota is turned on after quotacheck completes. Available options are:</para>
- <itemizedlist><listitem>
- <para>u -- checks the user disk quota information</para>
- </listitem>
-
-<listitem>
- <para>g -- checks the group disk quota information</para>
+ksocklnd 111812 1</screen>
+ </listitem>
+ </orderedlist>
+ <para>The Lustre mount command no longer recognizes the <literal>usrquota</literal> and <literal>grpquota</literal> options. If they were previously specified, remove them from <literal>/etc/fstab</literal>.</para>
+ <para>When quota is enabled, it is enabled for all file system clients (started automatically using <literal>quota_type</literal> or manually with <literal>lfs quotaon</literal>).</para>
+ <note>
+ <para>Lustre with the Linux kernel 2.4 does <emphasis>not</emphasis> support quotas.</para>
+ </note>
+ <para>To enable quotas automatically when the file system is started, you must set the <literal>mdt.quota_type</literal> and <literal>ost.quota_type</literal> parameters, respectively, on the MDT and OSTs. The parameters can be set to the string <literal>u</literal> (user), <literal>g</literal> (group) or <literal>ug</literal> for both users and groups.</para>
+ <para>You can enable quotas at <literal>mkfs</literal> time (<literal>mkfs.lustre --param mdt.quota_type=ug</literal>) or with <literal>tunefs.lustre</literal>. As an example:</para>
+ <screen>tunefs.lustre --param ost.quota_type=ug $ost_dev</screen>
+ <caution>
+ <para>If you are using <literal>mkfs.lustre --param mdt.quota_type=ug</literal> or <literal>tunefs.lustre --param ost.quota_type=ug</literal>, be sure to run the command on all OSTs and the MDT. Otherwise, abnormal results may occur.</para>
+ </caution>
+ <section remap="h4">
+ <title>21.2.0.1 Administrative and Operational Quotas</title>
+ <para>Lustre has two kinds of quota files:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Administrative quotas (for the MDT), which contain limits for users/groups for the entire cluster.</para>
</listitem>
-
-</itemizedlist>
- </listitem></orderedlist>
-
- <para>The lfsquotacheck command checks all objects on all OSTs and the MDS to sum up for every UID/GID. It reads all Lustre metadata and re-computes the number of blocks/inodes that each UID/GID has used. If there are many files in Lustre, it may take a long time to complete.</para>
-
- <note><para>User and group quotas are separate. If either quota limit is reached, a process with the corresponding UID/GID cannot allocate more space on the file system.</para></note>
-
- <note><para>When lfsquotacheck runs, it creates a quota file -- a sparse file with a size proportional to the highest UID in use and UID/GID distribution. As a general rule, if the highest UID in use is large, then the sparse file will be large, which may affect functions such as creating a snapshot.</para></note>
-
- <note><para>For Lustre 1.6 releases before version 1.6.5, and 1.4 releases before version 1.4.12, if the underlying ldiskfs file system has not unmounted gracefully (due to a crash, for example), re-run quotacheck to obtain accurate quota information. Lustre 1.6.5 and 1.4.12 use journaled quota, so it is not necessary to run quotacheck after an unclean shutdown.</para><para> In certain failure situations (e.g., when a broken Lustre installation or build is used), re-run quotacheck after checking the server kernel logs and fixing the root problem.</para></note>
-
- <para>The lfs command includes several command options to work with quotas:</para>
- <itemizedlist><listitem>
- <para><varname>quotaon</varname> -- enables disk quotas on the specified file system. The file system quota files must be present in the root directory of the file system.</para>
+ <listitem>
+ <para>Operational quotas (for the MDT and OSTs), which contain quota information dedicated to a cluster node.</para>
</listitem>
-
-<listitem>
- <para><varname>quotaoff</varname> -- disables disk quotas on the specified file system.</para>
- </listitem>
-
-<listitem>
- <para><varname>quota</varname> -- displays general quota information (disk usage and limits)</para>
- </listitem>
-
-<listitem>
- <para><varname>setquota</varname> -- specifies quota limits and tunes the grace period. By default, the grace period is one week.</para>
- </listitem>
-
-</itemizedlist>
- <para> Usage:</para>
- <screen>lfs quotaon [-ugf] <filesystem>
+ </itemizedlist>
+ <para>Lustre 1.6.5 introduced the v2 file format for administrative quota files, with continued support for the old file format (v1). The mdt.quota_type parameter also handles '1' and '2' options, to specify the Lustre quota versions that will be used. For example:</para>
+ <screen>--param mdt.quota_type=ug1
+--param mdt.quota_type=u2</screen>
+ <para>Lustre 1.6.6 introduced the v2 file format for operational quotas, with continued support for the old file format (v1). The ost.quota_type parameter handles '1' and '2' options, to specify the Lustre quota versions that will be used. For example:</para>
+ <screen>--param ost.quota_type=ug2
+--param ost.quota_type=u1</screen>
+ <para>For more information about the v1 and v2 formats, see <xref linkend="dbdoclet.50438217_66360">Quota File Formats</xref>.</para>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438217_49939">
+ <title>21.3 Creating Quota Files and Quota Administration</title>
+ <para>Once each quota-enabled file system is remounted, it is capable of working with disk quotas. However, the file system is not yet ready to support quotas. If <literal>umount</literal> has been done regularly, run the <literal>lfs</literal> command with the <literal>quotaon</literal> option. If <literal>umount</literal> has not been done, perform these steps:</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Take Lustre ''offline''.</emphasis></para>
+ <para>That is, verify that no write operations (append, write, truncate, create or delete) are being performed (preparing to run <literal>lfs quotacheck</literal>). Operations that do not change Lustre files (such as read or mount) are okay to run.</para>
+ <caution>
+ <para>When <literal>lfs quotacheck</literal> is run, Lustre must NOT be performing any write operations. Failure to follow this caution may cause the statistic information of quota to be inaccurate. For example, the number of blocks used by OSTs for users or groups will be inaccurate, which can cause unexpected quota problems.</para>
+ </caution>
+ </listitem>
+ <listitem>
+ <para> <emphasis role="bold">Run the <emphasis role="bold">
+ <literal>lfs</literal>
+ </emphasis> command with the <emphasis role="bold">
+ <literal>quotacheck</literal>
+ </emphasis> option:</emphasis></para>
+ <screen># lfs quotacheck -ug /mnt/lustre</screen>
+ <para>By default, quota is turned on after <literal>quotacheck</literal> completes. Available options are:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>u</literal> -- checks the user disk quota information</para>
+ </listitem>
+ <listitem>
+ <para><literal>g</literal> -- checks the group disk quota information</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ </orderedlist>
+ <para>The lfsquotacheck command checks all objects on all OSTs and the MDS to sum up for every UID/GID. It reads all Lustre metadata and re-computes the number of blocks/inodes that each UID/GID has used. If there are many files in Lustre, it may take a long time to complete.</para>
+ <note>
+ <para>User and group quotas are separate. If either quota limit is reached, a process with the corresponding UID/GID cannot allocate more space on the file system.</para>
+ </note>
+ <note>
+ <para>When <literal>lfs quotacheck</literal> runs, it creates a quota file -- a sparse file with a size proportional to the highest UID in use and UID/GID distribution. As a general rule, if the highest UID in use is large, then the sparse file will be large, which may affect functions such as creating a snapshot.</para>
+ </note>
+ <note>
+ <para>For Lustre 1.6 releases before version 1.6.5, and 1.4 releases before version 1.4.12, if the underlying <literal>ldiskfs</literal> file system has not unmounted gracefully (due to a crash, for example), re-run <literal>quotacheck</literal> to obtain accurate quota information. Lustre 1.6.5 and 1.4.12 use journaled quota, so it is not necessary to run <literal>quotacheck</literal> after an unclean shutdown.</para>
+ <para>In certain failure situations (e.g., when a broken Lustre installation or build is used), re-run <literal>quotacheck</literal> after checking the server kernel logs and fixing the root problem.</para>
+ </note>
+ <para>The <literal>lfs</literal> command includes several command options to work with quotas:</para>
+ <itemizedlist>
+ <listitem>
+ <para><varname>quotaon</varname> -- enables disk quotas on the specified file system. The file system quota files must be present in the root directory of the file system.</para>
+ </listitem>
+ <listitem>
+ <para><varname>quotaoff</varname> -- disables disk quotas on the specified file system.</para>
+ </listitem>
+ <listitem>
+ <para><varname>quota</varname> -- displays general quota information (disk usage and limits)</para>
+ </listitem>
+ <listitem>
+ <para><varname>setquota</varname> -- specifies quota limits and tunes the grace period. By default, the grace period is one week.</para>
+ </listitem>
+ </itemizedlist>
+ <para> Usage:</para>
+ <screen>lfs quotaon [-ugf] <filesystem>
lfs quotaoff [-ug] <filesystem>
lfs quota [-q] [-v] [-o obd_uuid] [-u|-g <uname>|uid|gname|gid>] <filesystem>
lfs quota -t <-u|-g> <filesystem>
lfs setquota <-u|--user|-g|--group> <username|groupname> [-b <block-softlimit>] [\
--B <block-hardlimit>] [-i <inode-softlimit>] [-I <inode-hardlimit>] <filesystem>
-</screen>
- <para>Examples:</para>
- <para>In all of the examples below, the file system is /mnt lustre.</para>
- <para>To turn on user and group quotas, run:</para>
- <screen>$ lfs quotaon -ug /mnt/lustre
-</screen>
- <para>To turn off user and group quotas, run:</para>
- <screen>$ lfs quotaoff -ug /mnt/lustre
-</screen>
- <para>To display general quota information (disk usage and limits) for the user running the command and his primary group, run:</para>
- <screen>$ lfs quota /mnt/lustre
-</screen>
- <para>To display general quota information for a specific user ("bob" in this example), run:</para>
- <screen>$ lfs quota -u bob /mnt/lustre
-</screen>
- <para>To display general quota information for a specific user ("bob" in this example) and detailed quota statistics for each MDT and OST, run:</para>
- <screen>$ lfs quota -u bob -v /mnt/lustre
-</screen>
- <para>To display general quota information for a specific group ("eng" in this example), run:</para>
- <screen>$ lfs quota -g eng /mnt/lustre
-</screen>
- <para>To display block and inode grace times for user quotas, run:</para>
- <screen>$ lfs quota -t -u /mnt/lustre
-</screen>
- <para>To set user and group quotas for a specific user ("bob" in this example), run:</para>
- <screen>$ lfs setquota -u bob 307200 309200 10000 11000 /mnt/lustre
-</screen>
- <para>In this example, the quota for user "bob" is set to 300 MB (309200*1024) and the hard limit is 11,000 files. Therefore, the inode hard limit should be 11000.</para>
-
- <note><para>For the Lustre command $lfssetquota/quota ... the qunit for block is KB (1024) and the qunit for inode is 1.</para></note>
-
- <para>The quota command displays the quota allocated and consumed for each Lustre device. Using the previous setquota example, running this lfs quota command:</para>
- <screen>$ lfs quota -u bob -v /mnt/lustre
-</screen>
- <para>displays this command output:</para>
- <screen>Disk quotas for user bob (uid 6000):
+-B <block-hardlimit>] [-i <inode-softlimit>] [-I <inode-hardlimit>] <filesystem></screen>
+ <para>Examples:</para>
+ <para>In all of the examples below, the file system is <literal>/mnt</literal> lustre.</para>
+ <para>To turn on user and group quotas, run:</para>
+ <screen>$ lfs quotaon -ug /mnt/lustre</screen>
+ <para>To turn off user and group quotas, run:</para>
+ <screen>$ lfs quotaoff -ug /mnt/lustre</screen>
+ <para>To display general quota information (disk usage and limits) for the user running the command and his primary group, run:</para>
+ <screen>$ lfs quota /mnt/lustre </screen>
+ <para>To display general quota information for a specific user ("<literal>bob</literal>" in this example), run:</para>
+ <screen>$ lfs quota -u bob /mnt/lustre</screen>
+ <para>To display general quota information for a specific user ("<literal>bob</literal>" in this example) and detailed quota statistics for each MDT and OST, run:</para>
+ <screen>$ lfs quota -u bob -v /mnt/lustre</screen>
+ <para>To display general quota information for a specific group ("<literal>eng</literal>" in this example), run:</para>
+ <screen>$ lfs quota -g eng /mnt/lustre</screen>
+ <para>To display block and inode grace times for user quotas, run:</para>
+ <screen>$ lfs quota -t -u /mnt/lustre</screen>
+ <para>To set user and group quotas for a specific user ("bob" in this example), run:</para>
+ <screen>$ lfs setquota -u bob 307200 309200 10000 11000 /mnt/lustre</screen>
+ <para>In this example, the quota for user "bob" is set to 300 MB (309200*1024) and the hard limit is 11,000 files. Therefore, the inode hard limit should be 11000.</para>
+ <note>
+ <para>For the Lustre command <literal>$lfssetquota/quota ...</literal> the qunit for block is KB (1024) and the qunit for inode is 1.</para>
+ </note>
+ <para>The quota command displays the quota allocated and consumed for each Lustre device. Using the previous <literal>setquota</literal> example, running this <literal>lfs</literal> quota command:</para>
+ <screen>$ lfs quota -u bob -v /mnt/lustre </screen>
+ <para>displays this command output:</para>
+ <screen>Disk quotas for user bob (uid 6000):
Filesystem kbytes quota limit grace \
files quota limit grace
/mnt/lustre 0 30720 30920 \
lustre-OST0000_UUID 0 - 16384 \
- 0 - 0 -
lustre-OST0001_UUID 0 - 16384 \
-- 0 - 0 -
+- 0 - 0 -</screen>
+ </section>
+ <section xml:id="dbdoclet.50438217_15106">
+ <title>21.4 Quota Allocation</title>
+ <para>In Lustre, quota must be properly allocated or users may experience unnecessary failures. The file system block quota is divided up among the OSTs within the file system. Each OST requests an allocation which is increased up to the quota limit. The quota allocation is then <emphasis role="italic">quantized</emphasis> to reduce the number of quota-related request traffic. By default, Lustre supports both user and group quotas to limit disk usage and file counts.</para>
+ <para>The quota system in Lustre is completely compatible with the quota systems used on other file systems. The Lustre quota system distributes quotas from the quota master. Generally, the MDS is the quota master for both inodes and blocks. All OSTs and the MDS are quota slaves to the OSS nodes. To reduce quota requests and get reasonably accurate quota distribution, the transfer quota unit (qunit) between quota master and quota slaves is changed dynamically by the lquota module. The default minimum value of qunit is 1 MB for blocks and 2 for inodes. The proc entries to set these values are: <literal>/proc/fs/lustre/mds/lustre-MDT*/quota_least_bunit</literal> and <literal>/proc/fs/lustre/mds/lustre-MDT*/quota_least_iunit</literal>. The default maximum value of <literal>qunit</literal> is 128 MB for blocks and 5120 for inodes. The proc entries to set these values are <literal>quota_bunit_sz</literal> and <literal>quota_iunit_sz</literal> in the MDT and OSTs.</para>
+ <note>
+ <para>In general, the <literal>quota_bunit_sz</literal> value should be larger than 1 MB. For testing purposes, it can be set to 4 KB, if necessary.</para>
+ </note>
+ <para>The file system block quota is divided up among the OSTs and the MDS within the file system. Only the MDS uses the file system inode quota.</para>
+ <para>This means that the minimum quota for block is 1 MB* (the number of OSTs + the number of MDSs), which is 1 MB* (number of OSTs + 1). If you attempt to assign a smaller quota, users maybe not be able to create files. As noted, the default minimum quota for inodes is 2. The default is established at file system creation time, but can be tuned via <literal>/proc</literal> values (described below). The inode quota is also allocated in a quantized manner on the MDS.</para>
+ <para>If we look at the <literal>setquota</literal> example again, running this <literal>lfs quota</literal> command:</para>
+ <screen># lfs quota -u bob -v /mnt/lustre
</screen>
- </section>
- <section xml:id="dbdoclet.50438217_15106">
- <title>21.4 Quota<anchor xml:id="dbdoclet.50438217_marker-1290226" xreflabel=""/> Allocation</title>
- <para>In Lustre, quota must be properly allocated or users may experience unnecessary failures. The file system block quota is divided up among the OSTs within the file system. Each OST requests an allocation which is increased up to the quota limit. The quota allocation is then quantized to reduce the number of quota-related request traffic. By default, Lustre supports both user and group quotas to limit disk usage and file counts.</para>
- <para>The quota system in Lustre is completely compatible with the quota systems used on other file systems. The Lustre quota system distributes quotas from the quota master. Generally, the MDS is the quota master for both inodes and blocks. All OSTs and the MDS are quota slaves to the OSS nodes. To reduce quota requests and get reasonably accurate quota distribution, the transfer quota unit (qunit) between quota master and quota slaves is changed dynamically by the lquota module. The default minimum value of qunit is 1 MB for blocks and 2 for inodes. The proc entries to set these values are: /proc/fs/lustre/mds/lustre-MDT*/quota_least_bunit and /proc/fs/lustre/mds/lustre-MDT*/quota_least_iunit. The default maximum value of qunit is 128 MB for blocks and 5120 for inodes. The proc entries to set these values are quota_bunit_sz and quota_iunit_sz in the MDT and OSTs.</para>
- <note><para>In general, the quota_bunit_sz value should be larger than 1 MB. For testing purposes, it can be set to 4 KB, if necessary.</para></note>
- <para>The file system block quota is divided up among the OSTs and the MDS within the file system. Only the MDS uses the file system inode quota.</para>
- <para>This means that the minimum quota for block is 1 MB* (the number of OSTs + the number of MDSs), which is 1 MB* (number of OSTs + 1). If you attempt to assign a smaller quota, users maybe not be able to create files. As noted, the default minimum quota for inodes is 2. The default is established at file system creation time, but can be tuned via /proc values (described below). The inode quota is also allocated in a quantized manner on the MDS.</para>
- <para>If we look at the setquota example again, running this lfsquota command:</para>
- <screen># lfs quota -u bob -v /mnt/lustre
-</screen>
- <para>displays this command output:</para>
- <screen>Disk quotas for user bob (uid 500):
+ <para>displays this command output:</para>
+ <screen>Disk quotas for user bob (uid 500):
Filesystem kbytes quota limit grace \
files quota limit grace
/mnt/lustre 30720* 30720 30920 \
lustre-OST0000_UUID 0 - 1024 \
- - - -
lustre-OST0001_UUID 30720* - 28872 \
-- - - -
-</screen>
- <para>The total quota limit of 30,920 is allotted to user bob, which is further distributed to two OSTs and one MDS.</para>
- <note><para>Values appended with '*' show the limit that has been over-used (exceeding the quota), and receives this message Disk quota exceeded. For example:</para><para> \</para><para>$ cp: writing `/mnt/lustre/var/cache/fontconfig/ beeeeb3dfe132a8a0633a017c99ce0-x86.cache': Disk quota exceeded.</para></note>
- <para>The requested quota of 300 MB is divided across the OSTs.</para>
- <note><para>It is very important to note that the block quota is consumed per OST and the MDS per block and inode (there is only one MDS for inodes). Therefore, when the quota is consumed on one OST, the client may not be able to create files regardless of the quota available on other OSTs.</para></note>
- <section remap="h5">
- <title>Additional information:</title>
- <para><emphasis role="bold">Grace period</emphasis> -- The period of time (in seconds) within which users are allowed to exceed their soft limit. There are four types of grace periods:</para>
- <itemizedlist><listitem>
- <para> user block soft limit</para>
- </listitem>
-
-<listitem>
- <para> user inode soft limit</para>
- </listitem>
-
-<listitem>
- <para> group block soft limit</para>
- </listitem>
-
-<listitem>
- <para> group inode soft limit</para>
- </listitem>
-
-</itemizedlist>
- <para>The grace periods are applied to all users. The user block soft limit is for all users who are using a blocks quota.</para>
- <para><emphasis role="bold">Soft limit</emphasis> -- Once you are beyond the soft limit, the quota module begins to time, but you still can write block and inode. When you are always beyond the soft limit and use up your grace time, you get the same result as the hard limit. For inodes and blocks, it is the same. Usually, the soft limit MUST be less than the hard limit; if not, the quota module never triggers the timing. If the soft limit is not needed, leave it as zero (0).</para>
- <para><emphasis role="bold">Hard limit</emphasis> -- When you are beyond the hard limit, you get -EQUOTA and cannot write inode/block any more. The hard limit is the absolute limit. When a grace period is set, you can exceed the soft limit within the grace period if are under the hard limits.</para>
- <para>Lustre quota allocation is controlled by two variables, quota_bunit_sz and quota_iunit_sz referring to KBs and inodes, respectively. These values can be accessed on the MDS as /proc/fs/lustre/mds/*/quota_* and on the OST as /proc/fs/lustre/obdfilter/*/quota_*. The quota_bunit_sz and quota_iunit_sz variables are the maximum qunit values for blocks and inodes, respectively. At any time, module lquota chooses a reasonable qunit between the minimum and maximum values.</para>
- <para>The /proc values are bounded by two other variables quota_btune_sz and quota_itune_sz. By default, the *tune_sz variables are set at 1/2 the *unit_sz variables, and you cannot set *tune_sz larger than *unit_sz. You must set bunit_sz first if it is increasing by more than 2x, and btune_sz first if it is decreasing by more than 2x.</para>
- <para><emphasis role="bold">Total number of inodes</emphasis> -- To determine the total number of inodes, use lfsdf-i (and also /proc/fs/lustre/*/*/filestotal). For more information on using the lfsdf-i command and the command output, see <link xl:href="ManagingStripingFreeSpace.html#50438209_35838">Checking File System Free Space</link>.</para>
- <para>Unfortunately, the statfs interface does not report the free inode count directly, but instead reports the total inode and used inode counts. The free inode count is calculated for df from (total inodes - used inodes).</para>
- <para>It is not critical to know a file system's total inode count. Instead, you should know (accurately), the free inode count and the used inode count for a file system. Lustre manipulates the total inode count in order to accurately report the other two values.</para>
- <para>The values set for the MDS must match the values set on the OSTs.</para>
- <para>The quota_bunit_sz parameter displays bytes, however lfs setquota uses KBs. The quota_bunit_sz parameter must be a multiple of 1024. A proper minimum KB size for lfs setquota can be calculated as:</para>
- <para><emphasis role="bold">Size in KBs = minimum_quota_bunit_sz * (number of OSTS + 1) = 1024 * (number of OSTs +1)</emphasis></para>
- <para>We add one (1) to the number of OSTs as the MDS also consumes KBs. As inodes are only consumed on the MDS, the minimum inode size for lfs setquota is equal to quota_iunit_sz.</para>
- <note><para>Setting the quota below this limit may prevent the user from all file creation.</para></note>
- </section>
+- - - -</screen>
+ <para>The total quota limit of 30,920 is allotted to user bob, which is further distributed to two OSTs and one MDS.</para>
+ <note>
+ <para>Values appended with '<literal>*</literal>' show the limit that has been over-used (exceeding the quota), and receives this message Disk quota exceeded. For example:</para>
+ <para><screen>$ cp: writing `/mnt/lustre/var/cache/fontconfig/ beeeeb3dfe132a8a0633a017c99ce0-x86.cache': Disk quota exceeded.</screen></para>
+ </note>
+ <para>The requested quota of 300 MB is divided across the OSTs.</para>
+ <note>
+ <para>It is very important to note that the block quota is consumed per OST and the MDS per block and inode (there is only one MDS for inodes). Therefore, when the quota is consumed on one OST, the client may not be able to create files regardless of the quota available on other OSTs.</para>
+ </note>
+ <section remap="h5">
+ <title>Additional information:</title>
+ <para><emphasis role="bold">Grace period</emphasis> -- The period of time (in seconds) within which users are allowed to exceed their soft limit. There are four types of grace periods:</para>
+ <itemizedlist>
+ <listitem>
+ <para> user block soft limit</para>
+ </listitem>
+ <listitem>
+ <para> user inode soft limit</para>
+ </listitem>
+ <listitem>
+ <para> group block soft limit</para>
+ </listitem>
+ <listitem>
+ <para> group inode soft limit</para>
+ </listitem>
+ </itemizedlist>
+ <para>The grace periods are applied to all users. The user block soft limit is for all users who are using a blocks quota.</para>
+ <para><emphasis role="bold">Soft limit</emphasis> -- Once you are beyond the soft limit, the quota module begins to time, but you still can write block and inode. When you are always beyond the soft limit and use up your grace time, you get the same result as the hard limit. For inodes and blocks, it is the same. Usually, the soft limit MUST be less than the hard limit; if not, the quota module never triggers the timing. If the soft limit is not needed, leave it as zero (0).</para>
+ <para><emphasis role="bold">Hard limit</emphasis> -- When you are beyond the hard limit, you get <literal>-EQUOTA</literal> and cannot write inode/block any more. The hard limit is the absolute limit. When a grace period is set, you can exceed the soft limit within the grace period if are under the hard limits.</para>
+ <para>Lustre quota allocation is controlled by two variables, <literal>quota_bunit_sz</literal> and <literal>quota_iunit_sz</literal> referring to KBs and inodes, respectively. These values can be accessed on the MDS as <literal>/proc/fs/lustre/mds/*/quota_*</literal> and on the OST as <literal>/proc/fs/lustre/obdfilter/*/quota_*</literal>. The <literal>quota_bunit_sz </literal>and <literal>quota_iunit_sz</literal> variables are the maximum qunit values for blocks and inodes, respectively. At any time, module lquota chooses a reasonable qunit between the minimum and maximum values.</para>
+ <para>The /proc values are bounded by two other variables <literal>quota_btune_sz</literal> and <literal>quota_itune_sz</literal>. By default, the <literal>*tune_sz</literal> variables are set at 1/2 the <literal>*unit_sz</literal> variables, and you cannot set <literal>*tune_sz</literal> larger than <literal>*unit_sz</literal>. You must set <literal>bunit_sz</literal> first if it is increasing by more than 2x, and <literal>btune_sz</literal> first if it is decreasing by more than 2x.</para>
+ <para><emphasis role="bold">Total number of inodes</emphasis> -- To determine the total number of inodes, use <literal>lfs df -i</literal> (and also <literal>/proc/fs/lustre/*/*/filestotal</literal>). For more information on using the <literal>lfs df -i</literal> command and the command output, see <xref linkend="dbdoclet.50438209_35838">Checking File System Free Space</xref>.</para>
+ <para>Unfortunately, the <literal>statfs</literal> interface does not report the free inode count directly, but instead reports the total inode and used inode counts. The free inode count is calculated for <literal>df</literal> from (total inodes - used inodes).</para>
+ <para>It is not critical to know a file system's total inode count. Instead, you should know (accurately), the free inode count and the used inode count for a file system. Lustre manipulates the total inode count in order to accurately report the other two values.</para>
+ <para>The values set for the MDS must match the values set on the OSTs.</para>
+ <para>The <literal>quota_bunit_sz</literal> parameter displays bytes, however <literal>lfs setquota</literal> uses KBs. The <literal>quota_bunit_sz</literal> parameter must be a multiple of 1024. A proper minimum KB size for <literal>lfs setquota</literal> can be calculated as:</para>
+ <para><example>
+ <emphasis role="bold">Size in KBs = minimum_quota_bunit_sz * (number of OSTS + 1) = 1024 * (number of OSTs +1)</emphasis>
+ </example></para>
+ <para>We add one (1) to the number of OSTs as the MDS also consumes KBs. As inodes are only consumed on the MDS, the minimum inode size for <literal>lfs setquota</literal> is equal to <literal>quota_iunit_sz</literal>.</para>
+ <note>
+ <para>Setting the quota below this limit may prevent the user from all file creation.</para>
+ </note>
</section>
- <section xml:id="dbdoclet.50438217_27895">
- <title>21.5 Known Issues <anchor xml:id="dbdoclet.50438217_marker-1290269" xreflabel=""/>with Quotas</title>
- <para>Using quotas in Lustre can be complex and there are several known issues.</para>
- <section remap="h3">
- <title>21.5.1 Granted<anchor xml:id="dbdoclet.50438217_marker-1290272" xreflabel=""/> Cache and Quota Limits</title>
- <para>In Lustre, granted cache does not respect quota limits. In this situation, OSTs grant cache to Lustre client to accelerate I/O. Granting cache causes writes to be successful in OSTs, even if they exceed the quota limits, and will overwrite them.</para>
- <para>The sequence is:</para>
- <orderedlist><listitem>
- <para>A user writes files to Lustre.</para>
- </listitem><listitem>
- <para>If the Lustre client has enough granted cache, then it returns 'success' to users and arranges the writes to the OSTs.</para>
- </listitem><listitem>
- <para>Because Lustre clients have delivered success to users, the OSTs cannot fail these writes.</para>
- </listitem></orderedlist>
- <para>Because of granted cache, writes always overwrite quota limitations. For example, if you set a 400 GB quota on user A and use IOR to write for user A from a bundle of clients, you will write much more data than 400 GB, and cause an out-of-quota error (-EDQUOT).</para>
- <note><para>The effect of granted cache on quota limits can be mitigated, but not eradicated. Reduce the max_dirty_buffer in the clients (can be set from 0 to 512). To set max_dirty_buffer to 0:</para><para> * In releases after Lustre 1.6.5, lctl set_param osc.*.max_dirty_mb=0.</para><para> * In releases before Lustre 1.6.5, proc/fs/lustre/osc/*/max_dirty_mb; do echo 512 > $O</para></note>
- </section>
- <section remap="h3">
- <title>21.5.2 <anchor xml:id="dbdoclet.50438217_50442" xreflabel=""/>Quota <anchor xml:id="dbdoclet.50438217_marker-1290282" xreflabel=""/>Limits</title>
- <para>Available quota limits depend on the Lustre version you are using.</para>
- <itemizedlist><listitem>
- <para> Lustre version 1.4.11 and earlier (for 1.4.x releases) and Lustre version 1.6.4 and earlier (for 1.6.x releases) support quota limits less than 4 TB.</para>
- </listitem>
-
-<listitem>
- <para> Lustre versions 1.4.12, 1.6.5 and later support quota limits of 4 TB and greater in Lustre configurations with OST storage limits of 4 TB and less.</para>
- </listitem>
-
-<listitem>
- <para> Future Lustre versions are expected to support quota limits of 4 TB and greater with no OST storage limits.</para>
- </listitem>
-<listitem>
- <informaltable frame="all">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Lustre Version</emphasis></para></entry>
- <entry><para><emphasis role="bold">Quota Limit Per User/Per Group</emphasis></para></entry>
- <entry><para><emphasis role="bold">OST Storage Limit</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> 1.4.11 and earlier</para></entry>
- <entry><para> < 4TB</para></entry>
- <entry><para> n/a</para></entry>
- </row>
- <row>
- <entry><para> 1.4.12</para></entry>
- <entry><para> => 4TB</para></entry>
- <entry><para> <= 4TB of storage</para></entry>
- </row>
- <row>
- <entry><para> 1.6.4 and earlier</para></entry>
- <entry><para> < 4TB</para></entry>
- <entry><para> n/a</para></entry>
- </row>
- <row>
- <entry><para> 1.6.5</para></entry>
- <entry><para> => 4TB</para></entry>
- <entry><para> <= 4TB of storage</para></entry>
- </row>
- <row>
- <entry><para> Future Lustre versions</para></entry>
- <entry><para> => 4TB</para></entry>
- <entry><para> No storage limit</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </listitem>
-</itemizedlist>
- </section>
- <section remap="h3">
- <title>21.5.3 <anchor xml:id="dbdoclet.50438217_66360" xreflabel=""/>Quota <anchor xml:id="dbdoclet.50438217_marker-1290326" xreflabel=""/>File Formats</title>
- <para>Lustre 1.6.5 introduced the v2 file format for administrative quotas, with 64-bit limits that support large-limits handling. The old quota file format (v1), with 32-bit limits, is also supported. Lustre 1.6.6 introduced the v2 file format for operational quotas. A few notes regarding the current quota file formats:</para>
- <para>Lustre 1.6.5 and later use mdt.quota_type to force a specific administrative quota version (v2 or v1).</para>
- <itemizedlist><listitem>
- <para> For the v2 quota file format, (OBJECTS/admin_quotafile_v2.{usr,grp})</para>
- </listitem>
-
-<listitem>
- <para> For the v1 quota file format, (OBJECTS/admin_quotafile.{usr,grp})</para>
- </listitem>
-
-</itemizedlist>
- <para>Lustre 1.6.6 and later use ost.quota_type to force a specific operational quota version (v2 or v1).</para>
- <itemizedlist><listitem>
- <para> For the v2 quota file format, (lquota_v2.{user,group})</para>
- </listitem>
-
-<listitem>
- <para> For the v1 quota file format, (lquota.{user,group})</para>
- </listitem>
-
-</itemizedlist>
- <para>The quota_type specifier can be used to set different combinations of administrative/operational quota file versions on a Lustre node:</para>
- <itemizedlist><listitem>
- <para> "1" - v1 (32-bit) administrative quota file, v1 (32-bit) operational quota file (default in releases before Lustre 1.6.5)</para>
- </listitem>
-
-<listitem>
- <para> "2" - v2 (64-bit) administrative quota file, v1 (32-bit) operational quota file (default in Lustre 1.6.5)</para>
- </listitem>
-
-<listitem>
- <para> "3" - v2 (64-bit) administrative quota file, v2 (64-bit) operational quota file (default in releases after Lustre 1.6.5)</para>
- </listitem>
-
-</itemizedlist>
- <para>If quotas do not exist or look broken, then quotacheck creates quota files of a required name and format.</para>
- <para>If Lustre is using the v2 quota file format when only v1 quota files exist, then quotacheck converts old v1 quota files to new v2 quota files. This conversion is triggered automatically, and is transparent to users. If an old quota file does not exist or looks broken, then the new v2 quota file will be empty. In case of an error, details can be found in the kernel log of the corresponding MDS/OST. During conversion of a v1 quota file to a v2 quota file, the v2 quota file is marked as broken, to avoid it being used if a crash occurs. The quota module does not use broken quota files (keeping quota off).</para>
- <para>In most situations, Lustre administrators do not need to set specific versioning options. Upgrading Lustre without using quota_type to force specific quota file versions results in quota files being upgraded automatically to the latest version. The option ensures backward compatibility, preventing a quota file upgrade to a version which is not supported by earlier Lustre versions.</para>
- </section>
+ </section>
+ <section xml:id="dbdoclet.50438217_27895">
+ <title>21.5 Known Issues with Quotas</title>
+ <para>Using quotas in Lustre can be complex and there are several known issues.</para>
+ <section remap="h3">
+ <title>21.5.1 Granted Cache and Quota Limits</title>
+ <para>In Lustre, granted cache does not respect quota limits. In this situation, OSTs grant cache to Lustre client to accelerate I/O. Granting cache causes writes to be successful in OSTs, even if they exceed the quota limits, and will overwrite them.</para>
+ <para>The sequence is:</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">A user writes files to Lustre.</emphasis></para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">If the Lustre client has enough granted cache, then it returns 'success' to users and arranges the writes to the OSTs.</emphasis></para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Because Lustre clients have delivered success to users, the OSTs cannot fail these writes.</emphasis></para>
+ </listitem>
+ </orderedlist>
+ <para>Because of granted cache, writes always overwrite quota limitations. For example, if you set a 400 GB quota on user A and use IOR to write for user A from a bundle of clients, you will write much more data than 400 GB, and cause an out-of-quota error (<literal>-EDQUOT</literal>).</para>
+ <note>
+ <para>The effect of granted cache on quota limits can be mitigated, but not eradicated. Reduce the <literal>max_dirty_buffer</literal> in the clients (can be set from 0 to 512). To set <literal>max_dirty_buffer</literal> to 0:</para>
+ <para> * In releases after Lustre 1.6.5, <literal>lctl set_param osc.*.max_dirty_mb=0</literal>.</para>
+ <para> * In releases before Lustre 1.6.5, <literal>proc/fs/lustre/osc/*/max_dirty_mb; do echo 512 > $O</literal></para>
+ </note>
</section>
- <section xml:id="dbdoclet.50438217_20772">
- <title>21.6 Lustre <anchor xml:id="dbdoclet.50438217_marker-1290343" xreflabel=""/>Quota Statistics</title>
- <para>Lustre includes statistics that monitor quota activity, such as the kinds of quota RPCs sent during a specific period, the average time to complete the RPCs, etc. These statistics are useful to measure performance of a Lustre file system.</para>
- <para>Each quota statistic consists of a quota event and min_time, max_time and sum_time values for the event.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Quota Event</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">sync_acq_req</emphasis></para></entry>
- <entry><para> Quota slaves send a acquiring_quota request and wait for its return.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">sync_rel_req</emphasis></para></entry>
- <entry><para> Quota slaves send a releasing_quota request and wait for its return.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">async_acq_req</emphasis></para></entry>
- <entry><para> Quota slaves send an acquiring_quota request and do not wait for its return.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">async_rel_req</emphasis></para></entry>
- <entry><para> Quota slaves send a releasing_quota request and do not wait for its return.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">wait_for_blk_quota (lquota_chkquota)</emphasis></para></entry>
- <entry><para> Before data is written to OSTs, the OSTs check if the remaining block quota is sufficient. This is done in the lquota_chkquota function.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">wait_for_ino_quota (lquota_chkquota)</emphasis></para></entry>
- <entry><para> Before files are created on the MDS, the MDS checks if the remaining inode quota is sufficient. This is done in the lquota_chkquota function.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">wait_for_blk_quota (lquota_pending_commit)</emphasis></para></entry>
- <entry><para> After blocks are written to OSTs, relative quota information is updated. This is done in the lquota_pending_commit function.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">wait_for_ino_quota (lquota_pending_commit)</emphasis></para></entry>
- <entry><para> After files are created, relative quota information is updated. This is done in the lquota_pending_commit function.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">wait_for_pending_blk_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
- <entry><para> On the MDS or OSTs, there is one thread sending a quota request for a specific UID/GID for block quota at any time. At that time, if other threads need to do this too, they should wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">wait_for_pending_ino_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
- <entry><para> On the MDS, there is one thread sending a quota request for a specific UID/GID for inode quota at any time. If other threads need to do this too, they should wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">nowait_for_pending_blk_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
- <entry><para> On the MDS or OSTs, there is one thread sending a quota request for a specific UID/GID for block quota at any time. When threads enter qctxt_wait_pending_dqacq, they do not need to wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">nowait_for_pending_ino_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
- <entry><para> On the MDS, there is one thread sending a quota request for a specific UID/GID for inode quota at any time. When threads enter qctxt_wait_pending_dqacq, they do not need to wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">quota_ctl</emphasis></para></entry>
- <entry><para> The quota_ctl statistic is generated when lfs setquota, lfs quota and so on, are issued.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">adjust_qunit</emphasis></para></entry>
- <entry><para> Each time qunit is adjusted, it is counted.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <section remap="h3">
- <title>21.6.1 Interpreting Quota Statistics</title>
- <para>Quota statistics are an important measure of a Lustre file system's performance. Interpreting these statistics correctly can help you diagnose problems with quotas, and may indicate adjustments to improve system performance.</para>
- <para>For example, if you run this command on the OSTs:</para>
- <screen>cat /proc/fs/lustre/lquota/lustre-OST0000/stats
-</screen>
- <para>You will get a result similar to this:</para>
- <screen>snapshot_time 1219908615.506895 secs.usecs
+ <section remap="h3">
+ <title>21.5.2 Quota Limits</title>
+ <para>Available quota limits depend on the Lustre version you are using.</para>
+ <itemizedlist>
+ <listitem>
+ <para> Lustre version 1.4.11 and earlier (for 1.4.x releases) and Lustre version 1.6.4 and earlier (for 1.6.x releases) support quota limits less than 4 TB.</para>
+ </listitem>
+ <listitem>
+ <para> Lustre versions 1.4.12, 1.6.5 and later support quota limits of 4 TB and greater in Lustre configurations with OST storage limits of 4 TB and less.</para>
+ </listitem>
+ <listitem>
+ <para> Future Lustre versions are expected to support quota limits of 4 TB and greater with no OST storage limits.</para>
+ </listitem>
+ <listitem>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="33*"/>
+ <colspec colname="c2" colwidth="33*"/>
+ <colspec colname="c3" colwidth="33*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Lustre Version</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Quota Limit Per User/Per Group</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">OST Storage Limit</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> 1.4.11 and earlier</para>
+ </entry>
+ <entry>
+ <para> < 4TB</para>
+ </entry>
+ <entry>
+ <para> n/a</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> 1.4.12</para>
+ </entry>
+ <entry>
+ <para> => 4TB</para>
+ </entry>
+ <entry>
+ <para> <= 4TB of storage</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> 1.6.4 and earlier</para>
+ </entry>
+ <entry>
+ <para> < 4TB</para>
+ </entry>
+ <entry>
+ <para> n/a</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> 1.6.5</para>
+ </entry>
+ <entry>
+ <para> => 4TB</para>
+ </entry>
+ <entry>
+ <para> <= 4TB of storage</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> Future Lustre versions</para>
+ </entry>
+ <entry>
+ <para> => 4TB</para>
+ </entry>
+ <entry>
+ <para> No storage limit</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="dbdoclet.50438217_66360">
+ <title>21.5.3 Quota File Formats</title>
+ <para>Lustre 1.6.5 introduced the v2 file format for administrative quotas, with 64-bit limits that support large-limits handling. The old quota file format (v1), with 32-bit limits, is also supported. Lustre 1.6.6 introduced the v2 file format for operational quotas. A few notes regarding the current quota file formats:</para>
+ <para>Lustre 1.6.5 and later use <literal>mdt.quota_type</literal> to force a specific administrative quota version (v2 or v1).</para>
+ <itemizedlist>
+ <listitem>
+ <para> For the v2 quota file format, (<literal>OBJECTS/admin_quotafile_v2.{usr,grp}</literal>)</para>
+ </listitem>
+ <listitem>
+ <para> For the v1 quota file format, (<literal>OBJECTS/admin_quotafile.{usr,grp}</literal>)</para>
+ </listitem>
+ </itemizedlist>
+ <para>Lustre 1.6.6 and later use <literal>ost.quota_type</literal> to force a specific operational quota version (v2 or v1).</para>
+ <itemizedlist>
+ <listitem>
+ <para>For the v2 quota file format, (<literal>lquota_v2.{user,group}</literal>)</para>
+ </listitem>
+ <listitem>
+ <para>For the v1 quota file format, (<literal>lquota.{user,group}</literal>)</para>
+ </listitem>
+ </itemizedlist>
+ <para>The quota_type specifier can be used to set different combinations of administrative/operational quota file versions on a Lustre node:</para>
+ <itemizedlist>
+ <listitem>
+ <para>"1" - v1 (32-bit) administrative quota file, v1 (32-bit) operational quota file (default in releases <emphasis role="underline">before</emphasis> Lustre 1.6.5)</para>
+ </listitem>
+ <listitem>
+ <para>"2" - v2 (64-bit) administrative quota file, v1 (32-bit) operational quota file (default in Lustre 1.6.5)</para>
+ </listitem>
+ <listitem>
+ <para>"3" - v2 (64-bit) administrative quota file, v2 (64-bit) operational quota file (default in releases <emphasis role="underline">after</emphasis> Lustre 1.6.5)</para>
+ </listitem>
+ </itemizedlist>
+ <para>If quotas do not exist or look broken, then <literal>quotacheck</literal> creates quota files of a required name and format.</para>
+ <para>If Lustre is using the v2 quota file format when only v1 quota files exist, then <literal>quotacheck</literal> converts old v1 quota files to new v2 quota files. This conversion is triggered automatically, and is transparent to users. If an old quota file does not exist or looks broken, then the new v2 quota file will be empty. In case of an error, details can be found in the kernel log of the corresponding MDS/OST. During conversion of a v1 quota file to a v2 quota file, the v2 quota file is marked as broken, to avoid it being used if a crash occurs. The quota module does not use broken quota files (keeping quota off).</para>
+ <para>In most situations, Lustre administrators do not need to set specific versioning options. Upgrading Lustre without using <literal>quota_type</literal> to force specific quota file versions results in quota files being upgraded automatically to the latest version. The option ensures backward compatibility, preventing a quota file upgrade to a version which is not supported by earlier Lustre versions.</para>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438217_20772">
+ <title>21.6 Lustre Quota Statistics</title>
+ <para>Lustre includes statistics that monitor quota activity, such as the kinds of quota RPCs sent during a specific period, the average time to complete the RPCs, etc. These statistics are useful to measure performance of a Lustre file system.</para>
+ <para>Each quota statistic consists of a quota event and <literal>min_time</literal>, <literal>max_time</literal> and <literal>sum_time</literal> values for the event.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Quota Event</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">sync_acq_req</emphasis></para>
+ </entry>
+ <entry>
+ <para> Quota slaves send a acquiring_quota request and wait for its return.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">sync_rel_req</emphasis></para>
+ </entry>
+ <entry>
+ <para> Quota slaves send a releasing_quota request and wait for its return.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">async_acq_req</emphasis></para>
+ </entry>
+ <entry>
+ <para> Quota slaves send an acquiring_quota request and do not wait for its return.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">async_rel_req</emphasis></para>
+ </entry>
+ <entry>
+ <para> Quota slaves send a releasing_quota request and do not wait for its return.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">wait_for_blk_quota (lquota_chkquota)</emphasis></para>
+ </entry>
+ <entry>
+ <para> Before data is written to OSTs, the OSTs check if the remaining block quota is sufficient. This is done in the lquota_chkquota function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">wait_for_ino_quota (lquota_chkquota)</emphasis></para>
+ </entry>
+ <entry>
+ <para> Before files are created on the MDS, the MDS checks if the remaining inode quota is sufficient. This is done in the lquota_chkquota function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">wait_for_blk_quota (lquota_pending_commit)</emphasis></para>
+ </entry>
+ <entry>
+ <para> After blocks are written to OSTs, relative quota information is updated. This is done in the lquota_pending_commit function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">wait_for_ino_quota (lquota_pending_commit)</emphasis></para>
+ </entry>
+ <entry>
+ <para> After files are created, relative quota information is updated. This is done in the lquota_pending_commit function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">wait_for_pending_blk_quota_req (qctxt_wait_pending_dqacq)</emphasis></para>
+ </entry>
+ <entry>
+ <para> On the MDS or OSTs, there is one thread sending a quota request for a specific UID/GID for block quota at any time. At that time, if other threads need to do this too, they should wait. This is done in the qctxt_wait_pending_dqacq function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">wait_for_pending_ino_quota_req (qctxt_wait_pending_dqacq)</emphasis></para>
+ </entry>
+ <entry>
+ <para> On the MDS, there is one thread sending a quota request for a specific UID/GID for inode quota at any time. If other threads need to do this too, they should wait. This is done in the qctxt_wait_pending_dqacq function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">nowait_for_pending_blk_quota_req (qctxt_wait_pending_dqacq)</emphasis></para>
+ </entry>
+ <entry>
+ <para> On the MDS or OSTs, there is one thread sending a quota request for a specific UID/GID for block quota at any time. When threads enter qctxt_wait_pending_dqacq, they do not need to wait. This is done in the qctxt_wait_pending_dqacq function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">nowait_for_pending_ino_quota_req (qctxt_wait_pending_dqacq)</emphasis></para>
+ </entry>
+ <entry>
+ <para> On the MDS, there is one thread sending a quota request for a specific UID/GID for inode quota at any time. When threads enter qctxt_wait_pending_dqacq, they do not need to wait. This is done in the qctxt_wait_pending_dqacq function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">quota_ctl</emphasis></para>
+ </entry>
+ <entry>
+ <para> The quota_ctl statistic is generated when lfs <literal>setquota</literal>, <literal>lfs quota</literal> and so on, are issued.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">adjust_qunit</emphasis></para>
+ </entry>
+ <entry>
+ <para> Each time qunit is adjusted, it is counted.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <section remap="h3">
+ <title>21.6.1 Interpreting Quota Statistics</title>
+ <para>Quota statistics are an important measure of a Lustre file system's performance. Interpreting these statistics correctly can help you diagnose problems with quotas, and may indicate adjustments to improve system performance.</para>
+ <para>For example, if you run this command on the OSTs:</para>
+ <screen>cat /proc/fs/lustre/lquota/lustre-OST0000/stats</screen>
+ <para>You will get a result similar to this:</para>
+ <screen>snapshot_time 1219908615.506895 secs.usecs
async_acq_req 1 samples [us] 32 32 32
async_rel_req 1 samples [us] 5 5 5
nowait_for_pending_blk_quota_req(qctxt_wait_pending_dqacq) 1 samples [us] 2\
2 2
quota_ctl 4 samples [us] 80 3470 4293
adjust_qunit 1 samples [us] 70 70 70
-....
-</screen>
- <para>In the first line, snapshot_time indicates when the statistics were taken. The remaining lines list the quota events and their associated data.</para>
- <para>In the second line, the async_acq_req event occurs one time. The min_time, max_time and sum_time statistics for this event are 32, 32 and 32, respectively. The unit is microseconds (s).</para>
- <para>In the fifth line, the quota_ctl event occurs four times. The min_time, max_time and sum_time statistics for this event are 80, 3470 and 4293, respectively. The unit is microseconds (s).</para>
- </section>
+....</screen>
+ <para>In the first line, <literal>snapshot_time</literal> indicates when the statistics were taken. The remaining lines list the quota events and their associated data.</para>
+ <para>In the second line, the <literal>async_acq_req</literal> event occurs one time. The <literal>min_time</literal>, <literal>max_time</literal> and <literal>sum_time</literal> statistics for this event are 32, 32 and 32, respectively. The unit is microseconds (μs).</para>
+ <para>In the fifth line, the quota_ctl event occurs four times. The <literal>min_time</literal>, <literal>max_time</literal> and <literal>sum_time</literal> statistics for this event are 80, 3470 and 4293, respectively. The unit is microseconds (μs).</para>
</section>
+ </section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='lnetselftest'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lnetselftest">
<info>
- <title xml:id='lnetselftest.title'>Testing Lustre Network Performance (LNET Self-Test)</title>
+ <title xml:id="lnetselftest.title">Testing Lustre Network Performance (LNET Self-Test)</title>
</info>
<para><anchor xml:id="dbdoclet.50438223_36273" xreflabel=""/>This chapter describes the LNET self-test, which is used by site administrators to confirm that Lustre Networking (LNET) has been properly installed and configured, and that underlying network software and hardware are performing according to expectations. The chapter includes:</para>
-
- <itemizedlist><listitem>
- <para><xref linkend="dbdoclet.50438223_91742"/></para>
- </listitem><listitem>
- <para><xref linkend="dbdoclet.50438223_48138"/></para>
- </listitem><listitem>
- <para><xref linkend="dbdoclet.50438223_27277"/></para>
- </listitem></itemizedlist>
-
- <section xml:id="dbdoclet.50438223_91742">
- <title>23.1 LNET Self-Test Overview</title>
- <para>LNET self-test is a kernel module that runs over LNET and the Lustre network drivers (LNDs. It is designed to:</para>
- <itemizedlist><listitem>
- <para> Test the connection ability of the Lustre network</para>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438223_91742"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438223_48138"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438223_27277"/></para>
+ </listitem>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438223_91742">
+ <title>23.1 LNET Self-Test Overview</title>
+ <para>LNET self-test is a kernel module that runs over LNET and the Lustre network drivers (LNDs. It is designed to:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Test the connection ability of the Lustre network</para>
+ </listitem>
+ <listitem>
+ <para> Run regression tests of the Lustre network</para>
+ </listitem>
+ <listitem>
+ <para> Test performance of the Lustre network</para>
+ </listitem>
+ </itemizedlist>
+ <para>After you have obtained performance results for your Lustre network, refer to <xref linkend="lustretuning"/> for information about parameters that can be used to tune LNET for optimum performance.</para>
+ <note>
+ <para>Apart from the performance impact, LNET self-test is invisible to Lustre.</para>
+ </note>
+ <para>An LNET self-test cluster includes two types of nodes:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">Console node</emphasis> - A node used to control and monitor an LNET self-test cluster. The console node serves as the user interface of the LNET self-test system and can be any node in the test cluster. All self-test commands are entered from the console node. From the console node, a user can control and monitor the status of the entire LNET self-test cluster (session). The console node is exclusive in that a user cannot control two different sessions from one console node.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Test nodes</emphasis> - The nodes on which the tests are run. Test nodes are controlled by the user from the console node; the user does not need to log into them directly.</para>
+ </listitem>
+ </itemizedlist>
+ <para>LNET self-test has two user utilities:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">
+ <literal>lst</literal>
+ </emphasis> - The user interface for the self-test console (run on the console node). It provides a list of commands to control the entire test system, including commands to create a session, create test groups, etc.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">
+ <literal>lstclient</literal>
+ </emphasis> - The userspace LNET self-test program (run on a test node). The lstclient utility is linked with userspace LNDs and LNET. This utility is not needed if only kernel space LNET and LNDs are used.</para>
+ </listitem>
+ </itemizedlist>
+ <note>
+ <para>Test nodes can be in either kernel or userspace. A console user can invite a kernel test node to join the test session by running lstadd_groupNID, but the console user cannot actively add a userspace test node to the test-session. However, the console user can passively accept a test node to the test session while the test node is running lstclient to connect to the console.</para>
+ </note>
+ <section remap="h3">
+ <title>23.1.1 Prerequisites</title>
+ <para>To run LNET self-test, these modules must be loaded:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>libcfs</literal></para>
</listitem>
-
-<listitem>
- <para> Run regression tests of the Lustre network</para>
+ <listitem>
+ <para><literal>net</literal></para>
</listitem>
-
-<listitem>
- <para> Test performance of the Lustre network</para>
+ <listitem>
+ <para><literal>lnet_selftest</literal></para>
</listitem>
-
-</itemizedlist>
-<para>After you have obtained performance results for your Lustre network, refer to <xref linkend='lustretuning'/> for information about parameters that can be used to tune LNET for optimum performance.</para>
- <note><para>Apart from the performance impact, LNET self-test is invisible to Lustre.</para></note>
-
- <para>An LNET self-test cluster includes two types of nodes:</para>
- <itemizedlist><listitem>
- <para><emphasis role="bold">Console node</emphasis> - A node used to control and monitor an LNET self-test cluster. The console node serves as the user interface of the LNET self-test system and can be any node in the test cluster. All self-test commands are entered from the console node. From the console node, a user can control and monitor the status of the entire LNET self-test cluster (session). The console node is exclusive in that a user cannot control two different sessions from one console node.</para>
+ <listitem>
+ <para> One of the <literal>klnds</literal> (i.e, <literal>ksocklnd</literal>, <literal>ko2iblnd</literal>...) as needed by your network configuration</para>
</listitem>
-
-<listitem>
- <para><emphasis role="bold">Test nodes</emphasis> - The nodes on which the tests are run. Test nodes are controlled by the user from the console node; the user does not need to log into them directly.</para>
+ </itemizedlist>
+ <para>To load the required modules, run:</para>
+ <screen>modprobe lnet_selftest </screen>
+ <para>This command recursively loads the modules on which LNET self-test depends.</para>
+ <note>
+ <para>While the console and test nodes require all the prerequisite modules to be loaded, userspace test nodes do not require these modules.</para>
+ </note>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438223_48138">
+ <title>23.2 Using LNET Self-Test</title>
+ <para>This section describes how to create and run an LNET self-test. The examples shown are for a test that simulates the traffic pattern of a set of Lustre servers on a TCP network accessed by Lustre clients on an InfiniBand network connected via LNET routers. In this example, half the clients are reading and half the clients are writing.</para>
+ <section remap="h3">
+ <title>23.2.1 Creating a Session</title>
+ <para>A <emphasis>session</emphasis> is a set of processes that run on a test node. Only one session can be run at a time on a test node to ensure that the session has exclusive use of the node. The console node is used to create, change or destroy a session (new_session, end_session, show_session). For more about session parameters, see <xref linkend="dbdoclet.50438223_91247">Session Commands</xref>.</para>
+ <para>Almost all operations should be performed within the context of a session. From the console node, a user can only operate nodes in his own session. If a session ends, the session context in all test nodes is stopped.</para>
+ <para>The following commands set the LST_SESSION environment variable to identify the session on the console node and create a session called read_write:</para>
+ <screen>export LST_SESSION=$$
+lst new_session read_write</screen>
+ </section>
+ <section remap="h3">
+ <title>23.2.2 Setting Up Groups</title>
+ <para>A <emphasis>group</emphasis> is a named collection of nodes. Any number of groups can exist in a single LNET self-test session. Group membership is not restricted in that a node can be included in any number of groups.</para>
+ <para>Each node in a group has a rank, determined by the order in which it was added to the group. The rank is used to establish test traffic patterns.</para>
+ <para>A user can only control nodes in his/her session. To allocate nodes to the session, the user needs to add nodes to a group (of the session). All nodes in a group can be referenced by the group name. A node can be allocated to multiple groups of a session.</para>
+ <para>In the following example, three groups are established:</para>
+ <screen>lst add_group servers 192.168.10.[8,10,12-16]@tcp
+lst add_group readers 192.168.1.[1-253/2]@o2ib
+lst add_group writers 192.168.1.[2-254/2]@o2ib</screen>
+ <para>These three groups include:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Nodes that will function as 'servers' to be accessed by 'clients' during the LNET self-test session</para>
</listitem>
-
-</itemizedlist>
- <para>LNET self-test has two user utilities:</para>
- <itemizedlist><listitem>
- <para><emphasis role="bold">lst</emphasis> - The user interface for the self-test console (run on the console node). It provides a list of commands to control the entire test system, including commands to create a session, create test groups, etc.</para>
+ <listitem>
+ <para>Nodes that will function as 'clients' that will simulate <emphasis>reading</emphasis> data from the 'servers'</para>
</listitem>
-
-<listitem>
- <para><emphasis role="bold">lstclient</emphasis> - The userspace LNET self-test program (run on a test node). The lstclient utility is linked with userspace LNDs and LNET. This utility is not needed if only kernel space LNET and LNDs are used.</para>
+ <listitem>
+ <para>Nodes that will function as 'clients' that will simulate <emphasis>writing</emphasis> data to the 'servers'</para>
</listitem>
-
-</itemizedlist>
- <note><para>Test nodes can be in either kernel or userspace. A console user can invite a kernel test node to join the test session by running lstadd_groupNID, but the console user cannot actively add a userspace test node to the test-session. However, the console user can passively accept a test node to the test session while the test node is running lstclient to connect to the console.</para></note>
-
- <section remap="h3">
- <title>23.1.1 Prerequisites</title>
- <para>To run LNET self-test, these modules must be loaded:</para>
- <itemizedlist><listitem>
- <para>libcfs</para>
- </listitem>
-
-<listitem>
- <para>net</para>
- </listitem>
-
-<listitem>
- <para>lnet_selftest</para>
- </listitem>
-
-<listitem>
- <para> One of the klnds (i.e, ksocklnd, ko2iblnd...) as needed by your network configuration</para>
- </listitem>
-
-</itemizedlist>
- <para>To load the required modules, run:</para>
- <screen>modprobe lnet_selftest
-</screen>
- <para>This command recursively loads the modules on which LNET self-test depends.</para>
- <note><para>While the console and test nodes require all the prerequisite modules to be loaded, userspace test nodes do not require these modules.</para></note>
-
- </section>
+ </itemizedlist>
+ <note>
+ <para>A console user can associate kernel space test nodes with the session by running lst <literal>add_group</literal> <literal>NIDs</literal>, but a userspace test node cannot be actively added to the session. However, the console user can passively "accept" a test node to associate with a test session while the test node running <literal>lstclient</literal> connects to the console node, i.e: <literal>lstclient --sesid CONSOLE_NID --group NAME</literal>).</para>
+ </note>
</section>
- <section xml:id="dbdoclet.50438223_48138">
- <title>23.2 Using LNET Self-Test</title>
- <para>This section describes how to create and run an LNET self-test. The examples shown are for a test that simulates the traffic pattern of a set of Lustre servers on a TCP network accessed by Lustre clients on an InfiniBand network connected via LNET routers. In this example, half the clients are reading and half the clients are writing.</para>
- <section remap="h3">
- <title>23.2.1 Creating a Session</title>
- <para>A <emphasis>session</emphasis> is a set of processes that run on a test node. Only one session can be run at a time on a test node to ensure that the session has exclusive use of the node. The console node is used to create, change or destroy a session (new_session, end_session, show_session). For more about session parameters, see <link xl:href="LNETSelfTest.html#50438223_91247">Session Commands</link>.</para>
- <para>Almost all operations should be performed within the context of a session. From the console node, a user can only operate nodes in his own session. If a session ends, the session context in all test nodes is stopped.</para>
- <para>The following commands set the LST_SESSION environment variable to identify the session on the console node and create a session called read_write:</para>
- <screen>export LST_SESSION=$$
-lst new_session read_write
-</screen>
- </section>
- <section remap="h3">
- <title>23.2.2 Setting Up Groups</title>
- <para>A <emphasis>group</emphasis> is a named collection of nodes. Any number of groups can exist in a single LNET self-test session. Group membership is not restricted in that a node can be included in any number of groups.</para>
- <para>Each node in a group has a rank, determined by the order in which it was added to the group. The rank is used to establish test traffic patterns.</para>
- <para>A user can only control nodes in his/her session. To allocate nodes to the session, the user needs to add nodes to a group (of the session). All nodes in a group can be referenced by the group name. A node can be allocated to multiple groups of a session.</para>
- <para>In the following example, three groups are established:</para>
- <screen>lst add_group servers 192.168.10.[8,10,12-16]@tcp
-lst add_group readers 192.168.1.[1-253/2]@o2ib
-lst add_group writers 192.168.1.[2-254/2]@o2ib
-</screen>
- <para>These three groups include:</para>
- <itemizedlist><listitem>
- <para> Nodes that will function as 'servers' to be accessed by 'clients' during the LNET self-test session</para>
- </listitem>
-
-<listitem>
- <para> Nodes that will function as 'clients' that will simulate <emphasis>reading</emphasis> data from the 'servers'</para>
- </listitem>
-
-<listitem>
- <para> Nodes that will function as 'clients' that will simulate <emphasis>writing</emphasis> data to the 'servers'</para>
- </listitem>
-
-</itemizedlist>
- <note><para>A console user can associate kernel space test nodes with the session by running lst add_group NIDs, but a userspace test node cannot be actively added to the session. However, the console user can passively "accept" a test node to associate with a test session while the test node running lstclient connects to the console node, i.e: lstclient --sesid CONSOLE_NID --group NAME).</para></note>
- </section>
- <section remap="h3">
- <title>23.2.3 <anchor xml:id="dbdoclet.50438223_42848" xreflabel=""/>Defining and Running the Tests</title>
- <para>A <emphasis>test</emphasis> generates a network load between two groups of nodes, a source group identified using the --from parameter and a target group identified using the --to parameter. When a test is running, each node in the --from<emphasis><group></emphasis> simulates a client by sending requests to nodes in the --to<emphasis><group></emphasis>, which are simulating a set of servers, and then receives responses in return. This activity is designed to mimic Lustre RPC traffic.</para>
- <para>A <emphasis>batch</emphasis> is a collection of tests that are started and stopped together and run in parallel. A test must always be run as part of a batch, even if it is just a single test. Users can only run or stop a test batch, not individual tests.</para>
- <para>Tests in a batch are non-destructive to the file system, and can be run in a normal Lustre environment (provided the performance impact is acceptable).</para>
- <para>A simple batch might contain a single test, for example, to determine whether the network bandwidth presents an I/O bottleneck. In this example, the --to<emphasis><group></emphasis> could be comprised of Lustre OSSs and --from<emphasis><group></emphasis> the compute nodes. A second test could be added to perform pings from a login node to the MDS to see how checkpointing affects the ls -l process.</para>
- <para>Two types of tests are available:</para>
- <itemizedlist><listitem>
- <para><emphasis role="bold">ping -</emphasis> A ping generates a short request message, which results in a short response. Pings are useful to determine latency and small message overhead and to simulate Lustre metadata traffic.</para>
- </listitem>
-
-<listitem>
- <para><emphasis role="bold">brw -</emphasis> In a brw ('bulk read write') test, data is transferred from the target to the source (brwread) or data is transferred from the source to the target (brwwrite). The size of the bulk transfer is set using the size parameter. A brw test is useful to determine network bandwidth and to simulate Lustre I/O traffic.</para>
- </listitem>
-
-</itemizedlist>
- <para>In the example below, a batch is created called bulk_rw. Then two brw tests are added. In the first test, 1M of data is sent from the servers to the clients as a simulated read operation with a simple data validation check. In the second test, 4K of data is sent from the clients to the servers as a simulated write operation with a full data validation check.</para>
- <screen>lst add_batch bulk_rw
+ <section xml:id='dbdoclet.50438223_42848'>
+ <title>23.2.3 Defining and Running the Tests</title>
+ <para>A <emphasis>test</emphasis> generates a network load between two groups of nodes, a source group identified using the <literal>--from</literal> parameter and a target group identified using the <literal>--to</literal> parameter. When a test is running, each node in the <literal>--from<emphasis><group></emphasis></literal> simulates a client by sending requests to nodes in the <literal>--to<emphasis><group></emphasis></literal>, which are simulating a set of servers, and then receives responses in return. This activity is designed to mimic Lustre RPC traffic.</para>
+ <para>A <emphasis>batch</emphasis> is a collection of tests that are started and stopped together and run in parallel. A test must always be run as part of a batch, even if it is just a single test. Users can only run or stop a test batch, not individual tests.</para>
+ <para>Tests in a batch are non-destructive to the file system, and can be run in a normal Lustre environment (provided the performance impact is acceptable).</para>
+ <para>A simple batch might contain a single test, for example, to determine whether the network bandwidth presents an I/O bottleneck. In this example, the <literal>--to<emphasis><group></emphasis></literal> could be comprised of Lustre OSSs and <literal>--from<emphasis><group></emphasis></literal> the compute nodes. A second test could be added to perform pings from a login node to the MDS to see how checkpointing affects the <literal>ls -l</literal> process.</para>
+ <para>Two types of tests are available:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold"><literal>ping</literal> -</emphasis> A <literal>ping</literal> generates a short request message, which results in a short response. Pings are useful to determine latency and small message overhead and to simulate Lustre metadata traffic.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold"><literal>brw</literal> -</emphasis> In a <literal>brw</literal> ('bulk read write') test, data is transferred from the target to the source (<literal>brwread</literal>) or data is transferred from the source to the target (<literal>brwwrite</literal>). The size of the bulk transfer is set using the <literal>size</literal> parameter. A brw test is useful to determine network bandwidth and to simulate Lustre I/O traffic.</para>
+ </listitem>
+ </itemizedlist>
+ <para>In the example below, a batch is created called <literal>bulk_rw</literal>. Then two <literal>brw</literal> tests are added. In the first test, 1M of data is sent from the servers to the clients as a simulated read operation with a simple data validation check. In the second test, 4K of data is sent from the clients to the servers as a simulated write operation with a full data validation check.</para>
+ <screen>lst add_batch bulk_rw
lst add_test --batch bulk_rw --from readers --to servers \
-brw read check=simple size=1M
+ brw read check=simple size=1M
lst add_test --batch bulk_rw --from writers --to servers \
-brw write check=full size=4K
-</screen>
-<para>The traffic pattern and test intensity is determined by several properties such as test type, distribution of test nodes, concurrency of test, and RDMA operation type. For more details, see <xref linkend='dbdoclet.50438223_36860'/>.</para>
- </section>
- <section remap="h3">
- <title>23.2.4 Sample Script</title>
- <para>This sample LNET self-test script simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an InfiniBand network (connected via LNET routers). In this example, half the clients are reading and half the clients are writing.</para>
- <para>Run this script on the console node:</para>
- <screen>#!/bin/bash
+ brw write check=full size=4K</screen>
+ <para>The traffic pattern and test intensity is determined by several properties such as test type, distribution of test nodes, concurrency of test, and RDMA operation type. For more details, see <xref linkend="dbdoclet.50438223_36860"/>.</para>
+ </section>
+ <section remap="h3">
+ <title>23.2.4 Sample Script</title>
+ <para>This sample LNET self-test script simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an InfiniBand network (connected via LNET routers). In this example, half the clients are reading and half the clients are writing.</para>
+ <para>Run this script on the console node:</para>
+ <screen>#!/bin/bash
export LST_SESSION=$$
lst new_session read/write
lst add_group servers 192.168.10.[8,10,12-16]@tcp
# display server stats for 30 seconds
lst stat servers & sleep 30; kill $!
# tear down
-lst end_session
-</screen>
- <note><para>This script can be easily adapted to pass the group NIDs by shell variables or command line arguments (making it good for general-purpose use).</para></note>
-
- </section>
+lst end_session</screen>
+ <note>
+ <para>This script can be easily adapted to pass the group NIDs by shell variables or command line arguments (making it good for general-purpose use).</para>
+ </note>
</section>
- <section xml:id="dbdoclet.50438223_27277">
- <title>23.3 LNET Self-Test <anchor xml:id="dbdoclet.50438223_marker-1298562" xreflabel=""/>Command Reference</title>
- <para>The LNET self-test (lst) utility is used to issue LNET self-test commands. The lst utility takes a number of command line arguments. The first argument is the command name and subsequent arguments are command-specific.</para>
- <section remap="h3">
- <title>23.3.1 <anchor xml:id="dbdoclet.50438223_91247" xreflabel=""/>Session Commands</title>
- <para>This section describes lst session commands.</para>
- <para><emphasis role="bold">LST_SESSION</emphasis></para>
- <para>The lst utility uses the LST_SESSION environmental variable to identify the session locally on the self-test console node. This should be a numeric value that uniquely identifies all session processes on the node. It is convenient to set this to the process ID of the shell both for interactive use and in shell scripts. Almost all lst commands require LST_SESSION to be set.</para>
- <para>Example:</para>
- <screen>export LST_SESSION=$$
-</screen>
- <para><emphasis role="bold">new_session [--timeout SECONDS] [--force] NAME</emphasis></para>
- <para>Creates a new session.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> --timeout<emphasis><seconds></emphasis></para></entry>
- <entry><para> Console timeout value of the session. The session ends automatically if it remains idle (i.e., no commands are issued) for this period.</para></entry>
- </row>
- <row>
- <entry><para> --force</para></entry>
- <entry><para> Ends conflicting sessions. This determines who 'wins' when one session conflicts with another. For example, if there is already an active session on this node, then the attempt to create a new session fails unless the -force flag is specified. If the -force flag is specified, then the active session is ended. Similarly, if a session attempts to add a node that is already 'owned' by another session, the -force flag allows this session to 'steal' the node.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis><name></emphasis></para></entry>
- <entry><para> A human-readable string to print when listing sessions or reporting session conflicts.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>$ lst new_session --force read_write
-</screen>
- <para><emphasis role="bold">end_session</emphasis></para>
- <para>Stops all operations and tests in the current session and clears the session'™s status.</para>
- <screen>$ lst end_session
-</screen>
- <para><emphasis role="bold">show_session</emphasis></para>
- <para>Shows the session information. This command prints information about the current session. It does not require LST_SESSION to be defined in the process environment.</para>
- <screen>$ lst show_session
-</screen>
- </section>
- <section remap="h3">
- <title>23.3.2 Group Commands</title>
- <para>This section describes lst group commands.</para>
- <para><emphasis role="bold">add_group</emphasis><emphasis><name> <NIDS> [<NIDs>...]</emphasis></para>
- <para>Creates the group and adds a list of test nodes to the group.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis><name></emphasis></para></entry>
- <entry><para> Name of the group.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis><NIDs></emphasis></para></entry>
- <entry><para> A string that may be expanded to include one or more LNET NIDs.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>$ lst add_group servers 192.168.10.[35,40-45]@tcp$ lst add_group clients 192.168.1.[10-100]@tcp 192.168.[2,4].\[10-20]@tcp</screen>
- <para> </para>
- <para><emphasis role="bold">update_group</emphasis><emphasis><name></emphasis><emphasis role="bold">[--refresh] [--clean</emphasis><emphasis><status></emphasis><emphasis role="bold">] [--remove</emphasis><emphasis><NIDs></emphasis><emphasis role="bold">]</emphasis></para>
- <para>Updates the state of nodes in a group or adjusts a group'™s membership. This command is useful if some nodes have crashed and should be excluded from the group.</para>
- <informaltable frame="all">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> --refresh</para></entry>
- <entry nameend="c3" namest="c2"><para> Refreshes the state of all inactive nodes in the group.</para></entry>
- </row>
- <row>
- <entry><para> --clean<emphasis><status></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> Removes nodes with a specified status from the group. Status may be:</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> active</para></entry>
- <entry><para> The node is in the current session.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> busy</para></entry>
- <entry><para> The node is now owned by another session.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> down</para></entry>
- <entry><para> The node has been marked down.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> unknown</para></entry>
- <entry><para> The node'™s status has yet to be determined.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> invalid</para></entry>
- <entry><para> Any state but active.</para></entry>
- </row>
- <row>
- <entry><para> --remove<emphasis><NIDs></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> Removes specified nodes from the group.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>$ lst update_group clients --refresh$ lst update_group clients --clean busy$ lst update_group clients --clean invalid // \invalid == busy || down || unknown$ lst update_group clients --remove \192.168.1.[10-20]@tcp</screen>
- <para> </para>
- <para><emphasis role="bold">list_group [</emphasis><emphasis><name></emphasis><emphasis role="bold">] [--active] [--busy] [--down] [--unknown] [--all]</emphasis></para>
- <para>Prints information about a group or lists all groups in the current session if no group is specified.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis><name></emphasis></para></entry>
- <entry><para> The name of the group.</para></entry>
- </row>
- <row>
- <entry><para> --active</para></entry>
- <entry><para> Lists the active nodes.</para></entry>
- </row>
- <row>
- <entry><para> --busy</para></entry>
- <entry><para> Lists the busy nodes.</para></entry>
- </row>
- <row>
- <entry><para> --down</para></entry>
- <entry><para> Lists the down nodes.</para></entry>
- </row>
- <row>
- <entry><para> --unknown</para></entry>
- <entry><para> Lists unknown nodes.</para></entry>
- </row>
- <row>
- <entry><para> --all</para></entry>
- <entry><para> Lists all nodes.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para> </para>
- <para>Example:</para>
- <screen>$ lst list_group1) clients2) serversTotal 2 groups$ lst list_group clientsACTIVE BUSY DOWN UNKNOWN TOTAL3 1 2 0 6$ lst list_group clients --all192.168.1.10@tcp Active192.168.1.11@tcp Active192.168.1.12@tcp Busy192.168.1.13@tcp Active192.168.1.14@tcp DOWN192.168.1.15@tcp DOWNTotal 6 nodes$ lst list_group clients --busy192.168.1.12@tcp BusyTotal 1 node</screen>
- <para> </para>
- <para><emphasis role="bold">del_group</emphasis><emphasis><name></emphasis></para>
- <para>Removes a group from the session. If the group is referred to by any test, then the operation fails. If nodes in the group are referred to only by this group, then they are kicked out from the current session; otherwise, they are still in the current session.</para>
- <screen>$ lst del_group clients
-</screen>
- <para><emphasis role="bold">lstclient --sesid</emphasis><emphasis><NID></emphasis><emphasis role="bold">--group</emphasis><emphasis><name></emphasis> [--server_mode]</para>
- <para>Use lstclient to run the userland self-test client. The lstclient command should be executed after creating a session on the console. There are only two mandatory options for lstclient:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> --sesid<emphasis><NID></emphasis></para></entry>
- <entry><para> The first console'™s NID.</para></entry>
- </row>
- <row>
- <entry><para> --group<emphasis><name></emphasis></para></entry>
- <entry><para> The test group to join.</para></entry>
- </row>
- <row>
- <entry><para> --server_mode</para></entry>
- <entry><para> When included, forces LNET to behave as a server, such as starting an acceptor if the underlying NID needs it or using privileged ports. Only root is allowed to use the --server_mode option.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>Console $ lst new_session testsession
+ </section>
+ <section xml:id="dbdoclet.50438223_27277">
+ <title>23.3 LNET Self-Test Command Reference</title>
+ <para>The LNET self-test (<literal>lst</literal>) utility is used to issue LNET self-test commands. The <literal>lst</literal> utility takes a number of command line arguments. The first argument is the command name and subsequent arguments are command-specific.</para>
+ <section xml:id="dbdoclet.50438223_91247">
+ <title>23.3.1 Session Commands</title>
+ <para>This section describes <literal>lst</literal> session commands.</para>
+ <para><emphasis role="bold">
+ <literal>LST_SESSION</literal>
+ </emphasis></para>
+ <para>The <literal>lst</literal> utility uses the <literal>LST_SESSION</literal> environmental variable to identify the session locally on the self-test console node. This should be a numeric value that uniquely identifies all session processes on the node. It is convenient to set this to the process ID of the shell both for interactive use and in shell scripts. Almost all <literal>lst</literal> commands require <literal>LST_SESSION</literal> to be set.</para>
+ <para>Example:</para>
+ <screen>export LST_SESSION=$$</screen>
+ <para><emphasis role="bold">
+ <literal>new_session [--timeout SECONDS] [--force] NAME</literal>
+ </emphasis></para>
+ <para>Creates a new session.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para><literal>--timeout<emphasis><seconds></emphasis></literal></para>
+ </entry>
+ <entry>
+ <para>Console timeout value of the session. The session ends automatically if it remains idle (i.e., no commands are issued) for this period.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><literal>--force</literal></para>
+ </entry>
+ <entry>
+ <para>Ends conflicting sessions. This determines who 'wins' when one session conflicts with another. For example, if there is already an active session on this node, then the attempt to create a new session fails unless the -force flag is specified. If the -force flag is specified, then the active session is ended. Similarly, if a session attempts to add a node that is already 'owned' by another session, the -force flag allows this session to 'steal' the node.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis>
+ <literal><name></literal>
+ </emphasis></para>
+ </entry>
+ <entry>
+ <para>A human-readable string to print when listing sessions or reporting session conflicts.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst new_session --force read_write</screen>
+ <para><literal>
+ <emphasis role="bold">end_session</emphasis>
+ </literal></para>
+ <para>Stops all operations and tests in the current session and clears the session's status.</para>
+ <screen>$ lst end_session</screen>
+ <para><literal>
+ <emphasis role="bold">show_session</emphasis>
+ </literal></para>
+ <para>Shows the session information. This command prints information about the current session. It does not require LST_SESSION to be defined in the process environment.</para>
+ <screen>$ lst show_session</screen>
+ </section>
+ <section remap="h3">
+ <title>23.3.2 Group Commands</title>
+ <para>This section describes <literal>lst</literal> group commands.</para>
+ <para><literal>
+ <emphasis role="bold">add_group</emphasis>
+ <emphasis> <name> <NIDS> [<NIDs>...]</emphasis>
+ </literal></para>
+ <para>Creates the group and adds a list of test nodes to the group.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>
+ <emphasis><name></emphasis>
+ </literal></para>
+ </entry>
+ <entry>
+ <para>Name of the group.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>
+ <emphasis><NIDs></emphasis>
+ </literal></para>
+ </entry>
+ <entry>
+ <para>A string that may be expanded to include one or more LNET NIDs.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst add_group servers 192.168.10.[35,40-45]@tcp$ lst add_group clients 192.168.1.[10-100]@tcp 192.168.[2,4].\[10-20]@tcp</screen>
+ <para><literal>
+ <emphasis role="bold">update_group</emphasis>
+ <emphasis><name></emphasis>
+ <emphasis role="bold">[--refresh] [--clean</emphasis>
+ <emphasis><status></emphasis>
+ <emphasis role="bold">] [--remove</emphasis>
+ <emphasis><NIDs></emphasis>
+ <emphasis role="bold">]</emphasis>
+ </literal></para>
+ <para>Updates the state of nodes in a group or adjusts a group's membership. This command is useful if some nodes have crashed and should be excluded from the group.</para>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="33*"/>
+ <colspec colname="c2" colwidth="33*"/>
+ <colspec colname="c3" colwidth="33*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <literal>
+ <para>--refresh</para>
+ </literal>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para> Refreshes the state of all inactive nodes in the group.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--clean<emphasis><status></emphasis></literal></para>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para> Removes nodes with a specified status from the group. Status may be:</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> </para>
+ </entry>
+ <entry>
+ <para> active</para>
+ </entry>
+ <entry>
+ <para> The node is in the current session.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> </para>
+ </entry>
+ <entry>
+ <para> busy</para>
+ </entry>
+ <entry>
+ <para> The node is now owned by another session.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> </para>
+ </entry>
+ <entry>
+ <para> down</para>
+ </entry>
+ <entry>
+ <para> The node has been marked down.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> </para>
+ </entry>
+ <entry>
+ <para> unknown</para>
+ </entry>
+ <entry>
+ <para> The node'™s status has yet to be determined.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> </para>
+ </entry>
+ <entry>
+ <para> invalid</para>
+ </entry>
+ <entry>
+ <para> Any state but active.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--remove<emphasis><NIDs></emphasis></literal></para>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para> Removes specified nodes from the group.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst update_group clients --refresh
+$ lst update_group clients --clean busy
+$ lst update_group clients --clean invalid // \
+ invalid == busy || down || unknown
+$ lst update_group clients --remove \192.168.1.[10-20]@tcp</screen>
+ <para><literal>
+ <emphasis role="bold">list_group [</emphasis>
+ <emphasis><name></emphasis>
+ <emphasis role="bold">] [--active] [--busy] [--down] [--unknown] [--all]</emphasis>
+ </literal></para>
+ <para>Prints information about a group or lists all groups in the current session if no group is specified.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para><literal>
+ <emphasis><name></emphasis>
+ </literal></para>
+ </entry>
+ <entry>
+ <para> The name of the group.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--active</para>
+ </literal>
+ </entry>
+ <entry>
+ <para> Lists the active nodes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--busy</para>
+ </literal>
+ </entry>
+ <entry>
+ <para> Lists the busy nodes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--down</para>
+ </literal>
+ </entry>
+ <entry>
+ <para> Lists the down nodes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--unknown</para>
+ </literal>
+ </entry>
+ <entry>
+ <para> Lists unknown nodes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--all</para>
+ </literal>
+ </entry>
+ <entry>
+ <para> Lists all nodes.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para>Example:</para>
+ <screen>$ lst list_group
+1) clients
+2) servers
+Total 2 groups
+$ lst list_group clients
+ACTIVE BUSY DOWN UNKNOWN TOTAL
+3 1 2 0 6
+$ lst list_group clients --all
+192.168.1.10@tcp Active
+192.168.1.11@tcp Active
+192.168.1.12@tcp Busy
+192.168.1.13@tcp Active
+192.168.1.14@tcp DOWN
+192.168.1.15@tcp DOWN
+Total 6 nodes
+$ lst list_group clients --busy
+192.168.1.12@tcp Busy
+Total 1 node</screen>
+ <para><literal>
+ <emphasis role="bold">del_group</emphasis>
+ <emphasis> <name></emphasis>
+ </literal></para>
+ <para>Removes a group from the session. If the group is referred to by any test, then the operation fails. If nodes in the group are referred to only by this group, then they are kicked out from the current session; otherwise, they are still in the current session.</para>
+ <screen>$ lst del_group clients</screen>
+ <para><literal><emphasis role="bold">lstclient --sesid</emphasis><emphasis> <NID> </emphasis><emphasis role="bold">--group</emphasis><emphasis> <name></emphasis> [--server_mode]</literal></para>
+ <para>Use <literal>lstclient</literal> to run the userland self-test client. The <literal>lstclient</literal> command should be executed after creating a session on the console. There are only two mandatory options for <literal>lstclient</literal>:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <literal>
+ <para> --sesid<emphasis><NID></emphasis></para>
+ </literal>
+ </entry>
+ <entry>
+ <para> The first console's NID.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para> --group<emphasis><name></emphasis></para>
+ </literal>
+ </entry>
+ <entry>
+ <para> The test group to join.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para> --server_mode</para>
+ </literal>
+ </entry>
+ <entry>
+ <para> When included, forces LNET to behave as a server, such as starting an acceptor if the underlying NID needs it or using privileged ports. Only root is allowed to use the <literal>--server_mode</literal> option.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>Console $ lst new_session testsession
Client1 $ lstclient --sesid 192.168.1.52@tcp --group clients</screen>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>Client1 $ lstclient --sesid 192.168.1.52@tcp |--group clients --server_mode
-</screen>
- </section>
- <section remap="h3">
- <title>23.3.3 <anchor xml:id="dbdoclet.50438223_36860" xreflabel=""/>Batch and Test Commands</title>
- <para>This section describes lst batch and test commands.</para>
- <para><emphasis role="bold">add_batch NAME</emphasis></para>
- <para>A default batch test set named batch is created when the session is started. You can specify a batch name by using add_batch:</para>
- <screen>$ lst add_batch bulkperf
-</screen>
- <para>Creates a batch test called bulkperf.</para>
- <para>add_test --batch<emphasis><batchname></emphasis> [--loop <emphasis><#>]</emphasis><emphasis role="bold">[--concurrency</emphasis><emphasis><#></emphasis><emphasis role="bold">] [--distribute</emphasis><emphasis><#:#></emphasis><emphasis role="bold">]</emphasis></para>
- <para>--from<emphasis><group></emphasis> --to <emphasis><group></emphasis> {brw|ping} <test options></para>
- <para>Adds a test to a batch. The parameters are described below.</para>
- <informaltable frame="all">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> --batch<emphasis><batchname></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> Names a group of tests for later execution.</para></entry>
- </row>
- <row>
- <entry><para> --loop<emphasis><#></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> Number of times to run the test.</para></entry>
- </row>
- <row>
- <entry><para> --concurrency<emphasis><#></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> The number of requests that are active at one time.</para></entry>
- </row>
- <row>
- <entry><para> --distribute<emphasis><#:#></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> Determines the ratio of client nodes to server nodes for the specified test. This allows you to specify a wide range of topologies, including one-to-one and all-to-all. Distribution divides the source group into subsets, which are paired with equivalent subsets from the target group so only nodes in matching subsets communicate.</para></entry>
- </row>
- <row>
- <entry><para> --from<emphasis><group></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> The source group (test client).</para></entry>
- </row>
- <row>
- <entry><para> --to<emphasis><group></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> The target group (test server).</para></entry>
- </row>
- <row>
- <entry><para> ping</para></entry>
- <entry nameend="c3" namest="c2"><para> Sends a small request message, resulting in a small reply message. For more details, see <link xl:href="LNETSelfTest.html#50438223_42848">Defining and Running the Tests</link></para></entry>
- </row>
- <row>
- <entry><para> brw</para></entry>
- <entry nameend="c3" namest="c2"><para> Sends a small request message followed by a bulk data transfer, resulting in a small reply message. <link xl:href="LNETSelfTest.html#50438223_42848">Defining and Running the Tests</link>. Options are:</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> read | write</para></entry>
- <entry><para> Read or write. The default is read.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> size=<#>| <#>K | <#>M</para></entry>
- <entry><para> I/O size in bytes, KB or MB (i.e., size=1024, size=4K, size=1M). The default is 4K bytes.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> check=full|simple</para></entry>
- <entry><para> A data validation check (checksum of data). The default is that no check is done.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Examples showing use of the distribute parameter:</emphasis></para>
- <screen>
-Clients: (C1, C2, C3, C4, C5, C6)
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>Client1 $ lstclient --sesid 192.168.1.52@tcp |--group clients --server_mode</screen>
+ </section>
+ <section xml:id="dbdoclet.50438223_36860">
+ <title>23.3.3 Batch and Test Commands</title>
+ <para>This section describes <literal>lst</literal> batch and test commands.</para>
+ <para><literal>
+ <emphasis role="bold">add_batch NAME</emphasis>
+ </literal></para>
+ <para>A default batch test set named batch is created when the session is started. You can specify a batch name by using add_batch:</para>
+ <screen>$ lst add_batch bulkperf</screen>
+ <para>Creates a batch test called <literal>bulkperf</literal>.</para>
+ <para><literal>
+ <emphasis role="bold">add_test --batch <emphasis><<emphasis role="italic">batchname</emphasis>></emphasis> [--loop<#>] <emphasis role="bold">[--concurrency</emphasis><emphasis><#></emphasis><emphasis role="bold">] [--distribute</emphasis><emphasis><#:#></emphasis><emphasis role="bold">]</emphasis><para>--from<emphasis> <<emphasis role="italic">group</emphasis>></emphasis> --to <emphasis><<emphasis role="italic">group</emphasis>></emphasis> {brw|ping} <test options></para></emphasis>
+ </literal></para>
+ <para>Adds a test to a batch. The parameters are described below.</para>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="33*"/>
+ <colspec colname="c2" colwidth="33*"/>
+ <colspec colname="c3" colwidth="33*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>--batch<emphasis><batchname></emphasis></literal></para>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para>Names a group of tests for later execution.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para> --loop<emphasis><#></emphasis></para>
+ </literal>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para>Number of times to run the test.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para> --concurrency<emphasis><#></emphasis></para>
+ </literal>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para>The number of requests that are active at one time.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para> --distribute<emphasis><#:#></emphasis></para>
+ </literal>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para>Determines the ratio of client nodes to server nodes for the specified test. This allows you to specify a wide range of topologies, including one-to-one and all-to-all. Distribution divides the source group into subsets, which are paired with equivalent subsets from the target group so only nodes in matching subsets communicate.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--from<emphasis><group></emphasis></para>
+ </literal>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para>The source group (test client).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--to<emphasis><group></emphasis></para>
+ </literal>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para>The target group (test server).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><literal>ping</literal></para>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para>Sends a small request message, resulting in a small reply message. For more details, see <xref linkend="dbdoclet.50438223_42848">Defining and Running the Tests</xref></para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><literal>brw</literal></para>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para>Sends a small request message followed by a bulk data transfer, resulting in a small reply message. <xref linkend="dbdoclet.50438223_42848">Defining and Running the Tests</xref>. Options are:</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> </para>
+ </entry>
+ <entry>
+ <literal>
+ <para>read | write</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Read or write. The default is read.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> </para>
+ </entry>
+ <entry>
+ <literal>
+ <para>size=<#>| <#>K | <#>M</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>I/O size in bytes, KB or MB (i.e., <literal>size=1024</literal>, <literal>size=4</literal>K, <literal>size=1M</literal>). The default is 4K bytes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> </para>
+ </entry>
+ <entry>
+ <literal>
+ <para>check=full|simple</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>A data validation check (checksum of data). The default is that no check is done.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Examples showing use of the distribute parameter:</emphasis></para>
+ <screen>Clients: (C1, C2, C3, C4, C5, C6)
Server: (S1, S2, S3)
--distribute 1:1 (C1->S1), (C2->S2), (C3->S3), (C4->S1), (C5->S2),
\(C6->S3) /* -> means test conversation */ --distribute 2:1 (C1,C2->S1), (C3,C4->S2), (C5,C6->S3)
--distribute 4:1 (C1,C2,C3,C4->S1), (C5,C6->S2), (NULL->S3)
--distribute 4:2 (C1,C2,C3,C4->S1,S2), (C5, C6->S3, S1)
--distribute 6:3 (C1,C2,C3,C4,C5,C6->S1,S2,S3)</screen>
- <para>The setting --distribute 1:1 is the default setting where each source node communicates with one target node.</para>
- <para>When the setting --distribute 1:<emphasis><n></emphasis> (where <emphasis><n></emphasis> is the size of the target group) is used, each source node communicates with every node in the target group.</para>
- <para>Note that if there are more source nodes than target nodes, some source nodes may share the same target nodes. Also, if there are more target nodes than source nodes, some higher-ranked target nodes will be idle.</para>
- <para><emphasis role="bold">Example showing a brw test:</emphasis></para>
- <screen>
-$ lst add_group clients 192.168.1.[10-17]@tcp
+ <para>The setting <literal>--distribute 1:1</literal> is the default setting where each source node communicates with one target node.</para>
+ <para>When the setting <literal>--distribute 1:<emphasis><n></emphasis></literal> (where <emphasis>
+ <literal><n></literal>
+ </emphasis> is the size of the target group) is used, each source node communicates with every node in the target group.</para>
+ <para>Note that if there are more source nodes than target nodes, some source nodes may share the same target nodes. Also, if there are more target nodes than source nodes, some higher-ranked target nodes will be idle.</para>
+ <para><emphasis role="bold">Example showing a <literal>brw</literal> test:</emphasis></para>
+ <screen>$ lst add_group clients 192.168.1.[10-17]@tcp
$ lst add_group servers 192.168.10.[100-103]@tcp
$ lst add_batch bulkperf
$ lst add_test --batch bulkperf --loop 100 --concurrency 4 \
---distribute 4:2 --from clients brw WRITE size=16K</screen>
- <para>In the example above, a batch test called bulkperf that will do a 16 kbyte bulk write request. In this test, two groups of four clients (sources) write to each of four servers (targets) as shown below:</para>
- <itemizedlist><listitem>
- <para> 192.168.1.[10-13] will write to 192.168.10.[100,101]</para>
- </listitem>
-
-<listitem>
- <para> 192.168.1.[14-17] will write to 192.168.10.[102,103]</para>
- </listitem>
-
-</itemizedlist>
- <para> </para>
- <para><emphasis role="bold">list_batch [</emphasis><emphasis><name></emphasis><emphasis role="bold">] [--test</emphasis><emphasis><index></emphasis><emphasis role="bold">] [--active] [--invalid]</emphasis> [--server | client<emphasis role="bold">]</emphasis></para>
- <para>Lists batches in the current session or lists client and server nodes in a batch or a test.</para>
- <informaltable frame="all">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> --test<emphasis><index></emphasis></para></entry>
- <entry nameend="c3" namest="c2"><para> Lists tests in a batch. If no option is used, all tests in the batch are listed. IIf one of these options are used, only specified tests in the batch are listed:</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> active</para></entry>
- <entry><para> Lists only active batch tests.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> invalid</para></entry>
- <entry><para> Lists only invalid batch tests.</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> server | client</para></entry>
- <entry><para> Lists client and server nodes in a batch test.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>$ lst list_batchbulkperf$ lst list_batch bulkperfBatch: bulkperf Tests: 1 State: IdleACTIVE BUSY DOWN UNKNOWN TOTALclient 8 0 0 0 8server 4 0 0 0 4Test 1(brw) (loop: 100, concurrency: 4)ACTIVE BUSY DOWN UNKNOWN TOTALclient 8 0 0 0 8server 4 0 0 0 4$ lst list_batch bulkperf --server --active192.168.10.100@tcp Active192.168.10.101@tcp Active192.168.10.102@tcp Active192.168.10.103@tcp Active</screen>
- <para> </para>
- <para><emphasis role="bold">run</emphasis><emphasis><name></emphasis></para>
- <para>Runs the batch.</para>
- <screen>$ lst run bulkperf
-</screen>
- <para><emphasis role="bold">stop</emphasis><emphasis><name></emphasis></para>
- <para>Stops the batch.</para>
- <screen>$ lst stop bulkperf
-</screen>
- <para><emphasis role="bold">query</emphasis><emphasis><name></emphasis><emphasis role="bold">[--test</emphasis><emphasis><index></emphasis><emphasis role="bold">] [--timeout</emphasis><emphasis><seconds></emphasis><emphasis role="bold">[--loop</emphasis><emphasis><#></emphasis><emphasis role="bold">] [--delay</emphasis><emphasis><seconds></emphasis><emphasis role="bold">] [--all]</emphasis></para>
- <para>Queries the batch status.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> --test<emphasis><index></emphasis></para></entry>
- <entry><para> Only queries the specified test. The test index starts from 1.</para></entry>
- </row>
- <row>
- <entry><para> --timeout<emphasis><seconds></emphasis></para></entry>
- <entry><para> The timeout value to wait for RPC. The default is 5 seconds.</para></entry>
- </row>
- <row>
- <entry><para> --loop<emphasis><#></emphasis></para></entry>
- <entry><para> The loop count of the query.</para></entry>
- </row>
- <row>
- <entry><para> --delay<emphasis><seconds></emphasis></para></entry>
- <entry><para> The interval of each query. The default is 5 seconds.</para></entry>
- </row>
- <row>
- <entry><para> --all</para></entry>
- <entry><para> The list status of all nodes in a batch or a test.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>$ lst run bulkperf$ lst query bulkperf --loop 5 --delay 3Batch is runningBatch is runningBatch is runningBatch is runningBatch is running$ lst query bulkperf --all192.168.1.10@tcp Running192.168.1.11@tcp Running192.168.1.12@tcp Running192.168.1.13@tcp Running192.168.1.14@tcp Running192.168.1.15@tcp Running192.168.1.16@tcp Running192.168.1.17@tcp Running$ lst stop bulkperf$ lst query bulkperfBatch is idle</screen>
- </section>
- <section remap="h3">
- <title>23.3.4 Other Commands</title>
- <para>This section describes other lst commands.</para>
- <para><emphasis role="bold">ping [-session] [--group</emphasis><emphasis><name></emphasis><emphasis role="bold">] [--nodes</emphasis><emphasis><NIDs></emphasis><emphasis role="bold">] [--batch</emphasis><emphasis><name></emphasis><emphasis role="bold">] [--server] [--timeout</emphasis><emphasis><seconds></emphasis><emphasis role="bold">]</emphasis></para>
- <para>Sends a 'hello' query to the nodes.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> --session</para></entry>
- <entry><para> Pings all nodes in the current session.</para></entry>
- </row>
- <row>
- <entry><para> --group<emphasis><name></emphasis></para></entry>
- <entry><para> Pings all nodes in a specified group.</para></entry>
- </row>
- <row>
- <entry><para> --nodes<emphasis><NIDs></emphasis></para></entry>
- <entry><para> Pings all specified nodes.</para></entry>
- </row>
- <row>
- <entry><para> --batch<emphasis><name></emphasis></para></entry>
- <entry><para> Pings all client nodes in a batch.</para></entry>
- </row>
- <row>
- <entry><para> --server</para></entry>
- <entry><para> Sends RPC to all server nodes instead of client nodes. This option is only used with --batch<emphasis><name></emphasis>.</para></entry>
- </row>
- <row>
- <entry><para> --timeout<emphasis><seconds></emphasis></para></entry>
- <entry><para> The RPC timeout value.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>$ lst ping 192.168.10.[15-20]@tcp192.168.1.15@tcp Active [session: liang id: 192.168.1.3@tcp]192.168.1.16@tcp Active [session: liang id: 192.168.1.3@tcp]192.168.1.17@tcp Active [session: liang id: 192.168.1.3@tcp]192.168.1.18@tcp Busy [session: Isaac id: 192.168.10.10@tcp]192.168.1.19@tcp Down [session: <NULL> id: LNET_NID_ANY]192.168.1.20@tcp Down [session: <NULL> id: LNET_NID_ANY]</screen>
- <para> </para>
- <para><emphasis role="bold">stat [--bw] [--rate] [--read] [--write] [--max] [--min] [--avg] " " [--timeout</emphasis><emphasis><seconds></emphasis><emphasis role="bold">] [--delay</emphasis><emphasis><seconds></emphasis><emphasis role="bold">]</emphasis><emphasis><group></emphasis><emphasis role="bold">|<</emphasis><emphasis>NIDs></emphasis><emphasis role="bold">[</emphasis><emphasis><group></emphasis><emphasis role="bold">|</emphasis><emphasis><NIDs></emphasis><emphasis role="bold">]</emphasis></para>
- <para>The collection performance and RPC statistics of one or more nodes.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> --bw</para></entry>
- <entry><para> Displays the bandwidth of the specified group/nodes.</para></entry>
- </row>
- <row>
- <entry><para> --rate</para></entry>
- <entry><para> Displays the rate of RPCs of the specified group/nodes.</para></entry>
- </row>
- <row>
- <entry><para> --read</para></entry>
- <entry><para> Displays the read statistics of the specified group/nodes.</para></entry>
- </row>
- <row>
- <entry><para> --write</para></entry>
- <entry><para> Displays the write statistics of the specified group/nodes.</para></entry>
- </row>
- <row>
- <entry><para> --max</para></entry>
- <entry><para> Displays the maximum value of the statistics.</para></entry>
- </row>
- <row>
- <entry><para> --min</para></entry>
- <entry><para> Displays the minimum value of the statistics.</para></entry>
- </row>
- <row>
- <entry><para> --avg</para></entry>
- <entry><para> Displays the average of the statistics.</para></entry>
- </row>
- <row>
- <entry><para> --timeout<emphasis><seconds></emphasis></para></entry>
- <entry><para> The timeout of the statistics RPC. The default is 5 seconds.</para></entry>
- </row>
- <row>
- <entry><para> --delay<emphasis><seconds></emphasis></para></entry>
- <entry><para> The interval of the statistics (in seconds).</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>$ lst run bulkperf$ lst stat clients[LNet Rates of clients][W] Avg: 1108 RPC/s Min: 1060 RPC/s Max: 1155 RPC/s[R] Avg: 2215 RPC/s Min: 2121 RPC/s Max: 2310 RPC/s[LNet Bandwidth of clients][W] Avg: 16.60 MB/s Min: 16.10 MB/s Max: 17.1 MB/s[R] Avg: 40.49 MB/s Min: 40.30 MB/s Max: 40.68 MB/s</screen>
- <para>Specifying a group name (<emphasis><group></emphasis>) causes statistics to be gathered for all nodes in a test group. For example:</para>
- <screen>$ lst stat servers
-</screen>
- <para>where servers is the name of a test group created by lst add_group</para>
- <para>Specifying a NID range (<emphasis><NIDs></emphasis>) causes statistics to be gathered for selected nodes. For example:</para>
- <screen>$ lst stat 192.168.0.[1-100/2]@tcp
-Only LNET performance statistics are available. By default, all statistics \
-information is displayed. Users can specify additional information with the\
-se options.
-
+ --distribute 4:2 --from clients brw WRITE size=16K</screen>
+ <para>In the example above, a batch test called bulkperf that will do a 16 kbyte bulk write request. In this test, two groups of four clients (sources) write to each of four servers (targets) as shown below:</para>
+ <itemizedlist>
+ <listitem>
+ <para> <literal>192.168.1.[10-13]</literal> will write to <literal>192.168.10.[100,101]</literal></para>
+ </listitem>
+ <listitem>
+ <para> <literal>192.168.1.[14-17]</literal> will write to <literal>192.168.10.[102,103]</literal></para>
+ </listitem>
+ </itemizedlist>
+ <para><emphasis role="bold">
+ <literal><emphasis role="bold">list_batch [</emphasis><emphasis><emphasis role="italic"><name</emphasis>></emphasis><emphasis role="bold">] [--test</emphasis><emphasis> <<emphasis role="italic">index</emphasis>></emphasis><emphasis role="bold">] [--active] [--invalid]</emphasis> [--server | client<emphasis role="bold">]</emphasis></literal>
+ </emphasis></para>
+ <para>Lists batches in the current session or lists client and server nodes in a batch or a test.</para>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="33*"/>
+ <colspec colname="c2" colwidth="33*"/>
+ <colspec colname="c3" colwidth="33*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <literal>
+ <para>--test<emphasis><index></emphasis></para>
+ </literal>
+ </entry>
+ <entry nameend="c3" namest="c2">
+ <para>Lists tests in a batch. If no option is used, all tests in the batch are listed. IIf one of these options are used, only specified tests in the batch are listed:</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> </para>
+ </entry>
+ <entry>
+ <para><literal>active</literal></para>
+ </entry>
+ <entry>
+ <para>Lists only active batch tests.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> </para>
+ </entry>
+ <entry>
+ <para><literal>invalid</literal></para>
+ </entry>
+ <entry>
+ <para>Lists only invalid batch tests.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> </para>
+ </entry>
+ <entry>
+ <literal>
+ <para>server | client</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Lists client and server nodes in a batch test.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst list_batchbulkperf
+$ lst list_batch bulkperf
+Batch: bulkperf Tests: 1 State: Idle
+ACTIVE BUSY DOWN UNKNOWN TOTAL
+client 8 0 0 0 8
+server 4 0 0 0 4
+Test 1(brw) (loop: 100, concurrency: 4)
+ACTIVE BUSY DOWN UNKNOWN TOTAL
+client 8 0 0 0 8
+server 4 0 0 0 4
+$ lst list_batch bulkperf --server --active
+192.168.10.100@tcp Active
+192.168.10.101@tcp Active
+192.168.10.102@tcp Active
+192.168.10.103@tcp Active</screen>
+ <para><literal>
+ <emphasis role="bold">run</emphasis>
+ <emphasis role="bold">
+ <emphasis> <<emphasis role="italic">name</emphasis>></emphasis>
+ </emphasis>
+ </literal></para>
+ <para>Runs the batch.</para>
+ <screen>$ lst run bulkperf</screen>
+ <para><literal>
+ <emphasis role="bold">
+ <emphasis role="bold">stop </emphasis>
+ <emphasis><<emphasis role="italic">name</emphasis>></emphasis>
+ </emphasis>
+ </literal></para>
+ <para>Stops the batch.</para>
+ <screen>$ lst stop bulkperf</screen>
+ <para><emphasis role="bold">
+ <literal>
+ <emphasis role="bold">query </emphasis>
+ <emphasis><<emphasis role="italic">name</emphasis>> </emphasis>
+ <emphasis role="bold">[--test</emphasis>
+ <emphasis> <<emphasis role="italic">index</emphasis>></emphasis>
+ <emphasis role="bold">] [--timeout</emphasis>
+ <emphasis> <<emphasis role="italic">seconds</emphasis>></emphasis>
+ <emphasis role="bold">] [--loop</emphasis>
+ <emphasis><#></emphasis>
+ <emphasis role="bold">] [--delay</emphasis>
+ <emphasis><<emphasis role="italic">seconds</emphasis>></emphasis>
+ <emphasis role="bold">] [--all]</emphasis>
+ </literal>
+ </emphasis></para>
+ <para>Queries the batch status.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <literal>
+ <para>--test<emphasis><index></emphasis></para>
+ </literal>
+ </entry>
+ <entry>
+ <para> Only queries the specified test. The test index starts from 1.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--timeout<emphasis><seconds></emphasis></para>
+ </literal>
+ </entry>
+ <entry>
+ <para> The timeout value to wait for RPC. The default is 5 seconds.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--loop<emphasis><#></emphasis></para>
+ </literal>
+ </entry>
+ <entry>
+ <para> The loop count of the query.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--delay<emphasis><seconds></emphasis></para>
+ </literal>
+ </entry>
+ <entry>
+ <para> The interval of each query. The default is 5 seconds.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--all</para>
+ </literal>
+ </entry>
+ <entry>
+ <para> The list status of all nodes in a batch or a test.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst run bulkperf
+$ lst query bulkperf --loop 5 --delay 3
+Batch is running
+Batch is running
+Batch is running
+Batch is running
+Batch is running
+$ lst query bulkperf --all
+192.168.1.10@tcp Running
+192.168.1.11@tcp Running
+192.168.1.12@tcp Running
+192.168.1.13@tcp Running
+192.168.1.14@tcp Running
+192.168.1.15@tcp Running
+192.168.1.16@tcp Running
+192.168.1.17@tcp Running
+$ lst stop bulkperf
+$ lst query bulkperf
+Batch is idle</screen>
+ </section>
+ <section remap="h3">
+ <title>23.3.4 Other Commands</title>
+ <para>This section describes other <literal>lst</literal> commands.</para>
+ <para><literal>
+ <emphasis role="bold">
+ <emphasis role="bold">ping [-session] [--group</emphasis>
+ <emphasis><<emphasis role="italic">name</emphasis>></emphasis>
+ <emphasis role="bold">] [--nodes</emphasis>
+ <emphasis> <NIDs></emphasis>
+ <emphasis role="bold">] [--batch</emphasis>
+ <emphasis> <<emphasis role="italic">name</emphasis>></emphasis>
+ <emphasis role="bold">] [--server] [--timeout</emphasis>
+ <emphasis> <<emphasis role="italic">seconds</emphasis>></emphasis>
+ <emphasis role="bold">]</emphasis>
+ </emphasis>
+ </literal></para>
+ <para>Sends a 'hello' query to the nodes.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <literal>
+ <para>--session</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Pings all nodes in the current session.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--group<emphasis><name></emphasis></para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Pings all nodes in a specified group.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--nodes<emphasis><NIDs></emphasis></para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Pings all specified nodes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--batch<emphasis><name></emphasis></para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Pings all client nodes in a batch.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--server</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Sends RPC to all server nodes instead of client nodes. This option is only used with <literal>--batch<emphasis><name></emphasis></literal>.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--timeout<emphasis><seconds></emphasis></para>
+ </literal>
+ </entry>
+ <entry>
+ <para>The RPC timeout value.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst ping 192.168.10.[15-20]@tcp
+192.168.1.15@tcp Active [session: liang id: 192.168.1.3@tcp]
+192.168.1.16@tcp Active [session: liang id: 192.168.1.3@tcp]
+192.168.1.17@tcp Active [session: liang id: 192.168.1.3@tcp]
+192.168.1.18@tcp Busy [session: Isaac id: 192.168.10.10@tcp]
+192.168.1.19@tcp Down [session: <NULL> id: LNET_NID_ANY]
+192.168.1.20@tcp Down [session: <NULL> id: LNET_NID_ANY]</screen>
+ <para><literal>
+ <emphasis role="bold">
+ <emphasis role="bold">stat [--bw] [--rate] [--read] [--write] [--max] [--min] [--avg] " " [--timeout</emphasis>
+ <emphasis> <<emphasis role="italic">seconds</emphasis>></emphasis>
+ <emphasis role="bold">] [--delay </emphasis>
+ <emphasis><<emphasis role="italic">seconds</emphasis>></emphasis>
+ <emphasis role="bold">] </emphasis>
+ <emphasis><<emphasis role="italic">group</emphasis>></emphasis>
+ <emphasis role="bold">|<</emphasis>
+ <emphasis><emphasis role="italic">NIDs</emphasis>> </emphasis>
+ <emphasis role="bold">[</emphasis>
+ <emphasis><<emphasis role="italic">group</emphasis>></emphasis>
+ <emphasis role="bold">|</emphasis>
+ <emphasis><<emphasis role="italic">NIDs</emphasis>></emphasis>
+ <emphasis role="bold">]</emphasis>
+ </emphasis>
+ </literal></para>
+ <para>The collection performance and RPC statistics of one or more nodes.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <literal>
+ <para>--bw</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Displays the bandwidth of the specified group/nodes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--rate</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Displays the rate of RPCs of the specified group/nodes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--read</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Displays the read statistics of the specified group/nodes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--write</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Displays the write statistics of the specified group/nodes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--max</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Displays the maximum value of the statistics.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--min</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Displays the minimum value of the statistics.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--avg</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Displays the average of the statistics.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--timeout<emphasis><seconds></emphasis></para>
+ </literal>
+ </entry>
+ <entry>
+ <para>The timeout of the statistics RPC. The default is 5 seconds.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal>
+ <para>--delay<emphasis><seconds></emphasis></para>
+ </literal>
+ </entry>
+ <entry>
+ <para>The interval of the statistics (in seconds).</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst run bulkperf
+$ lst stat clients
+[LNet Rates of clients]
+[W] Avg: 1108 RPC/s Min: 1060 RPC/s Max: 1155 RPC/s
+[R] Avg: 2215 RPC/s Min: 2121 RPC/s Max: 2310 RPC/s
+[LNet Bandwidth of clients]
+[W] Avg: 16.60 MB/s Min: 16.10 MB/s Max: 17.1 MB/s
+[R] Avg: 40.49 MB/s Min: 40.30 MB/s Max: 40.68 MB/s</screen>
+ <para>Specifying a group name (<emphasis>
+ <literal><group></literal>
+ </emphasis>) causes statistics to be gathered for all nodes in a test group. For example:</para>
+ <screen>$ lst stat servers
</screen>
- <para><emphasis role="bold">show_error [--session] [<group>|<NIDs>]...</emphasis></para>
- <para>Lists the number of failed RPCs on test nodes.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> --session</para></entry>
- <entry><para> Lists errors in the current test session. With this option, historical RPC errors are not listed.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>$ lst show_error clientsclients12345-192.168.1.15@tcp: [Session: 1 brw errors, 0 ping errors] \[RPC: 20 errors, 0 dropped,12345-192.168.1.16@tcp: [Session: 0 brw errors, 0 ping errors] \[RPC: 1 errors, 0 dropped, Total 2 error nodes in clients$ lst show_error --session clientsclients12345-192.168.1.15@tcp: [Session: 1 brw errors, 0 ping errors]Total 1 error nodes in clients</screen>
- </section>
+ <para>where servers is the name of a test group created by <literal>lst add_group</literal></para>
+ <para>Specifying a <literal>NID</literal> range (<emphasis><<literal>NIDs</literal>></emphasis>) causes statistics to be gathered for selected nodes. For example:</para>
+ <screen>$ lst stat 192.168.0.[1-100/2]@tcp</screen>
+ <para>Only LNET performance statistics are available. By default, all statistics
+information is displayed. Users can specify additional information with these options.</para>
+ <para><literal>
+ <emphasis role="bold">show_error [--session] [<group>|<NIDs>]...</emphasis>
+ </literal></para>
+ <para>Lists the number of failed RPCs on test nodes.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <literal>
+ <para>--session</para>
+ </literal>
+ </entry>
+ <entry>
+ <para>Lists errors in the current test session. With this option, historical RPC errors are not listed.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>$ lst show_error client
+sclients
+12345-192.168.1.15@tcp: [Session: 1 brw errors, 0 ping errors] \
+ [RPC: 20 errors, 0 dropped,
+12345-192.168.1.16@tcp: [Session: 0 brw errors, 0 ping errors] \
+ [RPC: 1 errors, 0 dropped, Total 2 error nodes in clients
+$ lst show_error --session clients
+clients
+12345-192.168.1.15@tcp: [Session: 1 brw errors, 0 ping errors]
+Total 1 error nodes in clients</screen>
</section>
+ </section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='lustremaintenance'>
- <info>
- <title xml:id='lustremaintenance.title'>Lustre Maintenance</title>
- </info>
- <para>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre maintenance tasks:</para>
-
- <itemizedlist><listitem>
- <para><xref linkend="dbdoclet.50438199_85142"/>Working with Inactive OSTs</para>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. -->
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremaintenance">
+ <info>
+ <title xml:id="lustremaintenance.title">Lustre Maintenance</title>
+ </info>
+ <para>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre maintenance tasks:</para>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438199_42877"/>Working with Inactive OSTs</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438199_15240"/>Finding Nodes in the Lustre File System</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438199_26070"/>Mounting a Server Without Lustre Service</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438199_54623"/>Regenerating Lustre Configuration Logs</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438199_31353"/>Changing a Server NID</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438199_22527"/>Adding a New OST to a Lustre File System</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438199_14978"/>Removing and Restoring OSTs</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438199_77819"/>Aborting Recovery</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438199_12607"/>Determining Which Machine is Serving an OST</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438199_62333"/>Changing the Address of a Failover Node</para>
+ </listitem>
+ <listitem>
+ <para> </para>
+ </listitem>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438199_42877">
+ <title>14.1 Working with Inactive OSTs</title>
+ <para>To mount a client or an MDT with one or more inactive OSTs, run commands similar to this:</para>
+ <screen>client> mount -o exclude=testfs-OST0000 -t lustre uml1:/testfs\ /mnt/testfs
+ client> cat /proc/fs/lustre/lov/testfs-clilov-*/target_obd</screen>
+ <para>To activate an inactive OST on a live client or MDT, use the <literal>lctl activate</literal> command on the OSC device. For example:</para>
+ <screen>lctl --device 7 activate
+ </screen>
+ <note>
+ <para>
+ A colon-separated list can also be specified. For example, <literal>exclude=testfs-OST0000:testfs-OST0001</literal>.</para>
+ </note>
+ <section xml:id="dbdoclet.50438199_15240">
+ <title>14.2 Finding Nodes in the Lustre File System</title>
+ <para>There may be situations in which you need to find all nodes in your Lustre file system or get the names of all OSTs.</para>
+ <para>To get a list of all Lustre nodes, run this command on the MGS:</para>
+ <screen># cat /proc/fs/lustre/mgs/MGS/live/*</screen>
+ <note>
+ <para>
+ This command must be run on the MGS.
+ </para>
+ </note>
+ <para>In this example, file system lustre has three nodes, <literal>lustre-MDT0000</literal>, <literal>lustre-OST0000</literal>, and <literal>lustre-OST0001</literal>.</para>
+ <screen>cfs21:/tmp# cat /proc/fs/lustre/mgs/MGS/live/*
+ fsname: lustre
+ flags: 0x0 gen: 26
+ lustre-MDT0000
+ lustre-OST0000
+ lustre-OST0001 </screen>
+ <para>To get the names of all OSTs, run this command on the MDS:</para>
+ <screen># cat /proc/fs/lustre/lov/<fsname>-mdtlov/target_obd </screen>
+ <note>
+ <para>
+ This command must be run on the MDS.
+ </para>
+ </note>
+ <para>In this example, there are two OSTs, lustre-OST0000 and lustre-OST0001, which are both active.</para>
+ <screen>cfs21:/tmp# cat /proc/fs/lustre/lov/lustre-mdtlov/target_obd
+0: lustre-OST0000_UUID ACTIVE
+1: lustre-OST0001_UUID ACTIVE </screen>
+ </section>
+ <section xml:id="dbdoclet.50438199_26070">
+ <title>14.3 Mounting a Server Without Lustre Service</title>
+ <para>If you are using a combined MGS/MDT, but you only want to start the MGS and not the MDT, run this command:</para>
+ <screen>mount -t lustre <MDT partition> -o nosvc <mount point></screen>
+ <para>The <literal><MDT partition></literal> variable is the combined MGS/MDT.</para>
+ <para>In this example, the combined MGS/MDT is <literal>testfs-MDT0000</literal> and the mount point is <literal>/mnt/test/md</literal>t.</para>
+ <screen>$ mount -t lustre -L testfs-MDT0000 -o nosvc /mnt/test/mdt</screen>
+ </section>
+ <section xml:id="dbdoclet.50438199_54623">
+ <title>14.4 Regenerating Lustre Configuration Logs</title>
+ <para>If the Lustre system's configuration logs are in a state where the file system cannot be started, use the <literal>writeconf</literal> command to erase them. After the <literal>writeconf</literal> command is run and the servers restart, the configuration logs are re-generated and stored on the MGS (as in a new file system).</para>
+ <para>You should only use the <literal>writeconf</literal> command if:</para>
+ <itemizedlist>
+ <listitem>
+ <para> The configuration logs are in a state where the file system cannot start</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_15240"/>Finding Nodes in the Lustre File System</para>
+ <para> A server NID is being changed</para>
</listitem>
+ </itemizedlist>
+ <para>The <literal>writeconf</literal> command is destructive to some configuration items (i.e., OST pools information and items set via <literal>conf_param</literal>), and should be used with caution. To avoid problems:</para>
+ <itemizedlist>
<listitem>
- <para><xref linkend="dbdoclet.50438199_26070"/>Mounting a Server Without Lustre Service</para>
+ <para> Shut down the file system before running the <literal>writeconf</literal> command</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_54623"/>Regenerating Lustre Configuration Logs</para>
+ <para> Run the <literal>writeconf</literal> command on all servers (MDT first, then OSTs)</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_31353"/>Changing a Server NID</para>
+ <para> Start the file system in this order:</para>
+ <itemizedlist>
+ <listitem>
+ <para> MGS (or the combined MGS/MDT)</para>
+ </listitem>
+ <listitem>
+ <para> MDT</para>
+ </listitem>
+ <listitem>
+ <para> OSTs</para>
+ </listitem>
+ <listitem>
+ <para> Lustre clients</para>
+ </listitem>
+ </itemizedlist>
</listitem>
+ </itemizedlist>
+ <caution>
+ <para>
+ The OST pools feature enables a group of OSTs to be named for file striping purposes. If you use OST pools, be aware that running the <literal>writeconf</literal> command erases <emphasis role="bold">all</emphasis> pools information (as well as any other parameters set via <literal>lctl conf_param</literal>). We recommend that the pools definitions (and <literal>conf_param</literal> settings) be executed via a script, so they can be reproduced easily after a <literal>writeconf</literal> is performed.</para>
+ </caution>
+ <para>To regenerate Lustre's system configuration logs:</para>
+ <orderedlist>
<listitem>
- <para><xref linkend="dbdoclet.50438199_22527"/>Adding a New OST to a Lustre File System</para>
+ <para><emphasis role="bold">Shut down the file system in this order.</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para>Unmount the clients.</para>
+ </listitem>
+ <listitem>
+ <para>Unmount the MDT.</para>
+ </listitem>
+ <listitem>
+ <para>Unmount all OSTs.</para>
+ </listitem>
+ </orderedlist>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_14978"/>Removing and Restoring OSTs</para>
+ <para><emphasis role="bold">Make sure the the MDT and OST devices are available.</emphasis></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_77819"/>Aborting Recovery</para>
+ <para><emphasis role="bold">Run the <literal>writeconf</literal> command on all servers.</emphasis></para>
+ <para>Run writeconf on the MDT first, and then the OSTs.</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">On the MDT, run:</emphasis></para>
+ <screen><mdt node>$ tunefs.lustre --writeconf <device></screen>
+ </listitem>
+ <listitem>
+ <emphasis role="bold">
+ <para>On each OST, run:</para>
+ </emphasis>
+ <screen><ost node>$ tunefs.lustre --writeconf <device></screen>
+ </listitem>
+ </orderedlist>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438199_12607"/>Determining Which Machine is Serving an OST</para>
+ <para><emphasis role="bold">Restart the file system in this order.</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para>Mount the MGS (or the combined MGS/MDT).</para>
+ </listitem>
+ <listitem>
+ <para>Mount the MDT.</para>
+ </listitem>
+ <listitem>
+ <para>Mount the OSTs.</para>
+ </listitem>
+ <listitem>
+ <para>Mount the clients.</para>
+ </listitem>
+ </orderedlist>
</listitem>
+ </orderedlist>
+ <para>After the <literal>writeconf</literal> command is run, the configuration logs are re-generated as servers restart.</para>
+ </section>
+ <section xml:id="dbdoclet.50438199_31353">
+ <title>14.5 Changing a Server NID</title>
+ <para>If you need to change the NID on the MDT or an OST, run the <literal>writeconf</literal> command to erase Lustre configuration information (including server NIDs), and then re-generate the system configuration using updated server NIDs.</para>
+ <para>Change a server NID in these situations:</para>
+ <itemizedlist>
<listitem>
- <para><xref linkend="dbdoclet.50438199_62333"/>Changing the Address of a Failover Node</para>
+ <para>New server hardware is added to the file system, and the MDS or an OSS is being moved to the new machine</para>
</listitem>
<listitem>
- <para> </para>
+ <para>New network card is installed in the server</para>
</listitem>
- </itemizedlist>
- <section xml:id="dbdoclet.50438199_42877">
- <title>14.1 <anchor xml:id="dbdoclet.50438199_85142" xreflabel=""/>Working with <anchor xml:id="dbdoclet.50438199_marker-1298888" xreflabel=""/>Inactive OSTs</title>
- <para>To mount a client or an MDT with one or more inactive OSTs, run commands similar to this:</para>
- <screen>client> mount -o exclude=testfs-OST0000 -t lustre uml1:/testfs\ /mnt/testfs
- client> cat /proc/fs/lustre/lov/testfs-clilov-*/target_obd
- </screen>
- <para>To activate an inactive OST on a live client or MDT, use the lctl activate command on the OSC device. For example:</para>
- <screen>lctl --device 7 activate
- </screen>
-
-
- <note>
- <para>
- A colon-separated list can also be specified. For example, exclude=testfs-OST0000:testfs-OST0001.</para>
- </note>
-
- <section xml:id="dbdoclet.50438199_15240">
- <title>14.2 Finding <anchor xml:id="dbdoclet.50438199_marker-1298897" xreflabel=""/>Nodes in the Lustre File System</title>
- <para>There may be situations in which you need to find all nodes in your Lustre file system or get the names of all OSTs.</para>
- <para>To get a list of all Lustre nodes, run this command on the MGS:</para>
- <screen># cat /proc/fs/lustre/mgs/MGS/live/*
- </screen>
- <note>
- <para>
- This command must be rund on the MGS.
- </para>
- </note>
-
- <para>In this example, file system lustre has three nodes, lustre-MDT0000, lustre-OST0000, and lustre-OST0001.</para>
- <screen>cfs21:/tmp# cat /proc/fs/lustre/mgs/MGS/live/*
- fsname: lustre
- flags: 0x0 gen: 26
- lustre-MDT0000
- lustre-OST0000
- lustre-OST0001
- </screen>
- <para>To get the names of all OSTs, run this command on the MDS:</para>
- <screen># cat /proc/fs/lustre/lov/<fsname>-mdtlov/target_obd
- </screen>
- <note>
- <para>
- This command must be rund on the MDS.
- </para>
- </note>
-
- <para>In this example, there are two OSTs, lustre-OST0000 and lustre-OST0001, which are both active.</para>
- <screen>cfs21:/tmp# cat /proc/fs/lustre/lov/lustre-mdtlov/target_obd
- 0: lustre-OST0000_UUID ACTIVE
- 1: lustre-OST0001_UUID ACTIVE
- </screen>
- </section>
- <section xml:id="dbdoclet.50438199_26070">
- <title>14.3 Mounting a Server Without <anchor xml:id="dbdoclet.50438199_marker-1298918" xreflabel=""/>Lustre Service</title>
- <para>If you are using a combined MGS/MDT, but you only want to start the MGS and not the MDT, run this command:</para>
- <screen>mount -t lustre <MDT partition> -o nosvc <mount point>
- </screen>
- <para>The <MDT partition> variable is the combined MGS/MDT.</para>
- <para>In this example, the combined MGS/MDT is testfs-MDT0000 and the mount point is mnt/test/mdt.</para>
- <screen>$ mount -t lustre -L testfs-MDT0000 -o nosvc /mnt/test/mdt
- </screen>
- </section>
- <section xml:id="dbdoclet.50438199_54623">
- <title>14.4 Regenerating Lustre <anchor xml:id="dbdoclet.50438199_marker-1305736" xreflabel=""/>Configuration Logs</title>
- <para>If the Lustre system's configuration logs are in a state where the file system cannot be started, use the writeconf command to erase them. After the writeconf command is run and the servers restart, the configuration logs are re-generated and stored on the MGS (as in a new file system).</para>
- <para>You should only use the writeconf command if:</para>
- <itemizedlist><listitem>
- <para> The configuration logs are in a state where the file system cannot start</para>
- </listitem>
- <listitem>
- <para> A server NID is being changed</para>
- </listitem>
- </itemizedlist>
- <para>The writeconf command is destructive to some configuration items (i.e., OST pools information and items set via conf_param), and should be used with caution. To avoid problems:</para>
- <itemizedlist><listitem>
- <para> Shut down the file system before running the writeconf command</para>
- </listitem>
- <listitem>
- <para> Run the writeconf command on all servers (MDT first, then OSTs)</para>
- </listitem>
- <listitem>
- <para> Start the file system in this order:</para>
- </listitem>
- <listitem>
- <orderedlist><listitem>
- <para> MGS (or the combined MGS/MDT)</para>
- </listitem>
- <listitem>
- <para> MDT</para>
- </listitem>
- <listitem>
- <para> OSTs</para>
- </listitem>
- <listitem>
- <para> Lustre clients</para>
- </listitem>
- </orderedlist>
- </listitem>
- </itemizedlist>
-
- <caution>
- <para>
- The OST pools feature enables a group of OSTs to be named for file striping purposes. If you use OST pools, be aware that running the writeconf command erases <emphasis role="bold">all</emphasis> pools information (as well as any other parameters set via lctl conf_param). We recommend that the pools definitions (and conf_param settings) be executed via a script, so they can be reproduced easily after a writeconf is performed.</para>
- </caution>
-
- <para>To regenerate Lustre's system configuration logs:</para>
- <orderedlist><listitem>
- <para>Shut down the file system in this order.</para>
- <orderedlist><listitem>
- <para>Unmount the clients.</para>
- </listitem><listitem>
- <para>Unmount the MDT.</para>
- </listitem><listitem>
- <para>Unmount all OSTs.</para>
- </listitem></orderedlist>
- </listitem><listitem>
- <para>Make sure the the MDT and OST devices are available.</para>
- </listitem><listitem>
- <para>Run the writeconf command on all servers.</para>
- <para>Run writeconf on the MDT first, and then the OSTs.</para>
- <orderedlist><listitem>
- <para>On the MDT, run:</para>
- <screen><mdt node>$ tunefs.lustre --writeconf <device>
- </screen>
- </listitem><listitem>
- <para>On each OST, run:</para>
- <screen><ost node>$ tunefs.lustre --writeconf <device>
- </screen>
- </listitem></orderedlist>
- <para>Restart the file system in this order.</para>
- <orderedlist><listitem>
- <para>Mount the MGS (or the combined MGS/MDT).</para>
- </listitem><listitem>
- <para>Mount the MDT.</para>
- </listitem><listitem>
- <para>Mount the OSTs.</para>
- </listitem><listitem>
- <para>Mount the clients.</para>
- </listitem></orderedlist>
- <para>After the writeconf command is run, the configuration logs are re-generated as servers restart.</para>
- </listitem></orderedlist>
- </section>
- <section xml:id="dbdoclet.50438199_31353">
- <title>14.5 Changing a <anchor xml:id="dbdoclet.50438199_marker-1305737" xreflabel=""/>Server NID</title>
- <para>If you need to change the NID on the MDT or an OST, run the writeconf command to erase Lustre configuration information (including server NIDs), and then re-generate the system configuration using updated server NIDs.</para>
- <para>Change a server NID in these situations:</para>
- <itemizedlist><listitem>
- <para> New server hardware is added to the file system, and the MDS or an OSS is being moved to the new machine</para>
- </listitem>
- <listitem>
- <para> New network card is installed in the server</para>
- </listitem>
- <listitem>
- <para> You want to reassign IP addresses</para>
- </listitem>
- </itemizedlist>
- <para><anchor xml:id="dbdoclet.50438199_DDE_LINK1" xreflabel=""/>To change a server NID:</para>
- <orderedlist><listitem>
- <para>Update the LNET configuration in the /etc/modprobe.conf file so the list of server NIDs (lctl list_nids) is correct.</para>
- <para>The lctl list_nids command indicates which network(s) are configured to work with Lustre.</para>
- </listitem><listitem>
- <para>Shut down the file system in this order.</para>
- <orderedlist><listitem>
- <para>Unmount the clients.</para>
- </listitem><listitem>
- <para>Unmount the MDT.</para>
- </listitem><listitem>
- <para>Unmount all OSTs.</para>
- </listitem></orderedlist>
- </listitem><listitem>
- <para>Run the writeconf command on all servers.</para>
- <para>Run writeconf on the MDT first, and then the OSTs.</para>
- <orderedlist><listitem>
- <para>On the MDT, run:</para>
- <screen><mdt node>$ tunefs.lustre --writeconf <device>
- </screen>
- </listitem><listitem>
- <para>On each OST, run:</para>
- <screen><ost node>$ tunefs.lustre --writeconf <device>
- </screen>
- </listitem><listitem>
- <para>If the NID on the MGS was changed, communicate the new MGS location to each server. Run:</para>
- <screen>tunefs.lustre --erase-param --mgsnode=<new_nid(s)> --writeconf /dev/..
- </screen>
- </listitem></orderedlist>
- </listitem><listitem>
- <para>Restart the file system in this order.</para>
- <orderedlist><listitem>
- <para>Mount the MGS (or the combined MGS/MDT).</para>
- </listitem><listitem>
- <para>Mount the MDT.</para>
- </listitem><listitem>
- <para>Mount the OSTs.</para>
- </listitem><listitem>
- <para>Mount the clients.</para>
- </listitem></orderedlist>
- </listitem></orderedlist>
- <para>After the writeconf command is run, the configuration logs are re-generated as servers restart, and server NIDs in the updated list_nids file are used.</para>
- </section>
- <section xml:id="dbdoclet.50438199_22527">
- <title>14.6 Adding a New <anchor xml:id="dbdoclet.50438199_marker-1306353" xreflabel=""/>OST to a Lustre File System</title>
- <para>To add an OST to existing Lustre file system:</para>
- <orderedlist><listitem>
- <para> 1. Add a new OST by passing on the following commands, run:</para>
- <screen>$ mkfs.lustre --fsname=spfs --ost --mgsnode=mds16@tcp0 /dev/sda
- $ mkdir -p /mnt/test/ost0
- $ mount -t lustre /dev/sda /mnt/test/ost0
- </screen>
- </listitem><listitem>
- <para> 2. Migrate the data (possibly).</para>
- </listitem></orderedlist>
-<para>The file system is quite unbalanced when new empty OSTs are added. New file creations are automatically balanced. If this is a scratch file system or files are pruned at a regular interval, then no further work may be needed.</para>
-<para>New files being created will preferentially be placed on the empty OST. As old files are deleted, they will release space on the old OST.</para>
-<para>Files existing prior to the expansion can optionally be rebalanced with an in-place copy, which can be done with a simple script. The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
-<para>For example, to rebalance all files within /mnt/lustre/dir, enter:</para>
-<screen>lfs_migrate /mnt/lustre/file
-</screen>
-<para>To migrate files within the /test filesystem on OST0004 that are larger than 4GB in size, enter:</para>
-<screen>lfs find /test -obd test-OST0004 -size +4G | lfs_migrate -y
-</screen>
-<para>See <xref linkend='userutilities'/> (lfs_migrate) for more details.</para>
-</section>
-<section xml:id="dbdoclet.50438199_14978">
-<title>14.7 Removing and Restoring OSTs</title>
-<para>OSTs can be removed from and restored to a Lustre file system. Currently in Lustre, removing an OST really means that the OST is 'deactivated' in the file system, not permanently removed. A removed OST still appears in the file system; do not create a new OST with the same name.</para>
-<para>You may want to remove (deactivate) an OST and prevent new files from being written to it in several situations:</para>
-<itemizedlist><listitem>
-<para> Hard drive has failed and a RAID resync/rebuild is underway</para>
-</listitem>
-<listitem>
-<para> OST is nearing its space capacity</para>
-</listitem>
-</itemizedlist>
-<section remap="h3">
-<title>14.7.1 Removing an OST from the File System</title>
-<para>OSTs can be removed from a Lustre file system. Currently in Lustre, removing an OST actually means that the OST is 'deactivated' from the file system, not permanently removed. A removed OST still appears in the device listing; you should not normally create a new OST with the same name.</para>
-<para>You may want to deactivate an OST and prevent new files from being written to it in several situations:</para>
-<itemizedlist><listitem>
-<para> OST is nearing its space capacity</para>
-</listitem>
-<listitem>
-<para> Hard drive has failed and a RAID resync/rebuild is underway</para>
-</listitem>
-<listitem>
-<para> OST storage has failed permanently</para>
-</listitem>
-</itemizedlist>
-<para>When removing an OST, remember that the MDT does not communicate directly with OSTs. Rather, each OST has a corresponding OSC which communicates with the MDT. It is necessary to determine the device number of the OSC that corresponds to the OST. Then, you use this device number to deactivate the OSC on the MDT.</para>
-<para>To remove an OST from the file system:</para>
-
-<orderedlist><listitem>
-<para>For the OST to be removed, determine the device number of the corresponding OSC on the MDT.</para>
-<orderedlist><listitem>
-<para> List all OSCs on the node, along with their device numbers. Run:</para>
-<screen>lctldl|grep " osc "
+ <listitem>
+ <para>You want to reassign IP addresses</para>
+ </listitem>
+ </itemizedlist>
+ <para>To change a server NID:</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Update the LNET configuration in the <literal>/etc/modprobe.conf</literal> file so the list of server NIDs (<literal>lctl list_nids</literal>) is correct.</emphasis></para>
+ <para>The <literal>lctl list_nids</literal> command indicates which network(s) are configured to work with Lustre.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Shut down the file system in this order.</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para>Unmount the clients.</para>
+ </listitem>
+ <listitem>
+ <para>Unmount the MDT.</para>
+ </listitem>
+ <listitem>
+ <para>Unmount all OSTs.</para>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Run the writeconf command on all servers.</emphasis></para>
+ <para>Run writeconf on the MDT first, and then the OSTs.</para>
+ <orderedlist>
+ <listitem>
+ <para>On the MDT, run:</para>
+ <screen><mdt node>$ tunefs.lustre --writeconf <device> </screen>
+ </listitem>
+ <listitem>
+ <para>On each OST, run:</para>
+ <screen><ost node>$ tunefs.lustre --writeconf <device></screen>
+ </listitem>
+ <listitem>
+ <para>If the NID on the MGS was changed, communicate the new MGS location to each server. Run:</para>
+ <screen>tunefs.lustre --erase-param --mgsnode=<new_nid(s)> --writeconf /dev/.. </screen>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Restart the file system in this order.</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para>Mount the MGS (or the combined MGS/MDT).</para>
+ </listitem>
+ <listitem>
+ <para>Mount the MDT.</para>
+ </listitem>
+ <listitem>
+ <para>Mount the OSTs.</para>
+ </listitem>
+ <listitem>
+ <para>Mount the clients.</para>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ </orderedlist>
+ <para>After the <literal>writeconf</literal> command is run, the configuration logs are re-generated as servers restart, and server NIDs in the updated <literal>list_nids</literal> file are used.</para>
+ </section>
+ <section xml:id="dbdoclet.50438199_22527">
+ <title>14.6 Adding a New OST to a Lustre File System</title>
+ <para>To add an OST to existing Lustre file system:</para>
+ <orderedlist>
+ <listitem>
+ <para> <emphasis role="bold">Add a new OST by passing on the following commands, run:</emphasis></para>
+ <screen>$ mkfs.lustre --fsname=spfs --ost --mgsnode=mds16@tcp0 /dev/sda
+$ mkdir -p /mnt/test/ost0
+$ mount -t lustre /dev/sda /mnt/test/ost0 </screen>
+ </listitem>
+ <listitem>
+ <para> <emphasis role="bold">Migrate the data (possibly).</emphasis></para>
+ <para>The file system is quite unbalanced when new empty OSTs are added. New file creations are automatically balanced. If this is a scratch file system or files are pruned at a regular interval, then no further work may be needed.</para>
+ <para>New files being created will preferentially be placed on the empty OST. As old files are deleted, they will release space on the old OST.</para>
+ <para>Files existing prior to the expansion can optionally be rebalanced with an in-place copy, which can be done with a simple script. The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
+ <para>For example, to rebalance all files within <literal>/mnt/lustre/dir</literal>, enter:</para>
+ <screen>lfs_migrate /mnt/lustre/file</screen>
+ <para>To migrate files within the <literal>/test</literal> filesystem on <literal>OST0004</literal> that are larger than 4GB in size, enter:</para>
+ <screen>lfs find /test -obd test-OST0004 -size +4G | lfs_migrate -y
</screen>
-<para>This is sample lctldl|grep</para>
-<screen>
-11 UP osc lustre-OST-0000-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
+ <para>See <xref linkend="dbdoclet.50438206_42260"/> for more details.</para>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section xml:id="dbdoclet.50438199_14978">
+ <title>14.7 Removing and Restoring OSTs</title>
+ <para>OSTs can be removed from and restored to a Lustre file system. Currently in Lustre, removing an OST really means that the OST is 'deactivated' in the file system, not permanently removed. A removed OST still appears in the file system; do not create a new OST with the same name.</para>
+ <para>You may want to remove (deactivate) an OST and prevent new files from being written to it in several situations:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Hard drive has failed and a RAID resync/rebuild is underway</para>
+ </listitem>
+ <listitem>
+ <para> OST is nearing its space capacity</para>
+ </listitem>
+ </itemizedlist>
+ <section remap="h3">
+ <title>14.7.1 Removing an OST from the File System</title>
+ <para>OSTs can be removed from a Lustre file system. Currently in Lustre, removing an OST actually means that the OST is 'deactivated' from the file system, not permanently removed. A removed OST still appears in the device listing; you should not normally create a new OST with the same name.</para>
+ <para>You may want to deactivate an OST and prevent new files from being written to it in several situations:</para>
+ <itemizedlist>
+ <listitem>
+ <para> OST is nearing its space capacity</para>
+ </listitem>
+ <listitem>
+ <para> Hard drive has failed and a RAID resync/rebuild is underway</para>
+ </listitem>
+ <listitem>
+ <para> OST storage has failed permanently</para>
+ </listitem>
+ </itemizedlist>
+ <para>When removing an OST, remember that the MDT does not communicate directly with OSTs. Rather, each OST has a corresponding OSC which communicates with the MDT. It is necessary to determine the device number of the OSC that corresponds to the OST. Then, you use this device number to deactivate the OSC on the MDT.</para>
+ <para>To remove an OST from the file system:</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">For the OST to be removed, determine the device number of the corresponding OSC on the MDT.</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para>List all OSCs on the node, along with their device numbers. Run:</para>
+ <screen>lctldl|grep " osc "</screen>
+ <para>This is sample <literal>lctl dl | grep</literal></para>
+ <screen>11 UP osc lustre-OST-0000-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
12 UP osc lustre-OST-0001-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
13 IN osc lustre-OST-0000-osc lustre-MDT0000-mdtlov_UUID 5
-14 UP osc lustre-OST-0001-osc lustre-MDT0000-mdtlov_UUID 5
-</screen>
-</listitem><listitem>
-<para>Determine the device number of the OSC that corresponds to the OST to be removed.</para>
-</listitem></orderedlist>
-</listitem><listitem>
-
-<para>Temporarily deactivate the OSC on the MDT. On the MDT, run: </para>
-
-<screen>
-$ mdt> lctl --device >devno< deactivate
-</screen>
-<para>
-For example, based on the command output in Step 1, to deactivate device 13 (the MDT’s OSC for OST-0000), the command would be:
+14 UP osc lustre-OST-0001-osc lustre-MDT0000-mdtlov_UUID 5</screen>
+ </listitem>
+ <listitem>
+ <para>Determine the device number of the OSC that corresponds to the OST to be removed.</para>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Temporarily deactivate the OSC on the MDT. On the MDT, run:</emphasis> </para>
+ <screen>$ mdt> lctl --device >devno< deactivate</screen>
+ <para>
+For example, based on the command output in Step 1, to deactivate device 13 (the MDT’s OSC for <literal>OST-0000</literal>), the command would be:
</para>
-
-<screen>
-$ mdt> lctl --device 13 deactivate
-</screen>
-
-<para>
+ <screen>$ mdt> lctl --device 13 deactivate</screen>
+ <para>
This marks the OST as inactive on the MDS, so no new objects are assigned to the OST. This does not prevent use of existing objects for reads or writes.
</para>
-
-<note><para>Do not deactivate the OST on the clients. Do so causes errors (EIOs), and the copy out to fail. </para></note>
-
-<caution><para>Do not use lctl conf_param to deactivate the OST. It permanently sets a parameter in the file system configuration.</para></caution>
-
-</listitem><listitem>
-<para>
-Discover all files that have objects residing on the deactivated OST.
-</para>
-
-<para>
-Depending on whether the deactivated OST is available or not, the data from that OST may be migrated to other OSTs, or may need to be restored from backup.
+ <note>
+ <para>Do not deactivate the OST on the clients. Do so causes errors (EIOs), and the copy out to fail. </para>
+ </note>
+ <caution>
+ <para>Do not use <literal>lctl conf_param</literal> to deactivate the OST. It permanently sets a parameter in the file system configuration.</para>
+ </caution>
+ </listitem>
+ <listitem>
+ <para>
+<emphasis role="bold">Discover all files that have objects residing on the deactivated OST. </emphasis></para>
+ <para>Depending on whether the deactivated OST is available or not, the data from that OST may be migrated to other OSTs, or may need to be restored from backup.
</para>
-
-<orderedlist><listitem>
-<para>
+ <orderedlist>
+ <listitem>
If the OST is still online and available, find all files with objects on the deactivated OST, and copy them to other OSTs in the file system to:
</para>
+ <screen>[client]# lfs find --obd <OST UUID> <mount_point> | lfs_migrate -y</screen>
+ </listitem>
+ <listitem> If the OST is no longer available, delete the files on that OST and restore them from backup: <screen>[client]# lfs find --obd <OST UUID> -print0 <mount_point> | \
+tee /tmp/files_to_restore | xargs -0 -n 1 unlink</screen> The list of files that need to be restored from backup is stored in <literal>/tmp/files_to_restore</literal>. Restoring these files is beyond the scope of this document. </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem> <emphasis role="bold">Deactivate the OST.</emphasis> <orderedlist>
+ <listitem> To temporarily disable the deactivated OST, enter: <screen>[client]# lctl set_param osc.<fsname>-<OST name>-*.active=0</screen>If there is expected to be a replacement OST in some short time (a few days), the OST can temporarily be deactivated on the clients: <note>
+ <para>This setting is only temporary and will be reset if the clients or MDS are rebooted. It needs to be run on all clients.</para>
+ </note></listitem>
+ <listitem> b. To permanently disable the deactivated OST, enter: <screen>[mgs]# lctl conf_param {OST name}.osc.active=0</screen></listitem>
+ </orderedlist>If there is not expected to be a replacement for this OST in the near future, permanently deactivate the OST on all clients and the MDS: <note>
+ <para>A removed OST still appears in the file system; do not create a new OST with the same name.</para>
+ </note></listitem>
+ </orderedlist>
+ </section>
+ <section remap="h3"><title>14.7.2 Backing Up OST Configuration Files</title>If the OST device is still accessible, then the Lustre configuration files on the OST should be backed up and saved for future use in order to avoid difficulties when a replacement OST is returned to service. These files rarely change, so they can and should be backed up while the OST is functional and accessible. If the deactivated OST is still available to mount (i.e. has not permanently failed or is unmountable due to severe corruption), an effort should be made to preserve these files. <orderedlist>
+ <listitem>
+ <emphasis role="bold">Mount the OST filesystem.</emphasis>
+ <screen>[oss]# mkdir -p /mnt/ost
+[oss]# mount -t ldiskfs {ostdev} /mnt/ost</screen>
+ </listitem>
+ <listitem>
+ <emphasis role="bold">Back up the OST configuration files.</emphasis>
+ <screen>[oss]# tar cvf {ostname}.tar -C /mnt/ost last_rcvd \
+CONFIGS/ O/0/LAST_ID</screen>
+ </listitem>
+ <listitem>
+ <emphasis role="bold">Unmount the OST filesystem.</emphasis>
+ <screen>[oss]# umount /mnt/ost</screen>
+ </listitem>
+ </orderedlist></section>
+ <section><title>14.7.3 Restoring OST Configuration Files</title>
-<screen>
-[client]# lfs find --obd <OST UUID> <mount_point> | lfs_migrate -y
-</screen>
-
-</listitem><listitem>
-If the OST is no longer available, delete the files on that OST and restore them from backup:
-
-<screen>
-[client]# lfs find --obd <OST UUID> -print0 <mount_point> | \
-tee /tmp/files_to_restore | xargs -0 -n 1 unlink
-</screen>
-
-The list of files that need to be restored from backup is stored in /tmp/files_to_restore. Restoring these files is beyond the scope of this document.
-
-</listitem></orderedlist>
-</listitem><listitem>
-Deactivate the OST.
-
-<orderedlist><listitem>
-To temporarily disable the deactivated OST, enter:
-
-<screen>
-[client]# lctl set_param osc.<fsname>-<OST name>-*.active=0
-</screen>
-
-If there is expected to be a replacement OST in some short time (a few days), the OST can temporarily be deactivated on the clients:
-<note><para> This setting is only temporary and will be reset if the clients or MDS are rebooted. It needs to be run on all clients.</para></note>
-
-</listitem><listitem>
-b. To permanently disable the deactivated OST, enter:
-
-<screen>
-[mgs]# lctl conf_param {OST name}.osc.active=0
-</screen>
-</listitem></orderedlist>
-
-If there is not expected to be a replacement for this OST in the near future, permanently deactivate the OST on all clients and the MDS:
-<note><para>A removed OST still appears in the file system; do not create a new OST with the same name.</para></note>
-</listitem></orderedlist>
-
-</section>
-<section remap="h3">
- <title>14.7.2 Backing Up OST Configuration Files</title>
-
-If the OST device is still accessible, then the Lustre configuration files on the OST should be backed up and saved for future use in order to avoid difficulties when a replacement OST is returned to service. These files rarely change, so they can and should be backed up while the OST is functional and accessible. If the deactivated OST is still available to mount (i.e. has not permanently failed or is unmountable due to severe corruption), an effort should be made to preserve these files.
-
-<orderedlist><listitem>
-1. Mount the OST filesystem.
-
-<screen>
-[oss]# mkdir -p /mnt/ost
-[oss]# mount -t ldiskfs {ostdev} /mnt/ost
-</screen>
-
-</listitem><listitem>
-2. Back up the OST configuration files.
-
-<screen>
-[oss]# tar cvf {ostname}.tar -C /mnt/ost last_rcvd \
-CONFIGS/ O/0/LAST_ID
-</screen>
-
-</listitem><listitem>
-3. Unmount the OST filesystem.
-
-<screen>
-[oss]# umount /mnt/ost
-</screen>
-</listitem></orderedlist>
-
-</section>
-<section>
- <title>14.7.3 Restoring OST Configuration Files</title>
-
-If the original OST is still available, it is best to follow the OST backup and restore procedure given in either Backing Up and Restoring an MDS or OST (Device Level), or Making a File-Level Backup of an OST File System and Restoring a File-Level Backup.
-
-To replace an OST that was removed from service due to corruption or hardware failure, the file system needs to be formatted for Lustre, and the Lustre configuration should be restored, if available.
-
-If the OST configuration files were not backed up, due to the OST file system being completely inaccessible, it is still possible to replace the failed OST with a new one at the same OST index.
-
-<orderedlist><listitem>
-1. Format the OST file system.
-
-<screen>
-[oss]# mkfs.lustre --ost --index {OST index} {other options} \
-{newdev}
-</screen>
-
-</listitem><listitem>
-2. Mount the OST filesystem.
-
-<screen>
-[oss]# mkdir /mnt/ost
-[oss]# mount -t ldiskfs {newdev} /mnt/ost
-</screen>
-
-</listitem><listitem>
-3. Restore the OST configuration files, if available.
-
-<screen>
-[oss]# tar xvf {ostname}.tar -C /mnt/ost
-</screen>
-
-</listitem><listitem>
-4. Recreate the OST configuration files, if unavailable.
-
-Follow the procedure in Fixing a Bad LAST_ID on an OST to recreate the LAST_ID file for this OST index. The last_rcvd file will be recreated when the OST is first mounted using the default parameters, which are normally correct for all file systems.
-
-The CONFIGS/mountdata file is created by mkfs.lustre at format time, but has flags set that request it to register itself with the MGS. It is possible to copy these flags from another working OST (which should be the same):
-
-<screen>
-[oss2]# debugfs -c -R "dump CONFIGS/mountdata /tmp/ldd" \
+ <para>
+ If the original OST is still available, it is best to follow the OST backup and restore procedure given in either <xref linkend='dbdoclet.50438207_71633'/>, or <xref linkend='dbdoclet.50438207_21638'/> and <xref linkend='dbdoclet.50438207_22325'/>.
+ </para>
+
+ <para>To replace an OST that was removed from service due to corruption or hardware failure, the file system needs to be formatted for Lustre, and the Lustre configuration should be restored, if available.
+
+ </para>
+ <para>If the OST configuration files were not backed up, due to the OST file system being completely inaccessible, it is still possible to replace the failed OST with a new one at the same OST index. </para>
+ <orderedlist>
+ <listitem>
+ <emphasis role="bold">Format the OST file system. </emphasis>
+ <screen>[oss]# mkfs.lustre --ost --index {OST index} {other options} \
+{newdev}</screen>
+ </listitem>
+ <listitem>
+ <emphasis role="bold">Mount the OST filesystem. </emphasis>
+ <screen>[oss]# mkdir /mnt/ost
+[oss]# mount -t ldiskfs {newdev} /mnt/ost</screen>
+ </listitem>
+ <listitem>
+ <emphasis role="bold">Restore the OST configuration files, if available. </emphasis>
+ <screen>[oss]# tar xvf {ostname}.tar -C /mnt/ost</screen>
+ </listitem>
+ <listitem>
+ <emphasis role="bold">Recreate the OST configuration files, if unavailable.</emphasis>
+ <para>Follow the procedure in <xref linkend='dbdoclet.50438198_69657'/> to recreate the LAST_ID file for this OST index. The <literal>last_rcvd</literal> file will be recreated when the OST is first mounted using the default parameters, which are normally correct for all file systems.
+
+The <literal>CONFIGS/mountdata</literal> file is created by <literal>mkfs.lustre</literal> at format time, but has flags set that request it to register itself with the MGS. It is possible to copy these flags from another working OST (which should be the same):
+
+<screen>[oss2]# debugfs -c -R "dump CONFIGS/mountdata /tmp/ldd" \
{other_osdev}
[oss2]# scp /tmp/ldd oss:/tmp/ldd
[oss]# dd if=/tmp/ldd of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 \
-seek=5 skip=5
-</screen>
-
-</listitem><listitem>
-5. Unmount the OST filesystem.
-
-<screen>
-[oss]# umount /mnt/ost
-</screen>
-</listitem></orderedlist>
-
-</section>
-<section>
- <title>14.7.4 Returning a Deactivated OST to Service</title>
-
-If the OST was permanently deactivated, it needs to be reactivated in the MGS configuration.
-
-<screen>
-[mgs]# lctl conf_param {OST name}.osc.active=1
-</screen>
-
-If the OST was temporarily deactivated, it needs to be reactivated on the MDS and clients.
-
-<screen>
-[mds]# lctl --device <devno> activate
-[client]# lctl set_param osc.<fsname>-<OST name>-*.active=0
-</screen>
-
-</section>
-</section>
- <section xml:id="dbdoclet.50438199_77819">
- <title>14.8 Aborting Recovery</title>
-
-You can abort recovery with either the lctl utility or by mounting the target with the abort_recov option (mount -o abort_recov). When starting a target, run:
-
-<screen>
-$ mount -t lustre -L <MDT name> -o abort_recov <mount_point>
-</screen>
-
-Note - The recovery process is blocked until all OSTs are available.
-
-</section>
- <section xml:id="dbdoclet.50438199_12607">
- <title>14.9 Determining Which Machine is Serving an OST </title>
-
-In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead.
-
-To identify the NID that is serving a specific OST, run one of the following commands on a client (you do not need to be a root user):
-
-<screen>
-client$ lctl get_param osc.${fsname}-${OSTname}*.ost_conn_uuid
-</screen>
-
-For example:
-
-<screen>
-client$ lctl get_param osc.*-OST0000*.ost_conn_uuid
-osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-</screen>
-
-- OR -
-
-<screen>
-client$ lctl get_param osc.*.ost_conn_uuid
+seek=5 skip=5</screen></para>
+ </listitem>
+ <listitem>
+ <emphasis role="bold">Unmount the OST filesystem.</emphasis>
+ <screen>[oss]# umount /mnt/ost</screen>
+ </listitem>
+ </orderedlist></section>
+ <section><title>14.7.4 Returning a Deactivated OST to Service</title> If the OST was permanently deactivated, it needs to be reactivated in the MGS configuration. <screen>[mgs]# lctl conf_param {OST name}.osc.active=1</screen> If the OST was temporarily deactivated, it needs to be reactivated on the MDS and clients. <screen>[mds]# lctl --device <devno> activate
+[client]# lctl set_param osc.<fsname>-<OST name>-*.active=0</screen></section>
+ </section>
+ <section xml:id="dbdoclet.50438199_77819"><title>14.8 Aborting Recovery</title>You can abort recovery with either the <literal>lctl</literal> utility or by mounting the target with the <literal>abort_recov</literal> option (<literal>mount -o abort_recov</literal>). When starting a target, run: <screen>$ mount -t lustre -L <MDT name> -o abort_recov <mount_point></screen> Note - The recovery process is blocked until all OSTs are available. </section>
+ <section xml:id="dbdoclet.50438199_12607"><title>14.9 Determining Which Machine is Serving an OST </title> In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the NID that is serving a specific OST, run one of the following commands on a client (you do not need to be a root user): <screen>client$ lctl get_param osc.${fsname}-${OSTname}*.ost_conn_uuid</screen> For example: <screen>client$ lctl get_param osc.*-OST0000*.ost_conn_uuid
+osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen> - OR - <screen>client$ lctl get_param osc.*.ost_conn_uuid
osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
osc.lustre-OST0001-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
osc.lustre-OST0002-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
osc.lustre-OST0003-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-osc.lustre-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-</screen>
-
-</section>
- <section xml:id="dbdoclet.50438199_62333">
- <title>14.10 Changing the Address of a Failover Node</title>
-
-To change the address of a failover node (e.g, to use node X instead of node Y), run this command on the OSS/OST partition:
-
-<screen>
-tunefs.lustre --erase-params --failnode=<NID> <device>
-</screen>
-
-
-
-
- </section>
- </section>
+osc.lustre-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen></section>
+ <section xml:id="dbdoclet.50438199_62333"><title>14.10 Changing the Address of a Failover Node</title>To change the address of a failover node (e.g, to use node X instead of node Y), run this command on the OSS/OST partition: <screen>tunefs.lustre --erase-params --failnode=<NID> <device> </screen></section>
+ </section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='lustreoperations'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustreoperations">
<info>
- <title xml:id='lustreoperations.title'>Lustre Operations</title>
+ <title xml:id="lustreoperations.title">Lustre Operations</title>
</info>
-
-
<para>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre administration tasks:</para>
- <itemizedlist><listitem>
+ <itemizedlist>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_42877"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_24122"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_84876"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_69255"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_57420"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_54138"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_88063"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_88980"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_41817"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_70905"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_16954"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_69998"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438194_30872"/></para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
-</itemizedlist>
- <section xml:id="dbdoclet.50438194_42877">
- <title>13.1 Mounting by Label</title>
- <para>The file system name is limited to 8 characters. We have encoded the file system and target information in the disk label, so you can mount by label. This allows system administrators to move disks around without worrying about issues such as SCSI disk reordering or getting the /dev/device wrong for a shared target. Soon, file system naming will be made as fail-safe as possible. Currently, Linux disk labels are limited to 16 characters. To identify the target within the file system, 8 characters are reserved, leaving 8 characters for the file system name:</para>
- <para><fsname>-MDT0000 or <fsname>-OST0a19</para>
- <para>To mount by label, use this command:</para>
- <screen>$ mount -t lustre -L <file system label> <mount point>
-</screen>
- <para>This is an example of mount-by-label:</para>
- <screen>$ mount -t lustre -L testfs-MDT0000 /mnt/mdt
-</screen>
- <caution><para>Mount-by-label should NOT be used in a multi-path environment.</para></caution>
-
- <para>Although the file system name is internally limited to 8 characters, you can mount the clients at any mount point, so file system users are not subjected to short names. Here is an example:</para>
- <screen>mount -t lustre uml1@tcp0:/shortfs /mnt/<long-file_system-name>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438194_42877">
+ <title>13.1 Mounting by Label</title>
+ <para>The file system name is limited to 8 characters. We have encoded the file system and target information in the disk label, so you can mount by label. This allows system administrators to move disks around without worrying about issues such as SCSI disk reordering or getting the <literal>/dev/device</literal> wrong for a shared target. Soon, file system naming will be made as fail-safe as possible. Currently, Linux disk labels are limited to 16 characters. To identify the target within the file system, 8 characters are reserved, leaving 8 characters for the file system name:</para>
+ <para><fsname>-MDT0000 or <fsname>-OST0a19</para>
+ <para>To mount by label, use this command:</para>
+ <screen>$ mount -t lustre -L <file system label> <mount point>
+</screen>
+ <para>This is an example of mount-by-label:</para>
+ <screen>$ mount -t lustre -L testfs-MDT0000 /mnt/mdt
+</screen>
+ <caution>
+ <para>Mount-by-label should NOT be used in a multi-path environment.</para>
+ </caution>
+ <para>Although the file system name is internally limited to 8 characters, you can mount the clients at any mount point, so file system users are not subjected to short names. Here is an example:</para>
+ <screen>mount -t lustre uml1@tcp0:/shortfs /mnt/<long-file_system-name>
</screen>
- </section>
- <section xml:id="dbdoclet.50438194_24122">
- <title>13.2 Starting <anchor xml:id="dbdoclet.50438194_marker-1305696" xreflabel=""/>Lustre</title>
- <para>The startup order of Lustre components depends on whether you have a combined MGS/MDT or these components are separate.</para>
- <itemizedlist><listitem>
- <para> If you have a combined MGS/MDT, the recommended startup order is OSTs, then the MGS/MDT, and then clients.</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para> If the MGS and MDT are separate, the recommended startup order is: MGS, then OSTs, then the MDT, and then clients.</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-</itemizedlist>
- <note><para>If an OST is added to a Lustre file system with a combined MGS/MDT, then the startup order changes slightly; the MGS must be started first because the OST needs to write its configuration data to it. In this scenario, the startup order is MGS/MDT, then OSTs, then the clients.</para></note>
- </section>
- <section xml:id="dbdoclet.50438194_84876">
- <title>13.3 Mounting a <anchor xml:id="dbdoclet.50438194_marker-1298863" xreflabel=""/>Server</title>
- <para>Starting a Lustre server is straightforward and only involves the mount command. Lustre servers can be added to /etc/fstab:</para>
- <screen>mount -t lustre
+ </section>
+ <section xml:id="dbdoclet.50438194_24122">
+ <title>13.2 Starting <anchor xml:id="dbdoclet.50438194_marker-1305696" xreflabel=""/>Lustre</title>
+ <para>The startup order of Lustre components depends on whether you have a combined MGS/MDT or these components are separate.</para>
+ <itemizedlist>
+ <listitem>
+ <para> If you have a combined MGS/MDT, the recommended startup order is OSTs, then the MGS/MDT, and then clients.</para>
+ </listitem>
+ <listitem>
+ <para> </para>
+ </listitem>
+ <listitem>
+ <para> If the MGS and MDT are separate, the recommended startup order is: MGS, then OSTs, then the MDT, and then clients.</para>
+ </listitem>
+ <listitem>
+ <para> </para>
+ </listitem>
+ </itemizedlist>
+ <note>
+ <para>If an OST is added to a Lustre file system with a combined MGS/MDT, then the startup order changes slightly; the MGS must be started first because the OST needs to write its configuration data to it. In this scenario, the startup order is MGS/MDT, then OSTs, then the clients.</para>
+ </note>
+ </section>
+ <section xml:id="dbdoclet.50438194_84876">
+ <title>13.3 Mounting a <anchor xml:id="dbdoclet.50438194_marker-1298863" xreflabel=""/>Server</title>
+ <para>Starting a Lustre server is straightforward and only involves the mount command. Lustre servers can be added to <literal>/etc/fstab</literal>:</para>
+ <screen>mount -t lustre
</screen>
-
<para>The mount command generates output similar to this:</para>
- <screen>/dev/sda1 on /mnt/test/mdt type lustre (rw)
+ <para>The mount command generates output similar to this:</para>
+ <screen>/dev/sda1 on /mnt/test/mdt type lustre (rw)
/dev/sda2 on /mnt/test/ost0 type lustre (rw)
192.168.0.21@tcp:/testfs on /mnt/testfs type lustre (rw)
</screen>
-
<para>In this example, the MDT, an OST (ost0) and file system (testfs) are mounted.</para>
- <screen>LABEL=testfs-MDT0000 /mnt/test/mdt lustre defaults,_netdev,noauto 0 0
+ <para>In this example, the MDT, an OST (ost0) and file system (testfs) are mounted.</para>
+ <screen>LABEL=testfs-MDT0000 /mnt/test/mdt lustre defaults,_netdev,noauto 0 0
LABEL=testfs-OST0000 /mnt/test/ost0 lustre defaults,_netdev,noauto 0 0
</screen>
- <para>In general, it is wise to specify noauto and let your high-availability (HA) package manage when to mount the device. If you are not using failover, make sure that networking has been started before mounting a Lustre server. RedHat, SuSE, Debian (and perhaps others) use the _netdev flag to ensure that these disks are mounted after the network is up.</para>
- <para>We are mounting by disk label here--the label of a device can be read with e2label. The label of a newly-formatted Lustre server ends in FFFF, meaning that it has yet to be assigned. The assignment takes place when the server is first started, and the disk label is updated.</para>
-
- <caution><para>Do not do this when the client and OSS are on the same node, as memory pressure between the client and OSS can lead to deadlocks.</para></caution>
-
- <caution><para>Mount-by-label should NOT be used in a multi-path environment.</para></caution>
-
- </section>
- <section xml:id="dbdoclet.50438194_69255">
- <title>13.4 Unmounting a<anchor xml:id="dbdoclet.50438194_marker-1298879" xreflabel=""/> Server</title>
- <para>To stop a Lustre server, use the umount <mount point> command.</para>
- <para>For example, to stop ost0 on mount point /mnt/test, run:</para>
- <screen>$ umount /mnt/test
-</screen>
- <para>Gracefully stopping a server with the umount command preserves the state of the connected clients. The next time the server is started, it waits for clients to reconnect, and then goes through the recovery procedure.</para>
- <para>If the force (-f) flag is used, then the server evicts all clients and stops WITHOUT recovery. Upon restart, the server does not wait for recovery. Any currently connected clients receive I/O errors until they reconnect.</para>
-
- <note><para>If you are using loopback devices, use the -d flag. This flag cleans up loop devices and can always be safely specified.</para></note>
-
- </section>
- <section xml:id="dbdoclet.50438194_57420">
- <title>13.5 Specifying Fail<anchor xml:id="dbdoclet.50438194_marker-1298926" xreflabel=""/>out/Failover Mode for OSTs</title>
- <para>Lustre uses two modes, failout and failover, to handle an OST that has become unreachable because it fails, is taken off the network, is unmounted, etc.</para>
- <itemizedlist><listitem>
- <para> In <emphasis>failout</emphasis> mode, Lustre clients immediately receive errors (EIOs) after a timeout, instead of waiting for the OST to recover.</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para> In <emphasis>failover</emphasis> mode, Lustre clients wait for the OST to recover.</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-</itemizedlist>
- <para>By default, the Lustre file system uses failover mode for OSTs. To specify failout mode instead, run this command:</para>
- <screen>$ mkfs.lustre --fsname=<fsname> --ost --mgsnode=<MGS node NID> --param="failover\
+ <para>In general, it is wise to specify noauto and let your high-availability (HA) package manage when to mount the device. If you are not using failover, make sure that networking has been started before mounting a Lustre server. RedHat, SuSE, Debian (and perhaps others) use the <literal>_netdev</literal> flag to ensure that these disks are mounted after the network is up.</para>
+ <para>We are mounting by disk label here--the label of a device can be read with <literal>e2label</literal>. The label of a newly-formatted Lustre server ends in <literal>FFFF</literal>, meaning that it has yet to be assigned. The assignment takes place when the server is first started, and the disk label is updated.</para>
+ <caution>
+ <para>Do not do this when the client and OSS are on the same node, as memory pressure between the client and OSS can lead to deadlocks.</para>
+ </caution>
+ <caution>
+ <para>Mount-by-label should NOT be used in a multi-path environment.</para>
+ </caution>
+ </section>
+ <section xml:id="dbdoclet.50438194_69255">
+ <title>13.4 Unmounting a<anchor xml:id="dbdoclet.50438194_marker-1298879" xreflabel=""/> Server</title>
+ <para>To stop a Lustre server, use the <literal>umount <mount point></literal> command.</para>
+ <para>For example, to stop <literal>ost0</literal> on mount point <literal>/mnt/test</literal>, run:</para>
+ <screen>$ umount /mnt/test
+</screen>
+ <para>Gracefully stopping a server with the <literal>umount</literal> command preserves the state of the connected clients. The next time the server is started, it waits for clients to reconnect, and then goes through the recovery procedure.</para>
+ <para>If the force (<literal>-f</literal>) flag is used, then the server evicts all clients and stops WITHOUT recovery. Upon restart, the server does not wait for recovery. Any currently connected clients receive I/O errors until they reconnect.</para>
+ <note>
+ <para>If you are using loopback devices, use the <literal>-d</literal> flag. This flag cleans up loop devices and can always be safely specified.</para>
+ </note>
+ </section>
+ <section xml:id="dbdoclet.50438194_57420">
+ <title>13.5 Specifying Fail<anchor xml:id="dbdoclet.50438194_marker-1298926" xreflabel=""/>out/Failover Mode for OSTs</title>
+ <para>Lustre uses two modes, failout and failover, to handle an OST that has become unreachable because it fails, is taken off the network, is unmounted, etc.</para>
+ <itemizedlist>
+ <listitem>
+ <para> In <emphasis>failout</emphasis> mode, Lustre clients immediately receive errors (EIOs) after a timeout, instead of waiting for the OST to recover.</para>
+ </listitem>
+ <listitem>
+ <para> </para>
+ </listitem>
+ <listitem>
+ <para> In <emphasis>failover</emphasis> mode, Lustre clients wait for the OST to recover.</para>
+ </listitem>
+ <listitem>
+ <para> </para>
+ </listitem>
+ </itemizedlist>
+ <para>By default, the Lustre file system uses failover mode for OSTs. To specify failout mode instead, run this command:</para>
+ <screen>$ mkfs.lustre --fsname=<fsname> --ost --mgsnode=<MGS node NID> --param="failover\
.mode=failout" <block device name>
</screen>
- <para>In this example, failout mode is specified for the OSTs on MGS uml1, file system testfs.</para>
- <screen>$ mkfs.lustre --fsname=testfs --ost --mgsnode=uml1 --param="failover.mode=fa\
+ <para>In this example, failout mode is specified for the OSTs on MGS <literal>uml1</literal>, file system <literal>testfs</literal>.</para>
+ <screen>$ mkfs.lustre --fsname=testfs --ost --mgsnode=uml1 --param="failover.mode=fa\
ilout" /dev/sdb
</screen>
-
- <caution><para>Before running this command, unmount all OSTs that will be affected by the change in the failover/failout mode.</para></caution>
- <note><para>After initial file system configuration, use the tunefs.lustre utility to change the failover/failout mode. For example, to set the failout mode, run:</para><para>$ tunefs.lustre --param failover.mode=failout <OST partition></para></note>
-
- </section>
- <section xml:id="dbdoclet.50438194_54138">
- <title>13.6 Handling <anchor xml:id="dbdoclet.50438194_marker-1307136" xreflabel=""/>Degraded OST RAID Arrays</title>
- <para>Lustre includes functionality that notifies Lustre if an external RAID array has degraded performance (resulting in reduced overall file system performance), either because a disk has failed and not been replaced, or because a disk was replaced and is undergoing a rebuild. To avoid a global performance slowdown due to a degraded OST, the MDS can avoid the OST for new object allocation if it is notified of the degraded state.</para>
- <para>A parameter for each OST, called degraded, specifies whether the OST is running in degraded mode or not.</para>
- <para>To mark the OST as degraded, use:</para>
- <screen>lctl set_param obdfilter.{OST_name}.degraded=1</screen>
- <para>To mark that the OST is back in normal operation, use:</para>
- <screen>lctl set_param obdfilter.{OST_name}.degraded=0
-</screen>
- <para>To determine if OSTs are currently in degraded mode, use:</para>
- <screen>lctl get_param obdfilter.*.degraded
-</screen>
- <para>If the OST is remounted due to a reboot or other condition, the flag resets to 0.</para>
- <para>It is recommended that this be implemented by an automated script that monitors the status of individual RAID devices.</para>
- </section>
- <section xml:id="dbdoclet.50438194_88063">
- <title>13.7 Running Multiple<anchor xml:id="dbdoclet.50438194_marker-1298939" xreflabel=""/> Lustre File Systems</title>
- <para>There may be situations in which you want to run multiple file systems. This is doable, as long as you follow specific naming conventions.</para>
- <para>By default, the mkfs.lustre command creates a file system named lustre. To specify a different file system name (limited to 8 characters), run this command:</para>
- <para>mkfs.lustre --fsname=<new file system name></para>
- <note><para>The MDT, OSTs and clients in the new file system must share the same name (prepended to the device name). For example, for a new file system named foo, the MDT and two OSTs would be named foo-MDT0000, foo-OST0000, and foo-OST0001.</para></note>
-
- <para>To mount a client on the file system, run:</para>
- <screen>mount -t lustre mgsnode:/<new fsname> <mountpoint>
-</screen>
- <para>For example, to mount a client on file system foo at mount point /mnt/lustre1, run:</para>
- <screen>mount -t lustre mgsnode:/foo /mnt/lustre1
-</screen>
- <note><para>If a client(s) will be mounted on several file systems, add the following line to /etc/xattr.conf file to avoid problems when files are moved between the file systems: lustre.* skip</para></note>
- <note><para>The MGS is universal; there is only one MGS per Lustre installation, not per file system.</para></note>
- <note><para>There is only one file system per MDT. Therefore, specify --mdt --mgs on one file system and --mdt --mgsnode=<MGS node NID> on the other file systems.</para></note>
-
- <para>A Lustre installation with two file systems (foo and bar) could look like this, where the MGS node is mgsnode@tcp0 and the mount points are /mnt/lustre1 and /mnt/lustre2.</para>
- <screen>mgsnode# mkfs.lustre --mgs /mnt/lustre1
+ <caution>
+ <para>Before running this command, unmount all OSTs that will be affected by the change in the failover/failout mode.</para>
+ </caution>
+ <note>
+ <para>After initial file system configuration, use the tunefs.lustre utility to change the failover/failout mode. For example, to set the failout mode, run:</para>
+ <para><screen>$ tunefs.lustre --param failover.mode=failout <OST partition></screen></para>
+ </note>
+ </section>
+ <section xml:id="dbdoclet.50438194_54138">
+ <title>13.6 Handling <anchor xml:id="dbdoclet.50438194_marker-1307136" xreflabel=""/>Degraded OST RAID Arrays</title>
+ <para>Lustre includes functionality that notifies Lustre if an external RAID array has degraded performance (resulting in reduced overall file system performance), either because a disk has failed and not been replaced, or because a disk was replaced and is undergoing a rebuild. To avoid a global performance slowdown due to a degraded OST, the MDS can avoid the OST for new object allocation if it is notified of the degraded state.</para>
+ <para>A parameter for each OST, called <literal>degraded</literal>, specifies whether the OST is running in degraded mode or not.</para>
+ <para>To mark the OST as degraded, use:</para>
+ <screen>lctl set_param obdfilter.{OST_name}.degraded=1</screen>
+ <para>To mark that the OST is back in normal operation, use:</para>
+ <screen>lctl set_param obdfilter.{OST_name}.degraded=0
+</screen>
+ <para>To determine if OSTs are currently in degraded mode, use:</para>
+ <screen>lctl get_param obdfilter.*.degraded
+</screen>
+ <para>If the OST is remounted due to a reboot or other condition, the flag resets to <literal>0</literal>.</para>
+ <para>It is recommended that this be implemented by an automated script that monitors the status of individual RAID devices.</para>
+ </section>
+ <section xml:id="dbdoclet.50438194_88063">
+ <title>13.7 Running Multiple<anchor xml:id="dbdoclet.50438194_marker-1298939" xreflabel=""/> Lustre File Systems</title>
+ <para>There may be situations in which you want to run multiple file systems. This is doable, as long as you follow specific naming conventions.</para>
+ <para>By default, the <literal>mkfs.lustre</literal> command creates a file system named <literal>lustre</literal>. To specify a different file system name (limited to 8 characters), run this command:</para>
+ <para><screen>mkfs.lustre --fsname=<new file system name></screen></para>
+ <note>
+ <para>The MDT, OSTs and clients in the new file system must share the same name (prepended to the device name). For example, for a new file system named <literal>foo</literal>, the MDT and two OSTs would be named <literal>foo-MDT0000</literal>, <literal>foo-OST0000</literal>, and <literal>foo-OST0001</literal>.</para>
+ </note>
+ <para>To mount a client on the file system, run:</para>
+ <screen>mount -t lustre mgsnode:/<new fsname> <mountpoint>
+</screen>
+ <para>For example, to mount a client on file system foo at mount point /mnt/lustre1, run:</para>
+ <screen>mount -t lustre mgsnode:/foo /mnt/lustre1
+</screen>
+ <note>
+ <para>If a client(s) will be mounted on several file systems, add the following line to <literal>/etc/xattr.conf</literal> file to avoid problems when files are moved between the file systems: <literal>lustre.* skip</literal></para>
+ </note>
+ <note>
+ <para>The MGS is universal; there is only one MGS per Lustre installation, not per file system.</para>
+ </note>
+ <note>
+ <para>There is only one file system per MDT. Therefore, specify <literal>--mdt --mgs</literal> on one file system and -<literal>-mdt --mgsnode=<MGS node NID></literal> on the other file systems.</para>
+ </note>
+ <para>A Lustre installation with two file systems (<literal>foo</literal> and <literal>bar</literal>) could look like this, where the MGS node is <literal>mgsnode@tcp0</literal> and the mount points are <literal>/mnt/lustre1</literal> and <literal>/mnt/lustre2</literal>.</para>
+ <screen>mgsnode# mkfs.lustre --mgs /mnt/lustre1
mdtfoonode# mkfs.lustre --fsname=foo --mdt --mgsnode=mgsnode@tcp0 /mnt/lust\
re1
ossfoonode# mkfs.lustre --fsname=foo --ost --mgsnode=mgsnode@tcp0 /mnt/lust\
ossbarnode# mkfs.lustre --fsname=bar --ost --mgsnode=mgsnode@tcp0 /mnt/lust\
re2
</screen>
- <para>To mount a client on file system foo at mount point /mnt/lustre1, run:</para>
- <screen>mount -t lustre mgsnode@tcp0:/foo /mnt/lustre1
+ <para>To mount a client on file system foo at mount point <literal>/mnt/lustre1</literal>, run:</para>
+ <screen>mount -t lustre mgsnode@tcp0:/foo /mnt/lustre1
</screen>
- <para>To mount a client on file system bar at mount point /mnt/lustre2, run:</para>
- <screen>mount -t lustre mgsnode@tcp0:/bar /mnt/lustre2
+ <para>To mount a client on file system bar at mount point <literal>/mnt/lustre2</literal>, run:</para>
+ <screen>mount -t lustre mgsnode@tcp0:/bar /mnt/lustre2
</screen>
+ </section>
+ <section xml:id="dbdoclet.50438194_88980">
+ <title>13.8 Setting <anchor xml:id="dbdoclet.50438194_marker-1302467" xreflabel=""/>and Retrieving Lustre Parameters</title>
+ <para>Several options are available for setting parameters in Lustre:</para>
+ <itemizedlist>
+ <listitem>
+ <para> When creating a file system, use mkfs.lustre. See <xref linkend="dbdoclet.50438194_17237"/> below.</para>
+ </listitem>
+ <listitem>
+ <para> When a server is stopped, use tunefs.lustre. See <xref linkend="dbdoclet.50438194_55253"/> below.</para>
+ </listitem>
+ <listitem>
+ <para> When the file system is running, use lctl to set or retrieve Lustre parameters. See <xref linkend="dbdoclet.50438194_51490"/> and <xref linkend="dbdoclet.50438194_63247"/> below.</para>
+ </listitem>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438194_17237">
+ <title>13.8.1 Setting Parameters with <literal>mkfs.lustre</literal></title>
+ <para>When the file system is created, parameters can simply be added as a <literal>--param</literal> option to the <literal>mkfs.lustre</literal> command. For example:</para>
+ <screen>$ mkfs.lustre --mdt --param="sys.timeout=50" /dev/sda
+</screen>
+ <para>For more details about creating a file system,see <xref linkend="configuringlustre"/>. For more details about <literal>mkfs.lustre</literal>, see <xref linkend="systemconfigurationutilities"/>.</para>
</section>
- <section xml:id="dbdoclet.50438194_88980">
- <title>13.8 Setting <anchor xml:id="dbdoclet.50438194_marker-1302467" xreflabel=""/>and Retrieving Lustre Parameters</title>
- <para>Several options are available for setting parameters in Lustre:</para>
- <itemizedlist><listitem>
- <para> When creating a file system, use mkfs.lustre. See <link xl:href="LustreOperations.html#50438194_17237">Setting Parameters with mkfs.lustre</link> below.</para>
- </listitem>
-<listitem>
- <para> When a server is stopped, use tunefs.lustre. See <link xl:href="LustreOperations.html#50438194_55253">Setting Parameters with tunefs.lustre</link> below.</para>
- </listitem>
-<listitem>
- <para> When the file system is running, use lctl to set or retrieve Lustre parameters. See <link xl:href="LustreOperations.html#50438194_51490">Setting Parameters with lctl</link> and <link xl:href="LustreOperations.html#50438194_63247">Reporting Current Parameter Values</link> below.</para>
- </listitem>
-</itemizedlist>
- <section remap="h3">
- <title>13.8.1 <anchor xml:id="dbdoclet.50438194_17237" xreflabel=""/>Setting Parameters with <anchor xml:id="dbdoclet.50438194_marker-1305722" xreflabel=""/>mkfs.lustre</title>
- <para>When the file system is created, parameters can simply be added as a --param option to the mkfs.lustre command. For example:</para>
- <screen>$ mkfs.lustre --mdt --param="sys.timeout=50" /dev/sda
+ <section xml:id="dbdoclet.50438194_55253">
+ <title>13.8.2 Setting Parameters with <literal>tunefs.lustre</literal></title>
+ <para>If a server (OSS or MDS) is stopped, parameters can be added using the <literal>--param</literal> option to the <literal>tunefs.lustre</literal> command. For example:</para>
+ <screen>$ tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda
</screen>
- <para>For more details about creating a file system,see <link xl:href="ConfiguringLustre.html#50438267_88428">Chapter 10</link><emphasis>:</emphasis><link xl:href="ConfiguringLustre.html#50438267_66186">Configuring Lustre</link>. For more details about mkfs.lustre, see <link xl:href="SystemConfigurationUtilities.html#50438219_66186">Chapter 36: System Configuration Utilities</link>.</para>
- </section>
- <section remap="h3">
- <title>13.8.2 <anchor xml:id="dbdoclet.50438194_55253" xreflabel=""/>Setting Parameters with <anchor xml:id="dbdoclet.50438194_marker-1305720" xreflabel=""/>tunefs.lustre</title>
- <para>If a server (OSS or MDS) is stopped, parameters can be added using the --param option to the tunefs.lustre command. For example:</para>
- <screen>$ tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda
-</screen>
- <para>With tunefs.lustre, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old tunefs.lustre parameters and just use newly-specified parameters, run:</para>
- <screen>$ tunefs.lustre --erase-params --param=<new parameters> </screen>
- <para>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <obd|fsname>.<obdtype>.<proc_file_name>=<value>. For example:</para>
- <screen>$ tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1
+ <para>With <literal>tunefs.lustre</literal>, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old <literal>tunefs.lustre</literal> parameters and just use newly-specified parameters, run:</para>
+ <screen>$ tunefs.lustre --erase-params --param=<new parameters> </screen>
+ <para>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <literal><obd|fsname>.<obdtype>.<proc_file_name>=<value></literal>. For example:</para>
+ <screen>$ tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1
</screen>
- <para>For more details about tunefs.lustre, see <link xl:href="SystemConfigurationUtilities.html#50438219_66186">Chapter 36: System Configuration Utilities</link>.</para>
- </section>
- <section remap="h3">
- <title>13.8.3 <anchor xml:id="dbdoclet.50438194_51490" xreflabel=""/>Setting Parameters <anchor xml:id="dbdoclet.50438194_marker-1305718" xreflabel=""/>with lctl</title>
- <para>When the file system is running, the lctl command can be used to set parameters (temporary or permanent) and report current parameter values. Temporary parameters are active as long as the server or client is not shut down. Permanent parameters live through server and client reboots.</para>
- <note><para>The lctl list_param command enables users to list all parameters that can be set. See <xref linkend='dbdoclet.50438194_88217'/>.</para></note>
- <para>For more details about the lctl command, see the examples in the sections below and <xref linkend='systemconfigurationutilities'/>.</para>
- <section remap="h4">
- <title>13.8.3.1 Setting Temporary Parameters</title>
- <para>Use lctl set_param to set temporary parameters on the node where it is run. These parameters map to items in /proc/{fs,sys}/{lnet,lustre}. The lctl set_param command uses this syntax:</para>
- <screen>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value>
-</screen>
- <para>For example:</para>
- <screen># lctl set_param osc.*.max_dirty_mb=1024
+ <para>For more details about tunefs.lustre, see <link xl:href="SystemConfigurationUtilities.html#50438219_66186">Chapter 36: System Configuration Utilities</link>.</para>
+ </section>
+ <section xml:id="dbdoclet.50438194_51490">
+ <title>13.8.3 Setting Parameters with <literal>lctl</literal></title>
+ <para>When the file system is running, the <literal>lctl</literal> command can be used to set parameters (temporary or permanent) and report current parameter values. Temporary parameters are active as long as the server or client is not shut down. Permanent parameters live through server and client reboots.</para>
+ <note>
+ <para>The lctl list_param command enables users to list all parameters that can be set. See <xref linkend="dbdoclet.50438194_88217"/>.</para>
+ </note>
+ <para>For more details about the <literal>lctl</literal> command, see the examples in the sections below and <xref linkend="systemconfigurationutilities"/>.</para>
+ <section remap="h4">
+ <title>13.8.3.1 Setting Temporary Parameters</title>
+ <para>Use <literal>lctl set_param</literal> to set temporary parameters on the node where it is run. These parameters map to items in <literal>/proc/{fs,sys}/{lnet,lustre}</literal>. The <literal>lctl set_param</literal> command uses this syntax:</para>
+ <screen>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value>
+</screen>
+ <para>For example:</para>
+ <screen># lctl set_param osc.*.max_dirty_mb=1024
osc.myth-OST0000-osc.max_dirty_mb=32
osc.myth-OST0001-osc.max_dirty_mb=32
osc.myth-OST0002-osc.max_dirty_mb=32
osc.myth-OST0003-osc.max_dirty_mb=32
osc.myth-OST0004-osc.max_dirty_mb=32
</screen>
- </section>
- <section remap="h4">
-
<title>13.8.3.2 <anchor xml:id="dbdoclet.50438194_64195" xreflabel=""/>Setting Permanent Parameters</title>
- <para>Use the lctl conf_param command to set permanent parameters. In general, the lctl conf_param command can be used to specify any parameter settable in a /proc/fs/lustre file, with its own OBD device. The lctl conf_param command uses this syntax (same as the mkfs.lustre and tunefs.lustre commands):</para>
- <screen><obd|fsname>.<obdtype>.<proc_file_name>=<value>)
+ </section>
+ <section remap="h4">
+ <title>13.8.3.2 <anchor xml:id="dbdoclet.50438194_64195" xreflabel=""/>Setting Permanent Parameters</title>
+ <para>Use the <literal>lctl conf_param</literal> command to set permanent parameters. In general, the <literal>lctl conf_param</literal> command can be used to specify any parameter settable in a <literal>/proc/fs/lustre</literal> file, with its own OBD device. The <literal>lctl conf_param</literal> command uses this syntax (same as the <literal>mkfs.lustre</literal> and <literal>tunefs.lustre</literal> commands):</para>
+ <screen><obd|fsname>.<obdtype>.<proc_file_name>=<value>)
</screen>
- <para>Here are a few examples of lctl conf_param commands:</para>
- <screen>$ mgs> lctl conf_param testfs-MDT0000.sys.timeout=40
+ <para>Here are a few examples of <literal>lctl conf_param</literal> commands:</para>
+ <screen>$ mgs> lctl conf_param testfs-MDT0000.sys.timeout=40
$ lctl conf_param testfs-MDT0000.mdt.group_upcall=NONE
$ lctl conf_param testfs.llite.max_read_ahead_mb=16
$ lctl conf_param testfs-MDT0000.lov.stripesize=2M
$ lctl conf_param testfs-OST0000.ost.client_cache_seconds=15
$ lctl conf_param testfs.sys.timeout=40
</screen>
- <caution><para>Parameters specified with the lctlconf_param command are set permanently in the file system's configuration file on the MGS.</para></caution>
- </section>
- <section remap="h4">
- <title>13.8.3.3 <anchor xml:id="dbdoclet.50438194_88217" xreflabel=""/>Listing Parameters</title>
- <para>To list Lustre or LNET parameters that are available to set, use the lctl list_param command. For example:</para>
- <screen>lctl list_param [-FR] <obdtype>.<obdname>
-</screen>
- <para>The following arguments are available for the lctl list_param command.</para>
- <para>-F Add '/', '@' or '=' for directories, symlinks and writeable files, respectively</para>
- <para>-R Recursively lists all parameters under the specified path</para>
- <para>For example:</para>
- <screen>$ lctl list_param obdfilter.lustre-OST0000
-</screen>
- </section>
- <section remap="h4">
- <title>13.8.3.4 <anchor xml:id="dbdoclet.50438194_63247" xreflabel=""/>Reporting Current <anchor xml:id="dbdoclet.50438194_marker-1302474" xreflabel=""/>Parameter Values</title>
- <para>To report current Lustre parameter values, use the lctl get_param command with this syntax:</para>
- <screen>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name>
-</screen>
- <para>This example reports data on RPC service times.</para>
- <screen>$ lctl get_param -n ost.*.ost_io.timeouts
+ <caution>
+ <para>Parameters specified with the <literal>lctl conf_param</literal> command are set permanently in the file system's configuration file on the MGS.</para>
+ </caution>
+ </section>
+ <section remap="h4">
+ <title>13.8.3.3 <anchor xml:id="dbdoclet.50438194_88217" xreflabel=""/>Listing Parameters</title>
+ <para>To list Lustre or LNET parameters that are available to set, use the <literal>lctl list_param</literal> command. For example:</para>
+ <screen>lctl list_param [-FR] <obdtype>.<obdname>
+</screen>
+ <para>The following arguments are available for the <literal>lctl list_param</literal> command.</para>
+ <para><literal>-F</literal> Add '<literal>/</literal>', '<literal>@</literal>' or '<literal>=</literal>' for directories, symlinks and writeable files, respectively</para>
+ <para><literal>-R</literal> Recursively lists all parameters under the specified path</para>
+ <para>For example:</para>
+ <screen>$ lctl list_param obdfilter.lustre-OST0000
+</screen>
+ </section>
+ <section xml:id="dbdoclet.50438194_63247">
+ <title>13.8.3.4 Reporting Current Parameter Values</title>
+ <para>To report current Lustre parameter values, use the <literal>lctl get_param</literal> command with this syntax:</para>
+ <screen>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name>
+</screen>
+ <para>This example reports data on RPC service times.</para>
+ <screen>$ lctl get_param -n ost.*.ost_io.timeouts
service : cur 1 worst 30 (at 1257150393, 85d23h58m54s ago) 1 1 1 1
</screen>
-
<para>This example reports the number of inodes available on each OST.</para>
- <screen># lctl get_param osc.*.filesfree
+ <para>This example reports the number of inodes available on each OST.</para>
+ <screen># lctl get_param osc.*.filesfree
osc.myth-OST0000-osc-ffff88006dd20000.filesfree=217623
osc.myth-OST0001-osc-ffff88006dd20000.filesfree=5075042
osc.myth-OST0002-osc-ffff88006dd20000.filesfree=3762034
osc.myth-OST0003-osc-ffff88006dd20000.filesfree=91052
osc.myth-OST0004-osc-ffff88006dd20000.filesfree=129651<anchor xml:id="dbdoclet.50438194_88030" xreflabel=""/><anchor xml:id="dbdoclet.50438194_54623" xreflabel=""/></screen>
- </section>
</section>
</section>
- <section xml:id="dbdoclet.50438194_41817">
- <title>13.9 <anchor xml:id="dbdoclet.50438194_42379" xreflabel=""/><anchor xml:id="dbdoclet.50438194_50129" xreflabel=""/>Specifying NIDs and Fail<anchor xml:id="dbdoclet.50438194_marker-1306313" xreflabel=""/>over</title>
- <para>If a node has multiple network interfaces, it may have multiple NIDs. When a node is specified, all of its NIDs must be listed, delimited by commas (,) so other nodes can choose the NID that is appropriate for their network interfaces. When failover nodes are specified, they are delimited by a colon (:) or by repeating a keyword (--mgsnode= or --failnode=). To obtain all NIDs from a node (while LNET is running), run:</para>
- <screen>lctl list_nids
-</screen>
- <para>This displays the server's NIDs (networks configured to work with Lustre).</para>
- <para>This example has a combined MGS/MDT failover pair on uml1 and uml2, and a OST failover pair on uml3 and uml4. There are corresponding Elan addresses on uml1 and uml2.</para>
- <screen>uml1> mkfs.lustre --fsname=testfs --mdt --mgs --failnode=uml2,2@elan /dev/sda1
+ </section>
+ <section xml:id="dbdoclet.50438194_41817">
+ <title>13.9 <anchor xml:id="dbdoclet.50438194_42379" xreflabel=""/><anchor xml:id="dbdoclet.50438194_50129" xreflabel=""/>Specifying NIDs and Failover</title>
+ <para>If a node has multiple network interfaces, it may have multiple NIDs. When a node is specified, all of its NIDs must be listed, delimited by commas (<literal>,</literal>) so other nodes can choose the NID that is appropriate for their network interfaces. When failover nodes are specified, they are delimited by a colon (<literal>:</literal>) or by repeating a keyword (<literal>--mgsnode=</literal> or <literal>--failnode=</literal>). To obtain all NIDs from a node (while LNET is running), run:</para>
+ <screen>lctl list_nids
+</screen>
+ <para>This displays the server's NIDs (networks configured to work with Lustre).</para>
+ <para>This example has a combined MGS/MDT failover pair on uml1 and uml2, and a OST failover pair on uml3 and uml4. There are corresponding Elan addresses on uml1 and uml2.</para>
+ <screen>uml1> mkfs.lustre --fsname=testfs --mdt --mgs --failnode=uml2,2@elan /dev/sda1
uml1> mount -t lustre /dev/sda1 /mnt/test/mdt
uml3> mkfs.lustre --fsname=testfs --ost --failnode=uml4 --mgsnode=uml1,1@ela\
n --mgsnode=uml2,2@elan /dev/sdb
uml2> mount -t lustre /dev/sda1 /mnt/test/mdt
uml2> cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status
</screen>
- <para>Where multiple NIDs are specified, comma-separation (for example, uml2,2@elan) means that the two NIDs refer to the same host, and that Lustre needs to choose the "best" one for communication. Colon-separation (for example, uml1:uml2) means that the two NIDs refer to two different hosts, and should be treated as failover locations (Lustre tries the first one, and if that fails, it tries the second one.)</para>
- <note><para>If you have an MGS or MDT configured for failover, perform these steps:</para><para> 1. On the OST, list the NIDs of all MGS nodes at mkfs time.</para><para>OST# mkfs.lustre --fsname sunfs --ost --mgsnode=10.0.0.1</para><para> --mgsnode=10.0.0.2 /dev/{device}</para><para> 2. On the client, mount the file system.</para><para>client# mount -t lustre 10.0.0.1:10.0.0.2:/sunfs /cfs/client/</para></note>
- </section>
- <section xml:id="dbdoclet.50438194_70905">
- <title>13.10 Erasing a <anchor xml:id="dbdoclet.50438194_marker-1307237" xreflabel=""/>File System</title>
- <para>If you want to erase a file system, run this command on your targets:</para>
- <screen>$ "mkfs.lustre -reformat"
-</screen>
- <para>If you are using a separate MGS and want to keep other file systems defined on that MGS, then set the writeconf flag on the MDT for that file system. The writeconf flag causes the configuration logs to be erased; they are regenerated the next time the servers start.</para>
- <para>To set the writeconf flag on the MDT:</para>
- <orderedlist><listitem>
- <para>Unmount all clients/servers using this file system, run:</para>
- <screen>$ umount /mnt/lustre
-</screen>
-</listitem><listitem>
- <para>Erase the file system and, presumably, replace it with another file system, run:</para>
- <screen>$ mkfs.lustre -reformat --fsname spfs --mdt --mgs /dev/sda
-</screen>
-</listitem><listitem>
- <para>If you have a separate MGS (that you do not want to reformat), then add the "writeconf" flag to mkfs.lustre on the MDT, run:</para>
- <screen>$ mkfs.lustre --reformat --writeconf -fsname spfs --mdt \ --mgs /dev/sda
-</screen>
-</listitem></orderedlist>
- <note><para>If you have a combined MGS/MDT, reformatting the MDT reformats the MGS as well, causing all configuration information to be lost; you can start building your new file system. Nothing needs to be done with old disks that will not be part of the new file system, just do not mount them.</para></note>
- </section>
- <section xml:id="dbdoclet.50438194_16954">
- <title>13.11 Reclaiming <anchor xml:id="dbdoclet.50438194_marker-1307251" xreflabel=""/>Reserved Disk Space</title>
- <para>All current Lustre installations run the ldiskfs file system internally on service nodes. By default, ldiskfs reserves 5% of the disk space for the root user. In order to reclaim this space, run the following command on your OSSs:</para>
- <screen>tune2fs [-m reserved_blocks_percent] [device]
-</screen>
- <para>You do not need to shut down Lustre before running this command or restart it afterwards.</para>
- </section>
- <section xml:id="dbdoclet.50438194_69998">
- <title>13.12 Replacing an Existing <anchor xml:id="dbdoclet.50438194_marker-1307278" xreflabel=""/>OST or MDS</title>
- <para>To copy the contents of an existing OST to a new OST (or an old MDS to a new MDS), use one of these methods:</para>
- <itemizedlist><listitem>
- <para> Connect the old OST disk and new OST disk to a single machine, mount both, and use rsync to copy all data between the OST file systems.</para>
+ <para>Where multiple NIDs are specified, comma-separation (for example, <literal>uml2,2@elan</literal>) means that the two NIDs refer to the same host, and that Lustre needs to choose the "best" one for communication. Colon-separation (for example, <literal>uml1:uml2</literal>) means that the two NIDs refer to two different hosts, and should be treated as failover locations (Lustre tries the first one, and if that fails, it tries the second one.)</para>
+ <note>
+ <para>If you have an MGS or MDT configured for failover, perform these steps:</para>
+ <orderedlist>
+ <listitem>
+ <para>On the OST, list the NIDs of all MGS nodes at mkfs time.</para>
+ <screen><para>OST# mkfs.lustre --fsname sunfs --ost --mgsnode=10.0.0.1</para><para> --mgsnode=10.0.0.2 /dev/{device}</para></screen>
+ </listitem>
+ <listitem>
+ <para>On the client, mount the file system.</para>
+ <para><screen>client# mount -t lustre 10.0.0.1:10.0.0.2:/sunfs /cfs/client/</screen></para>
</listitem>
-</itemizedlist>
- <para>For example:</para>
- <screen>mount -t ldiskfs /dev/old /mnt/ost_old
+ </orderedlist>
+ </note>
+ </section>
+ <section xml:id="dbdoclet.50438194_70905">
+ <title>13.10 Erasing a File System</title>
+ <para>If you want to erase a file system, run this command on your targets:</para>
+ <screen>$ "mkfs.lustre -reformat"
+</screen>
+ <para>If you are using a separate MGS and want to keep other file systems defined on that MGS, then set the <literal>writeconf</literal> flag on the MDT for that file system. The <literal>writeconf</literal> flag causes the configuration logs to be erased; they are regenerated the next time the servers start.</para>
+ <para>To set the <literal>writeconf</literal> flag on the MDT:</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Unmount all clients/servers using this file system, run:</emphasis></para>
+ <screen>$ umount /mnt/lustre
+</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Erase the file system and, presumably, replace it with another file system, run:</emphasis></para>
+ <screen>$ mkfs.lustre -reformat --fsname spfs --mdt --mgs /dev/sda
+</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">If you have a separate MGS (that you do not want to reformat), then add the "writeconf" flag to <literal>mkfs.lustre</literal> on the MDT, run:</emphasis></para>
+ <screen>$ mkfs.lustre --reformat --writeconf -fsname spfs --mdt \ --mgs /dev/sda
+</screen>
+ </listitem>
+ </orderedlist>
+ <note>
+ <para>If you have a combined MGS/MDT, reformatting the MDT reformats the MGS as well, causing all configuration information to be lost; you can start building your new file system. Nothing needs to be done with old disks that will not be part of the new file system, just do not mount them.</para>
+ </note>
+ </section>
+ <section xml:id="dbdoclet.50438194_16954">
+ <title>13.11 Reclaiming Reserved Disk Space</title>
+ <para>All current Lustre installations run the ldiskfs file system internally on service nodes. By default, ldiskfs reserves 5% of the disk space for the root user. In order to reclaim this space, run the following command on your OSSs:</para>
+ <screen>tune2fs [-m reserved_blocks_percent] [device]
+</screen>
+ <para>You do not need to shut down Lustre before running this command or restart it afterwards.</para>
+ </section>
+ <section xml:id="dbdoclet.50438194_69998">
+ <title>13.12 Replacing an Existing OST or MDS</title>
+ <para>To copy the contents of an existing OST to a new OST (or an old MDS to a new MDS), use one of these methods:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Connect the old OST disk and new OST disk to a single machine, mount both, and use rsync to copy all data between the OST file systems.</para>
+ </listitem>
+ </itemizedlist>
+ <para>For example:</para>
+ <screen>mount -t ldiskfs /dev/old /mnt/ost_old
mount -t ldiskfs /dev/new /mnt/ost_new
rsync -aSv /mnt/ost_old/ /mnt/ost_new
# note trailing slash on ost_old/
</screen>
- <itemizedlist><listitem>
- <para> If you are unable to connect both sets of disk to the same computer, use rsync to copy over the network using rsh (or ssh with -e ssh):</para>
- </listitem>
-</itemizedlist>
- <screen>rsync -aSvz /mnt/ost_old/ new_ost_node:/mnt/ost_new
-</screen>
- <itemizedlist><listitem>
- <para> Use the same procedure for the MDS, with one additional step:</para>
- </listitem>
-</itemizedlist>
- <screen>cd /mnt/mds_old; getfattr -R -e base64 -d . > /tmp/mdsea; \<copy all MDS file\
+ <itemizedlist>
+ <listitem>
+ <para> If you are unable to connect both sets of disk to the same computer, use <literal>rsync</literal> to copy over the network using <literal>rsh</literal> (or <literal>ssh</literal> with <literal>-e ssh</literal>):</para>
+ </listitem>
+ </itemizedlist>
+ <screen>rsync -aSvz /mnt/ost_old/ new_ost_node:/mnt/ost_new
+</screen>
+ <itemizedlist>
+ <listitem>
+ <para> Use the same procedure for the MDS, with one additional step:</para>
+ </listitem>
+ </itemizedlist>
+ <screen>cd /mnt/mds_old; getfattr -R -e base64 -d . > /tmp/mdsea; \<copy all MDS file\
s as above>; cd /mnt/mds_new; setfattr \--restore=/tmp/mdsea
</screen>
- </section>
- <section xml:id="dbdoclet.50438194_30872">
- <title>13.13 Identifying To Which Lustre File an OST Object Belongs</title>
- <para>Use this procedure to identify the file containing a given object on a given OST.</para>
- <orderedlist><listitem>
- <para>On the OST (as root), run debugfs to display the file identifier (FID) of the file associated with the object.</para>
- <para>For example, if the object is 34976 on /dev/lustre/ost_test2, the debug command is:</para>
- <screen># debugfs -c -R "stat /O/0/d$((34976 %32))/34976" /dev/lustre/ost_test2
-</screen>
- <para>The command output is:</para>
- <screen>debugfs 1.41.5.sun2 (23-Apr-2009)
+ </section>
+ <section xml:id="dbdoclet.50438194_30872">
+ <title>13.13 Identifying To Which Lustre File an OST Object Belongs</title>
+ <para>Use this procedure to identify the file containing a given object on a given OST.</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">On the OST (as root), run <literal>debugfs</literal> to display the file identifier (<literal>FID</literal>) of the file associated with the object.</emphasis></para>
+ <para>For example, if the object is <literal>34976</literal> on <literal>/dev/lustre/ost_test2</literal>, the debug command is:</para>
+ <screen># debugfs -c -R "stat /O/0/d$((34976 %32))/34976" /dev/lustre/ost_test2
+</screen>
+ <para>The command output is:</para>
+ <screen>debugfs 1.41.5.sun2 (23-Apr-2009)
/dev/lustre/ost_test2: catastrophic mode - not reading inode or group bitma\
ps
Inode: 352365 Type: regular Mode: 0666 Flags: 0x80000
(0-63):47968-48031
TOTAL: 64
</screen>
-</listitem><listitem>
- <para>Note the FID's EA and apply it to the osd_inode_id mapping.</para>
- <para>In this example, the FID's EA is:</para>
- <screen>e2001100000000002543c18700000000a0880000000000000000000000000000
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Note the FID's EA and apply it to the <literal>osd_inode_id</literal> mapping.</emphasis></para>
+ <para>In this example, the FID's EA is:</para>
+ <screen>e2001100000000002543c18700000000a0880000000000000000000000000000
struct osd_inode_id {
__u64 oii_ino; /* inode number */
__u32 oii_gen; /* inode generation */
__u32 oii_pad; /* alignment padding */
};
</screen>
- <para>After swapping, you get an inode number of 0x001100e2 and generation of 0.</para>
-</listitem><listitem>
- <para>On the MDT (as root), use debugfs to find the file associated with the inode.</para>
- <screen># debugfs -c -R "ncheck 0x001100e2" /dev/lustre/mdt_test
+ <para>After swapping, you get an inode number of <literal>0x001100e2</literal> and generation of <literal>0</literal>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">On the MDT (as root), use <literal>debugfs</literal> to find the file associated with the inode.</emphasis></para>
+ <screen># debugfs -c -R "ncheck 0x001100e2" /dev/lustre/mdt_test
</screen>
- <para>Here is the command output:</para>
- <screen>debugfs 1.41.5.sun2 (23-Apr-2009)
+
<para>Here is the command output:</para>
+ <screen>debugfs 1.41.5.sun2 (23-Apr-2009)
/dev/lustre/mdt_test: catastrophic mode - not reading inode or group bitmap\
s
Inode Pathname
1114338 /ROOT/brian-laptop-guest/clients/client11/~dmtmp/PWRPNT/ZD16.BMP
</screen>
-</listitem></orderedlist>
- <para>The command lists the inode and pathname associated with the object.</para>
- <note><para>Debugfs' ''ncheck'' is a brute-force search that may take a long time to complete.</para></note>
- <note><para>To find the Lustre file from a disk LBA, follow the steps listed in the document at this URL: <emphasis>http://smartmontools.sourceforge.net/badblockhowto.html. </emphasis> Then, follow the steps above to resolve the Lustre filename.</para></note>
-
+ </listitem>
+ </orderedlist>
+ <para>The command lists the inode and pathname associated with the object.</para>
+ <note>
+ <para><literal>Debugfs</literal>' ''ncheck'' is a brute-force search that may take a long time to complete.</para>
+ </note>
+ <note>
+ <para>To find the Lustre file from a disk LBA, follow the steps listed in the document at this URL: <emphasis><ulink>http://smartmontools.sourceforge.net/badblockhowto.html</ulink>. </emphasis> Then, follow the steps above to resolve the Lustre filename.</para>
+ </note>
</section>
</chapter>
<para>For example, for a 2 stripe file, stripe size = 1M, the bad OST is at index 0, and you have holes in the file at: [(2*N + 0)*1M, (2*N + 0)*1M + 1M - 1], N = { 0, 1, 2, ...}</para>
<para>If the file system cannot be mounted, currently there is no way that parses metadata directly from an MDS. If the bad OST does not start, options to mount the file system are to provide a loop device OST in its place or replace it with a newly-formatted OST. In that case, the missing objects are created and are read as zero-filled.</para>
</section>
- <section remap="h3">
- <title>26.3.4 <anchor xml:id="dbdoclet.50438198_69657" xreflabel=""/>Fixing a Bad LAST_ID on an OST</title>
+ <section xml:id="dbdoclet.50438198_69657">
+ <title>26.3.4 Fixing a Bad LAST_ID on an OST</title>
<para>Each OST contains a LAST_ID file, which holds the last object (pre-)created by the MDS <footnote><para>The contents of the LAST_ID file must be accurate regarding the actual objects that exist on the OST.</para></footnote>. The MDT contains a lov_objid file, with values that represent the last object the MDS has allocated to a file.</para>
<para>During normal operation, the MDT keeps some pre-created (but unallocated) objects on the OST, and the relationship between LAST_ID and lov_objid should be LAST_ID <= lov_objid. Any difference in the file values results in objects being created on the OST when it next connects to the MDS. These objects are never actually allocated to a file, since they are of 0 length (empty), but they do no harm. Creating empty objects enables the OST to catch up to the MDS, so normal operations resume.</para>
<para>However, in the case where lov_objid < LAST_ID, bad things can happen as the MDS is not aware of objects that have already been allocated on the OST, and it reallocates them to new files, overwriting their existing contents.</para>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='lustretuning'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. -->
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretuning">
<info>
- <title xml:id='lustretuning.title'>Lustre Tuning</title>
+ <title xml:id="lustretuning.title">Lustre Tuning</title>
</info>
<para>This chapter contains information about tuning Lustre for better performance and includes the following sections:</para>
-
- <itemizedlist><listitem>
+ <itemizedlist>
+ <listitem>
<para><xref linkend="dbdoclet.50438272_55226"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438272_73839"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438272_25884"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438272_80545"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438272_45406"/></para>
</listitem>
-
-</itemizedlist>
-
- <note><para>Many options in Lustre are set by means of kernel module parameters. These parameters are contained in the modprobe.conf file.</para></note>
-
- <section xml:id="dbdoclet.50438272_55226">
- <title>25.1 Optimizing the Number of Service Threads</title>
- <para>An OSS can have a minimum of 2 service threads and a maximum of 512 service threads. The number of service threads is a function of how much RAM and how many CPUs are on each OSS node (1 thread / 128MB * num_cpus). If the load on the OSS node is high, new service threads will be started in order to process more requests concurrently, up to 4x the initial number of threads (subject to the maximum of 512). For a 2GB 2-CPU system, the default thread count is 32 and the maximum thread count is 128.</para>
- <para>Increasing the size of the thread pool may help when:</para>
- <itemizedlist><listitem>
- <para> Several OSTs are exported from a single OSS</para>
- </listitem>
-
-<listitem>
- <para> Back-end storage is running synchronously</para>
- </listitem>
-
-<listitem>
- <para> I/O completions take excessive time due to slow storage</para>
- </listitem>
-
-</itemizedlist>
- <para>Decreasing the size of the thread pool may help if:</para>
- <itemizedlist><listitem>
- <para> Clients are overwhelming the storage capacity</para>
- </listitem>
-
-<listitem>
- <para> There are lots of "slow I/O" or similar messages</para>
- </listitem>
-
-</itemizedlist>
- <para>Increasing the number of I/O threads allows the kernel and storage to aggregate many writes together for more efficient disk I/O. The OSS thread pool is shared--each thread allocates approximately 1.5 MB (maximum RPC size + 0.5 MB) for internal I/O buffers.</para>
- <para>It is very important to consider memory consumption when increasing the thread pool size. Drives are only able to sustain a certain amount of parallel I/O activity before performance is degraded, due to the high number of seeks and the OST threads just waiting for I/O. In this situation, it may be advisable to decrease the load by decreasing the number of OST threads.</para>
- <para>Determining the optimum number of OST threads is a process of trial and error, and varies for each particular configuration. Variables include the number of OSTs on each OSS, number and speed of disks, RAID configuration, and available RAM. You may want to start with a number of OST threads equal to the number of actual disk spindles on the node. If you use RAID, subtract any dead spindles not used for actual data (e.g., 1 of N of spindles for RAID5, 2 of N spindles for RAID6), and monitor the performance of clients during usual workloads. If performance is degraded, increase the thread count and see how that works until performance is degraded again or you reach satisfactory performance.</para>
- <note><para>If there are too many threads, the latency for individual I/O requests can become very high and should be avoided. Set the desired maximum thread count permanently using the method described above.</para></note>
- <section remap="h3">
- <title>25.1.1 <anchor xml:id="dbdoclet.50438272_60005" xreflabel=""/>Specifying the OSS Service <anchor xml:id="dbdoclet.50438272_marker-1294858" xreflabel=""/>Thread Count</title>
- <para>The oss_num_threads parameter enables the number of OST service threads to be specified at module load time on the OSS nodes:</para>
- <screen>options ost oss_num_threads={N}</screen>
- <para>After startup, the minimum and maximum number of OSS thread counts can be set via the {service}.thread_{min,max,started} tunable. To change the tunable at runtime, run:</para>
- <para>lctl {get,set}_param {service}.thread_{min,max,started}</para>
- <para>For details, see <link xl:href="LustreProc.html#50438271_87260">Setting MDS and OSS Thread Counts</link>.</para>
- </section>
- <section remap="h3">
- <title>25.1.2 Specifying the MDS Service <anchor xml:id="dbdoclet.50438272_marker-1293755" xreflabel=""/>Thread Count</title>
- <para>The mds_num_threads parameter enables the number of MDS service threads to be specified at module load time on the MDS node:</para>
- <screen>options mds mds_num_threads={N}</screen>
- <para>After startup, the minimum and maximum number of MDS thread counts can be set via the {service}.thread_{min,max,started} tunable. To change the tunable at runtime, run:</para>
- <para>lctl {get,set}_param {service}.thread_{min,max,started}</para>
- <para>For details, see <link xl:href="LustreProc.html#50438271_87260">Setting MDS and OSS Thread Counts</link>.</para>
- <para>At this time, no testing has been done to determine the optimal number of MDS threads. The default value varies, based on server size, up to a maximum of 32. The maximum number of threads (MDS_MAX_THREADS) is 512.</para>
- <note><para>The OSS and MDS automatically start new service threads dynamically, in response to server load within a factor of 4. The default value is calculated the same way as before. Setting the _mu_threads module parameter disables automatic thread creation behavior.</para></note>
- </section>
+ </itemizedlist>
+ <note>
+ <para>Many options in Lustre are set by means of kernel module parameters. These parameters are contained in the <literal>modprobe.conf</literal> file.</para>
+ </note>
+ <section xml:id="dbdoclet.50438272_55226">
+ <title>25.1 Optimizing the Number of Service Threads</title>
+ <para>An OSS can have a minimum of 2 service threads and a maximum of 512 service threads. The number of service threads is a function of how much RAM and how many CPUs are on each OSS node (1 thread / 128MB * num_cpus). If the load on the OSS node is high, new service threads will be started in order to process more requests concurrently, up to 4x the initial number of threads (subject to the maximum of 512). For a 2GB 2-CPU system, the default thread count is 32 and the maximum thread count is 128.</para>
+ <para>Increasing the size of the thread pool may help when:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Several OSTs are exported from a single OSS</para>
+ </listitem>
+ <listitem>
+ <para>Back-end storage is running synchronously</para>
+ </listitem>
+ <listitem>
+ <para>I/O completions take excessive time due to slow storage</para>
+ </listitem>
+ </itemizedlist>
+ <para>Decreasing the size of the thread pool may help if:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Clients are overwhelming the storage capacity</para>
+ </listitem>
+ <listitem>
+ <para>There are lots of "slow I/O" or similar messages</para>
+ </listitem>
+ </itemizedlist>
+ <para>Increasing the number of I/O threads allows the kernel and storage to aggregate many writes together for more efficient disk I/O. The OSS thread pool is shared--each thread allocates approximately 1.5 MB (maximum RPC size + 0.5 MB) for internal I/O buffers.</para>
+ <para>It is very important to consider memory consumption when increasing the thread pool size. Drives are only able to sustain a certain amount of parallel I/O activity before performance is degraded, due to the high number of seeks and the OST threads just waiting for I/O. In this situation, it may be advisable to decrease the load by decreasing the number of OST threads.</para>
+ <para>Determining the optimum number of OST threads is a process of trial and error, and varies for each particular configuration. Variables include the number of OSTs on each OSS, number and speed of disks, RAID configuration, and available RAM. You may want to start with a number of OST threads equal to the number of actual disk spindles on the node. If you use RAID, subtract any dead spindles not used for actual data (e.g., 1 of N of spindles for RAID5, 2 of N spindles for RAID6), and monitor the performance of clients during usual workloads. If performance is degraded, increase the thread count and see how that works until performance is degraded again or you reach satisfactory performance.</para>
+ <note>
+ <para>If there are too many threads, the latency for individual I/O requests can become very high and should be avoided. Set the desired maximum thread count permanently using the method described above.</para>
+ </note>
+ <section remap="h3">
+ <title>25.1.1 Specifying the OSS Service Thread Count</title>
+ <para>The <literal>oss_num_threads</literal> parameter enables the number of OST service threads to be specified at module load time on the OSS nodes:</para>
+ <screen>options ost oss_num_threads={N}</screen>
+ <para>After startup, the minimum and maximum number of OSS thread counts can be set via the <literal>{service}.thread_{min,max,started}</literal> tunable. To change the tunable at runtime, run:</para>
+ <para><screen>lctl {get,set}_param {service}.thread_{min,max,started}</screen></para>
+ <para>For details, see <xref linkend="dbdoclet.50438271_87260">Setting MDS and OSS Thread Counts</xref>.</para>
</section>
- <section xml:id="dbdoclet.50438272_73839">
- <title>25.2 Tuning LNET Parameters</title>
- <para>This section describes LNET tunables. that may be necessary on some systems to improve performance. To test the performance of your Lustre network, see <link xl:href="LNETSelfTest.html#50438223_71556">Chapter 23</link>: <link xl:href="LNETSelfTest.html#50438223_21832">Testing Lustre Network Performance (LNET Self-Test)</link>.</para>
- <section remap="h3">
- <title>25.2.1 Transmit and Receive Buffer Size</title>
- <para>The kernel allocates buffers for sending and receiving messages on a network.</para>
- <para>ksocklnd has separate parameters for the transmit and receive buffers.</para>
- <screen>options ksocklnd tx_buffer_size=0 rx_buffer_size=0
-</screen>
- <para>If these parameters are left at the default value (0), the system automatically tunes the transmit and receive buffer size. In almost every case, this default produces the best performance. Do not attempt to tune these parameters unless you are a network expert.</para>
- </section>
- <section remap="h3">
- <title>25.2.2 Hardware Interrupts (enable_irq_affinity)</title>
- <para>The hardware interrupts that are generated by network adapters may be handled by any CPU in the system. In some cases, we would like network traffic to remain local to a single CPU to help keep the processor cache warm and minimize the impact of context switches. This is helpful when an SMP system has more than one network interface and ideal when the number of interfaces equals the number of CPUs. To enable the enable_irq_affinity parameter, enter:</para>
- <screen>options ksocklnd enable_irq_affinity=1
-</screen>
- <para>In other cases, if you have an SMP platform with a single fast interface such as 10Gb Ethernet and more than two CPUs, you may see performance improve by turning this parameter off.</para>
- <screen>options ksocklnd enable_irq_affinity=0
-</screen>
- <para>By default, this parameter is off. As always, you should test the performance to compare the impact of changing this parameter.</para>
- </section>
+ <section remap="h3">
+ <title>25.1.2 Specifying the MDS Service Thread Count</title>
+ <para>The <literal>mds_num_threads</literal> parameter enables the number of MDS service threads to be specified at module load time on the MDS node:</para>
+ <screen>options mds mds_num_threads={N}</screen>
+ <para>After startup, the minimum and maximum number of MDS thread counts can be set via the <literal>{service}.thread_{min,max,started}</literal> tunable. To change the tunable at runtime, run:</para>
+ <para><screen>lctl {get,set}_param {service}.thread_{min,max,started}</screen></para>
+ <para>For details, see <xref linkend="dbdoclet.50438271_87260">Setting MDS and OSS Thread Counts</xref>.</para>
+ <para>At this time, no testing has been done to determine the optimal number of MDS threads. The default value varies, based on server size, up to a maximum of 32. The maximum number of threads (<literal>MDS_MAX_THREADS</literal>) is 512.</para>
+ <note>
+ <para>The OSS and MDS automatically start new service threads dynamically, in response to server load within a factor of 4. The default value is calculated the same way as before. Setting the <literal>_mu_threads</literal> module parameter disables automatic thread creation behavior.</para>
+ </note>
</section>
- <section xml:id="dbdoclet.50438272_25884">
- <title>25.3 Lockless <anchor xml:id="dbdoclet.50438272_marker-1291703" xreflabel=""/>I/O Tunables</title>
- <para>The lockless I/O tunable feature allows servers to ask clients to do lockless I/O (liblustre-style where the server does the locking) on contended files.</para>
- <para>The lockless I/O patch introduces these tunables:</para>
- <itemizedlist><listitem>
- <para> OST-side:</para>
- </listitem>
-
-</itemizedlist>
- <screen>/proc/fs/lustre/ldlm/namespaces/filter-lustre-*
+ </section>
+ <section xml:id="dbdoclet.50438272_73839">
+ <title>25.2 Tuning LNET Parameters</title>
+ <para>This section describes LNET tunables. that may be necessary on some systems to improve performance. To test the performance of your Lustre network, see <link xl:href="LNETSelfTest.html#50438223_71556">Chapter 23</link>: <link xl:href="LNETSelfTest.html#50438223_21832">Testing Lustre Network Performance (LNET Self-Test)</link>.</para>
+ <section remap="h3">
+ <title>25.2.1 Transmit and Receive Buffer Size</title>
+ <para>The kernel allocates buffers for sending and receiving messages on a network.</para>
+ <para><literal>ksocklnd</literal> has separate parameters for the transmit and receive buffers.</para>
+ <screen>options ksocklnd tx_buffer_size=0 rx_buffer_size=0
</screen>
- <para>contended_locks - If the number of lock conflicts in the scan of granted and waiting queues at contended_locks is exceeded, the resource is considered to be contended.</para>
- <para>contention_seconds - The resource keeps itself in a contended state as set in the parameter.</para>
- <para>max_nolock_bytes - Server-side locking set only for requests less than the blocks set in the max_nolock_bytes parameter. If this tunable is set to zero (0), it disables server-side locking for read/write requests.</para>
- <itemizedlist><listitem>
- <para> Client-side:</para>
- </listitem>
-
-</itemizedlist>
- <screen>/proc/fs/lustre/llite/lustre-*
-</screen>
- <para>contention_seconds - llite inode remembers its contended state for the time specified in this parameter.</para>
- <itemizedlist><listitem>
- <para> Client-side statistics:</para>
- </listitem>
-
-</itemizedlist>
- <para>The /proc/fs/lustre/llite/lustre-*/stats file has new rows for lockless I/O statistics.</para>
- <para>lockless_read_bytes and lockless_write_bytes - To count the total bytes read or written, the client makes its own decisions based on the request size. The client does not communicate with the server if the request size is smaller than the min_nolock_size, without acquiring locks by the client.</para>
- </section>
- <section xml:id="dbdoclet.50438272_80545">
- <title>25.4 Improving Lustre <anchor xml:id="dbdoclet.50438272_marker-1295851" xreflabel=""/>Performance When Working with Small Files</title>
- <para>A Lustre environment where an application writes small file chunks from many clients to a single file will result in bad I/O performance. To improve Lustre'™s performance with small files:</para>
- <itemizedlist><listitem>
- <para> Have the application aggregate writes some amount before submitting them to Lustre. By default, Lustre enforces POSIX coherency semantics, so it results in lock ping-pong between client nodes if they are all writing to the same file at one time.</para>
- </listitem>
-
-<listitem>
- <para> Have the application do 4kB O_DIRECT sized I/O to the file and disable locking on the output file. This avoids partial-page IO submissions and, by disabling locking, you avoid contention between clients.</para>
- </listitem>
-
-<listitem>
- <para> Have the application write contiguous data.</para>
- </listitem>
-
-<listitem>
- <para> Add more disks or use SSD disks for the OSTs. This dramatically improves the IOPS rate. Consider creating larger OSTs rather than many smaller OSTs due to less overhead (journal, connections, etc).</para>
- </listitem>
-
-<listitem>
- <para> Use RAID-1+0 OSTs instead of RAID-5/6. There is RAID parity overhead for writing small chunks of data to disk.</para>
- </listitem>
-
-</itemizedlist>
+ <para>If these parameters are left at the default value (0), the system automatically tunes the transmit and receive buffer size. In almost every case, this default produces the best performance. Do not attempt to tune these parameters unless you are a network expert.</para>
</section>
- <section xml:id="dbdoclet.50438272_45406">
- <title>25.5 Understanding Why Write Performance Is Better Than Read Performance</title>
- <para>Typically, the performance of write operations on a Lustre cluster is better than read operations. When doing writes, all clients are sending write RPCs asynchronously. The RPCs are allocated, and written to disk in the order they arrive. In many cases, this allows the back-end storage to aggregate writes efficiently.</para>
- <para>In the case of read operations, the reads from clients may come in a different order and need a lot of seeking to get read from the disk. This noticeably hampers the read throughput.</para>
- <para>Currently, there is no readahead on the OSTs themselves, though the clients do readahead. If there are lots of clients doing reads it would not be possible to do any readahead in any case because of memory consumption (consider that even a single RPC (1 MB) readahead for 1000 clients would consume 1 GB of RAM).</para>
- <para>For file systems that use socklnd (TCP, Ethernet) as interconnect, there is also additional CPU overhead because the client cannot receive data without copying it from the network buffers. In the write case, the client CAN send data without the additional data copy. This means that the client is more likely to become CPU-bound during reads than writes.</para>
+ <section remap="h3">
+ <title>25.2.2 Hardware Interrupts (<literal>enable_irq_affinity</literal>)</title>
+ <para>The hardware interrupts that are generated by network adapters may be handled by any CPU in the system. In some cases, we would like network traffic to remain local to a single CPU to help keep the processor cache warm and minimize the impact of context switches. This is helpful when an SMP system has more than one network interface and ideal when the number of interfaces equals the number of CPUs. To enable the <literal>enable_irq_affinity</literal> parameter, enter:</para>
+ <screen>options ksocklnd enable_irq_affinity=1</screen>
+ <para>In other cases, if you have an SMP platform with a single fast interface such as 10Gb Ethernet and more than two CPUs, you may see performance improve by turning this parameter off.</para>
+ <screen>options ksocklnd enable_irq_affinity=0</screen>
+ <para>By default, this parameter is off. As always, you should test the performance to compare the impact of changing this parameter.</para>
</section>
+ </section>
+ <section xml:id="dbdoclet.50438272_25884">
+ <title>25.3 Lockless I/O Tunables</title>
+ <para>The lockless I/O tunable feature allows servers to ask clients to do lockless I/O (liblustre-style where the server does the locking) on contended files.</para>
+ <para>The lockless I/O patch introduces these tunables:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">OST-side:</emphasis></para>
+ <screen>/proc/fs/lustre/ldlm/namespaces/filter-lustre-*
+</screen>
+ <para><literal>contended_locks</literal> - If the number of lock conflicts in the scan of granted and waiting queues at contended_locks is exceeded, the resource is considered to be contended.</para>
+ <para><literal>contention_seconds</literal> - The resource keeps itself in a contended state as set in the parameter.</para>
+ <para><literal>max_nolock_bytes</literal> - Server-side locking set only for requests less than the blocks set in the <literal>max_nolock_bytes</literal> parameter. If this tunable is set to zero (0), it disables server-side locking for read/write requests.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Client-side:</emphasis></para>
+ <screen>/proc/fs/lustre/llite/lustre-*</screen>
+ <para><literal>contention_seconds</literal> - <literal>llite</literal> inode remembers its contended state for the time specified in this parameter.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Client-side statistics:</emphasis></para>
+ <para>The <literal>/proc/fs/lustre/llite/lustre-*/stats</literal> file has new rows for lockless I/O statistics.</para>
+ <para><literal>lockless_read_bytes</literal> and <literal>lockless_write_bytes</literal> - To count the total bytes read or written, the client makes its own decisions based on the request size. The client does not communicate with the server if the request size is smaller than the <literal>min_nolock_size</literal>, without acquiring locks by the client.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="dbdoclet.50438272_80545">
+ <title>25.4 Improving Lustre Performance When Working with Small Files</title>
+ <para>A Lustre environment where an application writes small file chunks from many clients to a single file will result in bad I/O performance. To improve Lustre'™s performance with small files:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Have the application aggregate writes some amount before submitting them to Lustre. By default, Lustre enforces POSIX coherency semantics, so it results in lock ping-pong between client nodes if they are all writing to the same file at one time.</para>
+ </listitem>
+ <listitem>
+ <para>Have the application do 4kB <literal>O_DIRECT</literal> sized I/O to the file and disable locking on the output file. This avoids partial-page IO submissions and, by disabling locking, you avoid contention between clients.</para>
+ </listitem>
+ <listitem>
+ <para>Have the application write contiguous data.</para>
+ </listitem>
+ <listitem>
+ <para>Add more disks or use SSD disks for the OSTs. This dramatically improves the IOPS rate. Consider creating larger OSTs rather than many smaller OSTs due to less overhead (journal, connections, etc).</para>
+ </listitem>
+ <listitem>
+ <para>Use RAID-1+0 OSTs instead of RAID-5/6. There is RAID parity overhead for writing small chunks of data to disk.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="dbdoclet.50438272_45406">
+ <title>25.5 Understanding Why Write Performance Is Better Than Read Performance</title>
+ <para>Typically, the performance of write operations on a Lustre cluster is better than read operations. When doing writes, all clients are sending write RPCs asynchronously. The RPCs are allocated, and written to disk in the order they arrive. In many cases, this allows the back-end storage to aggregate writes efficiently.</para>
+ <para>In the case of read operations, the reads from clients may come in a different order and need a lot of seeking to get read from the disk. This noticeably hampers the read throughput.</para>
+ <para>Currently, there is no readahead on the OSTs themselves, though the clients do readahead. If there are lots of clients doing reads it would not be possible to do any readahead in any case because of memory consumption (consider that even a single RPC (1 MB) readahead for 1000 clients would consume 1 GB of RAM).</para>
+ <para>For file systems that use socklnd (TCP, Ethernet) as interconnect, there is also additional CPU overhead because the client cannot receive data without copying it from the network buffers. In the write case, the client CAN send data without the additional data copy. This means that the client is more likely to become CPU-bound during reads than writes.</para>
+ </section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='managingfailover'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. -->
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingfailover">
<info>
- <title xml:id='managingfailover.title'>Managing Failover</title>
+ <title xml:id="managingfailover.title">Managing Failover</title>
</info>
<para>This chapter describes failover in a Lustre system and includes the following sections:</para>
-
- <itemizedlist><listitem>
-
<para><xref linkend="dbdoclet.50438213_13563"/></para>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438213_13563"/></para>
</listitem>
-</itemizedlist>
-
- <note><para>For information about high availability(HA) management software, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Red_Hat_Cluster_Manager_with_Lustre">Using Red Hat Cluster Manager with Lustre</link> or the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre">Using Pacemaker with Lustre</link>.</para></note>
-
- <section xml:id="dbdoclet.50438213_13563">
-
<title>20.1 Lustre Failover and <anchor xml:id="dbdoclet.50438213_marker-1301522" xreflabel=""/>Multiple-Mount Protection</title>
-
<para>The failover functionality in Lustre is implemented with the multiple-mount protection (MMP) feature, which protects the file system from being mounted simultaneously to more than one node. This feature is important in a shared storage environment (for example, when a failover pair of OSTs share a partition).</para>
- <para>Lustre's backend file system, ldiskfs, supports the MMP mechanism. A block in the file system is updated by a kmmpd daemon at one second intervals, and a sequence number is written in this block. If the file system is cleanly unmounted, then a special "clean" sequence is written to this block. When mounting the file system, ldiskfs checks if the MMP block has a clean sequence or not.</para>
- <para>Even if the MMP block has a clean sequence, ldiskfs waits for some interval to guard against the following situations:</para>
- <itemizedlist><listitem>
- <para> If I/O traffic is heavy, it may take longer for the MMP block to be updated.</para>
- </listitem>
-<listitem>
- <para> If another node is trying to mount the same file system, a "race" condition may occur.</para>
- </listitem>
-</itemizedlist>
- <para>With MMP enabled, mounting a clean file system takes at least 10 seconds. If the file system was not cleanly unmounted, then the file system mount may require additional time.</para>
-
- <note><para>The MMP feature is only supported on Linux kernel versions >= 2.6.9.</para></note>
-
- <section remap="h3">
- <title>20.1.1 Working with Multiple-Mount Protection</title>
- <para>On a new Lustre file system, MMP is automatically enabled by mkfs.lustre at format time if failover is being used and the kernel and e2fsprogs version support it. On an existing file system, a Lustre administrator can manually enable MMP when the file system is unmounted.</para>
- <para>Use the following commands to determine whether MMP is running in Lustre and to enable or disable the MMP feature.</para>
- <para>To determine if MMP is enabled, run:</para>
- <screen>dumpe2fs -h <device>|grep mmp
-</screen>
-
<para>Here is a sample command:</para>
- <screen>dumpe2fs -h /dev/sdc | grep mmp
+ </itemizedlist>
+ <note>
+ <para>For information about high availability(HA) management software, see the Lustre wiki topic <ulink xl:href="http://wiki.lustre.org/index.php/Using_Red_Hat_Cluster_Manager_with_Lustre">Using Red Hat Cluster Manager with Lustre</ulink> or the Lustre wiki topic <ulink xl:href="http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre">LuUsing Pacemaker with stre</ulink>.</para>
+ </note>
+ <section xml:id="dbdoclet.50438213_13563">
+ <title>20.1 Lustre Failover and <anchor xml:id="dbdoclet.50438213_marker-1301522" xreflabel=""/>Multiple-Mount Protection</title>
+ <para>The failover functionality in Lustre is implemented with the multiple-mount protection (MMP) feature, which protects the file system from being mounted simultaneously to more than one node. This feature is important in a shared storage environment (for example, when a failover pair of OSTs share a partition).</para>
+ <para>Lustre's backend file system, <literal>ldiskfs</literal>, supports the MMP mechanism. A block in the file system is updated by a <literal>kmmpd</literal> daemon at one second intervals, and a sequence number is written in this block. If the file system is cleanly unmounted, then a special "clean" sequence is written to this block. When mounting the file system, <literal>ldiskfs</literal> checks if the MMP block has a clean sequence or not.</para>
+ <para>Even if the MMP block has a clean sequence, <literal>ldiskfs</literal> waits for some interval to guard against the following situations:</para>
+ <itemizedlist>
+ <listitem>
+ <para> If I/O traffic is heavy, it may take longer for the MMP block to be updated.</para>
+ </listitem>
+ <listitem>
+ <para> If another node is trying to mount the same file system, a "race" condition may occur.</para>
+ </listitem>
+ </itemizedlist>
+ <para>With MMP enabled, mounting a clean file system takes at least 10 seconds. If the file system was not cleanly unmounted, then the file system mount may require additional time.</para>
+ <note>
+ <para>The MMP feature is only supported on Linux kernel versions >= 2.6.9.</para>
+ </note>
+ <section remap="h3">
+ <title>20.1.1 Working with Multiple-Mount Protection</title>
+ <para>On a new Lustre file system, MMP is automatically enabled by mkfs.lustre at format time if failover is being used and the kernel and <literal>e2fsprogs</literal> version support it. On an existing file system, a Lustre administrator can manually enable MMP when the file system is unmounted.</para>
+ <para>Use the following commands to determine whether MMP is running in Lustre and to enable or disable the MMP feature.</para>
+ <para>To determine if MMP is enabled, run:</para>
+ <screen>dumpe2fs -h <device>|grep mmp</screen>
+ <para>Here is a sample command:</para>
+ <screen>dumpe2fs -h /dev/sdc | grep mmp
Filesystem features: has_journal ext_attr resize_inode dir_index
-filetype extent mmp sparse_super large_file uninit_bg
-</screen>
- <para>To manually disable MMP, run:</para>
- <screen>tune2fs -O ^mmp <device>
-</screen>
- <para>To manually enable MMP, run:</para>
- <screen>tune2fs -O mmp <device>
-</screen>
- <para>When MMP is enabled, if ldiskfs detects multiple mount attempts after the file system is mounted, it blocks these later mount attempts and reports the time when the MMP block was last updated, the node name, and the device name of the node where the file system is currently mounted.</para>
- </section>
+filetype extent mmp sparse_super large_file uninit_bg</screen>
+ <para>To manually disable MMP, run:</para>
+ <screen>tune2fs -O ^mmp <device> </screen>
+ <para>To manually enable MMP, run:</para>
+ <screen>tune2fs -O mmp <device></screen>
+ <para>When MMP is enabled, if <literal>ldiskfs</literal> detects multiple mount attempts after the file system is mounted, it blocks these later mount attempts and reports the time when the MMP block was last updated, the node name, and the device name of the node where the file system is currently mounted.</para>
+ </section>
</section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='managingfilesystemio'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingfilesystemio">
<info>
- <title xml:id='managingfilesystemio.title'>Managing the File System and I/O</title>
+ <title xml:id="managingfilesystemio.title">Managing the File System and I/O</title>
</info>
<para>This chapter describes file striping and I/O options, and includes the following sections:</para>
-
- <itemizedlist><listitem>
-
<para><xref linkend="dbdoclet.50438211_17536"/></para>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438211_17536"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438211_75549"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438211_11204"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438211_80295"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438211_61024"/></para>
</listitem>
-</itemizedlist>
-
- <section xml:id="dbdoclet.50438211_17536">
- <title>19.1 Handling <anchor xml:id="dbdoclet.50438211_marker-1295529" xreflabel=""/>Full OSTs</title>
- <para>Sometimes a Lustre file system becomes unbalanced, often due to incorrectly-specified stripe settings, or when very large files are created that are not striped over all of the OSTs. If an OST is full and an attempt is made to write more information to the file system, an error occurs. The procedures below describe how to handle a full OST.</para>
- <para>The MDS will normally handle space balancing automatically at file creation time, and this procedure is normally not needed, but may be desirable in certain circumstances (e.g. when creating very large files that would consume more than the total free space of the full OSTs).</para>
- <section remap="h3">
- <title>19.1.1 <anchor xml:id="dbdoclet.50438211_54434" xreflabel=""/>Checking OST Space Usage</title>
- <para>The example below shows an unbalanced file system:</para>
- <screen>root@LustreClient01 ~]# lfs df -h
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438211_17536">
+ <title>19.1 Handling Full OSTs</title>
+ <para>Sometimes a Lustre file system becomes unbalanced, often due to incorrectly-specified stripe settings, or when very large files are created that are not striped over all of the OSTs. If an OST is full and an attempt is made to write more information to the file system, an error occurs. The procedures below describe how to handle a full OST.</para>
+ <para>The MDS will normally handle space balancing automatically at file creation time, and this procedure is normally not needed, but may be desirable in certain circumstances (e.g. when creating very large files that would consume more than the total free space of the full OSTs).</para>
+ <section remap="h3">
+ <title>19.1.1 Checking OST Space Usage</title>
+ <para>The example below shows an unbalanced file system:</para>
+ <screen>root@LustreClient01 ~]# lfs df -h
UUID bytes Used Available \
Use% Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G \
36% /mnt/lustre[OST:5]
filesystem summary: 11.8G 5.4G 5.8G \
-45% /mnt/lustre
-</screen>
- <para>In this case, OST:2 is almost full and when an attempt is made to write additional information to the file system (even with uniform striping over all the OSTs), the write command fails as follows:</para>
- <screen>[root@LustreClient01 ~]# lfs setstripe /mnt/lustre 4M 0 -1
+45% /mnt/lustre</screen>
+ <para>In this case, OST:2 is almost full and when an attempt is made to write additional information to the file system (even with uniform striping over all the OSTs), the write command fails as follows:</para>
+ <screen>[root@LustreClient01 ~]# lfs setstripe /mnt/lustre 4M 0 -1
[root@LustreClient01 ~]# dd if=/dev/zero of=/mnt/lustre/test_3 \ bs=10M cou\
nt=100
dd: writing `/mnt/lustre/test_3': No space left on device
98+0 records in
97+0 records out
-1017192448 bytes (1.0 GB) copied, 23.2411 seconds, 43.8 MB/s
-</screen>
- </section>
- <section remap="h3">
- <title>19.1.2 Taking a Full OST Offline</title>
- <para>To avoid running out of space in the file system, if the OST usage is imbalanced and one or more OSTs are close to being full while there are others that have a lot of space, the full OSTs may optionally be deactivated at the MDS to prevent the MDS from allocating new objects there.</para>
- <orderedlist><listitem>
- <para>Log into the MDS server:</para>
- <screen>[root@LustreClient01 ~]# ssh root@192.168.0.10
+1017192448 bytes (1.0 GB) copied, 23.2411 seconds, 43.8 MB/s</screen>
+ </section>
+ <section remap="h3">
+ <title>19.1.2 Taking a Full OST Offline</title>
+ <para>To avoid running out of space in the file system, if the OST usage is imbalanced and one or more OSTs are close to being full while there are others that have a lot of space, the full OSTs may optionally be deactivated at the MDS to prevent the MDS from allocating new objects there.</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Log into the MDS server:</emphasis></para>
+ <screen>[root@LustreClient01 ~]# ssh root@192.168.0.10
root@192.168.0.10's password:
-Last login: Wed Nov 26 13:35:12 2008 from 192.168.0.6
-</screen>
- </listitem><listitem>
- <para>Use the lctl dl command to show the status of all file system components:</para>
- <screen>[root@mds ~]# lctl dl
+Last login: Wed Nov 26 13:35:12 2008 from 192.168.0.6</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Use the <literal>lctl dl</literal> command to show the status of all file system components:</emphasis></para>
+ <screen>[root@mds ~]# lctl dl
0 UP mgs MGS MGS 9
1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
2 UP mdt MDS MDS_uuid 3
7 UP osc lustre-OST0002-osc lustre-mdtlov_UUID 5
8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
-10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5
-</screen>
- </listitem><listitem>
- <para>Use lctl deactivate to take the full OST offline:</para>
- <screen>[root@mds ~]# lctl --device 7 deactivate
-</screen>
- </listitem><listitem>
- <para>Display the status of the file system components:</para>
- <screen>[root@mds ~]# lctl dl
+10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Use <literal>lctl</literal> deactivate to take the full OST offline:</emphasis></para>
+ <screen>[root@mds ~]# lctl --device 7 deactivate</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Display the status of the file system components:</emphasis></para>
+ <screen>[root@mds ~]# lctl dl
0 UP mgs MGS MGS 9
1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
2 UP mdt MDS MDS_uuid 3
7 IN osc lustre-OST0002-osc lustre-mdtlov_UUID 5
8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
-10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5
-</screen>
- </listitem></orderedlist>
- <para>The device list shows that OST0002 is now inactive. When new files are created in the file system, they will only use the remaining active OSTs. Either manual space rebalancing can be done by migrating data to other OSTs, as shown in the next section, or normal file deletion and creation can be allowed to passively rebalance the space usage.</para>
- </section>
- <section remap="h3">
- <title>19.1.3 Migrating Data within a File System</title>
- <para>As stripes cannot be moved within the file system, data must be migrated manually by copying and renaming the file, removing the original file, and renaming the new file with the original file name. The simplest way to do this is to use the lfs_migrate command (see <link xl:href="UserUtilities.html#50438206_42260">lfs_migrate</link>). However, the steps for migrating a file by hand are also shown here for reference.</para>
- <orderedlist><listitem>
- <para>Identify the file(s) to be moved.</para>
- <para>In the example below, output from the getstripe command indicates that the file test_2 is located entirely on OST2:</para>
- <screen>[client]# lfs getstripe /mnt/lustre/test_2
+10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5</screen>
+ </listitem>
+ </orderedlist>
+ <para>The device list shows that OST0002 is now inactive. When new files are created in the file system, they will only use the remaining active OSTs. Either manual space rebalancing can be done by migrating data to other OSTs, as shown in the next section, or normal file deletion and creation can be allowed to passively rebalance the space usage.</para>
+ </section>
+ <section remap="h3">
+ <title>19.1.3 Migrating Data within a File System</title>
+ <para>As stripes cannot be moved within the file system, data must be migrated manually by copying and renaming the file, removing the original file, and renaming the new file with the original file name. The simplest way to do this is to use the <literal>lfs_migrate</literal> command (see <xref linkend="dbdoclet.50438206_42260">lfs_migrate</xref>). However, the steps for migrating a file by hand are also shown here for reference.</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Identify the file(s) to be moved.</emphasis></para>
+ <para>In the example below, output from the <literal>getstripe</literal> command indicates that the file <literal>test_2</literal> is located entirely on OST2:</para>
+ <screen>[client]# lfs getstripe /mnt/lustre/test_2
/mnt/lustre/test_2
obdidx objid objid group
- 2 8 0x8 0
-</screen>
- </listitem><listitem>
- <para>To move single object(s), create a new copy and remove the original. Enter:</para>
- <screen>[client]# cp -a /mnt/lustre/test_2 /mnt/lustre/test_2.tmp
-[client]# mv /mnt/lustre/test_2.tmp /mnt/lustre/test_2
-</screen>
- </listitem><listitem>
- <para>To migrate large files from one or more OSTs, enter:</para>
- <screen>[client]# lfs find --ost {OST_UUID} -size +1G | lfs_migrate -y
-</screen>
- </listitem><listitem>
- <para>Check the file system balance.</para>
- <para>The df output in the example below shows a more balanced system compared to the df output in the example in <link xl:href="ManagingFileSystemIO.html#50438211_54434">Checking OST Space Usage</link>.</para>
- <screen>[client]# lfs df -h
+ 2 8 0x8 0</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">To move single object(s), create a new copy and remove the original. Enter:</emphasis></para>
+ <screen>[client]# cp -a /mnt/lustre/test_2 /mnt/lustre/test_2.tmp
+[client]# mv /mnt/lustre/test_2.tmp /mnt/lustre/test_2</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">To migrate large files from one or more OSTs, enter:</emphasis></para>
+ <screen>[client]# lfs find --ost {OST_UUID} -size +1G | lfs_migrate -y</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Check the file system balance.</emphasis></para>
+ <para>The <literal>df</literal> output in the example below shows a more balanced system compared to the <literal>df</literal> output in the example in <xref linkend="dbdoclet.50438211_17536"/>.</para>
+ <screen>[client]# lfs df -h
UUID bytes Used Available Use% \
Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% \
/mnt/lustre[OST:5]
filesystem summary: 11.8G 7.3G 3.9G 61% \
-/mnt/lustre
-</screen>
- </listitem></orderedlist>
- </section>
- <section remap="h3">
- <title>19.1.4 Returning an Inactive OST Back Online</title>
-
<para>Once the deactivated OST(s) no longer are severely imbalanced, due to either active or passive data redistribution, they should be reactivated so they will again have new files allocated on them.</para>
- <screen>[mds]# lctl --device 7 activate
+/mnt/lustre</screen>
+ </listitem>
+ </orderedlist>
+ </section>
+ <section remap="h3">
+ <title>19.1.4 Returning an Inactive OST Back Online</title>
+ <para>Once the deactivated OST(s) no longer are severely imbalanced, due to either active or passive data redistribution, they should be reactivated so they will again have new files allocated on them.</para>
+ <screen>[mds]# lctl --device 7 activate
[mds]# lctl dl
0 UP mgs MGS MGS 9
1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-816dd1e813 5
9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID
</screen>
- </section>
</section>
- <section xml:id="dbdoclet.50438211_75549">
- <title>19.2 Creating and Managing <anchor xml:id="dbdoclet.50438211_marker-1295531" xreflabel=""/>OST Pools</title>
- <para><anchor xml:id="dbdoclet.50438211_81568" xreflabel=""/>The OST pools feature enables users to group OSTs together to make object placement more flexible. A 'pool' is the name associated with an arbitrary subset of OSTs in a Lustre cluster.</para>
- <para>OST pools follow these rules:</para>
- <itemizedlist><listitem>
- <para> An OST can be a member of multiple pools.</para>
- </listitem>
-<listitem>
- <para> No ordering of OSTs in a pool is defined or implied.</para>
+ </section>
+ <section xml:id="dbdoclet.50438211_75549">
+ <title>19.2 Creating and Managing OST Pools</title>
+ <para>The OST pools feature enables users to group OSTs together to make object placement more flexible. A 'pool' is the name associated with an arbitrary subset of OSTs in a Lustre cluster.</para>
+ <para>OST pools follow these rules:</para>
+ <itemizedlist>
+ <listitem>
+ <para> An OST can be a member of multiple pools.</para>
+ </listitem>
+ <listitem>
+ <para> No ordering of OSTs in a pool is defined or implied.</para>
+ </listitem>
+ <listitem>
+ <para> Stripe allocation within a pool follows the same rules as the normal stripe allocator.</para>
+ </listitem>
+ <listitem>
+ <para> OST membership in a pool is flexible, and can change over time.</para>
+ </listitem>
+ </itemizedlist>
+ <para>When an OST pool is defined, it can be used to allocate files. When file or directory striping is set to a pool, only OSTs in the pool are candidates for striping. If a stripe_index is specified which refers to an OST that is not a member of the pool, an error is returned.</para>
+ <para>OST pools are used only at file creation. If the definition of a pool changes (an OST is added or removed or the pool is destroyed), already-created files are not affected.</para>
+ <note>
+ <para>An error (<literal>EINVAL</literal>) results if you create a file using an empty pool.</para>
+ </note>
+ <note>
+ <para>If a directory has pool striping set and the pool is subsequently removed, the new files created in this directory have the (non-pool) default striping pattern for that directory applied and no error is returned.</para>
+ </note>
+ <section remap="h3">
+ <title>19.2.1 Working with OST Pools</title>
+ <para>OST pools are defined in the configuration log on the MGS. Use the lctl command to:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Create/destroy a pool</para>
</listitem>
-<listitem>
- <para> Stripe allocation within a pool follows the same rules as the normal stripe allocator.</para>
+ <listitem>
+ <para> Add/remove OSTs in a pool</para>
</listitem>
-<listitem>
- <para> OST membership in a pool is flexible, and can change over time.</para>
+ <listitem>
+ <para> List pools and OSTs in a specific pool</para>
</listitem>
-</itemizedlist>
- <para>When an OST pool is defined, it can be used to allocate files. When file or directory striping is set to a pool, only OSTs in the pool are candidates for striping. If a stripe_index is specified which refers to an OST that is not a member of the pool, an error is returned.</para>
- <para>OST pools are used only at file creation. If the definition of a pool changes (an OST is added or removed or the pool is destroyed), already-created files are not affected.</para>
- <note><para>An error (EINVAL) results if you create a file using an empty pool.</para></note>
- <note><para>If a directory has pool striping set and the pool is subsequently removed, the new files created in this directory have the (non-pool) default striping pattern for that directory applied and no error is returned.</para></note>
-
- <section remap="h3">
- <title>19.2.1 <anchor xml:id="dbdoclet.50438211_71392" xreflabel=""/>Working with OST Pools</title>
- <para>OST pools are defined in the configuration log on the MGS. Use the lctl command to:</para>
- <itemizedlist><listitem>
- <para> Create/destroy a pool</para>
- </listitem>
-<listitem>
- <para> Add/remove OSTs in a pool</para>
- </listitem>
-<listitem>
- <para> List pools and OSTs in a specific pool</para>
- </listitem>
-</itemizedlist>
- <para>The lctl command MUST be run on the MGS. Another requirement for managing OST pools is to either have the MDT and MGS on the same node or have a Lustre client mounted on the MGS node, if it is separate from the MDS. This is needed to validate the pool commands being run are correct.</para>
- <caution><para>Running the writeconf command on the MDS erases all pools information (as well as any other parameters set using lctl conf_param). We recommend that the pools definitions (and conf_param settings) be executed using a script, so they can be reproduced easily after a writeconf is performed.</para></caution>
-
- <para>To create a new pool, run:</para>
- <screen>lctl pool_new <fsname>.<poolname>
-</screen>
- <note><para>The pool name is an ASCII string up to 16 characters.</para></note>
-
- <para>To add the named OST to a pool, run:</para>
- <screen>lctl pool_add <fsname>.<poolname> <ost_list>
-</screen>
- <para>Where:</para>
- <itemizedlist><listitem>
- <para><ost_list> is <fsname->OST<index_range>[_UUID]</para>
- </listitem>
-<listitem>
- <para><index_range> is <ost_index_start>-<ost_index_end>[,<index_range>] or <ost_index_start>-<ost_index_end>/<step></para>
- </listitem>
-</itemizedlist>
- <para>If the leading <fsname> and/or ending _UUID are missing, they are automatically added.</para>
- <para> For example, to add even-numbered OSTs to pool1 on file system lustre, run a single command (add) to add many OSTs to the pool at one time:</para>
- <para>lctl pool_add lustre.pool1 OST[0-10/2]</para>
- <note><para>Each time an OST is added to a pool, a new llog configuration record is created. For convenience, you can run a single command.</para></note>
- <para>To remove a named OST from a pool, run:</para>
- <screen>lctl pool_remove <fsname>.<poolname> <ost_list>
+ </itemizedlist>
+ <para>The lctl command MUST be run on the MGS. Another requirement for managing OST pools is to either have the MDT and MGS on the same node or have a Lustre client mounted on the MGS node, if it is separate from the MDS. This is needed to validate the pool commands being run are correct.</para>
+ <caution>
+ <para>Running the <literal>writeconf</literal> command on the MDS erases all pools information (as well as any other parameters set using <literal>lctl conf_param</literal>). We recommend that the pools definitions (and <literal>conf_param</literal> settings) be executed using a script, so they can be reproduced easily after a <literal>writeconf</literal> is performed.</para>
+ </caution>
+ <para>To create a new pool, run:</para>
+ <screen>lctl pool_new <fsname>.<poolname>
</screen>
- <para>To destroy a pool, run:</para>
- <screen>lctl pool_destroy <fsname>.<poolname>
+ <note>
+ <para>The pool name is an ASCII string up to 16 characters.</para>
+ </note>
+ <para>To add the named OST to a pool, run:</para>
+ <screen>lctl pool_add <fsname>.<poolname> <ost_list>
</screen>
- <note><para>All OSTs must be removed from a pool before it can be destroyed.</para></note>
-
- <para>To list pools in the named file system, run:</para>
- <screen>lctl pool_list <fsname> | <pathname>
-</screen>
- <para>To list OSTs in a named pool, run:</para>
- <screen>lctl pool_list <fsname>.<poolname>
-</screen>
- <section remap="h4">
- <title>19.2.1.1 Using the lfs Command with OST Pools</title>
- <para>Several lfs commands can be run with OST pools. Use the lfs setstripe command to associate a directory with an OST pool. This causes all new regular files and directories in the directory to be created in the pool. The lfs command can be used to list pools in a file system and OSTs in a named pool.</para>
- <para>To associate a directory with a pool, so all new files and directories will be created in the pool, run:</para>
- <screen>lfs setstripe --pool|-p pool_name <filename|dirname>
-</screen>
- <para>To set striping patterns, run:</para>
- <screen>lfs setstripe [--size|-s stripe_size] [--offset|-o start_ost]
+ <para>Where:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal><ost_list> is <fsname->OST<index_range>[_UUID]</literal></para>
+ </listitem>
+ <listitem>
+ <para><literal><index_range> is <ost_index_start>-<ost_index_end>[,<index_range>]</literal> or <literal><ost_index_start>-<ost_index_end>/<step></literal></para>
+ </listitem>
+ </itemizedlist>
+ <para>If the leading <literal><fsname></literal> and/or ending <literal>_UUID</literal> are missing, they are automatically added.</para>
+ <para> For example, to add even-numbered OSTs to pool1 on file system <literal>lustre</literal>, run a single command (add) to add many OSTs to the pool at one time:</para>
+ <para><screen>lctl pool_add lustre.pool1 OST[0-10/2]</screen></para>
+ <note>
+ <para>Each time an OST is added to a pool, a new <literal>llog</literal> configuration record is created. For convenience, you can run a single command.</para>
+ </note>
+ <para>To remove a named OST from a pool, run:</para>
+ <screen>lctl pool_remove <fsname>.<poolname> <ost_list></screen>
+ <para>To destroy a pool, run:</para>
+ <screen>lctl pool_destroy <fsname>.<poolname></screen>
+ <note>
+ <para>All OSTs must be removed from a pool before it can be destroyed.</para>
+ </note>
+ <para>To list pools in the named file system, run:</para>
+ <screen>lctl pool_list <fsname> | <pathname></screen>
+ <para>To list OSTs in a named pool, run:</para>
+ <screen>lctl pool_list <fsname>.<poolname></screen>
+ <section remap="h4">
+ <title>19.2.1.1 Using the lfs Command with OST Pools</title>
+ <para>Several lfs commands can be run with OST pools. Use the <literal>lfs setstripe</literal> command to associate a directory with an OST pool. This causes all new regular files and directories in the directory to be created in the pool. The lfs command can be used to list pools in a file system and OSTs in a named pool.</para>
+ <para>To associate a directory with a pool, so all new files and directories will be created in the pool, run:</para>
+ <screen>lfs setstripe --pool|-p pool_name <filename|dirname> </screen>
+ <para>To set striping patterns, run:</para>
+ <screen>lfs setstripe [--size|-s stripe_size] [--offset|-o start_ost]
[--count|-c stripe_count] [--pool|-p pool_name]
- <dir|filename>
-</screen>
- <note><para>If you specify striping with an invalid pool name, because the pool does not exist or the pool name was mistyped, lfs setstripe returns an error. Run lfs pool_list to make sure the pool exists and the pool name is entered correctly.</para></note>
- <note><para>The --pool option for lfs setstripe is compatible with other modifiers. For example, you can set striping on a directory to use an explicit starting index.</para></note>
-
- </section>
- </section>
- <section remap="h3">
- <title>19.2.2 <anchor xml:id="dbdoclet.50438211_62780" xreflabel=""/>Tips for Using OST Pools</title>
- <para>Here are several suggestions for using OST pools.</para>
- <itemizedlist><listitem>
- <para> A directory or file can be given an extended attribute (EA), that restricts striping to a pool.</para>
- </listitem>
-<listitem>
- <para> Pools can be used to group OSTs with the same technology or performance (slower or faster), or that are preferred for certain jobs. Examples are SATA OSTs versus SAS OSTs or remote OSTs versus local OSTs.</para>
- </listitem>
-<listitem>
- <para> A file created in an OST pool tracks the pool by keeping the pool name in the file LOV EA.</para>
- </listitem>
-</itemizedlist>
+ <dir|filename></screen>
+ <note>
+ <para>If you specify striping with an invalid pool name, because the pool does not exist or the pool name was mistyped, <literal>lfs setstripe</literal> returns an error. Run <literal>lfs pool_list</literal> to make sure the pool exists and the pool name is entered correctly.</para>
+ </note>
+ <note>
+ <para>The <literal>--pool</literal> option for lfs setstripe is compatible with other modifiers. For example, you can set striping on a directory to use an explicit starting index.</para>
+ </note>
</section>
</section>
- <section xml:id="dbdoclet.50438211_11204">
- <title>19.3 Adding an OST to a Lustre File System</title>
- <para>To add an OST to existing Lustre file system:</para>
- <orderedlist><listitem>
- <para>Add a new OST by passing on the following commands, run:</para>
- <screen>$ mkfs.lustre --fsname=spfs --ost --mgsnode=mds16@tcp0 /dev/sda
-$ mkdir -p /mnt/test/ost0
-$ mount -t lustre /dev/sda /mnt/test/ost0
-</screen>
-
- </listitem><listitem>
- <para>Migrate the data (possibly).</para>
- <para>The file system is quite unbalanced when new empty OSTs are added. New file creations are automatically balanced. If this is a scratch file system or files are pruned at a regular interval, then no further work may be needed. Files existing prior to the expansion can be rebalanced with an in-place copy, which can be done with a simple script.</para>
- <para>The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
- <para>A very clever migration script would do the following:</para>
- <itemizedlist><listitem>
- <para> Examine the current distribution of data.</para>
- </listitem>
-<listitem>
- <para> Calculate how much data should move from each full OST to the empty ones.</para>
+ <section remap="h3">
+ <title>19.2.2 Tips for Using OST Pools</title>
+ <para>Here are several suggestions for using OST pools.</para>
+ <itemizedlist>
+ <listitem>
+ <para>A directory or file can be given an extended attribute (EA), that restricts striping to a pool.</para>
</listitem>
-<listitem>
- <para> Search for files on a given full OST (using lfs getstripe).</para>
+ <listitem>
+ <para>Pools can be used to group OSTs with the same technology or performance (slower or faster), or that are preferred for certain jobs. Examples are SATA OSTs versus SAS OSTs or remote OSTs versus local OSTs.</para>
</listitem>
-<listitem>
- <para> Force the new destination OST (using lfs setstripe).</para>
+ <listitem>
+ <para>A file created in an OST pool tracks the pool by keeping the pool name in the file LOV EA.</para>
</listitem>
-<listitem>
- <para> Copy only enough files to address the imbalance.</para>
- </listitem>
-</itemizedlist>
- </listitem></orderedlist>
- <para>If a Lustre administrator wants to explore this approach further, per-OST disk-usage statistics can be found under /proc/fs/lustre/osc/*/rpc_stats</para>
+ </itemizedlist>
</section>
- <section xml:id="dbdoclet.50438211_80295">
- <title>19.4 Performing <anchor xml:id="dbdoclet.50438211_marker-1291962" xreflabel=""/>Direct I/O</title>
- <para>Lustre supports the O_DIRECT flag to open.</para>
- <para>Applications using the read() and write() calls must supply buffers aligned on a page boundary (usually 4 K). If the alignment is not correct, the call returns -EINVAL. Direct I/O may help performance in cases where the client is doing a large amount of I/O and is CPU-bound (CPU utilization 100%).</para>
- <section remap="h3">
- <title>19.4.1 Making File System Objects Immutable</title>
- <para>An immutable file or directory is one that cannot be modified, renamed or removed. To do this:</para>
- <screen>chattr +i <file>
-</screen>
- <para>To remove this flag, use chattr -i</para>
- </section>
+ </section>
+ <section xml:id="dbdoclet.50438211_11204">
+ <title>19.3 Adding an OST to a Lustre File System</title>
+ <para>To add an OST to existing Lustre file system:</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Add a new OST by passing on the following commands, run:</emphasis></para>
+ <screen>$ mkfs.lustre --fsname=spfs --ost --mgsnode=mds16@tcp0 /dev/sda
+$ mkdir -p /mnt/test/ost0
+$ mount -t lustre /dev/sda /mnt/test/ost0</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Migrate the data (possibly).</emphasis></para>
+ <para>The file system is quite unbalanced when new empty OSTs are added. New file creations are automatically balanced. If this is a scratch file system or files are pruned at a regular interval, then no further work may be needed. Files existing prior to the expansion can be rebalanced with an in-place copy, which can be done with a simple script.</para>
+ <para>The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
+ <para>A very clever migration script would do the following:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Examine the current distribution of data.</para>
+ </listitem>
+ <listitem>
+ <para> Calculate how much data should move from each full OST to the empty ones.</para>
+ </listitem>
+ <listitem>
+ <para> Search for files on a given full OST (using <literal>lfs getstripe</literal>).</para>
+ </listitem>
+ <listitem>
+ <para> Force the new destination OST (using <literal>lfs setstripe</literal>).</para>
+ </listitem>
+ <listitem>
+ <para> Copy only enough files to address the imbalance.</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ </orderedlist>
+ <para>If a Lustre administrator wants to explore this approach further, per-OST disk-usage statistics can be found under <literal>/proc/fs/lustre/osc/*/rpc_stats</literal></para>
+ </section>
+ <section xml:id="dbdoclet.50438211_80295">
+ <title>19.4 Performing Direct I/O</title>
+ <para>Lustre supports the <literal>O_DIRECT</literal> flag to open.</para>
+ <para>Applications using the <literal>read()</literal> and <literal>write()</literal> calls must supply buffers aligned on a page boundary (usually 4 K). If the alignment is not correct, the call returns <literal>-EINVAL</literal>. Direct I/O may help performance in cases where the client is doing a large amount of I/O and is CPU-bound (CPU utilization 100%).</para>
+ <section remap="h3">
+ <title>19.4.1 Making File System Objects Immutable</title>
+ <para>An immutable file or directory is one that cannot be modified, renamed or removed. To do this:</para>
+ <screen>chattr +i <file></screen>
+ <para>To remove this flag, use <literal>chattr -i</literal></para>
</section>
- <section xml:id="dbdoclet.50438211_61024">
- <title>19.5 Other I/O Options</title>
- <para>This section describes other I/O options, including checksums.</para>
- <section remap="h3">
- <title>19.5.1 Lustre <anchor xml:id="dbdoclet.50438211_marker-1291974" xreflabel=""/>Checksums</title>
- <para>To guard against network data corruption, a Lustre client can perform two types of data checksums: in-memory (for data in client memory) and wire (for data sent over the network). For each checksum type, a 32-bit checksum of the data read or written on both the client and server is computed, to ensure that the data has not been corrupted in transit over the network. The ldiskfs backing file system does NOT do any persistent checksumming, so it does not detect corruption of data in the OST file system.</para>
- <para>The checksumming feature is enabled, by default, on individual client nodes. If the client or OST detects a checksum mismatch, then an error is logged in the syslog of the form:</para>
- <screen>LustreError: BAD WRITE CHECKSUM: changed in transit before arrival at OST: \
+ </section>
+ <section xml:id="dbdoclet.50438211_61024">
+ <title>19.5 Other I/O Options</title>
+ <para>This section describes other I/O options, including checksums.</para>
+ <section remap="h3">
+ <title>19.5.1 Lustre Checksums</title>
+ <para>To guard against network data corruption, a Lustre client can perform two types of data checksums: in-memory (for data in client memory) and wire (for data sent over the network). For each checksum type, a 32-bit checksum of the data read or written on both the client and server is computed, to ensure that the data has not been corrupted in transit over the network. The <literal>ldiskfs</literal> backing file system does NOT do any persistent checksumming, so it does not detect corruption of data in the OST file system.</para>
+ <para>The checksumming feature is enabled, by default, on individual client nodes. If the client or OST detects a checksum mismatch, then an error is logged in the syslog of the form:</para>
+ <screen>LustreError: BAD WRITE CHECKSUM: changed in transit before arrival at OST: \
from 192.168.1.1@tcp inum 8991479/2386814769 object 1127239/0 extent [10240\
-0-106495]
-</screen>
- <para>If this happens, the client will re-read or re-write the affected data up to five times to get a good copy of the data over the network. If it is still not possible, then an I/O error is returned to the application.</para>
- <para>To enable both types of checksums (in-memory and wire), run:</para>
- <screen>echo 1 > /proc/fs/lustre/llite/<emphasis><fsname></emphasis>/checksum_pages
-</screen>
- <para>To disable both types of checksums (in-memory and wire), run:</para>
- <screen>echo 0 > /proc/fs/lustre/llite/<emphasis><fsname></emphasis>/checksum_pages
-</screen>
- <para>To check the status of a wire checksum, run:</para>
- <screen>lctl get_param osc.*.checksums
-</screen>
- <section remap="h4">
- <title>19.5.1.1 Changing Checksum Algorithms</title>
- <para>By default, Lustre uses the adler32 checksum algorithm, because it is robust and has a lower impact on performance than crc32. The Lustre administrator can change the checksum algorithm via /proc, depending on what is supported in the kernel.</para>
- <para>To check which checksum algorithm is being used by Lustre, run:</para>
- <screen>$ cat /proc/fs/lustre/osc/<fsname>-OST<index>-osc-*/checksum_type
-</screen>
- <para>To change the wire checksum algorithm used by Lustre, run:</para>
- <screen>$ echo <algorithm name> /proc/fs/lustre/osc/<fsname>-OST<index>- \osc-*/checksum_\
-type
-</screen>
- <note><para>The in-memory checksum always uses the adler32 algorithm, if available, and only falls back to crc32 if adler32 cannot be used.</para></note>
-
- <para>In the following example, the cat command is used to determine that Lustre is using the adler32 checksum algorithm. Then the echo command is used to change the checksum algorithm to crc32. A second cat command confirms that the crc32 checksum algorithm is now in use.</para>
- <screen>$ cat /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/checksum_ty\
+0-106495]</screen>
+ <para>If this happens, the client will re-read or re-write the affected data up to five times to get a good copy of the data over the network. If it is still not possible, then an I/O error is returned to the application.</para>
+ <para>To enable both types of checksums (in-memory and wire), run:</para>
+ <screen>echo 1 > /proc/fs/lustre/llite/<emphasis><fsname></emphasis>/checksum_pages</screen>
+ <para>To disable both types of checksums (in-memory and wire), run:</para>
+ <screen>echo 0 > /proc/fs/lustre/llite/<emphasis><fsname></emphasis>/checksum_pages</screen>
+ <para>To check the status of a wire checksum, run:</para>
+ <screen>lctl get_param osc.*.checksums</screen>
+ <section remap="h4">
+ <title>19.5.1.1 Changing Checksum Algorithms</title>
+ <para>By default, Lustre uses the adler32 checksum algorithm, because it is robust and has a lower impact on performance than crc32. The Lustre administrator can change the checksum algorithm via /proc, depending on what is supported in the kernel.</para>
+ <para>To check which checksum algorithm is being used by Lustre, run:</para>
+ <screen>$ cat /proc/fs/lustre/osc/<fsname>-OST<index>-osc-*/checksum_type</screen>
+ <para>To change the wire checksum algorithm used by Lustre, run:</para>
+ <screen>$ echo <algorithm name> /proc/fs/lustre/osc/<fsname>-OST<index>- \osc-*/checksum_\
+type</screen>
+ <note>
+ <para>The in-memory checksum always uses the adler32 algorithm, if available, and only falls back to crc32 if adler32 cannot be used.</para>
+ </note>
+ <para>In the following example, the cat command is used to determine that Lustre is using the adler32 checksum algorithm. Then the echo command is used to change the checksum algorithm to crc32. A second cat command confirms that the crc32 checksum algorithm is now in use.</para>
+ <screen>$ cat /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/checksum_ty\
pe
crc32 [adler]
$ echo crc32 > /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/che\
cksum_type
$ cat /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/checksum_ty\
pe
-[crc32] adler
-</screen>
- </section>
+[crc32] adler</screen>
</section>
+ </section>
</section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='managinglnet'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. -->
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managinglnet">
<info>
- <title xml:id='managinglnet.title'>Managing Lustre Networking (LNET)</title>
+ <title xml:id="managinglnet.title">Managing Lustre Networking (LNET)</title>
</info>
-
<para>This chapter describes some tools for managing Lustre Networking (LNET) and includes the following sections:</para>
- <itemizedlist><listitem>
- <para><xref linkend="dbdoclet.50438203_51732"/>Updating the Health Status of a Peer or Router</para>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438203_51732"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438203_48703"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438203_82542"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438203_78227"/></para>
+ </listitem>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438203_51732">
+ <title>15.1 Updating the Health Status of a Peer or Router</title>
+ <para>There are two mechanisms to update the health status of a peer or a router:</para>
+ <itemizedlist>
+ <listitem>
+ <para>LNET can actively check health status of all routers and mark them as dead or alive automatically. By default, this is off. To enable it set <literal>auto_down</literal> and if desired <literal>check_routers_before_use</literal>. This initial check may cause a pause equal to <literal>router_ping_timeout</literal> at system startup, if there are dead routers in the system.</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438203_48703"/>Starting and Stopping LNET</para>
+ <para>When there is a communication error, all LNDs notify LNET that the peer (not necessarily a router) is down. This mechanism is always on, and there is no parameter to turn it off. However, if you set the LNET module parameter <literal>auto_down</literal> to <literal>0</literal>, LNET ignores all such peer-down notifications.</para>
</listitem>
+ </itemizedlist>
+ <para>Several key differences in both mechanisms:</para>
+ <itemizedlist>
<listitem>
- <para><xref linkend="dbdoclet.50438203_82542"/>Multi-Rail Configurations with LNET</para>
+ <para>The router pinger only checks routers for their health, while LNDs notices all dead peers, regardless of whether they are a router or not.</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438203_78227"/>Load Balancing with InfiniBand</para>
+ <para>The router pinger actively checks the router health by sending pings, but LNDs only notice a dead peer when there is network traffic going on.</para>
</listitem>
- </itemizedlist>
-
- <section xml:id="dbdoclet.50438203_51732">
- <title>15.1 Updating the Health Status of a Peer or <anchor xml:id="dbdoclet.50438203_marker-1288828" xreflabel=""/>Router</title>
- <para>There are two mechanisms to update the health status of a peer or a router:</para>
- <itemizedlist><listitem>
- <para>LNET can actively check health status of all routers and mark them as dead or alive automatically. By default, this is off. To enable it set auto_down and if desired check_routers_before_use. This initial check may cause a pause equal to router_ping_timeout at system startup, if there are dead routers in the system.</para>
- </listitem>
-<listitem>
- <para>When there is a communication error, all LNDs notify LNET that the peer (not necessarily a router) is down. This mechanism is always on, and there is no parameter to turn it off. However, if you set the LNET module parameter auto_down to 0, LNET ignores all such peer-down notifications.</para>
- </listitem>
-</itemizedlist>
- <para>Several key differences in both mechanisms:</para>
- <itemizedlist><listitem>
- <para>The router pinger only checks routers for their health, while LNDs notices all dead peers, regardless of whether they are a router or not.</para>
- </listitem>
-<listitem>
- <para>The router pinger actively checks the router health by sending pings, but LNDs only notice a dead peer when there is network traffic going on.</para>
- </listitem>
-<listitem>
- <para> The router pinger can bring a router from alive to dead or vice versa, but LNDs can only bring a peer down.</para>
- </listitem>
-</itemizedlist>
- </section>
- <section xml:id="dbdoclet.50438203_48703">
- <title>15.2 Starting and Stopping LNET</title>
- <para>Lustre automatically starts and stops LNET, but it can also be manually started in a standalone manner. This is particularly useful to verify that your networking setup is working correctly before you attempt to start Lustre.</para>
- <section remap="h3">
- <title>15.2.1 Starting <anchor xml:id="dbdoclet.50438203_marker-1287400" xreflabel=""/>LNET</title>
- <para>To start LNET, run:</para>
- <screen>$ modprobe lnet
-$ lctl network up
-</screen>
- <para>To see the list of local NIDs, run:</para>
- <screen>$ lctl list_nids
-</screen>
- <para>This command tells you the network(s) configured to work with Lustre</para>
- <para>If the networks are not correctly setup, see the modules.conf "networks=" line and make sure the network layer modules are correctly installed and configured.</para>
- <para>To get the best remote NID, run:</para>
- <screen>$ lctl which_nid <NID list>
-</screen>
- <para>where <NID list> is the list of available NIDs.</para>
- <para>This command takes the "best" NID from a list of the NIDs of a remote host. The "best" NID is the one that the local node uses when trying to communicate with the remote node.</para>
- <section remap="h4">
- <title>15.2.1.1 <anchor xml:id="dbdoclet.50438203_46145" xreflabel=""/>Starting Clients</title>
- <para>To start a TCP client, run:</para>
- <screen>mount -t lustre mdsnode:/mdsA/client /mnt/lustre/
-</screen>
- <para>To start an Elan client, run:</para>
- <screen>mount -t lustre 2@elan0:/mdsA/client /mnt/lustre
-</screen>
- </section>
- </section>
- <section remap="h3">
- <title>15.2.2 Stopping <anchor xml:id="dbdoclet.50438203_marker-1288543" xreflabel=""/>LNET</title>
- <para>Before the LNET modules can be removed, LNET references must be removed. In general, these references are removed automatically when Lustre is shut down, but for standalone routers, an explicit step is needed to stop LNET. Run:</para>
- <screen>lctl network unconfigure
-</screen>
-
-<note>
- <para>
-Attempting to remove Lustre modules prior to stopping the network may result in a crash or an LNET hang. if this occurs, the node must be rebooted (in most cases). Make sure that the Lustre network and Lustre are stopped prior to unloading the modules. Be extremely careful using rmmod -f.</para>
-</note>
-
- <para>To unconfigure the LNET network, run:</para>
- <screen>modprobe -r <any lnd and the lnet modules>
-</screen>
-
-<note>
- <para>
-To remove all Lustre modules, run:</para><para>$ lctl modules | awk '{print $2}' | xargs rmmod</para>
-</note>
-
-
+ <listitem>
+ <para> The router pinger can bring a router from alive to dead or vice versa, but LNDs can only bring a peer down.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="dbdoclet.50438203_48703">
+ <title>15.2 Starting and Stopping LNET</title>
+ <para>Lustre automatically starts and stops LNET, but it can also be manually started in a standalone manner. This is particularly useful to verify that your networking setup is working correctly before you attempt to start Lustre.</para>
+ <section remap="h3">
+ <title>15.2.1 Starting LNET</title>
+ <para>To start LNET, run:</para>
+ <screen>$ modprobe lnet
+$ lctl network up</screen>
+ <para>To see the list of local NIDs, run:</para>
+ <screen>$ lctl list_nids</screen>
+ <para>This command tells you the network(s) configured to work with Lustre</para>
+ <para>If the networks are not correctly setup, see the <literal>modules.conf</literal> "<literal>networks=</literal>" line and make sure the network layer modules are correctly installed and configured.</para>
+ <para>To get the best remote NID, run:</para>
+ <screen>$ lctl which_nid <NID list></screen>
+ <para>where <literal><NID list></literal> is the list of available NIDs.</para>
+ <para>This command takes the "best" NID from a list of the NIDs of a remote host. The "best" NID is the one that the local node uses when trying to communicate with the remote node.</para>
+ <section remap="h4">
+ <title>15.2.1.1 Starting Clients</title>
+ <para>To start a TCP client, run:</para>
+ <screen>mount -t lustre mdsnode:/mdsA/client /mnt/lustre/</screen>
+ <para>To start an Elan client, run:</para>
+ <screen>mount -t lustre 2@elan0:/mdsA/client /mnt/lustre</screen>
</section>
</section>
- <section xml:id="dbdoclet.50438203_72197">
- <title>15.3 <anchor xml:id="dbdoclet.50438203_82542" xreflabel=""/>Multi-Rail Configurations with <anchor xml:id="dbdoclet.50438203_Multi-rail-configurations-with-LNET-LNET" xreflabel=""/>LNET</title>
- <para>To aggregate bandwidth across both rails of a dual-rail IB cluster (o2iblnd) <footnote><para>Multi-rail configurations are only supported by o2iblnd; other IB LNDs do not support multiple interfaces.</para></footnote> using LNET, consider these points:</para>
- <itemizedlist><listitem>
- <para> LNET can work with multiple rails, however, it does not load balance across them. The actual rail used for any communication is determined by the peer NID.</para>
- </listitem>
-<listitem>
- <para> Multi-rail LNET configurations do not provide an additional level of network fault tolerance. The configurations described below are for bandwidth aggregation only. Network interface failover is planned as an upcoming Lustre feature.</para>
- </listitem>
-<listitem>
- <para> A Lustre node always uses the same local NID to communicate with a given peer NID. The criteria used to determine the local NID are:</para>
- <itemizedlist><listitem>
- <para> Fewest hops (to minimize routing), and</para>
- </listitem>
-<listitem>
- <para> Appears first in the "networks" or "ip2nets" LNET configuration strings</para>
- </listitem>
-</itemizedlist>
- </listitem>
-</itemizedlist>
+ <section remap="h3">
+ <title>15.2.2 Stopping LNET</title>
+ <para>Before the LNET modules can be removed, LNET references must be removed. In general, these references are removed automatically when Lustre is shut down, but for standalone routers, an explicit step is needed to stop LNET. Run:</para>
+ <screen>lctl network unconfigure</screen>
+ <note>
+ <para>
+Attempting to remove Lustre modules prior to stopping the network may result in a crash or an LNET hang. if this occurs, the node must be rebooted (in most cases). Make sure that the Lustre network and Lustre are stopped prior to unloading the modules. Be extremely careful using rmmod -f.</para>
+ </note>
+ <para>To unconfigure the LNET network, run:</para>
+ <screen>modprobe -r <any lnd and the lnet modules>
+</screen>
+ <note>
+ <para>
+To remove all Lustre modules, run:</para>
+ <para><literal>$ lctl modules | awk '{print $2}' | xargs rmmod</literal></para>
+ </note>
</section>
- <section xml:id="dbdoclet.50438203_78227">
- <title>15.4 Load Balancing with InfiniBand</title>
- <para>A Lustre file system contains OSSs with two InfiniBand HCAs. Lustre clients have only one InfiniBand HCA using OFED Infiniband ''o2ib'' drivers. Load balancing between the HCAs on the OSS is accomplished through LNET.</para>
- <section remap="h3">
- <title>15.4.1 Setting Up modprobe.conf<anchor xml:id="dbdoclet.50438203_77834" xreflabel=""/><anchor xml:id="dbdoclet.50438203_marker-1290316" xreflabel=""/> for Load Balancing</title>
- <para>To configure LNET for load balancing on clients and servers:</para>
- <para> 1. Set the modprobe.conf options.</para>
- <para>Depending on your configuration, set modprobe.conf options as follows:</para>
- <itemizedlist><listitem>
- <para> Dual HCA OSS server</para>
- </listitem>
-</itemizedlist>
- <screen>options lnet networks="o2ib0(ib0),o2ib1(ib1) 192.168.10.1.[101-102]
-</screen>
- <itemizedlist><listitem>
- <para> Client with the odd IP address</para>
+ </section>
+ <section xml:id="dbdoclet.50438203_82542">
+ <title>15.3 Multi-Rail Configurations with LNET</title>
+ <para>To aggregate bandwidth across both rails of a dual-rail IB cluster (o2iblnd) <footnote>
+ <para>Multi-rail configurations are only supported by o2iblnd; other IB LNDs do not support multiple interfaces.</para>
+ </footnote> using LNET, consider these points:</para>
+ <itemizedlist>
+ <listitem>
+ <para>LNET can work with multiple rails, however, it does not load balance across them. The actual rail used for any communication is determined by the peer NID.</para>
+ </listitem>
+ <listitem>
+ <para>Multi-rail LNET configurations do not provide an additional level of network fault tolerance. The configurations described below are for bandwidth aggregation only. Network interface failover is planned as an upcoming Lustre feature.</para>
+ </listitem>
+ <listitem>
+ <para>A Lustre node always uses the same local NID to communicate with a given peer NID. The criteria used to determine the local NID are:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Fewest hops (to minimize routing), and</para>
</listitem>
-</itemizedlist>
- <screen>options lnet networks=o2ib0(ib0) 192.168.10.[103-253/2]
-</screen>
- <itemizedlist><listitem>
- <para> Client with the even IP address</para>
+ <listitem>
+ <para> Appears first in the "<literal>networks</literal>" or "<literal>ip2nets</literal>" LNET configuration strings</para>
</listitem>
-</itemizedlist>
- <screen>options lnet networks=o2ib1(ib0) 192.168.10.[102-254/2]
+ </itemizedlist>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="dbdoclet.50438203_78227">
+ <title>15.4 Load Balancing with InfiniBand</title>
+ <para>A Lustre file system contains OSSs with two InfiniBand HCAs. Lustre clients have only one InfiniBand HCA using OFED Infiniband ''o2ib'' drivers. Load balancing between the HCAs on the OSS is accomplished through LNET.</para>
+ <section remap="h3">
+ <title>15.4.1 Setting Up <literal>modprobe.conf</literal> for Load Balancing</title>
+ <para>To configure LNET for load balancing on clients and servers:</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Set the <literal>modprobe.conf</literal> options.</emphasis></para>
+ <para>Depending on your configuration, set modprobe.conf options as follows:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Dual HCA OSS server</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet networks="o2ib0(ib0),o2ib1(ib1) 192.168.10.1.[101-102] </screen>
+ <itemizedlist>
+ <listitem>
+ <para> Client with the odd IP address</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet networks=o2ib0(ib0) 192.168.10.[103-253/2] </screen>
+ <itemizedlist>
+ <listitem>
+ <para> Client with the even IP address</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet networks=o2ib1(ib0) 192.168.10.[102-254/2]
</screen>
- <para> 2. Run the modprobe lnet command and create a combined MGS/MDT file system.</para>
- <para>The following commands create the MGS/MDT file system and mount the servers (MGS/MDT and OSS).</para>
- <screen>modprobe lnet
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Run the modprobe lnet command and create a combined MGS/MDT file system.</emphasis></para>
+ <para>The following commands create the MGS/MDT file system and mount the servers (MGS/MDT and OSS).</para>
+ <screen>modprobe lnet
$ mkfs.lustre --fsname lustre --mgs --mdt <block device name>
$ mkdir -p <mount point>
$ mount -t lustre <block device> <mount point>
$ mkfs.lustre --fsname lustre --mgs --mdt <block device name>
$ mkdir -p <mount point>
$ mount -t lustre <block device> <mount point>
-$ mount -t lustre <block device> <mount point>
-</screen>
- <para>For example:</para>
- <screen>modprobe lnet
+$ mount -t lustre <block device> <mount point> </screen>
+ <para>For example:</para>
+ <screen>modprobe lnet
$ mkfs.lustre --fsname lustre --mdt --mgs /dev/sda
$ mkdir -p /mnt/test/mdt
$ mount -t lustre /dev/sda /mnt/test/mdt
$ mkfs.lustre --fsname lustre --ost --mgsnode=mds@o2ib0 /dev/sda
$ mkdir -p /mnt/test/mdt
$ mount -t lustre /dev/sda /mnt/test/ost
-$ mount -t lustre mgs@o2ib0:/lustre /mnt/ost
-</screen>
- <para> 3. Mount the clients.</para>
- <screen>mount -t lustre <MGS node>:/<fsname> <mount point>
-</screen>
- <para>This example shows an IB client being mounted.</para>
- <screen>mount -t lustre
-192.168.10.101@o2ib0,192.168.10.102@o2ib1:/mds/client /mnt/lustre
-</screen>
- <para>As an example, consider a two-rail IB cluster running the OFA stack (OFED) with these IPoIB address assignments.</para>
- <screen> ib0 ib1
+$ mount -t lustre mgs@o2ib0:/lustre /mnt/ost</screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Mount the clients.</emphasis></para>
+ <screen>mount -t lustre <MGS node>:/<fsname> <mount point></screen>
+ <para>This example shows an IB client being mounted.</para>
+ <screen>mount -t lustre
+192.168.10.101@o2ib0,192.168.10.102@o2ib1:/mds/client /mnt/lustre</screen>
+ </listitem>
+ </orderedlist>
+ <para>As an example, consider a two-rail IB cluster running the OFA stack (OFED) with these IPoIB address assignments.</para>
+ <screen> ib0 ib1
Servers 192.168.0.* 192.168.1.*
-Clients 192.168.[2-127].* 192.168.[128-253].*
-</screen>
- <para>You could create these configurations:</para>
- <itemizedlist><listitem>
- <para> A cluster with more clients than servers. The fact that an individual client cannot get two rails of bandwidth is unimportant because the servers are the actual bottleneck.</para>
- </listitem>
-</itemizedlist>
- <screen>ip2nets="o2ib0(ib0), o2ib1(ib1) 192.168.[0-1].* \
+Clients 192.168.[2-127].* 192.168.[128-253].*</screen>
+ <para>You could create these configurations:</para>
+ <itemizedlist>
+ <listitem>
+ <para>A cluster with more clients than servers. The fact that an individual client cannot get two rails of bandwidth is unimportant because the servers are the actual bottleneck.</para>
+ </listitem>
+ </itemizedlist>
+ <screen>ip2nets="o2ib0(ib0), o2ib1(ib1) 192.168.[0-1].* \
#all servers;\
o2ib0(ib0) 192.168.[2-253].[0-252/2] #even cl\
ients;\
o2ib1(ib1) 192.168.[2-253].[1-253/2] #odd cli\
-ents"
-</screen>
- <para>This configuration gives every server two NIDs, one on each network, and statically load-balances clients between the rails.</para>
- <itemizedlist><listitem>
-
<para> A single client that must get two rails of bandwidth, and it does not matter if the maximum aggregate bandwidth is only (# servers) * (1 rail).</para>
- </listitem>
-</itemizedlist>
- <screen>ip2nets=" o2ib0(ib0) 192.168.[0-1].[0-252/2] \
+ents"</screen>
+ <para>This configuration gives every server two NIDs, one on each network, and statically load-balances clients between the rails.</para>
+ <itemizedlist>
+ <listitem>
+ <para> A single client that must get two rails of bandwidth, and it does not matter if the maximum aggregate bandwidth is only (# servers) * (1 rail).</para>
+ </listitem>
+ </itemizedlist>
+ <screen>ip2nets=" o2ib0(ib0) 192.168.[0-1].[0-252/2] \
#even servers;\
o2ib1(ib1) 192.168.[0-1].[1-253/2] \
#odd servers;\
o2ib0(ib0),o2ib1(ib1) 192.168.[2-253].* \
- #clients"
-</screen>
- <para>This configuration gives every server a single NID on one rail or the other. Clients have a NID on both rails.</para>
- <itemizedlist><listitem>
-
<para> All clients and all servers must get two rails of bandwidth.</para>
- </listitem>
-</itemizedlist>
- <screen>ip2nets=†o2ib0(ib0),o2ib2(ib1) 192.168.[0-1].[0-252/2] \
+ #clients"</screen>
+ <para>This configuration gives every server a single NID on one rail or the other. Clients have a NID on both rails.</para>
+ <itemizedlist>
+ <listitem>
+ <para> All clients and all servers must get two rails of bandwidth.</para>
+ </listitem>
+ </itemizedlist>
+ <screen>ip2nets=†o2ib0(ib0),o2ib2(ib1) 192.168.[0-1].[0-252/2] \
#even servers;\
o2ib1(ib0),o2ib3(ib1) 192.168.[0-1].[1-253/2] \
#odd servers;\
o2ib1(ib0),o2ib2(ib1) 192.168.[2-253].[1-253/2) \
#odd clients"
</screen>
- <para>This configuration includes two additional proxy o2ib networks to work around Lustre's simplistic NID selection algorithm. It connects "even" clients to "even" servers with o2ib0 on rail0, and "odd" servers with o2ib3 on rail1. Similarly, it connects "odd" clients to "odd" servers with o2ib1 on rail0, and "even" servers with o2ib2 on rail1.</para>
- </section>
+ <para>This configuration includes two additional proxy o2ib networks to work around Lustre's simplistic NID selection algorithm. It connects "even" clients to "even" servers with <literal>o2ib0</literal> on <literal>rail0</literal>, and "odd" servers with <literal>o2ib3</literal> on <literal>rail1</literal>. Similarly, it connects "odd" clients to "odd" servers with <literal>o2ib1</literal> on <literal>rail0</literal>, and "even" servers with <literal>o2ib2</literal> on <literal>rail1</literal>.</para>
+ </section>
</section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='managingsecurity'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingsecurity">
<info>
- <title xml:id='managingsecurity.title'>Managing Lustre Security</title>
+ <title xml:id="managingsecurity.title">Managing Lustre Security</title>
</info>
<para>This chapter describes Lustre security and includes the following sections:</para>
-
- <itemizedlist><listitem>
-
<para><xref linkend="dbdoclet.50438221_16221"/></para>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438221_16221"/></para>
</listitem>
-<listitem>
- <para><xref linkend="dbdoclet.50438221_64726"/></para>
+ <listitem>
+
<para><xref linkend="dbdoclet.50438221_64726"/></para>
</listitem>
-
-</itemizedlist>
-
- <section xml:id="dbdoclet.50438221_16221">
- <title>22.1 Using <anchor xml:id="dbdoclet.50438221_marker-1292306" xreflabel=""/>ACLs</title>
- <para>An access control list (ACL), is a set of data that informs an operating system about permissions or access rights that each user or group has to specific system objects, such as directories or files. Each object has a unique security attribute that identifies users who have access to it. The ACL lists each object and user access privileges such as read, write or execute.</para>
- <section remap="h3">
- <title>22.1.1 How ACLs Work</title>
- <para>Implementing ACLs varies between operating systems. Systems that support the Portable Operating System Interface (POSIX) family of standards share a simple yet powerful file system permission model, which should be well-known to the Linux/Unix administrator. ACLs add finer-grained permissions to this model, allowing for more complicated permission schemes. For a detailed explanation of ACLs on Linux, refer to the SuSE Labs article, <emphasis>Posix Access Control Lists on Linux</emphasis>:</para>
- <para><link xl:href="http://www.suse.de/~agruen/acl/linux-acls/online/">http://www.suse.de/~agruen/acl/linux-acls/online/</link></para>
- <para>We have implemented ACLs according to this model. Lustre works with the standard Linux ACL tools, setfacl, getfacl, and the historical chacl, normally installed with the ACL package.</para>
- <note><para>ACL support is a system-range feature, meaning that all clients have ACL enabled or not. You cannot specify which clients should enable ACL.</para></note>
- </section>
- <section remap="h3">
- <title>22.1.2 Using <anchor xml:id="dbdoclet.50438221_marker-1292314" xreflabel=""/>ACLs with Lustre</title>
- <para>POSIX Access Control Lists (ACLs) can be used with Lustre. An ACL consists of file entries representing permissions based on standard POSIX file system object permissions that define three classes of user (owner, group and other). Each class is associated with a set of permissions [read (r), write (w) and execute (x)].</para>
- <itemizedlist><listitem>
- <para> Owner class permissions define access privileges of the file owner.</para>
- </listitem>
-
-<listitem>
- <para> Group class permissions define access privileges of the owning group.</para>
- </listitem>
-
-<listitem>
- <para> Other class permissions define access privileges of all users not in the owner or group class.</para>
- </listitem>
-
-</itemizedlist>
- <para>The ls -l command displays the owner, group, and other class permissions in the first column of its output (for example, -rw-r- -- for a regular file with read and write access for the owner class, read access for the group class, and no access for others).</para>
- <para>Minimal ACLs have three entries. Extended ACLs have more than the three entries. Extended ACLs also contain a mask entry and may contain any number of named user and named group entries.</para>
- <para>The MDS needs to be configured to enable ACLs. Use --mountfsoptions to enable ACLs when creating your configuration:</para>
- <screen>$ mkfs.lustre --fsname spfs --mountfsoptions=acl --mdt -mgs /dev/sda
-</screen>
- <para>Alternately, you can enable ACLs at run time by using the --acl option with mkfs.lustre:</para>
- <screen>$ mount -t lustre -o acl /dev/sda /mnt/mdt
-</screen>
- <para>To check ACLs on the MDS:</para>
- <screen>$ lctl get_param -n mdc.home-MDT0000-mdc-*.connect_flags | grep acl acl
-</screen>
- <para>To mount the client with no ACLs:</para>
- <screen>$ mount -t lustre -o noacl ibmds2@o2ib:/home /home
-</screen>
- <para>ACLs are enabled in Lustre on a system-wide basis; either all clients enable ACLs or none do. Activating ACLs is controlled by MDS mount options acl / noacl (enable/disableACLs). Client-side mount options acl/noacl are ignored. You do not need to change the client configuration, and the 'acl' string will not appear in the client /etc/mtab. The client acl mount option is no longer needed. If a client is mounted with that option, then this message appears in the MDS syslog:</para>
- <screen>...MDS requires ACL support but client does not
-</screen>
- <para>The message is harmless but indicates a configuration issue, which should be corrected.</para>
- <para>If ACLs are not enabled on the MDS, then any attempts to reference an ACL on a client return an Operation not supported error.</para>
- </section>
- <section remap="h3">
- <title>22.1.3 <anchor xml:id="dbdoclet.50438221_marker-1292325" xreflabel=""/>Examples</title>
- <para>These examples are taken directly from the POSIX paper referenced above. ACLs on a Lustre file system work exactly like ACLs on any Linux file system. They are manipulated with the standard tools in the standard manner. Below, we create a directory and allow a specific user access.</para>
- <screen>[root@client lustre]# umask 027
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438221_16221">
+ <title>22.1 Using ACLs</title>
+ <para>An access control list (ACL), is a set of data that informs an operating system about permissions or access rights that each user or group has to specific system objects, such as directories or files. Each object has a unique security attribute that identifies users who have access to it. The ACL lists each object and user access privileges such as read, write or execute.</para>
+ <section remap="h3">
+ <title>22.1.1 How ACLs Work</title>
+ <para>Implementing ACLs varies between operating systems. Systems that support the Portable Operating System Interface (POSIX) family of standards share a simple yet powerful file system permission model, which should be well-known to the Linux/Unix administrator. ACLs add finer-grained permissions to this model, allowing for more complicated permission schemes. For a detailed explanation of ACLs on Linux, refer to the SuSE Labs article, <emphasis>Posix Access Control Lists on Linux</emphasis>:</para>
+ <para><ulink xl:href="http://www.suse.de/~agruen/acl/linux-acls/online/">http://www.suse.de/~agruen/acl/linux-acls/online/</ulink></para>
+ <para>We have implemented ACLs according to this model. Lustre works with the standard Linux ACL tools, setfacl, getfacl, and the historical chacl, normally installed with the ACL package.</para>
+ <note>
+ <para>ACL support is a system-range feature, meaning that all clients have ACL enabled or not. You cannot specify which clients should enable ACL.</para>
+ </note>
+ </section>
+ <section remap="h3">
+ <title>22.1.2 Using ACLs with Lustre</title>
+ <para>POSIX Access Control Lists (ACLs) can be used with Lustre. An ACL consists of file entries representing permissions based on standard POSIX file system object permissions that define three classes of user (owner, group and other). Each class is associated with a set of permissions [read (r), write (w) and execute (x)].</para>
+ <itemizedlist>
+ <listitem>
+ <para>Owner class permissions define access privileges of the file owner.</para>
+ </listitem>
+ <listitem>
+ <para>Group class permissions define access privileges of the owning group.</para>
+ </listitem>
+ <listitem>
+ <para>Other class permissions define access privileges of all users not in the owner or group class.</para>
+ </listitem>
+ </itemizedlist>
+ <para>The <literal>ls -l</literal> command displays the owner, group, and other class permissions in the first column of its output (for example, <literal>-rw-r- --</literal> for a regular file with read and write access for the owner class, read access for the group class, and no access for others).</para>
+ <para>Minimal ACLs have three entries. Extended ACLs have more than the three entries. Extended ACLs also contain a mask entry and may contain any number of named user and named group entries.</para>
+ <para>The MDS needs to be configured to enable ACLs. Use <literal>--mountfsoptions</literal> to enable ACLs when creating your configuration:</para>
+ <screen>$ mkfs.lustre --fsname spfs --mountfsoptions=acl --mdt -mgs /dev/sda</screen>
+ <para>Alternately, you can enable ACLs at run time by using the <literal>--acl</literal> option with <literal>mkfs.lustre</literal>:</para>
+ <screen>$ mount -t lustre -o acl /dev/sda /mnt/mdt</screen>
+ <para>To check ACLs on the MDS:</para>
+ <screen>$ lctl get_param -n mdc.home-MDT0000-mdc-*.connect_flags | grep acl acl</screen>
+ <para>To mount the client with no ACLs:</para>
+ <screen>$ mount -t lustre -o noacl ibmds2@o2ib:/home /home</screen>
+ <para>ACLs are enabled in Lustre on a system-wide basis; either all clients enable ACLs or none do. Activating ACLs is controlled by MDS mount options <literal>acl</literal> / <literal>noacl</literal> (enable/disableACLs). Client-side mount options acl/noacl are ignored. You do not need to change the client configuration, and the 'acl' string will not appear in the client /etc/mtab. The client acl mount option is no longer needed. If a client is mounted with that option, then this message appears in the MDS syslog:</para>
+ <screen>...MDS requires ACL support but client does not</screen>
+ <para>The message is harmless but indicates a configuration issue, which should be corrected.</para>
+ <para>If ACLs are not enabled on the MDS, then any attempts to reference an ACL on a client return an Operation not supported error.</para>
+ </section>
+ <section remap="h3">
+ <title>22.1.3 Examples</title>
+ <para>These examples are taken directly from the POSIX paper referenced above. ACLs on a Lustre file system work exactly like ACLs on any Linux file system. They are manipulated with the standard tools in the standard manner. Below, we create a directory and allow a specific user access.</para>
+ <screen>[root@client lustre]# umask 027
[root@client lustre]# mkdir rain
[root@client lustre]# ls -ld rain
drwxr-x--- 2 root root 4096 Feb 20 06:50 rain
user:chirag:rwx
group::r-x
mask::rwx
-other::---
-</screen>
- </section>
+other::---</screen>
</section>
- <section xml:id="dbdoclet.50438221_64726">
- <title>22.2 Using <anchor xml:id="dbdoclet.50438221_marker-1294644" xreflabel=""/>Root Squash</title>
- <para>Lustre 1.6 introduced root squash functionality, a security feature which controls super user access rights to an Lustre file system. Before the root squash feature was added, Lustre users could run rm -rf * as root, and remove data which should not be deleted. Using the root squash feature prevents this outcome.</para>
- <para>The root squash feature works by re-mapping the user ID (UID) and group ID (GID) of the root user to a UID and GID specified by the system administrator, via the Lustre configuration management server (MGS). The root squash feature also enables the Lustre administrator to specify a set of client for which UID/GID re-mapping does not apply.</para>
- <section remap="h3">
- <title>22.2.1 Configuring <anchor xml:id="dbdoclet.50438221_marker-1294610" xreflabel=""/>Root Squash</title>
- <para>Root squash functionality is managed by two configuration parameters, root_squash and nosquash_nids.</para>
- <itemizedlist><listitem>
- <para> The root_squash parameter specifies the UID and GID with which the root user accesses the Lustre file system.</para>
- </listitem>
-
-<listitem>
- <para> The nosquash_nids parameter specifies the set of clients to which root squash does not apply. LNET NID range syntax is used for this parameter (see the NID range syntax rules described in <xref linkend="dbdoclet.50438221_48757"/>). For example:</para>
- </listitem>
-
-</itemizedlist>
- <screen>nosquash_nids=172.16.245.[0-255/2]@tcp
-</screen>
- <para>In this example, root squash does not apply to TCP clients on subnet 172.16.245.0 that have an even number as the last component of their IP address.</para>
- </section>
- <section remap="h3">
- <title>22.2.2 <anchor xml:id="dbdoclet.50438221_48757" xreflabel=""/>Enabling and <anchor xml:id="dbdoclet.50438221_marker-1294611" xreflabel=""/>Tuning Root Squash</title>
- <para>The default value for nosquash_nids is NULL, which means that root squashing applies to all clients. Setting the root squash UID and GID to 0 turns root squash off.</para>
- <para>Root squash parameters can be set when the MDT is created (mkfs.lustre --mdt). For example:</para>
- <screen>mkfs.lustre --reformat --fsname=Lustre --mdt --mgs \
+ </section>
+ <section xml:id="dbdoclet.50438221_64726">
+ <title>22.2 Using Root Squash</title>
+ <para>Lustre 1.6 introduced root squash functionality, a security feature which controls super user access rights to an Lustre file system. Before the root squash feature was added, Lustre users could run <literal>rm -rf *</literal> as root, and remove data which should not be deleted. Using the root squash feature prevents this outcome.</para>
+ <para>The root squash feature works by re-mapping the user ID (UID) and group ID (GID) of the root user to a UID and GID specified by the system administrator, via the Lustre configuration management server (MGS). The root squash feature also enables the Lustre administrator to specify a set of client for which UID/GID re-mapping does not apply.</para>
+ <section remap="h3">
+ <title>22.2.1 Configuring Root Squash</title>
+ <para>Root squash functionality is managed by two configuration parameters, <literal>root_squash</literal> and <literal>nosquash_nids</literal>.</para>
+ <itemizedlist>
+ <listitem>
+ <para>The <literal>root_squash</literal> parameter specifies the UID and GID with which the root user accesses the Lustre file system.</para>
+ </listitem>
+ <listitem>
+ <para>The <literal>nosquash_nids</literal> parameter specifies the set of clients to which root squash does not apply. LNET NID range syntax is used for this parameter (see the NID range syntax rules described in <xref linkend="dbdoclet.50438221_48757"/>). For example:</para>
+ </listitem>
+ </itemizedlist>
+ <screen>nosquash_nids=172.16.245.[0-255/2]@tcp</screen>
+ <para>In this example, root squash does not apply to TCP clients on subnet 172.16.245.0 that have an even number as the last component of their IP address.</para>
+ </section>
+ <section xml:id="dbdoclet.50438221_48757">
+ <title >22.2.2 Enabling and Tuning Root Squash</title>
+ <para>The default value for <literal>nosquash_nids</literal> is NULL, which means that root squashing applies to all clients. Setting the root squash UID and GID to 0 turns root squash off.</para>
+ <para>Root squash parameters can be set when the MDT is created (<literal>mkfs.lustre --mdt</literal>). For example:</para>
+ <screen>mkfs.lustre --reformat --fsname=Lustre --mdt --mgs \
--param "mds.root_squash=500:501" \
- --param "mds.nosquash_nids='0@elan1 192.168.1.[10,11]'" /dev/sda1
-</screen>
- <para>Root squash parameters can also be changed on an unmounted device with tunefs.lustre. For example:</para>
- <screen>tunefs.lustre --param "mds.root_squash=65534:65534" \
+ --param "mds.nosquash_nids='0@elan1 192.168.1.[10,11]'" /dev/sda1</screen>
+ <para>Root squash parameters can also be changed on an unmounted device with <literal>tunefs.lustre</literal>. For example:</para>
+ <screen>tunefs.lustre --param "mds.root_squash=65534:65534" \
--param "mds.nosquash_nids=192.168.0.13@tcp0" /dev/sda1
</screen>
- <para>Root squash parameters can also be changed with the lctl conf_param command. For example:</para>
- <screen>lctl conf_param Lustre.mds.root_squash="1000:100"
-lctl conf_param Lustre.mds.nosquash_nids="*@tcp"
-</screen>
- <note><para>When using the lctl conf_param command, keep in mind:</para><para> * lctl conf_param must be run on a live MGS</para><para> * lctl conf_param causes the parameter to change on all MDSs</para><para> * lctl conf_param is to be used once per a parameter</para></note>
- <para>The nosquash_nids list can be cleared with:</para>
- <screen>lctl conf_param Lustre.mds.nosquash_nids="NONE"
-</screen>
- <para>- OR -</para>
- <screen>lctl conf_param Lustre.mds.nosquash_nids="clear"
-</screen>
- <para>If the nosquash_nids value consists of several NID ranges (e.g. 0@elan, 1@elan1), the list of NID ranges must be quoted with single (') or double ('') quotation marks. List elements must be separated with a space. For example:</para>
- <screen>mkfs.lustre ... --param "mds.nosquash_nids='0@elan1 1@elan2'" /dev/sda1
-lctl conf_param Lustre.mds.nosquash_nids="24@elan 15@elan1"
-</screen>
- <para>These are examples of incorrect syntax:</para>
- <screen>mkfs.lustre ... --param "mds.nosquash_nids=0@elan1 1@elan2" /dev/sda1
-lctl conf_param Lustre.mds.nosquash_nids=24@elan 15@elan1
-</screen>
- <para>To check root squash parameters, use the lctl get_param command:</para>
- <screen>lctl get_param mds.Lustre-MDT0000.root_squash
-lctl get_param mds.Lustre-MDT000*.nosquash_nids
-</screen>
- <note><para>An empty nosquash_nids list is reported as NONE.</para></note>
- </section>
- <section remap="h3">
- <title>22.2.3 Tips on Using <anchor xml:id="dbdoclet.50438221_marker-1296366" xreflabel=""/>Root Squash</title>
- <para>Lustre configuration management limits root squash in several ways.</para>
- <itemizedlist><listitem>
- <para> The lctl conf_param value overwrites the parameter's previous value. If the new value uses an incorrect syntax, then the system continues with the old parameters and the previously-correct value is lost on remount. That is, be careful doing root squash tuning.</para>
+ <para>Root squash parameters can also be changed with the <literal>lctl conf_param</literal> command. For example:</para>
+ <screen>lctl conf_param Lustre.mds.root_squash="1000:100"
+lctl conf_param Lustre.mds.nosquash_nids="*@tcp"</screen>
+ <note>
+ <para>When using the lctl conf_param command, keep in mind:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>lctl conf_param</literal> must be run on a live MGS</para>
</listitem>
-
-<listitem>
- <para>mkfs.lustre and tunefs.lustre do not perform syntax checking. If the root squash parameters are incorrect, they are ignored on mount and the default values are used instead.</para>
+ <listitem>
+ <para><literal>lctl conf_param</literal> causes the parameter to change on all MDSs</para>
</listitem>
-
-<listitem>
- <para> Root squash parameters are parsed with rigorous syntax checking. The root_squash parameter should be specified as <decnum>':'<decnum>. The nosquash_nids parameter should follow LNET NID range list syntax.</para>
+ <listitem>
+ <para><literal>lctl conf_param</literal> is to be used once per a parameter</para>
</listitem>
-
-</itemizedlist>
- <para>LNET NID range syntax:</para>
- <screen><nidlist> :== <nidrange> [ ' ' <nidrange> ]
+ </itemizedlist>
+ </note>
+ <para>The <literal>nosquash_nids</literal> list can be cleared with:</para>
+ <screen>lctl conf_param Lustre.mds.nosquash_nids="NONE"</screen>
+ <para>- OR -</para>
+ <screen>lctl conf_param Lustre.mds.nosquash_nids="clear"</screen>
+ <para>If the <literal>nosquash_nids</literal> value consists of several NID ranges (e.g. <literal>0@elan</literal>, <literal>1@elan1</literal>), the list of NID ranges must be quoted with single (') or double ('') quotation marks. List elements must be separated with a space. For example:</para>
+ <screen>mkfs.lustre ... --param "mds.nosquash_nids='0@elan1 1@elan2'" /dev/sda1
+lctl conf_param Lustre.mds.nosquash_nids="24@elan 15@elan1"</screen>
+ <para>These are examples of incorrect syntax:</para>
+ <screen>mkfs.lustre ... --param "mds.nosquash_nids=0@elan1 1@elan2" /dev/sda1
+lctl conf_param Lustre.mds.nosquash_nids=24@elan 15@elan1</screen>
+ <para>To check root squash parameters, use the lctl get_param command:</para>
+ <screen>lctl get_param mds.Lustre-MDT0000.root_squash
+lctl get_param mds.Lustre-MDT000*.nosquash_nids</screen>
+ <note>
+ <para>An empty nosquash_nids list is reported as NONE.</para>
+ </note>
+ </section>
+ <section remap="h3">
+ <title>22.2.3 Tips on Using Root Squash</title>
+ <para>Lustre configuration management limits root squash in several ways.</para>
+ <itemizedlist>
+ <listitem>
+ <para>The <literal>lctl conf_param</literal> value overwrites the parameter's previous value. If the new value uses an incorrect syntax, then the system continues with the old parameters and the previously-correct value is lost on remount. That is, be careful doing root squash tuning.</para>
+ </listitem>
+ <listitem>
+ <para><literal>mkfs.lustre</literal> and <literal>tunefs.lustre</literal> do not perform syntax checking. If the root squash parameters are incorrect, they are ignored on mount and the default values are used instead.</para>
+ </listitem>
+ <listitem>
+ <para>Root squash parameters are parsed with rigorous syntax checking. The root_squash parameter should be specified as <literal><decnum>':'<decnum></literal>. The <literal>nosquash_nids</literal> parameter should follow LNET NID range list syntax.</para>
+ </listitem>
+ </itemizedlist>
+ <para>LNET NID range syntax:</para>
+ <screen><nidlist> :== <nidrange> [ ' ' <nidrange> ]
<nidrange> :== <addrrange> '@' <net>
<addrrange> :== '*' |
<ipaddr_range> |
<net> :== <netname> | <netname><number>
<netname> :== "lo" | "tcp" | "o2ib" | "cib" | "openib" | "iib" |
"vib" | "ra" | "elan" | "gm" | "mx" | "ptl"
-<number> :== <nonnegative decimal> | <hexadecimal>
-</screen>
- <note><para>For networks using numeric addresses (e.g. elan), the address range must be specified in the <numaddr_range> syntax. For networks using IP addresses, the address range must be in the <ipaddr_range>. For example, if elan is using numeric addresses, 1.2.3.4@elan is incorrect.</para></note>
- </section>
+<number> :== <nonnegative decimal> | <hexadecimal></screen>
+ <note>
+ <para>For networks using numeric addresses (e.g. elan), the address range must be specified in the <literal><numaddr_range></literal> syntax. For networks using IP addresses, the address range must be in the <literal><ipaddr_range></literal>. For example, if elan is using numeric addresses, <literal>1.2.3.4@elan</literal> is incorrect.</para>
+ </note>
</section>
+ </section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='managingstripingfreespace'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. -->
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingstripingfreespace">
<info>
- <title xml:id='managingstripingfreespace.title'>Managing File Striping and Free Space</title>
+ <title xml:id="managingstripingfreespace.title">Managing File Striping and Free Space</title>
</info>
-
<para>This chapter describes file striping and I/O options, and includes the following sections:</para>
-
- <itemizedlist><listitem>
+ <itemizedlist>
+ <listitem>
<para><xref linkend="dbdoclet.50438209_79324"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438209_48033"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438209_78664"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438209_44776"/></para>
</listitem>
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438209_10424"/></para>
</listitem>
-</itemizedlist>
-
- <section xml:id="dbdoclet.50438209_79324">
- <title>18.1 How Lustre Striping Works</title>
- <para>Lustre uses a round-robin algorithm for selecting the next OST to which a stripe is to be written. Normally the usage of OSTs is well balanced. However, if users create a small number of exceptionally large files or incorrectly specify striping parameters, imbalanced OST usage may result.</para>
- <para>The MDS allocates objects on seqential OSTs. Periodically, it will adjust the striping layout to eliminate some degenerated cases where applications that create very regular file layouts (striping patterns) would preferentially use a particular OST in the sequence.</para>
- <para>Stripes are written to sequential OSTs until free space across the OSTs differs by more than 20%. The MDS will then use weighted random allocations with a preference for allocating objects on OSTs with more free space. This can reduce I/O performance until space usage is rebalanced to within 20% again.</para>
- <para>For a more detailed description of stripe assignments, see <xref linkend='dbdoclet.50438209_10424'/>.</para>
- </section>
- <section xml:id="dbdoclet.50438209_48033">
- <title>18.2 Lustre File <anchor xml:id="dbdoclet.50438209_marker-1291832" xreflabel=""/>Striping Considerations</title>
- <para>Whether you should set up file striping and what parameter values you select depends on your need. A good rule of thumb is to stripe over as few objects as will meet those needs and no more.</para>
- <para>Some reasons for using striping include:</para>
- <itemizedlist><listitem>
- <para><emphasis role="bold">Providing high-bandwidth access</emphasis> - Many applications require high-bandwidth access to a single file - more bandwidth than can be provided by a single OSS. For example, scientific applications that write to a single file from hundreds of nodes, or a binary executable that is loaded by many nodes when an application starts.</para>
-
- <para>In cases like these, a file can be striped over as many OSSs as it takes to achieve the required peak aggregate bandwidth for that file. Striping across a larger number of OSSs should only be used when the file size is very large and/or is accessed by many nodes at a time. Currently, Lustre files can be striped across up to 160 OSSs, the maximum stripe count for an ldiskfs file system.</para>
- </listitem><listitem>
- <para><emphasis role="bold">Improving performance when OSS bandwidth is exceeded</emphasis> - Striping across many OSSs can improve performance if the aggregate client bandwidth exceeds the server bandwidth and the application reads and writes data fast enough to take advantage of the additional OSS bandwidth. The largest useful stripe count is bounded by the I/O rate of the clients/jobs divided by the performance per OSS.</para>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438209_79324">
+ <title>18.1 How Lustre Striping Works</title>
+ <para>Lustre uses a round-robin algorithm for selecting the next OST to which a stripe is to be written. Normally the usage of OSTs is well balanced. However, if users create a small number of exceptionally large files or incorrectly specify striping parameters, imbalanced OST usage may result.</para>
+ <para>The MDS allocates objects on seqential OSTs. Periodically, it will adjust the striping layout to eliminate some degenerated cases where applications that create very regular file layouts (striping patterns) would preferentially use a particular OST in the sequence.</para>
+ <para>Stripes are written to sequential OSTs until free space across the OSTs differs by more than 20%. The MDS will then use weighted random allocations with a preference for allocating objects on OSTs with more free space. This can reduce I/O performance until space usage is rebalanced to within 20% again.</para>
+ <para>For a more detailed description of stripe assignments, see <xref linkend="dbdoclet.50438209_10424"/>.</para>
+ </section>
+ <section xml:id="dbdoclet.50438209_48033">
+ <title>18.2 Lustre File <anchor xml:id="dbdoclet.50438209_marker-1291832" xreflabel=""/>Striping Considerations</title>
+ <para>Whether you should set up file striping and what parameter values you select depends on your need. A good rule of thumb is to stripe over as few objects as will meet those needs and no more.</para>
+ <para>Some reasons for using striping include:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">Providing high-bandwidth access</emphasis> - Many applications require high-bandwidth access to a single file - more bandwidth than can be provided by a single OSS. For example, scientific applications that write to a single file from hundreds of nodes, or a binary executable that is loaded by many nodes when an application starts.</para>
+ <para>In cases like these, a file can be striped over as many OSSs as it takes to achieve the required peak aggregate bandwidth for that file. Striping across a larger number of OSSs should only be used when the file size is very large and/or is accessed by many nodes at a time. Currently, Lustre files can be striped across up to 160 OSSs, the maximum stripe count for an ldiskfs file system.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Improving performance when OSS bandwidth is exceeded</emphasis> - Striping across many OSSs can improve performance if the aggregate client bandwidth exceeds the server bandwidth and the application reads and writes data fast enough to take advantage of the additional OSS bandwidth. The largest useful stripe count is bounded by the I/O rate of the clients/jobs divided by the performance per OSS.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Providing space for very large files.</emphasis> Striping is also useful when a single OST does not have enough free space to hold the entire file.</para>
+ </listitem>
+ </itemizedlist>
+ <para>Some reasons to minimize or avoid striping:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">Increased overhead</emphasis> - Striping results in more locks and extra network operations during common operations such as stat and unlink. Even when these operations are performed in parallel, one network operation takes less time than 100 operations.</para>
+ <para>Increased overhead also results from server contention. Consider a cluster with 100 clients and 100 OSSs, each with one OST. If each file has exactly one object and the load is distributed evenly, there is no contention and the disks on each server can manage sequential I/O. If each file has 100 objects, then the clients all compete with one another for the attention of the servers, and the disks on each node seek in 100 different directions. In this case, there is needless contention.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Increased risk</emphasis> - When a file is striped across all servers and one of the servers breaks down, a small part of each striped file is lost. By comparison, if each file has exactly one stripe, you lose fewer files, but you lose them in their entirety. Many users would prefer to lose some of their files entirely than all of their files partially.</para>
+ </listitem>
+ </itemizedlist>
+ <section remap="h3">
+ <title>18.2.1 Choosing a Stripe Size</title>
+ <para>Choosing a stripe size is a small balancing act, but there are reasonable defaults.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">The stripe size must be a multiple of the page size.</emphasis> Lustre's tools enforce a multiple of 64 KB (the maximum page size on ia64 and PPC64 nodes) so that users on platforms with smaller pages do not accidentally create files that might cause problems for ia64 clients.</para>
</listitem>
-<listitem>
- <para><emphasis role="bold">Providing space for very large files.</emphasis> Striping is also useful when a single OST does not have enough free space to hold the entire file.</para>
+ <listitem>
+ <para><emphasis role="bold">The smallest recommended stripe size is 512 KB.</emphasis> Although you can create files with a stripe size of 64 KB, the smallest practical stripe size is 512 KB because Lustre sends 1MB chunks over the network. Choosing a smaller stripe size may result in inefficient I/O to the disks and reduced performance.</para>
</listitem>
-</itemizedlist>
- <para>Some reasons to minimize or avoid striping:</para>
- <itemizedlist><listitem>
- <para><emphasis role="bold">Increased overhead</emphasis> - Striping results in more locks and extra network operations during common operations such as stat and unlink. Even when these operations are performed in parallel, one network operation takes less time than 100 operations.</para>
- <para>Increased overhead also results from server contention. Consider a cluster with 100 clients and 100 OSSs, each with one OST. If each file has exactly one object and the load is distributed evenly, there is no contention and the disks on each server can manage sequential I/O. If each file has 100 objects, then the clients all compete with one another for the attention of the servers, and the disks on each node seek in 100 different directions. In this case, there is needless contention.</para>
- </listitem><listitem>
- <para><emphasis role="bold">Increased risk</emphasis> - When a file is striped across all servers and one of the servers breaks down, a small part of each striped file is lost. By comparison, if each file has exactly one stripe, you lose fewer files, but you lose them in their entirety. Many users would prefer to lose some of their files entirely than all of their files partially.</para>
+ <listitem>
+ <para><emphasis role="bold">A good stripe size for sequential I/O using high-speed networks is between 1 MB and 4 MB.</emphasis> In most situations, stripe sizes larger than 4 MB may result in longer lock hold times and contention on shared file access.</para>
</listitem>
-</itemizedlist>
- <section remap="h3">
- <title>18.2.1 Choosing a Stripe <anchor xml:id="dbdoclet.50438209_marker-1291859" xreflabel=""/>Size</title>
- <para>Choosing a stripe size is a small balancing act, but there are reasonable defaults.</para>
- <itemizedlist><listitem>
- <para><emphasis role="bold">The stripe size must be a multiple of the page size.</emphasis> Lustre's tools enforce a multiple of 64 KB (the maximum page size on ia64 and PPC64 nodes) so that users on platforms with smaller pages do not accidentally create files that might cause problems for ia64 clients.</para>
- </listitem>
-<listitem>
- <para><emphasis role="bold">The smallest recommended stripe size is 512 KB.</emphasis> Although you can create files with a stripe size of 64 KB, the smallest practical stripe size is 512 KB because Lustre sends 1MB chunks over the network. Choosing a smaller stripe size may result in inefficient I/O to the disks and reduced performance.</para>
- </listitem>
-<listitem>
- <para><emphasis role="bold">A good stripe size for sequential I/O using high-speed networks is between 1 MB and 4 MB.</emphasis> In most situations, stripe sizes larger than 4 MB may result in longer lock hold times and contention on shared file access.</para>
- </listitem>
-<listitem>
- <para><emphasis role="bold">The maximum stripe size is 4GB.</emphasis> Using a large stripe size can improve performance when accessing very large files. It allows each client to have exclusive access to its own part of a file. However, it can be counterproductive in some cases if it does not match your I/O pattern.</para>
- </listitem>
-<listitem>
- <para><emphasis role="bold">Choose a stripe pattern that takes into account your application's write patterns.</emphasis> Writes that cross an object boundary are slightly less efficient than writes that go entirely to one server. If the file is written in a very consistent and aligned way, make the stripe size a multiple of the write() size.</para>
- </listitem>
-<listitem>
- <para><emphasis role="bold">The choice of stripe size has no effect on a single-stripe file.</emphasis></para>
- </listitem>
-</itemizedlist>
- </section>
+ <listitem>
+ <para><emphasis role="bold">The maximum stripe size is 4GB.</emphasis> Using a large stripe size can improve performance when accessing very large files. It allows each client to have exclusive access to its own part of a file. However, it can be counterproductive in some cases if it does not match your I/O pattern.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Choose a stripe pattern that takes into account your application's write patterns.</emphasis> Writes that cross an object boundary are slightly less efficient than writes that go entirely to one server. If the file is written in a very consistent and aligned way, make the stripe size a multiple of the write() size.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">The choice of stripe size has no effect on a single-stripe file.</emphasis></para>
+ </listitem>
+ </itemizedlist>
</section>
- <section xml:id="dbdoclet.50438209_78664">
- <title>18.3 Setting the File Layout/Striping Configuration (lfs setstripe)</title>
- <para>Use the lfs setstripe command to create new files with a specific file layout (stripe pattern) configuration.</para>
- <screen>lfs setstripe [--size|-s stripe_size] [--count|-c stripe_count]
-[--index|-i start_ost] [--pool|-p pool_name] <filename|dirname>
-</screen>
- <para><emphasis role="bold">stripe_size</emphasis></para>
- <para>The stripe size indicates how much data to write to one OST before moving to the next OST. The default stripe_size is 1 MB, and passing a stripe_size of 0 causes the default stripe size to be used. Otherwise, the stripe_size value must be a multiple of 64 KB.</para>
- <para><emphasis role="bold">stripe_count</emphasis></para>
- <para>The stripe count indicates how many OSTs to use. The default stripe_count value is 1. Setting stripe_count to 0 causes the default stripe count to be used. Setting stripe_count to -1 means stripe over all available OSTs (full OSTs are skipped).</para>
- <para><emphasis role="bold">start_ost</emphasis></para>
- <para>The start OST is the first OST to which files are written. The default value for start_ost is -1 , which allows the MDS to choose the starting index. This setting is strongly recommended, as it allows space and load balancing to be done by the MDS as needed. Otherwise, the file starts on the specified OST index. The numbering of the OSTs starts at 0.</para>
- <note><para>If you pass a start_ost value of 0 and a stripe_count value of <emphasis>1</emphasis>, all files are written to OST 0, until space is exhausted. This is probably not what you meant to do. If you only want to adjust the stripe count and keep the other parameters at their default settings, do not specify any of the other parameters:</para><para>lfs setstripe -c <stripe_count> <file></para></note>
- <para><emphasis role="bold">pool_name</emphasis></para>
- <para>Specify the OST pool on which the file will be written. This allows limiting the OSTs used to a subset of all OSTs in the file system. For more details about using OST pools, see <link xl:href="ManagingFileSystemIO.html#50438211_75549">Creating and Managing OST Pools</link>.</para>
- <section remap="h3">
- <title>18.3.1 <anchor xml:id="dbdoclet.50438209_48100" xreflabel=""/>Using a Specific Striping Pattern/File Layout for a Single File</title>
- <para>It is possible to specify the file layout when a new file is created using the command lfssetstripe. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an lfssetstripe command fails if the file already exists.</para>
- <section remap="h4">
- <title>18.3.1.1 <anchor xml:id="dbdoclet.50438209_60155" xreflabel=""/>Setting the Stripe Size</title>
- <para>The command to create a new file with a specified stripe size is similar to:</para>
- <screen>[client]# lfs setstripe -s 4M /mnt/lustre/new_file
-</screen>
- <para>This example command creates the new file /mnt/lustre/new_file with a stripe size of 4 MB.</para>
- <para>Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:</para>
- <screen>
-[client]# lfs getstripe /mnt/lustre/new_file
+ </section>
+ <section xml:id="dbdoclet.50438209_78664">
+ <title>18.3 Setting the File Layout/Striping Configuration (<literal>lfs setstripe</literal>)</title>
+ <para>Use the <literal>lfs setstripe</literal> command to create new files with a specific file layout (stripe pattern) configuration.</para>
+ <screen>lfs setstripe [--size|-s stripe_size] [--count|-c stripe_count]
+[--index|-i start_ost] [--pool|-p pool_name] <filename|dirname> </screen>
+ <para><literal>
+ <emphasis role="bold">stripe_size</emphasis>
+ </literal></para>
+ <para>The <literal>stripe_size</literal> indicates how much data to write to one OST before moving to the next OST. The default <literal>stripe_size</literal> is 1 MB, and passing a stripe_size of 0 causes the default stripe size to be used. Otherwise, the <literal>stripe_size</literal> value must be a multiple of 64 KB.</para>
+ <para><literal>
+ <emphasis role="bold">stripe_count</emphasis>
+ </literal></para>
+ <para>The <literal>stripe_count</literal> indicates how many OSTs to use. The default <literal>stripe_count</literal> value is 1. Setting <literal>stripe_count</literal> to 0 causes the default stripe count to be used. Setting <literal>stripe_count</literal> to -1 means stripe over all available OSTs (full OSTs are skipped).</para>
+ <para><literal>
+ <emphasis role="bold">start_ost</emphasis>
+ </literal></para>
+ <para>The start OST is the first OST to which files are written. The default value for <literal>start_ost</literal> is -1, which allows the MDS to choose the starting index. This setting is strongly recommended, as it allows space and load balancing to be done by the MDS as needed. Otherwise, the file starts on the specified OST index. The numbering of the OSTs starts at 0.</para>
+ <note>
+ <para>If you pass a <literal>start_ost</literal> value of 0 and a <literal>stripe_count</literal> value of <emphasis>1</emphasis>, all files are written to OST 0, until space is exhausted. This is probably not what you meant to do. If you only want to adjust the stripe count and keep the other parameters at their default settings, do not specify any of the other parameters:</para>
+ <para><screen>lfs setstripe -c <stripe_count> <file></screen></para>
+ </note>
+ <para><literal>
+ <emphasis role="bold">pool_name</emphasis>
+ </literal></para>
+ <para>Specify the OST pool on which the file will be written. This allows limiting the OSTs used to a subset of all OSTs in the file system. For more details about using OST pools, see <link xl:href="ManagingFileSystemIO.html#50438211_75549">Creating and Managing OST Pools</link>.</para>
+ <section remap="h3">
+ <title>18.3.1 Using a Specific Striping Pattern/File Layout for a Single File</title>
+ <para>It is possible to specify the file layout when a new file is created using the command <literal>lfs setstripe</literal>. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an <literal>lfs setstripe</literal> command fails if the file already exists.</para>
+ <section xml:id="dbdoclet.50438209_60155">
+ <title>18.3.1.1 Setting the Stripe Size</title>
+ <para>The command to create a new file with a specified stripe size is similar to:</para>
+ <screen>[client]# lfs setstripe -s 4M /mnt/lustre/new_file</screen>
+ <para>This example command creates the new file <literal>/mnt/lustre/new_file</literal> with a stripe size of 4 MB.</para>
+ <para>Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:</para>
+ <screen> [client]# lfs getstripe /mnt/lustre/new_file
/mnt/lustre/4mb_file
lmm_stripe_count: 1
lmm_stripe_size: 4194304
lmm_stripe_offset: 1
obdidx objid objid group
-1 690550 0xa8976 0
-</screen>
- <para>As can be seen, the stripe size is 4 MB.</para>
- </section>
- <section remap="h4">
- <title>18.3.1.2 Setting the Stripe Count</title>
- <para>The command below creates a new file with a stripe count of -1 to specify striping over all available OSTs:</para>
- <screen>[client]# lfs setstripe -c -1 /mnt/lustre/full_stripe
-</screen>
- <para>The example below indicates that the file full_stripe is striped over all six active OSTs in the configuration:</para>
- <screen>[client]# lfs getstripe /mnt/lustre/full_stripe
+1 690550 0xa8976 0 </screen>
+ <para>As can be seen, the stripe size is 4 MB.</para>
+ </section>
+ <section remap="h4">
+ <title>18.3.1.2 Setting the Stripe Count</title>
+ <para>The command below creates a new file with a stripe count of -1 to specify striping over all available OSTs:</para>
+ <screen>[client]# lfs setstripe -c -1 /mnt/lustre/full_stripe</screen>
+ <para>The example below indicates that the file full_stripe is striped over all six active OSTs in the configuration:</para>
+ <screen>[client]# lfs getstripe /mnt/lustre/full_stripe
/mnt/lustre/full_stripe
obdidx objid objid group
0 8 0x8 0
2 5 0x5 0
3 5 0x5 0
4 4 0x4 0
-5 2 0x2 0
-</screen>
-<para> This is in contrast to the output in <xref linkend='dbdoclet.50438209_60155'/> that shows only a single object for the file.</para>
- </section>
+5 2 0x2 0</screen>
+ <para> This is in contrast to the output in <xref linkend="dbdoclet.50438209_60155"/> that shows only a single object for the file.</para>
</section>
- <section remap="h3">
- <title>18.3.2 Changing Striping for a Directory</title>
- <para>In a directory, the lfssetstripe command sets a default striping configuration for files created in the directory. The usage is the same as lfssetstripe for a regular file, except that the directory must exist prior to setting the default striping configuration. If a file is created in a directory with a default stripe configuration (without otherwise specifying striping), Lustre uses those striping parameters instead of the file system default for the new file.</para>
- <para>To change the striping pattern (file layout) for a sub-directory, create a directory with desired file layout as described above. Sub-directories inherit the file layout of the root/parent directory.</para>
- </section>
- <section remap="h3">
- <title>18.3.3 Changing Striping for a File System</title>
- <para>Change the striping on the file system root will change the striping for all newly created files that would otherwise have a striping parameter from the parent directory or explicitly on the command line.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis>Striping of new files and sub-directories is done per the striping parameter settings of the root directory. Once you set striping on the root directory, then, by default, it applies to any new child directories created in that root directory (unless they have their own striping settings).</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h3">
- <title>18.3.4 Creating a File on a Specific OST</title>
- <para>You can use lfs setstripe to create a file on a specific OST. In the following example, the file "bob" will be created on the first OST (id 0).</para>
- <screen>$ lfs setstripe --count 1 --index 0 bob
+ </section>
+ <section remap="h3">
+ <title>18.3.2 Changing Striping for a Directory</title>
+ <para>In a directory, the <literal>lfs setstripe</literal> command sets a default striping configuration for files created in the directory. The usage is the same as <literal>lfs setstripe</literal> for a regular file, except that the directory must exist prior to setting the default striping configuration. If a file is created in a directory with a default stripe configuration (without otherwise specifying striping), Lustre uses those striping parameters instead of the file system default for the new file.</para>
+ <para>To change the striping pattern (file layout) for a sub-directory, create a directory with desired file layout as described above. Sub-directories inherit the file layout of the root/parent directory.</para>
+ </section>
+ <section remap="h3">
+ <title>18.3.3 Changing Striping for a File System</title>
+ <para>Change the striping on the file system root will change the striping for all newly created files that would otherwise have a striping parameter from the parent directory or explicitly on the command line.</para>
+ <note>
+ <para>Striping of new files and sub-directories is done per the striping parameter settings of the root directory. Once you set striping on the root directory, then, by default, it applies to any new child directories created in that root directory (unless they have their own striping settings).</para>
+ </note>
+ </section>
+ <section remap="h3">
+ <title>18.3.4 Creating a File on a Specific OST</title>
+ <para>You can use <literal>lfs setstripe</literal> to create a file on a specific OST. In the following example, the file "<literal>bob</literal>" will be created on the first OST (id 0).</para>
+ <screen>$ lfs setstripe --count 1 --index 0 bob
$ dd if=/dev/zero of=bob count=1 bs=100M
1+0 records in
1+0 records out
-$ lfs getstripe bob
-</screen>
- <para>OBDS:</para>
- <screen>0: home-OST0000_UUID ACTIVE
+$ lfs getstripe bob</screen>
+ <para>OBDS:</para>
+ <screen>0: home-OST0000_UUID ACTIVE
[...]
bob
obdidx objid objid group
- 0 33459243 0x1fe8c2b 0
-</screen>
- </section>
+ 0 33459243 0x1fe8c2b 0</screen>
</section>
- <section xml:id="dbdoclet.50438209_44776">
- <title>18.4 Retrieving File Layout/Striping Information (getstripe)</title>
- <para>The lfsgetstripe command is used to display information that shows over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along with the OST index and object ID for each stripe in the file. For directories, the default settings for files created in that directory are printed.</para>
- <section remap="h3">
- <title>18.4.1 Displaying the Current Stripe Size</title>
- <para>To see the current stripe size, use the lfsgetstripe command on a Lustre file or directory. For example:</para>
- <screen>[client]# lfs getstripe /mnt/lustre
-</screen>
- <para>This command produces output similar to this:</para>
- <screen>/mnt/lustre
-(Default) stripe_count: 1 stripe_size: 1M stripe_offset: -1
-</screen>
- <para>In this example, the default stripe count is 1 (data blocks are striped over a single OSTs), the default stripe size is 1 MB, and objects are created over all available OSTs.</para>
- </section>
- <section remap="h3">
- <title>18.4.2 Inspecting the File Tree</title>
- <para>To inspect an entire tree of files, use the lfs find command:</para>
- <screen>lfs find [--recursive | -r] <file or directory> ...
-</screen>
- <para>You can also use ls -l /proc/<emphasis><pid></emphasis>/fd/ to find open files using Lustre. For example:</para>
- <screen>$ lfs getstripe $(readlink /proc/$(pidof cat)/fd/1)
-</screen>
- <para>Typical output is:</para>
- <screen>/mnt/lustre/foo
+ </section>
+ <section xml:id="dbdoclet.50438209_44776">
+ <title>18.4 Retrieving File Layout/Striping Information (<literal>getstripe</literal>)</title>
+ <para>The <literal>lfs getstripe</literal> command is used to display information that shows over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along with the OST index and object ID for each stripe in the file. For directories, the default settings for files created in that directory are printed.</para>
+ <section remap="h3">
+ <title>18.4.1 Displaying the Current Stripe Size</title>
+ <para>To see the current stripe size, use the <literal>lfs getstripe</literal> command on a Lustre file or directory. For example:</para>
+ <screen>[client]# lfs getstripe /mnt/lustre </screen>
+ <para>This command produces output similar to this:</para>
+ <screen>/mnt/lustre
+(Default) stripe_count: 1 stripe_size: 1M stripe_offset: -1</screen>
+ <para>In this example, the default stripe count is 1 (data blocks are striped over a single OSTs), the default stripe size is 1 MB, and objects are created over all available OSTs.</para>
+ </section>
+ <section remap="h3">
+ <title>18.4.2 Inspecting the File Tree</title>
+ <para>To inspect an entire tree of files, use the <literal>lfs</literal> find command:</para>
+ <screen>lfs find [--recursive | -r] <file or directory> ...</screen>
+ <para>You can also use <literal>ls -l /proc/<emphasis><pid></emphasis>/fd/</literal> to find open files using Lustre. For example:</para>
+ <screen>$ lfs getstripe $(readlink /proc/$(pidof cat)/fd/1)</screen>
+ <para>Typical output is:</para>
+ <screen>/mnt/lustre/foo
obdidx objid objid \
group
-2 835487 0xcbf9f 0
-</screen>
- <para>In this example, the file lives on obdidx 2, which is lustre-OST0002. To see which node is serving that OST, run:</para>
- <screen>$ lctl get_param osc.lustre-OST0002-osc.ost_conn_uuid
-</screen>
- <para>Typical output is:</para>
- <screen>osc.lustre-OST0002-osc.ost_conn_uuid=192.168.20.1@tcp
-</screen>
- </section>
+2 835487 0xcbf9f 0</screen>
+ <para>In this example, the file lives on <literal>obdidx</literal><literal> 2</literal>, which is <literal>lustre-OST0002</literal>. To see which node is serving that OST, run:</para>
+ <screen>$ lctl get_param osc.lustre-OST0002-osc.ost_conn_uuid</screen>
+ <para>Typical output is:</para>
+ <screen>osc.lustre-OST0002-osc.ost_conn_uuid=192.168.20.1@tcp</screen>
</section>
- <section xml:id="dbdoclet.50438209_10424">
- <title>18.5 Managing Free <anchor xml:id="dbdoclet.50438209_marker-1295520" xreflabel=""/>Space</title>
- <para>The MDT assigns file stripes to OSTs based on location (which OSS) and size considerations (free space) to optimize file system performance. Emptier OSTs are preferentially selected for stripes, and stripes are preferentially spread out between OSSs to increase network bandwidth utilization. The weighting factor between these two optimizations can be adjusted by the user.</para>
- <section remap="h3">
- <title>18.5.1 <anchor xml:id="dbdoclet.50438209_35838" xreflabel=""/>Checking File System Free Space</title>
- <para>Free space is an important consideration in assigning file stripes. The lfsdf command shows available disk space on the mounted Lustre file system and space consumption per OST. If multiple Lustre file systems are mounted, a path may be specified, but is not required.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">-h</emphasis></para></entry>
- <entry><para> Human-readable print sizes in human readable format (for example: 1K, 234M, 5G).</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">-i, --inodes</emphasis></para></entry>
- <entry><para> Lists inodes instead of block usage.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis>The df-i and lfsdf-i commands show the minimum number of inodes that can be created in the file system. Depending on the configuration, it may be possible to create more inodes than initially reported by df-i. Later, df-i operations will show the current, estimated free inode count.</para><para> If the underlying file system has fewer free blocks than inodes, then the total inode count for the file system reports only as many inodes as there are free blocks. This is done because Lustre may need to store an external attribute for each new inode, and it is better to report a free inode count that is the guaranteed, minimum number of inodes that can be created.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="bold">Examples</emphasis></para>
- <screen>[lin-cli1] $ lfs df
+ </section>
+ <section xml:id="dbdoclet.50438209_10424">
+ <title>18.5 Managing Free Space</title>
+ <para>The MDT assigns file stripes to OSTs based on location (which OSS) and size considerations (free space) to optimize file system performance. Emptier OSTs are preferentially selected for stripes, and stripes are preferentially spread out between OSSs to increase network bandwidth utilization. The weighting factor between these two optimizations can be adjusted by the user.</para>
+ <section xml:id='dbdoclet.50438209_35838'>
+ <title>18.5.1 Checking File System Free Space</title>
+ <para>Free space is an important consideration in assigning file stripes. The <literal>lfs df</literal> command shows available disk space on the mounted Lustre file system and space consumption per OST. If multiple Lustre file systems are mounted, a path may be specified, but is not required.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">
+ <literal>-h</literal>
+ </emphasis></para>
+ </entry>
+ <entry>
+ <para> Human-readable print sizes in human readable format (for example: 1K, 234M, 5G).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>
+ <emphasis role="bold">-i, --inodes</emphasis>
+ </literal></para>
+ </entry>
+ <entry>
+ <para> Lists inodes instead of block usage.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <note>
+ <para>The <literal>df -i</literal> and <literal>lfs df -i</literal> commands show the minimum number of inodes that can be created in the file system. Depending on the configuration, it may be possible to create more inodes than initially reported by <literal>df -i</literal>. Later, <literal>df -i</literal> operations will show the current, estimated free inode count.</para>
+ <para> If the underlying file system has fewer free blocks than inodes, then the total inode count for the file system reports only as many inodes as there are free blocks. This is done because Lustre may need to store an external attribute for each new inode, and it is better to report a free inode count that is the guaranteed, minimum number of inodes that can be created.</para>
+ </note>
+ <para><emphasis role="bold">Examples</emphasis></para>
+ <screen>[lin-cli1] $ lfs df
UUID 1K-blockS Used \
Available Use% Mounted on
mds-lustre-0_UUID 9174328 1020024 8154304 \
ost-lustre-2_UUID 737280 12214 725066 \
1% /mnt/lustre[OST:2]
filesystem summary: 2211572 41924 \
-2169648 1% /mnt/lustre[OST:2]
-</screen>
- </section>
- <section remap="h3">
- <title>18.5.2 Using Stripe Allocations</title>
- <para>Two stripe allocation methods are provided: <emphasis>round-robin</emphasis> and <emphasis>weighted</emphasis>. By default, the allocation method is determined by the amount of free-space imbalance on the OSTs. The weighted allocator is used when any two OSTs are imbalanced by more than 20%. Otherwise, the faster round-robin allocator is used. (The round-robin order maximizes network balancing.)</para>
- <itemizedlist><listitem>
- <para><emphasis role="bold">Round-robin allocator</emphasis><anchor xml:id="dbdoclet.50438209_marker-1293988" xreflabel=""/> - When OSTs have approximately the same amount of free space (within 20%), an efficient round-robin allocator is used. The round-robin allocator alternates stripes between OSTs on different OSSs, so the OST used for stripe 0 of each file is evenly distributed among OSTs, regardless of the stripe count. Here are several sample round-robin stripe orders (each letter represents a different OST on a single OSS):</para>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <tbody>
- <row>
- <entry><para> 3: AAA</para></entry>
- <entry><para> One 3-OST OSS</para></entry>
- </row>
- <row>
- <entry><para> 3x3: ABABAB</para></entry>
- <entry><para> Two 3-OST OSSs</para></entry>
- </row>
- <row>
- <entry><para> 3x4: BBABABA</para></entry>
- <entry><para> One 3-OST OSS (A) and one 4-OST OSS (B)</para></entry>
- </row>
- <row>
- <entry><para> 3x5: BBABBABA</para></entry>
- <entry><para> One 3-OST OSS (A) and one 5-OST OSS (B)</para></entry>
- </row>
- <row>
- <entry><para> 3x3x3: ABCABCABC</para></entry>
- <entry><para> Three 3-OST OSSs</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </listitem>
-<listitem>
- <para><emphasis role="bold">Weighted allocator</emphasis><anchor xml:id="dbdoclet.50438209_marker-1294020" xreflabel=""/> - When the free space difference between the OSTs is significant (by default, 20% of the free space), then a weighting algorithm is used to influence OST ordering based on size and location. Note that these are weightings for a random algorithm, so the OST with the most free space is not necessarily chosen each time. On average, the weighted allocator fills the emptier OSTs faster.</para>
- </listitem>
-</itemizedlist>
- </section>
- <section remap="h3">
- <title>18.5.3 Adjusting the <anchor xml:id="dbdoclet.50438209_marker-1294033" xreflabel=""/>Weighting Between Free Space and Location</title>
- <para>The weighting priority can be adjusted in the /proc file /proc/fs/lustre/lov/lustre-mdtlov/qos_prio_free proc. The default value is 90%. Use this command on the MGS to permanently change this weighting:</para>
- <screen>lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90
-</screen>
- <para>Increasing this value puts more weighting on free space. When the free space priority is set to 100%, then location is no longer used in stripe-ordering calculations and weighting is based entirely on free space.</para>
- <note><para>Setting the priority to 100% means that OSS distribution does not count in the weighting, but the stripe assignment is still done via a weighting. For example, if OST2 has twice as much free space as OST1, then OST2 is twice as likely to be used, but it is not guaranteed to be used.</para></note>
- </section>
+2169648 1% /mnt/lustre[OST:2]</screen>
+ </section>
+ <section remap="h3">
+ <title>18.5.2 Using Stripe Allocations</title>
+ <para>Two stripe allocation methods are provided: <emphasis>round-robin</emphasis> and <emphasis>weighted</emphasis>. By default, the allocation method is determined by the amount of free-space imbalance on the OSTs. The weighted allocator is used when any two OSTs are imbalanced by more than 20%. Otherwise, the faster round-robin allocator is used. (The round-robin order maximizes network balancing.)</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">Round-robin allocator</emphasis> - When OSTs have approximately the same amount of free space (within 20%), an efficient round-robin allocator is used. The round-robin allocator alternates stripes between OSTs on different OSSs, so the OST used for stripe 0 of each file is evenly distributed among OSTs, regardless of the stripe count. Here are several sample round-robin stripe orders (each letter represents a different OST on a single OSS):</para>
+ <informaltable frame="none">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <tbody>
+ <row>
+ <entry>
+ <para> 3: AAA</para>
+ </entry>
+ <entry>
+ <para> One 3-OST OSS</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> 3x3: ABABAB</para>
+ </entry>
+ <entry>
+ <para> Two 3-OST OSSs</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> 3x4: BBABABA</para>
+ </entry>
+ <entry>
+ <para> One 3-OST OSS (A) and one 4-OST OSS (B)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> 3x5: BBABBABA</para>
+ </entry>
+ <entry>
+ <para> One 3-OST OSS (A) and one 5-OST OSS (B)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> 3x3x3: ABCABCABC</para>
+ </entry>
+ <entry>
+ <para> Three 3-OST OSSs</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Weighted allocator</emphasis> - When the free space difference between the OSTs is significant (by default, 20% of the free space), then a weighting algorithm is used to influence OST ordering based on size and location. Note that these are weightings for a random algorithm, so the OST with the most free space is not necessarily chosen each time. On average, the weighted allocator fills the emptier OSTs faster.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section remap="h3">
+ <title>18.5.3 Adjusting the Weighting Between Free Space and Location</title>
+ <para>The weighting priority can be adjusted in the <literal>/proc</literal> file <literal>/proc/fs/lustre/lov/lustre-mdtlov/qos_prio_free proc</literal>. The default value is 90%. Use this command on the MGS to permanently change this weighting:</para>
+ <screen>lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90</screen>
+ <para>Increasing this value puts more weighting on free space. When the free space priority is set to 100%, then location is no longer used in stripe-ordering calculations and weighting is based entirely on free space.</para>
+ <note>
+ <para>Setting the priority to 100% means that OSS distribution does not count in the weighting, but the stripe assignment is still done via a weighting. For example, if OST2 has twice as much free space as OST1, then OST2 is twice as likely to be used, but it is not guaranteed to be used.</para>
+ </note>
</section>
+ </section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='upgradinglustre'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="upgradinglustre">
<info>
- <title xml:id='upgradinglustre.title'>Upgrading Lustre</title>
+ <title xml:id="upgradinglustre.title">Upgrading Lustre</title>
</info>
<para>This chapter describes Lustre interoperability and how to upgrade to Lustre 2.0, and includes the following sections:</para>
-
- <itemizedlist><listitem>
-
<para><xref linkend="dbdoclet.50438205_82079"/>Lustre Interoperability</para>
- </listitem>
- <listitem>
-
<para><xref linkend="dbdoclet.50438205_51369"/>Upgrading Lustre 1.8.x to 2.0</para>
- </listitem>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438205_82079"/>Lustre Interoperability</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438205_51369"/>Upgrading Lustre 1.8.x to 2.0</para>
+ </listitem>
</itemizedlist>
-
- <section xml:id="dbdoclet.50438205_82079">
- <title>16.1 Lustre <anchor xml:id="dbdoclet.50438205_marker-1293321" xreflabel=""/>Interoperability</title>
- <para>Lustre 2.0 is built on a new architectural code base, which is different than the one used with Lustre 1.8. These architectural changes require existing Lustre 1.8.x users to follow a slightly different procedure to upgrade to Lustre 2.0 - requiring clients to be unmounted and the file system be shut down. Once the servers are upgraded and restarted, then the clients can be remounted. After the upgrade, Lustre 2.0 servers can interoperate with compatible 1.8 clients and servers. Lustre 2.0 does <emphasis>not</emphasis> support 2.0 clients interoperating with 1.8 servers.</para>
-
- <note><para>Lustre 1.8 clients support a mix of 1.8 and 2.0 OSTs, not all OSSs need to be upgraded at the same time.</para></note>
- <note><para>Lustre 2.0 is compatible with version 1.8.4 and above. If you are planning a heterogeneous environment (mixed 1.8 and 2.0 servers), make sure that version 1.8.4 is installed on the client and server nodes that are not upgraded to 2.0.</para></note>
-
- </section>
- <section xml:id="dbdoclet.50438205_51369">
- <title>16.2 Upgrading <anchor xml:id="dbdoclet.50438205_marker-1294652" xreflabel=""/>Lustre 1.8.x to 2.0</title>
- <para>Upgrading to Lustre 2.0 involves shutting down the file system and upgrading servers, and optionally clients, all at the same time. Lustre 2.0 does not support a rolling upgrade in which the file system operates continuously while individual servers (or their failover partners) and clients are upgraded one at a time.</para>
-
- <note><para>Although the Lustre 1.8 to 2.0 upgrade path has been tested, for best results we recommend performing a fresh Lustre 2.0 installation, rather than upgrading from 1.8 to 2.0.</para></note>
-
- <section remap="h3">
- <title>16.2.1 <anchor xml:id="dbdoclet.50438205_80840" xreflabel=""/>Performing a <anchor xml:id="dbdoclet.50438205_marker-1293327" xreflabel=""/>File System Upgrade</title>
- <para>This procedure describes a file system upgrade in which Lustre 2.0 packages are installed on multiple 1.8.x servers and, optionally, clients, requiring a file system shut down. You can choose to upgrade the entire Lustre file system to 2.0 or just upgrade the Lustre servers to 2.0 and leave the clients at 1.8.x. Lustre 2.0 servers can interoperate with compatible 1.8 clients and servers.</para>
- <tip><para>In a Lustre upgrade, the package install and file system unmount steps are reversible; you can do either step first. To minimize downtime, this procedure first performs the 2.0 package installation, and then unmounts the file system.</para></tip>
-
- <orderedlist><listitem>
- <para>Make a complete, restorable file system backup before upgrading Lustre.</para>
-
- </listitem><listitem>
- <para>If any Lustre nodes will not be upgraded to 2.0, make sure that these client and server nodes are at version 1.8.4.</para>
- <para>Lustre 2.0 is compatible with version 1.8.4 and above. If you are planning a heterogeneous environment (mixed 1.8 and 2.0 clients and servers), make sure that version 1.8.4 is installed on nodes that are not upgraded to 2.0.</para>
- </listitem><listitem>
- <para>Install the 2.0 packages on the Lustre servers and, optionally, the clients.</para>
- <para>Some or all servers can be upgraded. Some or all clients can be upgraded.</para>
- <para>For help determining where to install a specific package, see <link xl:href="InstallingLustre.html#50438261_21654">TABLE 8-1</link> (Lustre packages, descriptions and installation guidance).</para>
- <orderedlist><listitem>
- <para>Install the kernel, modules and ldiskfs packages. For example:</para>
- <screen>$ rpm -ivh
+ <section xml:id="dbdoclet.50438205_82079">
+ <title>16.1 Lustre Interoperability</title>
+ <para>Lustre 2.0 is built on a new architectural code base, which is different than the one used with Lustre 1.8. These architectural changes require existing Lustre 1.8.x users to follow a slightly different procedure to upgrade to Lustre 2.0 - requiring clients to be unmounted and the file system be shut down. Once the servers are upgraded and restarted, then the clients can be remounted. After the upgrade, Lustre 2.0 servers can interoperate with compatible 1.8 clients and servers. Lustre 2.0 does <emphasis>not</emphasis> support 2.0 clients interoperating with 1.8 servers.</para>
+ <note>
+ <para>Lustre 1.8 clients support a mix of 1.8 and 2.0 OSTs, not all OSSs need to be upgraded at the same time.</para>
+ </note>
+ <note>
+ <para>Lustre 2.0 is compatible with version 1.8.4 and above. If you are planning a heterogeneous environment (mixed 1.8 and 2.0 servers), make sure that version 1.8.4 is installed on the client and server nodes that are not upgraded to 2.0.</para>
+ </note>
+ </section>
+ <section xml:id="dbdoclet.50438205_51369">
+ <title>16.2 Upgrading Lustre 1.8.x to 2.0</title>
+ <para>Upgrading to Lustre 2.0 involves shutting down the file system and upgrading servers, and optionally clients, all at the same time. Lustre 2.0 does not support a rolling upgrade in which the file system operates continuously while individual servers (or their failover partners) and clients are upgraded one at a time.</para>
+ <note>
+ <para>Although the Lustre 1.8 to 2.0 upgrade path has been tested, for best results we recommend performing a fresh Lustre 2.0 installation, rather than upgrading from 1.8 to 2.0.</para>
+ </note>
+ <section remap="h3">
+ <title>16.2.1 Performing a File System Upgrade</title>
+ <para>This procedure describes a file system upgrade in which Lustre 2.0 packages are installed on multiple 1.8.x servers and, optionally, clients, requiring a file system shut down. You can choose to upgrade the entire Lustre file system to 2.0 or just upgrade the Lustre servers to 2.0 and leave the clients at 1.8.x. Lustre 2.0 servers can interoperate with compatible 1.8 clients and servers.</para>
+ <tip>
+ <para>In a Lustre upgrade, the package install and file system unmount steps are reversible; you can do either step first. To minimize downtime, this procedure first performs the 2.0 package installation, and then unmounts the file system.</para>
+ </tip>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Make a complete, restorable file system backup before upgrading Lustre.</emphasis></para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">If any Lustre nodes will not be upgraded to 2.0, make sure that these client and server nodes are at version 1.8.4.</emphasis></para>
+ <para>Lustre 2.0 is compatible with version 1.8.4 and above. If you are planning a heterogeneous environment (mixed 1.8 and 2.0 clients and servers), make sure that version 1.8.4 is installed on nodes that are not upgraded to 2.0.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Install the 2.0 packages on the Lustre servers and, optionally, the clients.</emphasis></para>
+ <para>Some or all servers can be upgraded. Some or all clients can be upgraded.</para>
+ <para>For help determining where to install a specific package, see <xref linkend="installinglustre.tab.req"/>.</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Install the kernel, modules and ldiskfs packages. For example:</emphasis></para>
+ <screen>$ rpm -ivh
kernel-lustre-smp-<ver> \
kernel-ib-<ver> \
lustre-modules-<ver> \
-lustre-ldiskfs-<ver>
-</screen>
- </listitem><listitem>
- <para>Upgrade the utilities/userspace packages. For example:</para>
- <screen>$ rpm -Uvh lustre-<ver>
-</screen>
- </listitem><listitem>
- <para>If a new e2fsprogs package is available, upgrade it. For example:</para>
- <screen>$ rpm -Uvh e2fsprogs-<ver>
-</screen>
- <para>Use e2fsprogs-1.41-10 or later, available at:</para>
- <para><link xl:href="http://downloads.lustre.org/public/tools/e2fsprogs/">http://downloads.lustre.org/public/tools/e2fsprogs/</link></para>
- </listitem><listitem>
- <para>(Optional) If you want to add optional packages to your Lustre system, install them now.</para>
- </listitem></orderedlist>
- </listitem><listitem>
- <para>Shut down the file system.</para>
- <para>Shut down the components in this order: clients, then the MDT, then OSTs. Unmounting a block device causes Lustre to be shut down on that node.</para>
- <orderedlist><listitem>
- <para>Unmount the clients. On each client node, run:</para>
- <screen>umount <mount point>
-</screen>
- </listitem><listitem>
- <para>Unmount the MDT. On the MDS node, run:</para>
- <screen>umount <mount point>
+lustre-ldiskfs-<ver></screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Upgrade the utilities/userspace packages. For example:</emphasis></para>
+ <screen>$ rpm -Uvh lustre-<ver></screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">If a new <literal>e2fsprogs</literal> package is available, upgrade it. For example:</emphasis></para>
+ <screen>$ rpm -Uvh e2fsprogs-<ver>
</screen>
- </listitem><listitem>
- <para>Unmount the OSTs (be sure to unmount all OSTs). On each OSS node, run:</para>
- <screen>umount <mount point>
-</screen>
- </listitem></orderedlist>
- </listitem><listitem>
- <para>Unload the old Lustre modules by rebooting the node or manually removing the Lustre modules.</para>
- <para>Run lustre_rmmod several times and use lsmod to check the currently loaded modules.</para>
- </listitem><listitem>
- <para>Start the upgraded file system.</para>
- <para>Start the components in this order: OSTs, then the MDT, then clients.</para>
- <orderedlist><listitem>
- <para>Mount the OSTs (be sure to mount all OSTs). On each OSS node, run:</para>
- <screen>mount -t lustre <block device name> <mount point>
-</screen>
- </listitem><listitem>
- <para>Mount the MDT. On the MDS node, run:</para>
- <screen>mount -t lustre <block device name> <mount point>
-</screen>
- </listitem><listitem>
- <para>Mount the file system on the clients. On each client node, run:</para>
- <screen>mount -t lustre <MGS node>:/<fsname> <mount point>
-</screen>
- </listitem></orderedlist></listitem></orderedlist>
- <para>If you have a problem upgrading Lustre, contact us via the <link xl:href="https://jira.whamcloud.com">Whamcloud Jira</link> bug tracker.</para>
- <para> </para>
- </section>
+ <para>Use e2fsprogs-1.41-10 or later, available at:</para>
+ <para><link xl:href="http://downloads.lustre.org/public/tools/e2fsprogs/">http://downloads.lustre.org/public/tools/e2fsprogs/</link></para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">(Optional) If you want to add optional packages to your Lustre system, install them now.</emphasis></para>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Shut down the file system.</emphasis></para>
+ <para>Shut down the components in this order: clients, then the MDT, then OSTs. Unmounting a block device causes Lustre to be shut down on that node.</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Unmount the clients. On each client node, run:</emphasis></para>
+ <screen>umount <mount point></screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Unmount the MDT. On the MDS node, run:</emphasis></para>
+ <screen>umount <mount point></screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Unmount the OSTs (be sure to unmount all OSTs). On each OSS node, run:</emphasis></para>
+ <screen>umount <mount point></screen>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Unload the old Lustre modules by rebooting the node or manually removing the Lustre modules.</emphasis></para>
+ <para>Run <literal>lustre_rmmod</literal> several times and use <literal>lsmod</literal> to check the currently loaded modules.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold"><emphasis role="bold">Start the upgraded file system</emphasis>.</emphasis></para>
+ <para>Start the components in this order: OSTs, then the MDT, then clients.</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Mount the OSTs (be sure to mount all OSTs). On each OSS node, run:</emphasis></para>
+ <screen>mount -t lustre <block device name> <mount point></screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Mount the MDT. On the MDS node, run:</emphasis></para>
+ <screen>mount -t lustre <block device name> <mount point> </screen>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Mount the file system on the clients. On each client node, run:</emphasis></para>
+ <screen>mount -t lustre <MGS node>:/<fsname> <mount point> </screen>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ </orderedlist>
+ <para>If you have a problem upgrading Lustre, contact us via the <ulink xl:href="https://jira.whamcloud.com">Whamcloud Jira</ulink> bug tracker.</para>
+ </section>
</section>
</chapter>
git://git.whamcloud.com / doc / manual.git / commitdiff