</listitem>
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438216_pgfId-1304876" xreflabel=""/>If the router checker does not get a reply message from the router within router_ping_timeout seconds, it considers the router to be down.</para>
- <para><anchor xml:id="dbdoclet.50438216_pgfId-1304878" xreflabel=""/>If a router is marked “up†and responds to a ping, the timeout is reset.</para>
+ <para><anchor xml:id="dbdoclet.50438216_pgfId-1304878" xreflabel=""/>If a router is marked 'up' and responds to a ping, the timeout is reset.</para>
<para><anchor xml:id="dbdoclet.50438216_pgfId-1304881" xreflabel=""/>If 100 packets have been sent successfully through a router, the sent-packets counter for that router will have a value of 100.</para>
</section>
<section xml:id="dbdoclet.50438216_15200">
<row>
<entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292895" xreflabel=""/>start_ost</para></entry>
<entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292897" xreflabel=""/>-1</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292899" xreflabel=""/>The first OST where objects are created for each file. The default -1 allows the MDS to choose the starting index based on available space and load balancing. <emphasis>It’s strongly recommended not to change the default for this parameter to a value other than -1.</emphasis></para></entry>
+ <entry><para> <anchor xml:id="dbdoclet.50438267_pgfId-1292899" xreflabel=""/>The first OST where objects are created for each file. The default -1 allows the MDS to choose the starting index based on available space and load balancing. <emphasis>It's strongly recommended not to change the default for this parameter to a value other than -1.</emphasis></para></entry>
</row>
</tbody>
</tgroup>
</listitem>
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290159" xreflabel=""/>Lustre 1.6.5 introduced the v2 file format for administrative quota files, with continued support for the old file format (v1). The mdt.quota_type parameter also handles ‘1’ and ‘2’ options, to specify the Lustre quota versions that will be used. For example:</para>
+ <para><anchor xml:id="dbdoclet.50438217_pgfId-1290159" xreflabel=""/>Lustre 1.6.5 introduced the v2 file format for administrative quota files, with continued support for the old file format (v1). The mdt.quota_type parameter also handles '1' and '2' options, to specify the Lustre quota versions that will be used. For example:</para>
<screen><anchor xml:id="dbdoclet.50438217_pgfId-1290160" xreflabel=""/>--param mdt.quota_type=ug1
<anchor xml:id="dbdoclet.50438217_pgfId-1290161" xreflabel=""/>--param mdt.quota_type=u2
</screen>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290162" xreflabel=""/>Lustre 1.6.6 introduced the v2 file format for operational quotas, with continued support for the old file format (v1). The ost.quota_type parameter handles ‘1’ and ‘2’ options, to specify the Lustre quota versions that will be used. For example:</para>
+ <para><anchor xml:id="dbdoclet.50438217_pgfId-1290162" xreflabel=""/>Lustre 1.6.6 introduced the v2 file format for operational quotas, with continued support for the old file format (v1). The ost.quota_type parameter handles '1' and '2' options, to specify the Lustre quota versions that will be used. For example:</para>
<screen><anchor xml:id="dbdoclet.50438217_pgfId-1290163" xreflabel=""/>--param ost.quota_type=ug2
<anchor xml:id="dbdoclet.50438217_pgfId-1290164" xreflabel=""/>--param ost.quota_type=u1
</screen>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290256" xreflabel=""/>The /proc values are bounded by two other variables quota_btune_sz and quota_itune_sz. By default, the *tune_sz variables are set at 1/2 the *unit_sz variables, and you cannot set *tune_sz larger than *unit_sz. You must set bunit_sz first if it is increasing by more than 2x, and btune_sz first if it is decreasing by more than 2x.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290257" xreflabel=""/><emphasis role="bold">Total number of inodes</emphasis> -- To determine the total number of inodes, use lfsdf-i (and also /proc/fs/lustre/*/*/filestotal). For more information on using the lfsdf-i command and the command output, see <link xl:href="ManagingStripingFreeSpace.html#50438209_35838">Checking File System Free Space</link>.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290261" xreflabel=""/>Unfortunately, the statfs interface does not report the free inode count directly, but instead reports the total inode and used inode counts. The free inode count is calculated for df from (total inodes - used inodes).</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290262" xreflabel=""/>It is not critical to know a file system’s total inode count. Instead, you should know (accurately), the free inode count and the used inode count for a file system. Lustre manipulates the total inode count in order to accurately report the other two values.</para>
+ <para><anchor xml:id="dbdoclet.50438217_pgfId-1290262" xreflabel=""/>It is not critical to know a file system's total inode count. Instead, you should know (accurately), the free inode count and the used inode count for a file system. Lustre manipulates the total inode count in order to accurately report the other two values.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290263" xreflabel=""/>The values set for the MDS must match the values set on the OSTs.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290264" xreflabel=""/>The quota_bunit_sz parameter displays bytes, however lfs setquota uses KBs. The quota_bunit_sz parameter must be a multiple of 1024. A proper minimum KB size for lfs setquota can be calculated as:</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290265" xreflabel=""/><emphasis role="bold">Size in KBs = minimum_quota_bunit_sz * (number of OSTS + 1) = 1024 * (number of OSTs +1)</emphasis></para>
<orderedlist><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290276" xreflabel=""/>A user writes files to Lustre.</para>
</listitem><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290277" xreflabel=""/>If the Lustre client has enough granted cache, then it returns ‘success’ to users and arranges the writes to the OSTs.</para>
+ <para><anchor xml:id="dbdoclet.50438217_pgfId-1290277" xreflabel=""/>If the Lustre client has enough granted cache, then it returns 'success' to users and arranges the writes to the OSTs.</para>
</listitem><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290278" xreflabel=""/>Because Lustre clients have delivered success to users, the OSTs cannot fail these writes.</para>
</listitem></orderedlist>
</informaltable>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438217_pgfId-1290408" xreflabel=""/>21.6.1 Interpreting Quota Statistics</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290409" xreflabel=""/>Quota statistics are an important measure of a Lustre file system’s performance. Interpreting these statistics correctly can help you diagnose problems with quotas, and may indicate adjustments to improve system performance.</para>
+ <para><anchor xml:id="dbdoclet.50438217_pgfId-1290409" xreflabel=""/>Quota statistics are an important measure of a Lustre file system's performance. Interpreting these statistics correctly can help you diagnose problems with quotas, and may indicate adjustments to improve system performance.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290410" xreflabel=""/>For example, if you run this command on the OSTs:</para>
<screen><anchor xml:id="dbdoclet.50438217_pgfId-1290411" xreflabel=""/>cat /proc/fs/lustre/lquota/lustre-OST0000/stats
</screen>
<glossterm><anchor xml:id="dbdoclet.50438338_pgfId-1000650" xreflabel=""/>Fileset
</glossterm>
<glossdef>
- <para><anchor xml:id="dbdoclet.50438338_pgfId-1000651" xreflabel=""/>A group of files that are defined through a directory that represents a file system’s start point.</para>
+ <para><anchor xml:id="dbdoclet.50438338_pgfId-1000651" xreflabel=""/>A group of files that are defined through a directory that represents a file system's start point.</para>
</glossdef>
</glossentry>
<glossentry xml:id="fldb">
</section>
<section xml:id="dbdoclet.50438199_54623">
<title>14.4 Regenerating Lustre <anchor xml:id="dbdoclet.50438199_marker-1305736" xreflabel=""/>Configuration Logs</title>
- <para><anchor xml:id="dbdoclet.50438199_pgfId-1304951" xreflabel=""/>If the Lustre system’s configuration logs are in a state where the file system cannot be started, use the writeconf command to erase them. After the writeconf command is run and the servers restart, the configuration logs are re-generated and stored on the MGS (as in a new file system).</para>
+ <para><anchor xml:id="dbdoclet.50438199_pgfId-1304951" xreflabel=""/>If the Lustre system's configuration logs are in a state where the file system cannot be started, use the writeconf command to erase them. After the writeconf command is run and the servers restart, the configuration logs are re-generated and stored on the MGS (as in a new file system).</para>
<para><anchor xml:id="dbdoclet.50438199_pgfId-1304970" xreflabel=""/>You should only use the writeconf command if:</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438199_pgfId-1304971" xreflabel=""/> The configuration logs are in a state where the file system cannot start</para>
<para><anchor xml:id="dbdoclet.50438273_pgfId-1297847" xreflabel=""/>To register a new changelog user, run:</para>
<screen><anchor xml:id="dbdoclet.50438273_pgfId-1297848" xreflabel=""/>lctl --device <mdt_device> changelog_register
</screen>
- <para><anchor xml:id="dbdoclet.50438273_pgfId-1297849" xreflabel=""/>Changelog entries are not purged beyond a registered user’s set point (see lfs changelog_clear).</para>
+ <para><anchor xml:id="dbdoclet.50438273_pgfId-1297849" xreflabel=""/>Changelog entries are not purged beyond a registered user's set point (see lfs changelog_clear).</para>
</section>
<section remap="h5">
<title><anchor xml:id="dbdoclet.50438273_pgfId-1297785" xreflabel=""/>lfs changelog</title>
<anchor xml:id="dbdoclet.50438194_pgfId-1302355" xreflabel=""/>$ lctl conf_param testfs-OST0000.ost.client_cache_seconds=15
<anchor xml:id="dbdoclet.50438194_pgfId-1302356" xreflabel=""/>$ lctl conf_param testfs.sys.timeout=40
</screen>
- <caution><para>Parameters specified with the lctlconf_param command are set permanently in the file system’s configuration file on the MGS.</para></caution>
+ <caution><para>Parameters specified with the lctlconf_param command are set permanently in the file system's configuration file on the MGS.</para></caution>
</section>
<section remap="h4">
<title><anchor xml:id="dbdoclet.50438194_pgfId-1305661" xreflabel=""/>13.8.3.3 <anchor xml:id="dbdoclet.50438194_88217" xreflabel=""/>Listing Parameters</title>
</section>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438198_pgfId-1291367" xreflabel=""/>26.3.3 Identifying a <anchor xml:id="dbdoclet.50438198_marker-1291366" xreflabel=""/>Missing OST</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291368" xreflabel=""/>If an OST is missing for any reason, you may need to know what files are affected. Although an OST is missing, the files system should be operational. From any mounted client node, generate a list of files that reside on the affected OST. It is advisable to mark the missing OST as ’unavailable’ so clients and the MDS do not time out trying to contact it.</para>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291369" xreflabel=""/> 1. Generate a list of devices and determine the OST’s device number. Run:</para>
+ <para><anchor xml:id="dbdoclet.50438198_pgfId-1291368" xreflabel=""/>If an OST is missing for any reason, you may need to know what files are affected. Although an OST is missing, the files system should be operational. From any mounted client node, generate a list of files that reside on the affected OST. It is advisable to mark the missing OST as 'unavailable' so clients and the MDS do not time out trying to contact it.</para>
+ <para><anchor xml:id="dbdoclet.50438198_pgfId-1291369" xreflabel=""/> 1. Generate a list of devices and determine the OST's device number. Run:</para>
<screen><anchor xml:id="dbdoclet.50438198_pgfId-1291370" xreflabel=""/>$ lctl dl
</screen>
<para><anchor xml:id="dbdoclet.50438198_pgfId-1293115" xreflabel=""/>The lctl dl command output lists the device name and number, along with the device UUID and the number of references on the device.</para>
<para><anchor xml:id="dbdoclet.50438198_pgfId-1291896" xreflabel=""/>After the file system has been running for some time, it contains more data in cache and hence, the variability caused by reading critical metadata from disk is mostly eliminated. The file system now reads data from the cache.</para>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438198_pgfId-1291922" xreflabel=""/>26.3.13 Log Message ‘Out of <anchor xml:id="dbdoclet.50438198_marker-1292113" xreflabel=""/>Memory’ on OST</title>
- <para><anchor xml:id="dbdoclet.50438198_pgfId-1291934" xreflabel=""/>When planning the hardware for an OSS node, consider the memory usage of several components in the Lustre system. If insufficient memory is available, an ‘out of memory’ message can be logged.</para>
+ <title><anchor xml:id="dbdoclet.50438198_pgfId-1291922" xreflabel=""/>26.3.13 Log Message 'Out of <anchor xml:id="dbdoclet.50438198_marker-1292113" xreflabel=""/>Memory' on OST</title>
+ <para><anchor xml:id="dbdoclet.50438198_pgfId-1291934" xreflabel=""/>When planning the hardware for an OSS node, consider the memory usage of several components in the Lustre system. If insufficient memory is available, an 'out of memory' message can be logged.</para>
<para><anchor xml:id="dbdoclet.50438198_pgfId-1292123" xreflabel=""/>During normal operation, several conditions indicate insufficient RAM on a server node:</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438198_pgfId-1291969" xreflabel=""/> kernel "Out of memory" and/or "oom-killer" messages</para>
<para><anchor xml:id="dbdoclet.50438221_pgfId-1292593" xreflabel=""/>To mount the client with no ACLs:</para>
<screen><anchor xml:id="dbdoclet.50438221_pgfId-1292597" xreflabel=""/>$ mount -t lustre -o noacl ibmds2@o2ib:/home /home
</screen>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1292321" xreflabel=""/>ACLs are enabled in Lustre on a system-wide basis; either all clients enable ACLs or none do. Activating ACLs is controlled by MDS mount options acl / noacl (enable/disableACLs). Client-side mount options acl/noacl are ignored. You do not need to change the client configuration, and the “acl†string will not appear in the client /etc/mtab. The client acl mount option is no longer needed. If a client is mounted with that option, then this message appears in the MDS syslog:</para>
+ <para><anchor xml:id="dbdoclet.50438221_pgfId-1292321" xreflabel=""/>ACLs are enabled in Lustre on a system-wide basis; either all clients enable ACLs or none do. Activating ACLs is controlled by MDS mount options acl / noacl (enable/disableACLs). Client-side mount options acl/noacl are ignored. You do not need to change the client configuration, and the 'acl' string will not appear in the client /etc/mtab. The client acl mount option is no longer needed. If a client is mounted with that option, then this message appears in the MDS syslog:</para>
<screen><anchor xml:id="dbdoclet.50438221_pgfId-1292322" xreflabel=""/>...MDS requires ACL support but client does not
</screen>
<para><anchor xml:id="dbdoclet.50438221_pgfId-1292323" xreflabel=""/>The message is harmless but indicates a configuration issue, which should be corrected.</para>
<title><anchor xml:id="dbdoclet.50438221_pgfId-1293871" xreflabel=""/>22.2.3 Tips on Using <anchor xml:id="dbdoclet.50438221_marker-1296366" xreflabel=""/>Root Squash</title>
<para><anchor xml:id="dbdoclet.50438221_pgfId-1296298" xreflabel=""/>Lustre configuration management limits root squash in several ways.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438221_pgfId-1296299" xreflabel=""/> The lctl conf_param value overwrites the parameter’s previous value. If the new value uses an incorrect syntax, then the system continues with the old parameters and the previously-correct value is lost on remount. That is, be careful doing root squash tuning.</para>
+ <para><anchor xml:id="dbdoclet.50438221_pgfId-1296299" xreflabel=""/> The lctl conf_param value overwrites the parameter's previous value. If the new value uses an incorrect syntax, then the system continues with the old parameters and the previously-correct value is lost on remount. That is, be careful doing root squash tuning.</para>
</listitem>
<listitem>
<title><anchor xml:id="dbdoclet.50438209_pgfId-1291860" xreflabel=""/>18.2.1 Choosing a Stripe <anchor xml:id="dbdoclet.50438209_marker-1291859" xreflabel=""/>Size</title>
<para><anchor xml:id="dbdoclet.50438209_pgfId-1296612" xreflabel=""/>Choosing a stripe size is a small balancing act, but there are reasonable defaults.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296443" xreflabel=""/><emphasis role="bold">The stripe size must be a multiple of the page size.</emphasis> Lustre’s tools enforce a multiple of 64 KB (the maximum page size on ia64 and PPC64 nodes) so that users on platforms with smaller pages do not accidentally create files that might cause problems for ia64 clients.</para>
+ <para><anchor xml:id="dbdoclet.50438209_pgfId-1296443" xreflabel=""/><emphasis role="bold">The stripe size must be a multiple of the page size.</emphasis> Lustre's tools enforce a multiple of 64 KB (the maximum page size on ia64 and PPC64 nodes) so that users on platforms with smaller pages do not accidentally create files that might cause problems for ia64 clients.</para>
</listitem>
<listitem>
<para><anchor xml:id="dbdoclet.50438209_pgfId-1291862" xreflabel=""/><emphasis role="bold">The smallest recommended stripe size is 512 KB.</emphasis> Although you can create files with a stripe size of 64 KB, the smallest practical stripe size is 512 KB because Lustre sends 1MB chunks over the network. Choosing a smaller stripe size may result in inefficient I/O to the disks and reduced performance.</para>
<para><anchor xml:id="dbdoclet.50438209_pgfId-1297206" xreflabel=""/><emphasis role="bold">The maximum stripe size is 4GB.</emphasis> Using a large stripe size can improve performance when accessing very large files. It allows each client to have exclusive access to its own part of a file. However, it can be counterproductive in some cases if it does not match your I/O pattern.</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291865" xreflabel=""/><emphasis role="bold">Choose a stripe pattern that takes into account your application’s write patterns.</emphasis> Writes that cross an object boundary are slightly less efficient than writes that go entirely to one server. If the file is written in a very consistent and aligned way, make the stripe size a multiple of the write() size.</para>
+ <para><anchor xml:id="dbdoclet.50438209_pgfId-1291865" xreflabel=""/><emphasis role="bold">Choose a stripe pattern that takes into account your application's write patterns.</emphasis> Writes that cross an object boundary are slightly less efficient than writes that go entirely to one server. If the file is written in a very consistent and aligned way, make the stripe size a multiple of the write() size.</para>
</listitem>
<listitem>
<para><anchor xml:id="dbdoclet.50438209_pgfId-1291866" xreflabel=""/><emphasis role="bold">The choice of stripe size has no effect on a single-stripe file.</emphasis></para>
</section>
<section xml:id="dbdoclet.50438225_12316">
<title>27.3 Recovering from an <anchor xml:id="dbdoclet.50438225_marker-1292768" xreflabel=""/>Unavailable OST</title>
- <para><anchor xml:id="dbdoclet.50438225_pgfId-1292771" xreflabel=""/>One of the most common problems encountered in a Lustre environment is when an OST becomes unavailable, because of a network partition, OSS node crash, etc. When this happens, the OST’s clients pause and wait for the OST to become available again, either on the primary OSS or a failover OSS. When the OST comes back online, Lustre starts a recovery process to enable clients to reconnect to the OST. Lustre servers put a limit on the time they will wait in recovery for clients to reconnect. The timeout length is determined by the obd_timeout parameter.</para>
+ <para><anchor xml:id="dbdoclet.50438225_pgfId-1292771" xreflabel=""/>One of the most common problems encountered in a Lustre environment is when an OST becomes unavailable, because of a network partition, OSS node crash, etc. When this happens, the OST's clients pause and wait for the OST to become available again, either on the primary OSS or a failover OSS. When the OST comes back online, Lustre starts a recovery process to enable clients to reconnect to the OST. Lustre servers put a limit on the time they will wait in recovery for clients to reconnect. The timeout length is determined by the obd_timeout parameter.</para>
<para><anchor xml:id="dbdoclet.50438225_pgfId-1292775" xreflabel=""/>During recovery, clients reconnect and replay their requests serially, in the same order they were done originally. Until a client receives a confirmation that a given transaction has been written to stable storage, the client holds on to the transaction, in case it needs to be replayed. Periodically, a progress message prints to the log, stating how_many/expected clients have reconnected. If the recovery is aborted, this log shows how many clients managed to reconnect. When all clients have completed recovery, or if the recovery timeout is reached, the recovery period ends and the OST resumes normal request processing.</para>
<para><anchor xml:id="dbdoclet.50438225_pgfId-1292779" xreflabel=""/>If some clients fail to replay their requests during the recovery period, this will not stop the recovery from completing. You may have a situation where the OST recovers, but some clients are not able to participate in recovery (e.g. network problems or client failure), so they are evicted and their requests are not replayed. This would result in any operations on the evicted clients failing, including in-progress writes, which would cause cached writes to be lost. This is a normal outcome; the recovery cannot wait indefinitely, or the file system would be hung any time a client failed. The lost transactions are an unfortunate result of the recovery process.</para>
<note><para>The version-based recovery (VBR) feature enables a failed client to be ''skipped'', so remaining clients can replay their requests, resulting in a more successful recovery from a downed OST. For more information about the VBR feature, see <xref linkend='lustrerecovery'/>(Version-based Recovery).</para></note>
<para><anchor xml:id="dbdoclet.50438146_pgfId-25402" xreflabel=""/></para>
<para>improving Lustre performance when working with small files, <link xl:href="LustreTuning.html#50438272_marker-1295851">1</link></para>
<para><anchor xml:id="dbdoclet.50438146_pgfId-25404" xreflabel=""/></para>
- <para>log message ’out of memory’ on OST, <link xl:href="LustreTroubleshooting.html#50438198_marker-1292113">1</link></para>
+ <para>log message 'out of memory' on OST, <link xl:href="LustreTroubleshooting.html#50438198_marker-1292113">1</link></para>
<para><anchor xml:id="dbdoclet.50438146_pgfId-25406" xreflabel=""/>Lustre Error</para>
<para><anchor xml:id="dbdoclet.50438146_pgfId-25407" xreflabel=""/></para>
<para>"slow start_page_write", <link xl:href="LustreTroubleshooting.html#50438198_marker-1291517">1</link></para>
git://git.whamcloud.com / doc / manual.git / commitdiff