<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink">
+<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='lustreproc'>
<info>
- <title>LustreProc</title>
+ <title xml:id='lustreproc.title'>LustreProc</title>
</info>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1301083" xreflabel=""/>The /proc file system acts as an interface to internal data structures in the kernel. The /proc variables can be used to control aspects of Lustre performance and provide information.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290340" xreflabel=""/>This chapter describes Lustre /proc entries and includes the following sections:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290344" xreflabel=""/><link xl:href="LustreProc.html#50438271_90999">Proc Entries for Lustre</link></para>
+ <para><xref linkend="dbdoclet.50438271_90999"/></para>
</listitem>
+
<listitem>
- <para> </para>
+ <para><xref linkend="dbdoclet.50438271_78950"/></para>
</listitem>
+
<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290348" xreflabel=""/><link xl:href="LustreProc.html#50438271_78950">Lustre I/O Tunables</link></para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1290352" xreflabel=""/><link xl:href="LustreProc.html#50438271_83523">Debug</link></para>
- </listitem>
-<listitem>
- <para> </para>
+ <para><xref linkend="dbdoclet.50438271_83523"/></para>
</listitem>
+
</itemizedlist>
- <section remap="h2">
- <title><anchor xml:id="dbdoclet.50438271_pgfId-1290359" xreflabel=""/></title>
- <section remap="h2">
- <title>31.1 <anchor xml:id="dbdoclet.50438271_90999" xreflabel=""/>Proc Entries for Lustre</title>
+
+ <section xml:id="dbdoclet.50438271_90999">
+ <title>31.1 Proc Entries for Lustre</title>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290360" xreflabel=""/>This section describes /proc entries for Lustre.</para>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438271_pgfId-1290361" xreflabel=""/>31.1.1 Locating Lustre <anchor xml:id="dbdoclet.50438271_marker-1296151" xreflabel=""/>File Systems and Servers</title>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290363" xreflabel=""/> All known file systems</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1290364" xreflabel=""/># cat /proc/fs/lustre/mgs/MGS/filesystems
<anchor xml:id="dbdoclet.50438271_pgfId-1291584" xreflabel=""/>spfs
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290367" xreflabel=""/> The server names participating in a file system (for each file system that has at least one server running)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1290368" xreflabel=""/># cat /proc/fs/lustre/mgs/MGS/live/spfs
<anchor xml:id="dbdoclet.50438271_pgfId-1291593" xreflabel=""/>fsname: spfs
<title><anchor xml:id="dbdoclet.50438271_pgfId-1290389" xreflabel=""/>31.1.2 Lustre <anchor xml:id="dbdoclet.50438271_marker-1296153" xreflabel=""/>Timeouts</title>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1294163" xreflabel=""/>Lustre uses two types of timeouts.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294195" xreflabel=""/> LND timeouts that ensure point-to-point communications complete in finite time in the presence of failures. These timeouts are logged with the S_LND flag set. They may <emphasis>not</emphasis> be printed as console messages, so you should check the Lustre log for D_NETERROR messages, or enable printing of D_NETERROR messages to the console (echo + neterror > /proc/sys/lnet/printk).</para>
- </listitem>
-<listitem>
- <para> </para>
+ <para><anchor xml:id="dbdoclet.50438271_pgfId-1294195" xreflabel=""/>LND timeouts that ensure point-to-point communications complete in finite time in the presence of failures. These timeouts are logged with the S_LND flag set. They may <emphasis>not</emphasis> be printed as console messages, so you should check the Lustre log for D_NETERROR messages, or enable printing of D_NETERROR messages to the console (echo + neterror > /proc/sys/lnet/printk).</para>
</listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1294211" xreflabel=""/>Congested routers can be a source of spurious LND timeouts. To avoid this, increase the number of LNET router buffers to reduce back-pressure and/or increase LND timeouts on all nodes on all connected networks. You should also consider increasing the total number of LNET router nodes in the system so that the aggregate router bandwidth matches the aggregate server bandwidth.</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1294177" xreflabel=""/> Lustre timeouts that ensure Lustre RPCs complete in finite time in the presence of failures. These timeouts should <emphasis>always</emphasis> be printed as console messages. If Lustre timeouts are not accompanied by LNET timeouts, then you need to increase the lustre timeout on both servers and clients.</para>
- </listitem>
-<listitem>
- <para> </para>
+ <para><anchor xml:id="dbdoclet.50438271_pgfId-1294177" xreflabel=""/>Lustre timeouts that ensure Lustre RPCs complete in finite time in the presence of failures. These timeouts should <emphasis>always</emphasis> be printed as console messages. If Lustre timeouts are not accompanied by LNET timeouts, then you need to increase the lustre timeout on both servers and clients.</para>
</listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1294236" xreflabel=""/>Specific Lustre timeouts are described below.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290390" xreflabel=""/><emphasis role="bold">/proc/sys/lustre/timeout</emphasis></para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290391" xreflabel=""/>This is the time period that a client waits for a server to complete an RPC (default is 100s). Servers wait half of this time for a normal client RPC to complete and a quarter of this time for a single bulk request (read or write of up to 1 MB) to complete. The client pings recoverable targets (MDS and OSTs) at one quarter of the timeout, and the server waits one and a half times the timeout before evicting a client for being "stale."</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438271_pgfId-1292928" xreflabel=""/>Lustre sends periodic ‘PING’ messages to servers with which it had no communication for a specified period of time. Any network activity on the file system that triggers network traffic toward servers also works as a health check.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>Lustre sends periodic 'PING' messages to servers with which it had no communication for a specified period of time. Any network activity on the file system that triggers network traffic toward servers also works as a health check.</para></note>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1292930" xreflabel=""/><emphasis role="bold">/proc/sys/lustre/ldlm_timeout</emphasis></para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290393" xreflabel=""/>This is the time period for which a server will wait for a client to reply to an initial AST (lock cancellation request) where default is 20s for an OST and 6s for an MDS. If the client replies to the AST, the server will give it a normal timeout (half of the client timeout) to flush any dirty data and release the lock.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290394" xreflabel=""/><emphasis role="bold">/proc/sys/lustre/fail_loc</emphasis></para>
<title><anchor xml:id="dbdoclet.50438271_pgfId-1292947" xreflabel=""/>31.1.3.1 Configuring <anchor xml:id="dbdoclet.50438271_marker-1293381" xreflabel=""/>Adaptive Timeouts</title>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1292948" xreflabel=""/>One of the goals of adaptive timeouts is to relieve users from having to tune the obd_timeout value. In general, obd_timeout should no longer need to be changed. However, there are several parameters related to adaptive timeouts that users can set. In most situations, the default values should be used.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299111" xreflabel=""/>The following parameters can be set persistently system-wide using lctl conf_param on the MGS. For example, lctl conf_param work1.sys.at_max=1500 sets the at_max value for all servers and clients using the work1 file system.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438271_pgfId-1299130" xreflabel=""/>Nodes using multiple Lustre file systems must use the same at_* values for all file systems.)</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>Nodes using multiple Lustre file systems must use the same at_* values for all file systems.)</para></note>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
</row>
<row>
<entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294381" xreflabel=""/><emphasis role="bold">at_max</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1299206" xreflabel=""/>Sets the maximum adaptive timeout (in seconds). The at_max parameter is an upper-limit on the service time estimate, and is used as a 'failsafe' in case of rogue/bad/buggy code that would lead to never-ending estimate increases. If at_max is reached, an RPC request is considered 'broken' and should time out.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1294383" xreflabel=""/>Setting at_max to 0 causes adaptive timeouts to be disabled and the old fixed-timeout method (obd_timeout) to be used. This is the default value in Lustre 1.6.5.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1294387" xreflabel=""/> </para><para><anchor xml:id="dbdoclet.50438271_pgfId-1294388" xreflabel=""/><emphasis role="bold">NOTE</emphasis>: It is possible that slow hardware might validly cause the service estimate to increase beyond the default value of at_max. In this case, you should increase at_max to the maximum time you are willing to wait for an RPC completion.</para></entry>
+ <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1299206" xreflabel=""/>Sets the maximum adaptive timeout (in seconds). The at_max parameter is an upper-limit on the service time estimate, and is used as a 'failsafe' in case of rogue/bad/buggy code that would lead to never-ending estimate increases. If at_max is reached, an RPC request is considered 'broken' and should time out.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1294383" xreflabel=""/>Setting at_max to 0 causes adaptive timeouts to be disabled and the old fixed-timeout method (obd_timeout) to be used. This is the default value in Lustre 1.6.5.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1294387" xreflabel=""/> </para>
+
+ <note><para>It is possible that slow hardware might validly cause the service estimate to increase beyond the default value of at_max. In this case, you should increase at_max to the maximum time you are willing to wait for an RPC completion.</para></note></entry>
</row>
<row>
<entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1294390" xreflabel=""/><emphasis role="bold">at_history</emphasis></para></entry>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1293587" xreflabel=""/>Adaptive timeouts are enabled, by default. To disable adaptive timeouts, at run time, set at_max to 0. On the MGS, run:</para>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1294471" xreflabel=""/>$ lctl conf_param <fsname>.sys.at_max=0
</screen>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438271_pgfId-1294472" xreflabel=""/>Changing adaptive timeouts status at runtime may cause transient timeout, reconnect, recovery, etc.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>Changing adaptive timeouts status at runtime may cause transient timeout, reconnect, recovery, etc.</para></note>
</section>
<section remap="h4">
<title><anchor xml:id="dbdoclet.50438271_pgfId-1292959" xreflabel=""/>31.1.3.2 Interpreting <anchor xml:id="dbdoclet.50438271_marker-1293383" xreflabel=""/>Adaptive Timeouts Information</title>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296531" xreflabel=""/><emphasis role="bold">QOS</emphasis></para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297308" xreflabel=""/>Quality of Service (QOS) considers an OST’s available blocks, speed, and the number of existing objects, etc. Using these criteria, the MDS selects OSTs with more free space more often than OSTs with less free space.</para>
+ <para><anchor xml:id="dbdoclet.50438271_pgfId-1297308" xreflabel=""/>Quality of Service (QOS) considers an OST's available blocks, speed, and the number of existing objects, etc. Using these criteria, the MDS selects OSTs with more free space more often than OSTs with less free space.</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1297309" xreflabel=""/><emphasis role="bold">RR</emphasis></para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296534" xreflabel=""/>Round-Robin (RR) allocates objects evenly across all OSTs. The RR stripe allocator is faster than QOS, and used often because it distributes space usage/load best in most situations, maximizing network balancing and improving performance.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296535" xreflabel=""/>Whether QOS or RR is used depends on the setting of the qos_threshold_rr proc tunable. The qos_threshold_rr variable specifies a percentage threshold where the use of QOS or RR becomes more/less likely. The qos_threshold_rr tunable can be set as an integer, from 0 to 100, and results in this stripe allocation behavior:</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296536" xreflabel=""/> If qos_threshold_rr is set to 0, then QOS is always used</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296537" xreflabel=""/> If qos_threshold_rr is set to 100, then RR is always used</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296538" xreflabel=""/> The larger the qos_threshold_rr setting, the greater the possibility that RR is used instead of QOS</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
</section>
</section>
</section>
- <section remap="h2">
- <title>31.2
<anchor xml:id="dbdoclet.50438271_78950" xreflabel=""/>Lustre I/O <anchor xml:id="dbdoclet.50438271_marker-1290508" xreflabel=""/>Tunables</title>
+ <section xml:id="dbdoclet.50438271_78950">
+ <title>31.2 Lustre I/O <anchor xml:id="dbdoclet.50438271_marker-1290508" xreflabel=""/>Tunables</title>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290510" xreflabel=""/>The section describes I/O tunables.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290511" xreflabel=""/><emphasis role="bold">/proc/fs/lustre/llite/<fsname>-<uid>/max_cache_mb</emphasis></para>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1290512" xreflabel=""/># cat /proc/fs/lustre/llite/lustre-ce63ca00/max_cached_mb 128
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290531" xreflabel=""/><emphasis role="bold">/proc/fs/lustre/osc/<object name>/max_rpcs_in_flight</emphasis></para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296189" xreflabel=""/>This tunable is the maximum number of concurrent RPCs in flight from an OSC to its OST. If the OSC tries to initiate an RPC but finds that it already has the same number of RPCs outstanding, it will wait to issue further RPCs until some complete. The minimum setting is 1 and maximum setting is 32. If you are looking to improve small file I/O performance, increase the max_rpcs_in_flight value.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290533" xreflabel=""/>To maximize performance, the value for max_dirty_mb is recommended to be 4 * max_pages_per_rpc * max_rpcs_in_flight.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438271_pgfId-1290534" xreflabel=""/>The <object name> varies depending on the specific Lustre configuration. For <object name> examples, refer to the sample command output.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>The <object name> varies depending on the specific Lustre configuration. For <object name> examples, refer to the sample command output.</para></note>
</section>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438271_pgfId-1290536" xreflabel=""/>31.2.2 Watching the <anchor xml:id="dbdoclet.50438271_marker-1290535" xreflabel=""/>Client RPC Stream</title>
</row>
<row>
<entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292066" xreflabel=""/><emphasis role="bold">Offset</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292068" xreflabel=""/>Difference from the previous range end to the current range start.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292069" xreflabel=""/>For example, Smallest-Extent indicates that the writes in the range 100 to 1110 were sequential, with a minimum write of 10 and a maximum write of 500. This range was started with an offset of -150. That means this is the difference between the last entry
’s range-end and this entry’s range-start for the same file.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292070" xreflabel=""/>The rw_offset_stats file can be cleared by writing to it:</para><screen><anchor xml:id="dbdoclet.50438271_pgfId-1292071" xreflabel=""/>
+ <entry><para> <anchor xml:id="dbdoclet.50438271_pgfId-1292068" xreflabel=""/>Difference from the previous range end to the current range start.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292069" xreflabel=""/>For example, Smallest-Extent indicates that the writes in the range 100 to 1110 were sequential, with a minimum write of 10 and a maximum write of 500. This range was started with an offset of -150. That means this is the difference between the last entry
's range-end and this entry's range-start for the same file.</para><para><anchor xml:id="dbdoclet.50438271_pgfId-1292070" xreflabel=""/>The rw_offset_stats file can be cleared by writing to it:</para><screen><anchor xml:id="dbdoclet.50438271_pgfId-1292071" xreflabel=""/>
<anchor xml:id="dbdoclet.50438271_pgfId-1292072" xreflabel=""/>echo > /proc/fs/lustre/llite/lustre-f57dee00/rw_offset_stats
</screen></entry>
</row>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299055" xreflabel=""/> Number of requests</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299083" xreflabel=""/> Request wait time (avg, min, max and std dev)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299086" xreflabel=""/> Service idle time (% of elapsed time)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299058" xreflabel=""/>Additionally, data on each Lustre service is provided by service type:</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299059" xreflabel=""/> Number of requests of this type</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299092" xreflabel=""/> Request service time (avg, min, max and std dev)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
</section>
<section remap="h3">
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1295517" xreflabel=""/> Many clients are accessing the same data set (as in HPC applications and when diskless clients boot from Lustre)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1295521" xreflabel=""/> One client is storing data while another client is reading it (essentially exchanging data via the OST)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1295528" xreflabel=""/> A client has very limited caching of its own</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1295515" xreflabel=""/>OSS read cache offers these benefits:</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1295469" xreflabel=""/> Allows OSTs to cache read data more frequently</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1295485" xreflabel=""/> Improves repeated reads to match network speeds instead of disk speeds</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1295488" xreflabel=""/> Provides the building blocks for OST write cache (small-write aggregation)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<section remap="h4">
<title><anchor xml:id="dbdoclet.50438271_pgfId-1295497" xreflabel=""/>31.2.7.1 Using OSS Read Cache</title>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296609" xreflabel=""/>read_cache_enable controls whether data read from disk during a read request is kept in memory and available for later read requests for the same data, without having to re-read it from disk. By default, read cache is enabled (read_cache_enable = 1).</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296624" xreflabel=""/>When the OSS receives a read request from a client, it reads data from disk into its memory and sends the data as a reply to the requests. If read cache is enabled, this data stays in memory after the client’s request is finished, and the OSS skips reading data from disk when subsequent read requests for the same are received. The read cache is managed by the Linux kernel globally across all OSTs on that OSS, and the least recently used cache pages will be dropped from memory when the amount of free memory is running low.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1295918" xreflabel=""/>If read cache is disabled (read_cache_enable = 0), then the OSS will discard the data after the client’s read requests are serviced and, for subsequent read requests, the OSS must read the data from disk.</para>
+ <para><anchor xml:id="dbdoclet.50438271_pgfId-1296624" xreflabel=""/>When the OSS receives a read request from a client, it reads data from disk into its memory and sends the data as a reply to the requests. If read cache is enabled, this data stays in memory after the client's request is finished, and the OSS skips reading data from disk when subsequent read requests for the same are received. The read cache is managed by the Linux kernel globally across all OSTs on that OSS, and the least recently used cache pages will be dropped from memory when the amount of free memory is running low.</para>
+ <para><anchor xml:id="dbdoclet.50438271_pgfId-1295918" xreflabel=""/>If read cache is disabled (read_cache_enable = 0), then the OSS will discard the data after the client's read requests are serviced and, for subsequent read requests, the OSS must read the data from disk.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296677" xreflabel=""/>To disable read cache on all OSTs of an OSS, run:</para>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1296709" xreflabel=""/>root@oss1# lctl set_param obdfilter.*.read_cache_enable=0
</screen>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296775" xreflabel=""/>writethrough_cache_enable controls whether data sent to the OSS as a write request is kept in the read cache and available for later reads, or if it is discarded from cache when the write is completed. By default, writethrough cache is enabled (writethrough_cache_enable = 1).</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296844" xreflabel=""/>When the OSS receives write requests from a client, it receives data from the client into its memory and writes the data to disk. If writethrough cache is enabled, this data stays in memory after the write request is completed, allowing the OSS to skip reading this data from disk if a later read request, or partial-page write request, for the same data is received.</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1296871" xreflabel=""/>If writethrough cache is disabled (writethrough_cache_enabled = 0), then the OSS discards the data after the client’s write request is completed, and for subsequent read request, or partial-page write request, the OSS must re-read the data from disk.</para>
+ <para><anchor xml:id="dbdoclet.50438271_pgfId-1296871" xreflabel=""/>If writethrough cache is disabled (writethrough_cache_enabled = 0), then the OSS discards the data after the client's write request is completed, and for subsequent read request, or partial-page write request, the OSS must re-read the data from disk.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296892" xreflabel=""/>Enabling writethrough cache is advisable if clients are doing small or unaligned writes that would cause partial-page updates, or if the files written by one node are immediately being accessed by other nodes. Some examples where this might be useful include producer-consumer I/O models or shared-file writes with a different node doing I/O not aligned on 4096-byte boundaries. Disabling writethrough cache is advisable in the case where files are mostly written to the file system but are not re-read within a short time period, or files are only written and re-read by the same node, regardless of whether the I/O is aligned or not.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1296922" xreflabel=""/>To disable writethrough cache on all OSTs of an OSS, run:</para>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1296924" xreflabel=""/>root@oss1# lctl set_param obdfilter.*.writethrough_cache_enable=0
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1297052" xreflabel=""/>readcache_max_filesize controls the maximum size of a file that both the read cache and writethrough cache will try to keep in memory. Files larger than readcache_max_filesize will not be kept in cache for either reads or writes.</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1297105" xreflabel=""/>This can be very useful for workloads where relatively small files are repeatedly accessed by many clients, such as job startup files, executables, log files, etc., but large files are read or written only once. By not putting the larger files into the cache, it is much more likely that more of the smaller files will remain in cache for a longer time.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1297126" xreflabel=""/>When setting readcache_max_filesize, the input value can be specified in bytes, or can have a suffix to indicate other binary units such as <emphasis role="bold">K</emphasis>ilobytes, <emphasis role="bold">M</emphasis>egabytes, <emphasis role="bold">G</emphasis>igabytes, <emphasis role="bold">T</emphasis>erabytes, or <emphasis role="bold">P</emphasis>etabytes.</para>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438271_pgfId-1300258" xreflabel=""/>31.2.8 OSS Asynchronous Journal Commit</title>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1300265" xreflabel=""/>The OSS asynchronous journal commit feature synchronously writes data to disk without forcing a journal flush. This reduces the number of seeks and significantly improves performance on some hardware.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438271_pgfId-1300763" xreflabel=""/>Asynchronous journal commit cannot work with O_DIRECT writes, a journal flush is still forced.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>Asynchronous journal commit cannot work with O_DIRECT writes, a journal flush is still forced.</para></note>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1300718" xreflabel=""/>When asynchronous journal commit is enabled, client nodes keep data in the page cache (a page reference). Lustre clients monitor the last committed transaction number (transno) in messages sent from the OSS to the clients. When a client sees that the last committed transno reported by the OSS is >=bulk write transno, it releases the reference on the corresponding pages. To avoid page references being held for too long on clients after a bulk write, a 7 second ping request is scheduled (jbd commit time is 5 seconds) after the bulk write reply is received, so the OSS has an opportunity to report the last committed transno.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1300661" xreflabel=""/>If the OSS crashes before the journal commit occurs, then the intermediate data is lost. However, new OSS recovery functionality (introduced in the asynchronous journal commit feature), causes clients to replay their write requests and compensate for the missing disk updates by restoring the state of the file system.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1300287" xreflabel=""/>To enable asynchronous journal commit, set the sync_journal parameter to zero (sync_journal=0):</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290801" xreflabel=""/> Pre-allocation for single files (helps to resist fragmentation)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290802" xreflabel=""/> Pre-allocation for a group of files (helps to pack small files into large, contiguous chunks)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290803" xreflabel=""/> Stream allocation (helps to decrease the seek rate)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290830" xreflabel=""/>The following mballoc3 tunables are available:</para>
<informaltable frame="all">
<title><anchor xml:id="dbdoclet.50438271_pgfId-1290875" xreflabel=""/>31.2.11 <anchor xml:id="dbdoclet.50438271_13474" xreflabel=""/>Lo<anchor xml:id="dbdoclet.50438271_marker-1290874" xreflabel=""/>cking</title>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290876" xreflabel=""/><emphasis role="bold">/proc/fs/lustre/ldlm/ldlm/namespaces/<OSC name|MDC name>/lru_size</emphasis></para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290877" xreflabel=""/>The lru_size parameter is used to control the number of client-side locks in an LRU queue. LRU size is dynamic, based on load. This optimizes the number of locks available to nodes that have different workloads (e.g., login/build nodes vs. compute nodes vs. backup nodes).</para>
- <para><anchor xml:id="dbdoclet.50438271_pgfId-1297289" xreflabel=""/>The total number of locks available is a function of the server’s RAM. The default limit is 50 locks/1 MB of RAM. If there is too much memory pressure, then the LRU size is shrunk. The number of locks on the server is limited to {number of OST/MDT on node} * {number of clients} * {client lru_size}.</para>
+ <para><anchor xml:id="dbdoclet.50438271_pgfId-1297289" xreflabel=""/>The total number of locks available is a function of the server's RAM. The default limit is 50 locks/1 MB of RAM. If there is too much memory pressure, then the LRU size is shrunk. The number of locks on the server is limited to {number of OST/MDT on node} * {number of clients} * {client lru_size}.</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1294671" xreflabel=""/> To enable automatic LRU sizing, set the lru_size parameter to 0. In this case, the lru_size parameter shows the current number of locks being used on the export. (In Lustre 1.6.5.1 and later, LRU sizing is enabled, by default.)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1294717" xreflabel=""/> To specify a maximum number of locks, set the lru_size parameter to a value > 0 (former numbers are okay, 100 * CPU_NR). We recommend that you only increase the LRU size on a few login nodes where users access the file system interactively.</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290878" xreflabel=""/>To clear the LRU on a single client, and as a result flush client cache, without changing the lru_size value:</para>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1294737" xreflabel=""/>$ lctl set_param ldlm.namespaces.<osc_name|mdc_name>.lru_size=clear
</screen>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290880" xreflabel=""/>If you shrink the LRU size below the number of existing unused locks, then the unused locks are canceled immediately. Use echo clear to cancel all locks without changing the value.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438271_pgfId-1298435" xreflabel=""/>Currently, the lru_size parameter can only be set temporarily with lctl set_param; it cannot be set permanently.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>Currently, the lru_size parameter can only be set temporarily with lctl set_param; it cannot be set permanently.</para></note>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1294998" xreflabel=""/>To disable LRU sizing, run this command on the Lustre clients:</para>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1295001" xreflabel=""/>$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
</screen>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299411" xreflabel=""/> To temporarily set this tunable, run:</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1299412" xreflabel=""/># lctl {get,set}_param {service}.thread_{min,max,started}
</screen>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299413" xreflabel=""/> To permanently set this tunable, run:</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1299414" xreflabel=""/># lctl conf_param {service}.thread_{min,max,started} </screen>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299409" xreflabel=""/>The following examples show how to set thread counts and get the number of running threads for the ost_io service.</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299429" xreflabel=""/> To get the number of running threads, run:</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1299270" xreflabel=""/># lctl get_param ost.OSS.ost_io.threads_started</screen>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299276" xreflabel=""/>The command output will be similar to this:</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299257" xreflabel=""/> To set the maximum number of threads (512), run:</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1299294" xreflabel=""/># lctl get_param ost.OSS.ost_io.threads_max
</screen>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299306" xreflabel=""/> To set the maximum thread count to 256 instead of 512 (to avoid overloading the storage or for an array with requests), run:</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1299316" xreflabel=""/># lctl set_param ost.OSS.ost_io.threads_max=256
</screen>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299368" xreflabel=""/> To check if the new threads_max setting is active, run:</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1299375" xreflabel=""/># lctl get_param ost.OSS.ost_io.threads_max
</screen>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1299381" xreflabel=""/>The command output will be similar to this:</para>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1299382" xreflabel=""/>ost.OSS.ost_io.threads_max=256
</screen>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438271_pgfId-1298512" xreflabel=""/>Currently, the maximum thread count setting is advisory because Lustre does not reduce the number of service threads in use, even if that number exceeds the threads_max value. Lustre does not stop service threads once they are started.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>Currently, the maximum thread count setting is advisory because Lustre does not reduce the number of service threads in use, even if that number exceeds the threads_max value. Lustre does not stop service threads once they are started.</para></note>
</section>
</section>
- <section remap="h2">
- <title>31.3 <anchor xml:id="dbdoclet.50438271_83523" xreflabel=""/>Debug</title>
+ <section xml:id="dbdoclet.50438271_83523">
+ <title>31.3 Debug</title>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290884" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/debug</emphasis></para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1298213" xreflabel=""/>By default, Lustre generates a detailed log of all operations to aid in debugging. The level of debugging can affect the performance or speed you achieve with Lustre. Therefore, it is useful to reduce this overhead by turning down the debug level<footnote><para><anchor xml:id="dbdoclet.50438271_pgfId-1298216" xreflabel=""/>This controls the level of Lustre debugging kept in the internal log buffer. It does not alter the level of debugging that goes to syslog.</para></footnote> to improve performance. Raise the debug level when you need to collect the logs for debugging problems. The debugging mask can be set with "symbolic names" instead of the numerical values that were used in prior releases. The new symbolic format is shown in the examples below.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438271_pgfId-1298217" xreflabel=""/>All of the commands below must be run as root; note the # nomenclature.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>All of the commands below must be run as root; note the # nomenclature.</para></note>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1298218" xreflabel=""/>To verify the debug level used by examining the sysctl that controls debugging, run:</para>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1297891" xreflabel=""/># sysctl lnet.debug
<anchor xml:id="dbdoclet.50438271_pgfId-1297995" xreflabel=""/>lnet.debug = ioctl neterror warning error emerg ha config console
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290899" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/debug_path</emphasis></para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290900" xreflabel=""/>This indicates the location where debugging symbols should be stored for gdb. The default is set to /r/tmp/lustre-log-localhost.localdomain.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1290901" xreflabel=""/>These values can also be set via sysctl -w lnet.debug={value}</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438271_pgfId-1290902" xreflabel=""/>The above entries only exist when Lustre has already been loaded.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>The above entries only exist when Lustre has already been loaded.</para></note>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1294968" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/panic_on_lbug</emphasis></para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1294969" xreflabel=""/>This causes Lustre to call ''panic'' when it detects an internal problem (an LBUG); panic crashes the node. This is particularly useful when a kernel crash dump utility is configured. The crash dump is triggered when the internal inconsistency is detected by Lustre.</para>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1294977" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/upcall</emphasis></para>
</screen>
<section remap="h4">
<title><anchor xml:id="dbdoclet.50438271_pgfId-1290921" xreflabel=""/>31.3.1.1 Interpreting OST Statistics</title>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438271_pgfId-1297352" xreflabel=""/>See also <link xl:href="SystemConfigurationUtilities.html#50438219_84890">llobdstat</link> and <link xl:href="LustreMonitoring.html#50438273_80593">CollectL</link>.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>See also <xref linkend="dbdoclet.50438219_84890"/> (llobdstat) and <xref linend="dbdoclet.50438273_80593"/> (CollectL).</para></note>
+
<para><anchor xml:id="dbdoclet.50438271_pgfId-1301139" xreflabel=""/>The OST .../stats files can be used to track client statistics (client activity) for each OST. It is possible to get a periodic dump of values from these file (for example, every 10 seconds), that show the RPC rates (similar to iostat) by using the llstat.pl tool:</para>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1290922" xreflabel=""/># llstat /proc/fs/lustre/osc/lustre-OST0000-osc/stats
<anchor xml:id="dbdoclet.50438271_pgfId-1290923" xreflabel=""/>/usr/bin/llstat: STATS on 09/14/07 /proc/fs/lustre/osc/lustre-OST0000-osc/s\
</section>
<section remap="h4">
<title><anchor xml:id="dbdoclet.50438271_pgfId-1291461" xreflabel=""/>31.3.1.2 Interpreting MDT Statistics</title>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438271_pgfId-1301159" xreflabel=""/>See also <link xl:href="SystemConfigurationUtilities.html#50438219_84890">llobdstat</link> and <link xl:href="LustreMonitoring.html#50438273_80593">CollectL</link>.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>See also <xref linkend="dbdoclet.50438219_84890"/> (llobdstat) and <xref linkend="dbdoclet.50438273_80593"/> (CollectL).</para></note>
<para><anchor xml:id="dbdoclet.50438271_pgfId-1297382" xreflabel=""/>The MDT .../stats files can be used to track MDT statistics for the MDS. Here is sample output for an MDT stats file:</para>
<screen><anchor xml:id="dbdoclet.50438271_pgfId-1297438" xreflabel=""/># cat /proc/fs/lustre/mds/*-MDT0000/stats
<anchor xml:id="dbdoclet.50438271_pgfId-1297466" xreflabel=""/>snapshot_time 1244832003.676892 secs.usecs
</section>
</section>
</section>
- </section>
</chapter>
git://git.whamcloud.com / doc / manual.git / commitdiff