<?xml version='1.0' encoding='UTF-8'?>
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0"
xml:lang="en-US" xml:id="lustreproc">
- <title xml:id="lustreproc.title">LustreProc</title>
- <para>The <literal>/proc</literal> file system acts as an interface to internal data structures in
- the kernel. This chapter describes entries in <literal>/proc</literal> that are useful for
- tuning and monitoring aspects of a Lustre file system. It includes these sections:</para>
+ <title xml:id="lustreproc.title">Lustre Parameters</title>
+ <para>The <literal>/proc</literal> and <literal>/sys</literal> file systems
+ acts as an interface to internal data structures in the kernel. This chapter
+ describes parameters and tunables that are useful for optimizing and
+ monitoring aspects of a Lustre file system. It includes these sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438271_83523"/></para>
</listitem>
</itemizedlist>
<section>
- <title>Introduction to <literal>/proc</literal></title>
- <para>The <literal>/proc</literal> directory provides an interface to internal data structures
- in the kernel that enables monitoring and tuning of many aspects of Lustre file system and
- application performance These data structures include settings and metrics for components such
- as memory, networking, file systems, and kernel housekeeping routines, which are available
- throughout the hierarchical file layout in <literal>/proc.</literal>
+ <title>Introduction to Lustre Parameters</title>
+ <para>Lustre parameters and statistics files provide an interface to
+ internal data structures in the kernel that enables monitoring and
+ tuning of many aspects of Lustre file system and application performance.
+ These data structures include settings and metrics for components such
+ as memory, networking, file systems, and kernel housekeeping routines,
+ which are available throughout the hierarchical file layout.
</para>
- <para>Typically, metrics are accessed by reading from <literal>/proc</literal> files and
- settings are changed by writing to <literal>/proc</literal> files. Some data is server-only,
- some data is client-only, and some data is exported from the client to the server and is thus
- duplicated in both locations.</para>
+ <para>Typically, metrics are accessed via <literal>lctl get_param</literal>
+ files and settings are changed by via <literal>lctl set_param</literal>.
+ Some data is server-only, some data is client-only, and some data is
+ exported from the client to the server and is thus duplicated in both
+ locations.</para>
<note>
- <para>In the examples in this chapter, <literal>#</literal> indicates a command is entered as
- root. Servers are named according to the convention
- <literal><replaceable>fsname</replaceable>-<replaceable>MDT|OSTnumber</replaceable></literal>.
+ <para>In the examples in this chapter, <literal>#</literal> indicates
+ a command is entered as root. Lustre servers are named according to the
+ convention <literal><replaceable>fsname</replaceable>-<replaceable>MDT|OSTnumber</replaceable></literal>.
The standard UNIX wildcard designation (*) is used.</para>
</note>
- <para>In most cases, information is accessed using the <literal>lctl get_param</literal> command
- and settings are changed using the <literal>lctl set_param</literal> command. Some examples
- are shown below:</para>
+ <para>Some examples are shown below:</para>
<itemizedlist>
<listitem>
<para> To obtain data from a Lustre client:</para>
osc.testfs-OST0006-osc-ffff881071d5cc00
osc.testfs-OST0007-osc-ffff881071d5cc00
osc.testfs-OST0008-osc-ffff881071d5cc00</screen>
- <para>In this example, information about OST connections available on a client is displayed
- (indicated by "osc").</para>
+ <para>In this example, information about OST connections available
+ on a client is displayed (indicated by "osc").</para>
</listitem>
</itemizedlist>
<itemizedlist>
</itemizedlist>
<itemizedlist>
<listitem>
- <para> To view a specific file, use <literal>lctl get_param</literal>
- :<screen># lctl get_param osc.lustre-OST0000-osc-ffff881071d5cc00.rpc_stats</screen></para>
+ <para> To view a specific file, use <literal>lctl get_param</literal>:
+ <screen># lctl get_param osc.lustre-OST0000*.rpc_stats</screen></para>
</listitem>
</itemizedlist>
<para>For more information about using <literal>lctl</literal>, see <xref
xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438194_51490"/>.</para>
- <para>Data can also be viewed using the <literal>cat</literal> command with the full path to the
- file. The form of the <literal>cat</literal> command is similar to that of the <literal>lctl
- get_param</literal> command with these differences. In the <literal>cat</literal> command: </para>
+ <para>Data can also be viewed using the <literal>cat</literal> command
+ with the full path to the file. The form of the <literal>cat</literal>
+ command is similar to that of the <literal>lctl get_param</literal>
+ command with some differences. Unfortunately, as the Linux kernel has
+ changed over the years, the location of statistics and parameter files
+ has also changed, which means that the Lustre parameter files may be
+ located in either the <literal>/proc</literal> directory, in the
+ <literal>/sys</literal> directory, and/or in the
+ <literal>/sys/kernel/debug</literal> directory, depending on the kernel
+ version and the Lustre version being used. The <literal>lctl</literal>
+ command insulates scripts from these changes and is preferred over direct
+ file access, unless as part of a high-performance monitoring system.
+ In the <literal>cat</literal> command:</para>
<itemizedlist>
<listitem>
- <para> Replace the dots in the path with slashes.</para>
+ <para>Replace the dots in the path with slashes.</para>
</listitem>
<listitem>
- <para> Prepend the path with the following as
- appropriate:<screen>/proc/{fs,sys}/{lustre,lnet}</screen></para>
+ <para>Prepend the path with the following as appropriate:
+ <screen>/{proc,sys}/{fs,sys}/{lustre,lnet}</screen></para>
</listitem>
</itemizedlist>
<para>For example, an <literal>lctl get_param</literal> command may look like
osc.testfs-OST0000-osc-ffff881071d5cc00.uuid=594db456-0685-bd16-f59b-e72ee90e9819
osc.testfs-OST0001-osc-ffff881071d5cc00.uuid=594db456-0685-bd16-f59b-e72ee90e9819
...</screen></para>
- <para>The equivalent <literal>cat</literal> command looks like
- this:<screen># cat /proc/fs/lustre/osc/*/uuid
+ <para>The equivalent <literal>cat</literal> command may look like this:
+ <screen># cat /proc/fs/lustre/osc/*/uuid
594db456-0685-bd16-f59b-e72ee90e9819
594db456-0685-bd16-f59b-e72ee90e9819
...</screen></para>
- <para>The <literal>llstat</literal> utility can be used to monitor some Lustre file system I/O
- activity over a specified time period. For more details, see <xref
- xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438219_23232"/></para>
- <para>Some data is imported from attached clients and is available in a directory called
- <literal>exports</literal> located in the corresponding per-service directory on a Lustre
- server. For
- example:<screen># ls /proc/fs/lustre/obdfilter/testfs-OST0000/exports/192.168.124.9\@o2ib1/
+ <para>or like this:
+ <screen># cat /sys/fs/lustre/osc/*/uuid
+594db456-0685-bd16-f59b-e72ee90e9819
+594db456-0685-bd16-f59b-e72ee90e9819
+...</screen></para>
+ <para>The <literal>llstat</literal> utility can be used to monitor some
+ Lustre file system I/O activity over a specified time period. For more
+ details, see
+ <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438219_23232"/></para>
+ <para>Some data is imported from attached clients and is available in a
+ directory called <literal>exports</literal> located in the corresponding
+ per-service directory on a Lustre server. For example:
+ <screen>oss:/root# lctl list_param obdfilter.testfs-OST0000.exports.*
# hash ldlm_stats stats uuid</screen></para>
<section remap="h3">
<title>Identifying Lustre File Systems and Servers</title>
- <para>Several <literal>/proc</literal> files on the MGS list existing Lustre file systems and
- file system servers. The examples below are for a Lustre file system called
+ <para>Several <literal>/proc</literal> files on the MGS list existing
+ Lustre file systems and file system servers. The examples below are for
+ a Lustre file system called
<literal>testfs</literal> with one MDT and three OSTs.</para>
<itemizedlist>
<listitem>
</tbody>
</tgroup>
</informaltable>
+ <para>Buddy group cache information found in
+ <literal>/proc/fs/ldiskfs/<replaceable>disk_device</replaceable>/mb_groups</literal> may
+ be useful for assessing on-disk fragmentation. For
+ example:<screen>cat /proc/fs/ldiskfs/loop0/mb_groups
+#group: free free frags first pa [ 2^0 2^1 2^2 2^3 2^4 2^5 2^6 2^7 2^8 2^9
+ 2^10 2^11 2^12 2^13]
+#0 : 2936 2936 1 42 0 [ 0 0 0 1 1 1 1 2 0 1
+ 2 0 0 0 ]</screen></para>
+ <para>In this example, the columns show:<itemizedlist>
+ <listitem>
+ <para>#group number</para>
+ </listitem>
+ <listitem>
+ <para>Available blocks in the group</para>
+ </listitem>
+ <listitem>
+ <para>Blocks free on a disk</para>
+ </listitem>
+ <listitem>
+ <para>Number of free fragments</para>
+ </listitem>
+ <listitem>
+ <para>First free block in the group</para>
+ </listitem>
+ <listitem>
+ <para>Number of preallocated chunks (not blocks)</para>
+ </listitem>
+ <listitem>
+ <para>A series of available chunks of different sizes</para>
+ </listitem>
+ </itemizedlist></para>
</section>
<section>
<title>Monitoring Lustre File System I/O</title>
<para>The header information includes:</para>
<itemizedlist>
<listitem>
- <para><literal>snapshot_time</literal> - UNIX* epoch instant the file was read.</para>
+ <para><literal>snapshot_time</literal> - UNIX epoch instant the file was read.</para>
</listitem>
<listitem>
<para><literal>read RPCs in flight</literal> - Number of read RPCs issued by the OSC, but
<para>For information about optimizing the client I/O RPC stream, see <xref
xmlns:xlink="http://www.w3.org/1999/xlink" linkend="TuningClientIORPCStream"/>.</para>
</section>
- <section remap="h3">
- <title><indexterm>
- <primary>proc</primary>
- <secondary>read/write survey</secondary>
- </indexterm>Monitoring Client Read-Write Offset Statistics</title>
- <para>The <literal>offset_stats</literal> parameter maintains statistics for occurrences of a
- series of read or write calls from a process that did not access the next sequential
- location. The <literal>OFFSET</literal> field is reset to 0 (zero) whenever a different file
- is read or written.</para>
- <para>Read/write offset statistics are "off" by default. The statistics can be activated by
- writing anything into the <literal>offset_stats</literal> file.</para>
- <para>The <literal>offset_stats</literal> file can be cleared by
- entering:<screen>lctl set_param llite.*.offset_stats=0</screen></para>
- <para><emphasis role="italic"><emphasis role="bold">Example:</emphasis></emphasis></para>
- <screen># lctl get_param llite.testfs-f57dee0.offset_stats
-snapshot_time: 1155748884.591028 (secs.usecs)
- RANGE RANGE SMALLEST LARGEST
-R/W PID START END EXTENT EXTENT OFFSET
-R 8385 0 128 128 128 0
-R 8385 0 224 224 224 -128
-W 8385 0 250 50 100 0
-W 8385 100 1110 10 500 -150
-W 8384 0 5233 5233 5233 0
-R 8385 500 600 100 100 -610</screen>
- <para>In this example, <literal>snapshot_time</literal> is the UNIX epoch instant the file was
- read. The tabular data is described in the table below.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry>
- <para><emphasis role="bold">Field</emphasis></para>
- </entry>
- <entry>
- <para><emphasis role="bold">Description</emphasis></para>
- </entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry>
- <para>R/W</para>
- </entry>
- <entry>
- <para>Indicates if the non-sequential call was a read or write</para>
- </entry>
- </row>
- <row>
- <entry>
- <para>PID </para>
- </entry>
- <entry>
- <para>Process ID of the process that made the read/write call.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para>RANGE START/RANGE END</para>
- </entry>
- <entry>
- <para>Range in which the read/write calls were sequential.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para>SMALLEST EXTENT </para>
- </entry>
- <entry>
- <para>Smallest single read/write in the corresponding range (in bytes).</para>
- </entry>
- </row>
- <row>
- <entry>
- <para>LARGEST EXTENT </para>
- </entry>
- <entry>
- <para>Largest single read/write in the corresponding range (in bytes).</para>
- </entry>
- </row>
- <row>
- <entry>
- <para>OFFSET </para>
- </entry>
- <entry>
- <para>Difference between the previous range end and the current range start.</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><emphasis role="italic"><emphasis role="bold">Analysis:</emphasis></emphasis></para>
- <para>This data provides an indication of how contiguous or fragmented the data is. For
- example, the fourth entry in the example above shows the writes for this RPC were sequential
- in the range 100 to 1110 with the minimum write 10 bytes and the maximum write 500 bytes.
- The range started with an offset of -150 from the <literal>RANGE END</literal> of the
- previous entry in the example.</para>
- </section>
<section xml:id="lustreproc.clientstats" remap="h3">
<title><indexterm>
<primary>proc</primary>
<para>The <literal>stats</literal> file maintains statistics accumulate during typical
operation of a client across the VFS interface of the Lustre file system. Only non-zero
parameters are displayed in the file. </para>
- <para>Client statistics are enabled by default. The statistics can be cleared by echoing an
- empty string into the <literal>stats</literal> file or by using the command:
- <screen>lctl set_param llite.*.stats=0</screen></para>
+ <para>Client statistics are enabled by default.</para>
<note>
<para>Statistics for all mounted file systems can be discovered by
entering:<screen>lctl get_param llite.*.stats</screen></para>
setxattr 19059 samples [regs]
getxattr 61169 samples [regs]
</screen>
+ <para> The statistics can be cleared by echoing an empty string into the
+ <literal>stats</literal> file or by using the command:
+ <screen>lctl set_param llite.*.stats=0</screen></para>
<para>The statistics displayed are described in the table below.</para>
<informaltable frame="all">
<tgroup cols="2">
<title><indexterm>
<primary>proc</primary>
<secondary>read/write survey</secondary>
+ </indexterm>Monitoring Client Read-Write Offset Statistics</title>
+ <para>When the <literal>offset_stats</literal> parameter is set, statistics are maintained for
+ occurrences of a series of read or write calls from a process that did not access the next
+ sequential location. The <literal>OFFSET</literal> field is reset to 0 (zero) whenever a
+ different file is read or written.</para>
+ <note>
+ <para>By default, statistics are not collected in the <literal>offset_stats</literal>,
+ <literal>extents_stats</literal>, and <literal>extents_stats_per_process</literal> files
+ to reduce monitoring overhead when this information is not needed. The collection of
+ statistics in all three of these files is activated by writing
+ anything, except for 0 (zero) and "disable", into any one of the
+ files.</para>
+ </note>
+ <para><emphasis role="italic"><emphasis role="bold">Example:</emphasis></emphasis></para>
+ <screen># lctl get_param llite.testfs-f57dee0.offset_stats
+snapshot_time: 1155748884.591028 (secs.usecs)
+ RANGE RANGE SMALLEST LARGEST
+R/W PID START END EXTENT EXTENT OFFSET
+R 8385 0 128 128 128 0
+R 8385 0 224 224 224 -128
+W 8385 0 250 50 100 0
+W 8385 100 1110 10 500 -150
+W 8384 0 5233 5233 5233 0
+R 8385 500 600 100 100 -610</screen>
+ <para>In this example, <literal>snapshot_time</literal> is the UNIX epoch instant the file was
+ read. The tabular data is described in the table below.</para>
+ <para>The <literal>offset_stats</literal> file can be cleared by
+ entering:<screen>lctl set_param llite.*.offset_stats=0</screen></para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Field</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para>R/W</para>
+ </entry>
+ <entry>
+ <para>Indicates if the non-sequential call was a read or write</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>PID </para>
+ </entry>
+ <entry>
+ <para>Process ID of the process that made the read/write call.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>RANGE START/RANGE END</para>
+ </entry>
+ <entry>
+ <para>Range in which the read/write calls were sequential.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>SMALLEST EXTENT </para>
+ </entry>
+ <entry>
+ <para>Smallest single read/write in the corresponding range (in bytes).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>LARGEST EXTENT </para>
+ </entry>
+ <entry>
+ <para>Largest single read/write in the corresponding range (in bytes).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>OFFSET </para>
+ </entry>
+ <entry>
+ <para>Difference between the previous range end and the current range start.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para><emphasis role="italic"><emphasis role="bold">Analysis:</emphasis></emphasis></para>
+ <para>This data provides an indication of how contiguous or fragmented the data is. For
+ example, the fourth entry in the example above shows the writes for this RPC were sequential
+ in the range 100 to 1110 with the minimum write 10 bytes and the maximum write 500 bytes.
+ The range started with an offset of -150 from the <literal>RANGE END</literal> of the
+ previous entry in the example.</para>
+ </section>
+ <section remap="h3">
+ <title><indexterm>
+ <primary>proc</primary>
+ <secondary>read/write survey</secondary>
</indexterm>Monitoring Client Read-Write Extent Statistics</title>
<para>For in-depth troubleshooting, client read-write extent statistics can be accessed to
obtain more detail about read/write I/O extents for the file system or for a particular
process.</para>
+ <note>
+ <para>By default, statistics are not collected in the <literal>offset_stats</literal>,
+ <literal>extents_stats</literal>, and <literal>extents_stats_per_process</literal> files
+ to reduce monitoring overhead when this information is not needed. The collection of
+ statistics in all three of these files is activated by writing
+ anything, except for 0 (zero) and "disable", into any one of the
+ files.</para>
+ </note>
<section remap="h3">
<title>Client-Based I/O Extent Size Survey</title>
- <para>The <literal>rw_extent_stats</literal> histogram in the <literal>llite</literal>
- directory shows the statistics for the sizes of the read?write I/O extents. This file does
- not maintain the per-process statistics. The file can be cleared by issuing the following
- command:<screen># lctl set_param llite.testfs-*.extents_stats=0</screen></para>
+ <para>The <literal>extents_stats</literal> histogram in the
+ <literal>llite</literal> directory shows the statistics for the sizes
+ of the read/write I/O extents. This file does not maintain the per
+ process statistics.</para>
<para><emphasis role="italic"><emphasis role="bold">Example:</emphasis></emphasis></para>
<screen># lctl get_param llite.testfs-*.extents_stats
snapshot_time: 1213828728.348516 (secs.usecs)
read | write
extents calls % cum% | calls % cum%
-
+
0K - 4K : 0 0 0 | 2 2 2
4K - 8K : 0 0 0 | 0 0 2
8K - 16K : 0 0 0 | 0 0 2
was read. The table shows cumulative extents organized according to size with statistics
provided separately for reads and writes. Each row in the table shows the number of RPCs
for reads and writes respectively (<literal>calls</literal>), the relative percentage of
- total calls (<literal>%</literal>), and the cumulative percentage to that point in the
- table of calls (<literal>cum %</literal>). </para>
+ total calls (<literal>%</literal>), and the cumulative percentage to
+ that point in the table of calls (<literal>cum %</literal>). </para>
+ <para> The file can be cleared by issuing the following command:
+ <screen># lctl set_param llite.testfs-*.extents_stats=1</screen></para>
</section>
<section>
<title>Per-Process Client I/O Statistics</title>
<primary>proc</primary>
<secondary>readahead</secondary>
</indexterm>Tuning File Readahead and Directory Statahead</title>
- <para>File readahead and directory statahead enable reading of data into memory before a
- process requests the data. File readahead reads file content data into memory and directory
- statahead reads metadata into memory. When readahead and statahead work well, a process that
- accesses data finds that the information it needs is available immediately when requested in
- memory without the delay of network I/O.</para>
- <para condition="l22">In Lustre release 2.2.0, the directory statahead feature was improved to
- enhance directory traversal performance. The improvements primarily addressed two
- issues:</para>
- <orderedlist condition="l22">
- <listitem>
- <para>A race condition existed between the statahead thread and other VFS operations while
- processing asynchronous <literal>getattr</literal> RPC replies, causing duplicate
- entries in dcache. This issue was resolved by using statahead local dcache. </para>
- </listitem>
- <listitem>
- <para>File size/block attributes pre-fetching was not supported, so the traversing thread
- had to send synchronous glimpse size RPCs to OST(s). This issue was resolved by using
- asynchronous glimpse lock (AGL) RPCs to pre-fetch file size/block attributes from
- OST(s).</para>
- </listitem>
- </orderedlist>
+ <para>File readahead and directory statahead enable reading of data
+ into memory before a process requests the data. File readahead prefetches
+ file content data into memory for <literal>read()</literal> related
+ calls, while directory statahead fetches file metadata into memory for
+ <literal>readdir()</literal> and <literal>stat()</literal> related
+ calls. When readahead and statahead work well, a process that accesses
+ data finds that the information it needs is available immediately in
+ memory on the client when requested without the delay of network I/O.
+ </para>
<section remap="h4">
<title>Tuning File Readahead</title>
- <para>File readahead is triggered when two or more sequential reads by an application fail
- to be satisfied by data in the Linux buffer cache. The size of the initial readahead is 1
- MB. Additional readaheads grow linearly and increment until the readahead cache on the
- client is full at 40 MB.</para>
+ <para>File readahead is triggered when two or more sequential reads
+ by an application fail to be satisfied by data in the Linux buffer
+ cache. The size of the initial readahead is 1 MB. Additional
+ readaheads grow linearly and increment until the readahead cache on
+ the client is full at 40 MB.</para>
<para>Readahead tunables include:</para>
<itemizedlist>
<listitem>
- <para><literal>llite.<replaceable>fsname-instance</replaceable>.max_read_ahead_mb</literal>
- - Controls the maximum amount of data readahead on a file. Files are read ahead in
- RPC-sized chunks (1 MB or the size of the <literal>read()</literal> call, if larger)
- after the second sequential read on a file descriptor. Random reads are done at the
- size of the <literal>read()</literal> call only (no readahead). Reads to
- non-contiguous regions of the file reset the readahead algorithm, and readahead is not
- triggered again until sequential reads take place again. </para>
- <para>To disable readahead, set this tunable to 0. The default value is 40 MB.</para>
+ <para><literal>llite.<replaceable>fsname-instance</replaceable>.max_read_ahead_mb</literal> -
+ Controls the maximum amount of data readahead on a file.
+ Files are read ahead in RPC-sized chunks (1 MB or the size of
+ the <literal>read()</literal> call, if larger) after the second
+ sequential read on a file descriptor. Random reads are done at
+ the size of the <literal>read()</literal> call only (no
+ readahead). Reads to non-contiguous regions of the file reset
+ the readahead algorithm, and readahead is not triggered again
+ until sequential reads take place again.
+ </para>
+ <para>To disable readahead, set
+ <literal>max_read_ahead_mb=0</literal>. The default value is 40 MB.
+ </para>
</listitem>
<listitem>
- <para><literal>llite.<replaceable>fsname-instance</replaceable>.max_read_ahead_whole_mb</literal>
- - Controls the maximum size of a file that is read in its entirety, regardless of the
- size of the <literal>read()</literal>.</para>
+ <para><literal>llite.<replaceable>fsname-instance</replaceable>.max_read_ahead_whole_mb</literal> -
+ Controls the maximum size of a file that is read in its entirety,
+ regardless of the size of the <literal>read()</literal>. This
+ avoids multiple small read RPCs on relatively small files, when
+ it is not possible to efficiently detect a sequential read
+ pattern before the whole file has been read.
+ </para>
</listitem>
</itemizedlist>
</section>
<section>
<title>Tuning Directory Statahead and AGL</title>
- <para>Many system commands, such as <literal>ls –l</literal>, <literal>du</literal>, and
- <literal>find</literal>, traverse a directory sequentially. To make these commands run
- efficiently, the directory statahead and asynchronous glimpse lock (AGL) can be enabled to
- improve the performance of traversing.</para>
+ <para>Many system commands, such as <literal>ls –l</literal>,
+ <literal>du</literal>, and <literal>find</literal>, traverse a
+ directory sequentially. To make these commands run efficiently, the
+ directory statahead can be enabled to improve the performance of
+ directory traversal.</para>
<para>The statahead tunables are:</para>
<itemizedlist>
<listitem>
- <para><literal>statahead_max</literal> - Controls whether directory statahead is enabled
- and the maximum statahead window size (i.e., how many files can be pre-fetched by the
- statahead thread). By default, statahead is enabled and the value of
- <literal>statahead_max</literal> is 32.</para>
- <para>To disable statahead, run:</para>
+ <para><literal>statahead_max</literal> -
+ Controls the maximum number of file attributes that will be
+ prefetched by the statahead thread. By default, statahead is
+ enabled and <literal>statahead_max</literal> is 32 files.</para>
+ <para>To disable statahead, set <literal>statahead_max</literal>
+ to zero via the following command on the client:</para>
<screen>lctl set_param llite.*.statahead_max=0</screen>
- <para>To set the maximum statahead window size (<replaceable>n</replaceable>),
- run:</para>
+ <para>To change the maximum statahead window size on a client:</para>
<screen>lctl set_param llite.*.statahead_max=<replaceable>n</replaceable></screen>
- <para>The maximum value of <replaceable>n</replaceable> is 8192.</para>
- <para>The AGL can be controlled by entering:</para>
- <screen>lctl set_param llite.*.statahead_agl=<replaceable>n</replaceable></screen>
- <para>The default value for <replaceable>n</replaceable> is 1, which enables the AGL. If
- <replaceable>n</replaceable> is 0, the AGL is disabled.</para>
+ <para>The maximum <literal>statahead_max</literal> is 8192 files.
+ </para>
+ <para>The directory statahead thread will also prefetch the file
+ size/block attributes from the OSTs, so that all file attributes
+ are available on the client when requested by an application.
+ This is controlled by the asynchronous glimpse lock (AGL) setting.
+ The AGL behaviour can be disabled by setting:</para>
+ <screen>lctl set_param llite.*.statahead_agl=0</screen>
</listitem>
<listitem>
- <para><literal>statahead_stats</literal> - A read-only interface that indicates the
- current statahead and AGL statistics, such as how many times statahead/AGL has been
- triggered since the last mount, how many statahead/AGL failures have occurred due to
- an incorrect prediction or other causes.</para>
+ <para><literal>statahead_stats</literal> -
+ A read-only interface that provides current statahead and AGL
+ statistics, such as how many times statahead/AGL has been triggered
+ since the last mount, how many statahead/AGL failures have occurred
+ due to an incorrect prediction or other causes.</para>
<note>
- <para>The AGL is affected by statahead because the inodes processed by AGL are built
- by the statahead thread, which means the statahead thread is the input of the AGL
- pipeline. So if statahead is disabled, then the AGL is disabled by force.</para>
+ <para>AGL behaviour is affected by statahead since the inodes
+ processed by AGL are built by the statahead thread. If
+ statahead is disabled, then AGL is also disabled.</para>
</note>
</listitem>
</itemizedlist>
<para>To re-enable the writethrough cache on one OST, run:</para>
<screen>root@oss1# lctl set_param obdfilter.{OST_name}.writethrough_cache_enable=1</screen>
<para>To check if the writethrough cache is enabled, run:</para>
- <screen>root@oss1# lctl set_param obdfilter.*.writethrough_cache_enable=1</screen>
+ <screen>root@oss1# lctl get_param obdfilter.*.writethrough_cache_enable</screen>
</listitem>
<listitem>
<para><literal>readcache_max_filesize</literal> - Controls the maximum size of a file
<screen>$ lctl get_param obdfilter.*.sync_on_lock_cancel
obdfilter.lol-OST0001.sync_on_lock_cancel=never</screen>
</section>
+ <section xml:id="dbdoclet.TuningModRPCs" condition='l28'>
+ <title>
+ <indexterm>
+ <primary>proc</primary>
+ <secondary>client metadata performance</secondary>
+ </indexterm>
+ Tuning the Client Metadata RPC Stream
+ </title>
+ <para>The client metadata RPC stream represents the metadata RPCs issued
+ in parallel by a client to a MDT target. The metadata RPCs can be split
+ in two categories: the requests that do not modify the file system
+ (like getattr operation), and the requests that do modify the file system
+ (like create, unlink, setattr operations). To help optimize the client
+ metadata RPC stream, several tuning variables are provided to adjust
+ behavior according to network conditions and cluster size.</para>
+ <para>Note that increasing the number of metadata RPCs issued in parallel
+ might improve the performance metadata intensive parallel applications,
+ but as a consequence it will consume more memory on the client and on
+ the MDS.</para>
+ <section>
+ <title>Configuring the Client Metadata RPC Stream</title>
+ <para>The MDC <literal>max_rpcs_in_flight</literal> parameter defines
+ the maximum number of metadata RPCs, both modifying and
+ non-modifying RPCs, that can be sent in parallel by a client to a MDT
+ target. This includes every file system metadata operations, such as
+ file or directory stat, creation, unlink. The default setting is 8,
+ minimum setting is 1 and maximum setting is 256.</para>
+ <para>To set the <literal>max_rpcs_in_flight</literal> parameter, run
+ the following command on the Lustre client:</para>
+ <screen>client$ lctl set_param mdc.*.max_rpcs_in_flight=16</screen>
+ <para>The MDC <literal>max_mod_rpcs_in_flight</literal> parameter
+ defines the maximum number of file system modifying RPCs that can be
+ sent in parallel by a client to a MDT target. For example, the Lustre
+ client sends modify RPCs when it performs file or directory creation,
+ unlink, access permission modification or ownership modification. The
+ default setting is 7, minimum setting is 1 and maximum setting is
+ 256.</para>
+ <para>To set the <literal>max_mod_rpcs_in_flight</literal> parameter,
+ run the following command on the Lustre client:</para>
+ <screen>client$ lctl set_param mdc.*.max_mod_rpcs_in_flight=12</screen>
+ <para>The <literal>max_mod_rpcs_in_flight</literal> value must be
+ strictly less than the <literal>max_rpcs_in_flight</literal> value.
+ It must also be less or equal to the MDT
+ <literal>max_mod_rpcs_per_client</literal> value. If one of theses
+ conditions is not enforced, the setting fails and an explicit message
+ is written in the Lustre log.</para>
+ <para>The MDT <literal>max_mod_rpcs_per_client</literal> parameter is a
+ tunable of the kernel module <literal>mdt</literal> that defines the
+ maximum number of file system modifying RPCs in flight allowed per
+ client. The parameter can be updated at runtime, but the change is
+ effective to new client connections only. The default setting is 8.
+ </para>
+ <para>To set the <literal>max_mod_rpcs_per_client</literal> parameter,
+ run the following command on the MDS:</para>
+ <screen>mds$ echo 12 > /sys/module/mdt/parameters/max_mod_rpcs_per_client</screen>
+ </section>
+ <section>
+ <title>Monitoring the Client Metadata RPC Stream</title>
+ <para>The <literal>rpc_stats</literal> file contains histogram data
+ showing information about modify metadata RPCs. It can be helpful to
+ identify the level of parallelism achieved by an application doing
+ modify metadata operations.</para>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>client$ lctl get_param mdc.*.rpc_stats
+snapshot_time: 1441876896.567070 (secs.usecs)
+modify_RPCs_in_flight: 0
+
+ modify
+rpcs in flight rpcs % cum %
+0: 0 0 0
+1: 56 0 0
+2: 40 0 0
+3: 70 0 0
+4 41 0 0
+5: 51 0 1
+6: 88 0 1
+7: 366 1 2
+8: 1321 5 8
+9: 3624 15 23
+10: 6482 27 50
+11: 7321 30 81
+12: 4540 18 100</screen>
+ <para>The file information includes:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>snapshot_time</literal> - UNIX epoch instant the
+ file was read.</para>
+ </listitem>
+ <listitem>
+ <para><literal>modify_RPCs_in_flight</literal> - Number of modify
+ RPCs issued by the MDC, but not completed at the time of the
+ snapshot. This value should always be less than or equal to
+ <literal>max_mod_rpcs_in_flight</literal>.</para>
+ </listitem>
+ <listitem>
+ <para><literal>rpcs in flight</literal> - Number of modify RPCs
+ that are pending when a RPC is sent, the relative percentage
+ (<literal>%</literal>) of total modify RPCs, and the cumulative
+ percentage (<literal>cum %</literal>) to that point.</para>
+ </listitem>
+ </itemizedlist>
+ <para>If a large proportion of modify metadata RPCs are issued with a
+ number of pending metadata RPCs close to the
+ <literal>max_mod_rpcs_in_flight</literal> value, it means the
+ <literal>max_mod_rpcs_in_flight</literal> value could be increased to
+ improve the modify metadata performance.</para>
+ </section>
+ </section>
</section>
<section>
<title>Configuring Timeouts in a Lustre File System</title>
messages or enable printing of <literal>D_NETERROR</literal> messages to the console
using:<screen>lctl set_param printk=+neterror</screen></para>
<para>Congested routers can be a source of spurious LND timeouts. To avoid this
- situation, increase the number of LNET router buffers to reduce back-pressure and/or
+ situation, increase the number of LNet router buffers to reduce back-pressure and/or
increase LND timeouts on all nodes on all connected networks. Also consider increasing
- the total number of LNET router nodes in the system so that the aggregate router
+ the total number of LNet router nodes in the system so that the aggregate router
bandwidth matches the aggregate server bandwidth.</para>
</listitem>
<listitem>
<section remap="h3">
<title><indexterm>
<primary>proc</primary>
- <secondary>LNET</secondary>
+ <secondary>LNet</secondary>
</indexterm><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>proc</secondary>
- </indexterm>Monitoring LNET</title>
- <para>LNET information is located in <literal>/proc/sys/lnet</literal> in these files:<itemizedlist>
+ </indexterm>Monitoring LNet</title>
+ <para>LNet information is located in <literal>/proc/sys/lnet</literal> in these files:<itemizedlist>
<listitem>
<para><literal>peers</literal> - Shows all NIDs known to this node and provides
information on the queue state.</para>
</tgroup>
</informaltable>
<para>Credits are initialized to allow a certain number of operations (in the example
- above the table, eight as shown in the <literal>max</literal> column. LNET keeps track
+ above the table, eight as shown in the <literal>max</literal> column. LNet keeps track
of the minimum number of credits ever seen over time showing the peak congestion that
has occurred during the time monitored. Fewer available credits indicates a more
congested resource. </para>
credits (<literal>rtr/tx</literal>) that is less than <literal>max</literal> indicates
operations are in progress. If the ratio <literal>rtr/tx</literal> is greater than
<literal>max</literal>, operations are blocking.</para>
- <para>LNET also limits concurrent sends and number of router buffers allocated to a single
+ <para>LNet also limits concurrent sends and number of router buffers allocated to a single
peer so that no peer can occupy all these resources.</para>
</listitem>
<listitem>
</listitem>
</itemizedlist></para>
</section>
- <section remap="h3">
+ <section remap="h3" xml:id="dbdoclet.balancing_free_space">
<title><indexterm>
<primary>proc</primary>
<secondary>free space</secondary>
</indexterm>Allocating Free Space on OSTs</title>
- <para>Free space is allocated using either a round-robin or a weighted algorithm. The allocation
- method is determined by the maximum amount of free-space imbalance between the OSTs. When free
- space is relatively balanced across OSTs, the faster round-robin allocator is used, which
- maximizes network balancing. The weighted allocator is used when any two OSTs are out of
- balance by more than a specified threshold.</para>
- <para>Free space distribution can be tuned using these two <literal>/proc</literal>
- tunables:</para>
+ <para>Free space is allocated using either a round-robin or a weighted
+ algorithm. The allocation method is determined by the maximum amount of
+ free-space imbalance between the OSTs. When free space is relatively
+ balanced across OSTs, the faster round-robin allocator is used, which
+ maximizes network balancing. The weighted allocator is used when any two
+ OSTs are out of balance by more than a specified threshold.</para>
+ <para>Free space distribution can be tuned using these two
+ <literal>/proc</literal> tunables:</para>
<itemizedlist>
<listitem>
- <para><literal>qos_threshold_rr</literal> - The threshold at which the allocation method
- switches from round-robin to weighted is set in this file. The default is to switch to the
- weighted algorithm when any two OSTs are out of balance by more than 17 percent.</para>
+ <para><literal>qos_threshold_rr</literal> - The threshold at which
+ the allocation method switches from round-robin to weighted is set
+ in this file. The default is to switch to the weighted algorithm when
+ any two OSTs are out of balance by more than 17 percent.</para>
+ </listitem>
+ <listitem>
+ <para><literal>qos_prio_free</literal> - The weighting priority used
+ by the weighted allocator can be adjusted in this file. Increasing the
+ value of <literal>qos_prio_free</literal> puts more weighting on the
+ amount of free space available on each OST and less on how stripes are
+ distributed across OSTs. The default value is 91 percent weighting for
+ free space rebalancing and 9 percent for OST balancing. When the
+ free space priority is set to 100, weighting is based entirely on free
+ space and location is no longer used by the striping algorithm.</para>
</listitem>
<listitem>
- <para><literal>qos_prio_free</literal> - The weighting priority used by the weighted
- allocator can be adjusted in this file. Increasing the value of
- <literal>qos_prio_free</literal> puts more weighting on the amount of free space
- available on each OST and less on how stripes are distributed across OSTs. The default
- value is 91 percent. When the free space priority is set to 100, weighting is based
- entirely on free space and location is no longer used by the striping algorthm.</para>
+ <para condition="l29"><literal>reserved_mb_low</literal> - The low
+ watermark used to stop object allocation if available space is less
+ than it. The default is 0.1 percent of total OST size.</para>
+ </listitem>
+ <listitem>
+ <para condition="l29"><literal>reserved_mb_high</literal> - The high watermark used to start
+ object allocation if available space is more than it. The default is 0.2 percent of total
+ OST size.</para>
</listitem>
</itemizedlist>
<para>For more information about monitoring and managing free space, see <xref
</row>
<row>
<entry>
- <literal> mdt.MDS.mds </literal>
+ <literal> mds.MDS.mdt </literal>
</entry>
<entry>
<para>Main metadata operations service</para>
</row>
<row>
<entry>
- <literal> mdt.MDS.mds_readpage </literal>
+ <literal> mds.MDS.mdt_readpage </literal>
</entry>
<entry>
<para>Metadata <literal>readdir</literal> service</para>
</row>
<row>
<entry>
- <literal> mdt.MDS.mds_setattr </literal>
+ <literal> mds.MDS.mdt_setattr </literal>
</entry>
<entry>
<para>Metadata <literal>setattr/close</literal> operations service </para>
</tgroup>
</informaltable>
<para>For each service, an entry as shown below is
- created:<screen>/proc/fs/lustre/<replaceable>{service}</replaceable>/*/thread_<replaceable>{min,max,started}</replaceable></screen></para>
- <para>To temporarily set this tunable, run:</para>
- <screen># lctl <replaceable>{get,set}</replaceable>_param <replaceable>{service}</replaceable>.thread_<replaceable>{min,max,started}</replaceable> </screen>
- <para>To permanently set this tunable, run:</para>
- <screen># lctl conf_param <replaceable>{service}</replaceable>.thread_<replaceable>{min,max,started}</replaceable> </screen>
- <para>The following examples show how to set thread counts and get the number of running threads
- for the service <literal>ost_io</literal> using the tunable
- <literal>{service}.thread_{min,max,started}</literal>.</para>
+ created:<screen>/proc/fs/lustre/<replaceable>service</replaceable>/*/threads_<replaceable>min|max|started</replaceable></screen></para>
+ <itemizedlist>
+ <listitem>
+ <para>To temporarily set this tunable, run:</para>
+ <screen># lctl <replaceable>get|set</replaceable>_param <replaceable>service</replaceable>.threads_<replaceable>min|max|started</replaceable> </screen>
+ </listitem>
+ <listitem>
+ <para>To permanently set this tunable, run:</para>
+ <screen># lctl conf_param <replaceable>obdname|fsname.obdtype</replaceable>.threads_<replaceable>min|max|started</replaceable> </screen>
+ <para condition='l25'>For version 2.5 or later, run:
+ <screen># lctl set_param -P <replaceable>service</replaceable>.threads_<replaceable>min|max|started</replaceable></screen></para>
+ </listitem>
+ </itemizedlist>
+ <para>The following examples show how to set thread counts and get the number of running threads
+ for the service <literal>ost_io</literal> using the tunable
+ <literal><replaceable>service</replaceable>.threads_<replaceable>min|max|started</replaceable></literal>.</para>
<itemizedlist>
<listitem>
<para>To get the number of running threads, run:</para>
ost.OSS.ost_io.threads_max=256</screen>
</listitem>
<listitem>
+ <para>To set the maximum thread count to 256 instead of 512 permanently, run:</para>
+ <screen># lctl conf_param testfs.ost.ost_io.threads_max=256</screen>
+ <para condition='l25'>For version 2.5 or later, run:
+ <screen># lctl set_param -P ost.OSS.ost_io.threads_max=256
+ost.OSS.ost_io.threads_max=256 </screen> </para>
+ </listitem>
+ <listitem>
<para> To check if the <literal>threads_max</literal> setting is active, run:</para>
<screen># lctl get_param ost.OSS.ost_io.threads_max
ost.OSS.ost_io.threads_max=256</screen>
<para>See also <xref linkend="dbdoclet.50438219_84890"/> (<literal>llobdstat</literal>) and
<xref linkend="dbdoclet.50438273_80593"/> (<literal>collectl</literal>).</para>
</note>
- <para>MDT <literal>stats</literal> files can be used to track MDT statistics for the MDS. The
- example below shows sample output from an MDT <literal>stats</literal> file.</para>
+ <para>MDT <literal>stats</literal> files can be used to track MDT
+ statistics for the MDS. The example below shows sample output from an
+ MDT <literal>stats</literal> file.</para>
<screen># lctl get_param mds.*-MDT0000.stats
snapshot_time 1244832003.676892 secs.usecs
open 2 samples [reqs]
git://git.whamcloud.com / doc / manual.git / blobdiff