<?xml version='1.0' encoding='UTF-8'?>
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0"
xml:lang="en-US" xml:id="lustreproc">
- <title xml:id="lustreproc.title">LustreProc</title>
- <para>The <literal>/proc</literal> file system acts as an interface to internal data structures in
- the kernel. This chapter describes entries in <literal>/proc</literal> that are useful for
- tuning and monitoring aspects of a Lustre file system. It includes these sections:</para>
+ <title xml:id="lustreproc.title">Lustre Parameters</title>
+ <para>The <literal>/proc</literal> and <literal>/sys</literal> file systems
+ acts as an interface to internal data structures in the kernel. This chapter
+ describes parameters and tunables that are useful for optimizing and
+ monitoring aspects of a Lustre file system. It includes these sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438271_83523"/></para>
</listitem>
</itemizedlist>
<section>
- <title>Introduction to <literal>/proc</literal></title>
- <para>The <literal>/proc</literal> directory provides an interface to internal data structures
- in the kernel that enables monitoring and tuning of many aspects of Lustre file system and
- application performance These data structures include settings and metrics for components such
- as memory, networking, file systems, and kernel housekeeping routines, which are available
- throughout the hierarchical file layout in <literal>/proc.</literal>
+ <title>Introduction to Lustre Parameters</title>
+ <para>Lustre parameters and statistics files provide an interface to
+ internal data structures in the kernel that enables monitoring and
+ tuning of many aspects of Lustre file system and application performance.
+ These data structures include settings and metrics for components such
+ as memory, networking, file systems, and kernel housekeeping routines,
+ which are available throughout the hierarchical file layout.
</para>
- <para>Typically, metrics are accessed by reading from <literal>/proc</literal> files and
- settings are changed by writing to <literal>/proc</literal> files. Some data is server-only,
- some data is client-only, and some data is exported from the client to the server and is thus
- duplicated in both locations.</para>
+ <para>Typically, metrics are accessed via <literal>lctl get_param</literal>
+ files and settings are changed by via <literal>lctl set_param</literal>.
+ While it is possible to access parameters in <literal>/proc</literal>
+ and <literal>/sys</literal> directly, the location of these parameters may
+ change between releases, so it is recommended to always use
+ <literal>lctl</literal> to access the parameters from userspace scripts.
+ Some data is server-only, some data is client-only, and some data is
+ exported from the client to the server and is thus duplicated in both
+ locations.</para>
<note>
- <para>In the examples in this chapter, <literal>#</literal> indicates a command is entered as
- root. Servers are named according to the convention
- <literal><replaceable>fsname</replaceable>-<replaceable>MDT|OSTnumber</replaceable></literal>.
+ <para>In the examples in this chapter, <literal>#</literal> indicates
+ a command is entered as root. Lustre servers are named according to the
+ convention <literal><replaceable>fsname</replaceable>-<replaceable>MDT|OSTnumber</replaceable></literal>.
The standard UNIX wildcard designation (*) is used.</para>
</note>
- <para>In most cases, information is accessed using the <literal>lctl get_param</literal> command
- and settings are changed using the <literal>lctl set_param</literal> command. Some examples
- are shown below:</para>
+ <para>Some examples are shown below:</para>
<itemizedlist>
<listitem>
<para> To obtain data from a Lustre client:</para>
osc.testfs-OST0006-osc-ffff881071d5cc00
osc.testfs-OST0007-osc-ffff881071d5cc00
osc.testfs-OST0008-osc-ffff881071d5cc00</screen>
- <para>In this example, information about OST connections available on a client is displayed
- (indicated by "osc").</para>
+ <para>In this example, information about OST connections available
+ on a client is displayed (indicated by "osc").</para>
</listitem>
</itemizedlist>
<itemizedlist>
</itemizedlist>
<itemizedlist>
<listitem>
- <para> To view a specific file, use <literal>lctl get_param</literal>
- :<screen># lctl get_param osc.lustre-OST0000-osc-ffff881071d5cc00.rpc_stats</screen></para>
+ <para> To view a specific file, use <literal>lctl get_param</literal>:
+ <screen># lctl get_param osc.lustre-OST0000*.rpc_stats</screen></para>
</listitem>
</itemizedlist>
<para>For more information about using <literal>lctl</literal>, see <xref
xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438194_51490"/>.</para>
- <para>Data can also be viewed using the <literal>cat</literal> command with the full path to the
- file. The form of the <literal>cat</literal> command is similar to that of the <literal>lctl
- get_param</literal> command with these differences. In the <literal>cat</literal> command: </para>
+ <para>Data can also be viewed using the <literal>cat</literal> command
+ with the full path to the file. The form of the <literal>cat</literal>
+ command is similar to that of the <literal>lctl get_param</literal>
+ command with some differences. Unfortunately, as the Linux kernel has
+ changed over the years, the location of statistics and parameter files
+ has also changed, which means that the Lustre parameter files may be
+ located in either the <literal>/proc</literal> directory, in the
+ <literal>/sys</literal> directory, and/or in the
+ <literal>/sys/kernel/debug</literal> directory, depending on the kernel
+ version and the Lustre version being used. The <literal>lctl</literal>
+ command insulates scripts from these changes and is preferred over direct
+ file access, unless as part of a high-performance monitoring system.
+ In the <literal>cat</literal> command:</para>
<itemizedlist>
<listitem>
- <para> Replace the dots in the path with slashes.</para>
+ <para>Replace the dots in the path with slashes.</para>
</listitem>
<listitem>
- <para> Prepend the path with the following as
- appropriate:<screen>/proc/{fs,sys}/{lustre,lnet}</screen></para>
+ <para>Prepend the path with the appropriate directory component:
+ <screen>/{proc,sys}/{fs,sys}/{lustre,lnet}</screen></para>
</listitem>
</itemizedlist>
<para>For example, an <literal>lctl get_param</literal> command may look like
osc.testfs-OST0000-osc-ffff881071d5cc00.uuid=594db456-0685-bd16-f59b-e72ee90e9819
osc.testfs-OST0001-osc-ffff881071d5cc00.uuid=594db456-0685-bd16-f59b-e72ee90e9819
...</screen></para>
- <para>The equivalent <literal>cat</literal> command looks like
- this:<screen># cat /proc/fs/lustre/osc/*/uuid
+ <para>The equivalent <literal>cat</literal> command may look like this:
+ <screen># cat /proc/fs/lustre/osc/*/uuid
594db456-0685-bd16-f59b-e72ee90e9819
594db456-0685-bd16-f59b-e72ee90e9819
...</screen></para>
- <para>The <literal>llstat</literal> utility can be used to monitor some Lustre file system I/O
- activity over a specified time period. For more details, see <xref
- xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438219_23232"/></para>
- <para>Some data is imported from attached clients and is available in a directory called
- <literal>exports</literal> located in the corresponding per-service directory on a Lustre
- server. For
- example:<screen># ls /proc/fs/lustre/obdfilter/testfs-OST0000/exports/192.168.124.9\@o2ib1/
+ <para>or like this:
+ <screen># cat /sys/fs/lustre/osc/*/uuid
+594db456-0685-bd16-f59b-e72ee90e9819
+594db456-0685-bd16-f59b-e72ee90e9819
+...</screen></para>
+ <para>The <literal>llstat</literal> utility can be used to monitor some
+ Lustre file system I/O activity over a specified time period. For more
+ details, see
+ <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438219_23232"/></para>
+ <para>Some data is imported from attached clients and is available in a
+ directory called <literal>exports</literal> located in the corresponding
+ per-service directory on a Lustre server. For example:
+ <screen>oss:/root# lctl list_param obdfilter.testfs-OST0000.exports.*
# hash ldlm_stats stats uuid</screen></para>
<section remap="h3">
<title>Identifying Lustre File Systems and Servers</title>
- <para>Several <literal>/proc</literal> files on the MGS list existing Lustre file systems and
- file system servers. The examples below are for a Lustre file system called
+ <para>Several parameter files on the MGS list existing
+ Lustre file systems and file system servers. The examples below are for
+ a Lustre file system called
<literal>testfs</literal> with one MDT and three OSTs.</para>
<itemizedlist>
<listitem>
notify_count: 4</screen>
</listitem>
<listitem>
- <para>To view the names of all live servers in the file system as listed in
- <literal>/proc/fs/lustre/devices</literal>, enter:</para>
+ <para>To list all configured devices on the local node, enter:</para>
<screen># lctl device_list
0 UP mgs MGS MGS 11
1 UP mgc MGC192.168.10.34@tcp 1f45bb57-d9be-2ddb-c0b0-5431a49226705
<row>
<entry>
<para>
- <literal>mb_prealloc_table</literal></para>
+ <literal>prealloc_table</literal></para>
</entry>
<entry>
<para>A table of values used to preallocate space when a new request is received. By
</tgroup>
</informaltable>
<para>Buddy group cache information found in
- <literal>/proc/fs/ldiskfs/<replaceable>disk_device</replaceable>/mb_groups</literal> may
+ <literal>/sys/fs/ldiskfs/<replaceable>disk_device</replaceable>/mb_groups</literal> may
be useful for assessing on-disk fragmentation. For
example:<screen>cat /proc/fs/ldiskfs/loop0/mb_groups
#group: free free frags first pa [ 2^0 2^1 2^2 2^3 2^4 2^5 2^6 2^7 2^8 2^9
</itemizedlist></para>
</section>
<section>
- <title>Monitoring Lustre File System I/O</title>
+ <title>Monitoring Lustre File System I/O</title>
<para>A number of system utilities are provided to enable collection of data related to I/O
activity in a Lustre file system. In general, the data collected describes:</para>
<itemizedlist>
<para>By default, statistics are not collected in the <literal>offset_stats</literal>,
<literal>extents_stats</literal>, and <literal>extents_stats_per_process</literal> files
to reduce monitoring overhead when this information is not needed. The collection of
- statistics in all three of these files is activated by writing anything into any one of
- the files.</para>
+ statistics in all three of these files is activated by writing
+ anything, except for 0 (zero) and "disable", into any one of the
+ files.</para>
</note>
<para><emphasis role="italic"><emphasis role="bold">Example:</emphasis></emphasis></para>
<screen># lctl get_param llite.testfs-f57dee0.offset_stats
snapshot_time: 1155748884.591028 (secs.usecs)
- RANGE RANGE SMALLEST LARGEST
+ RANGE RANGE SMALLEST LARGEST
R/W PID START END EXTENT EXTENT OFFSET
R 8385 0 128 128 128 0
R 8385 0 224 224 224 -128
<para>By default, statistics are not collected in the <literal>offset_stats</literal>,
<literal>extents_stats</literal>, and <literal>extents_stats_per_process</literal> files
to reduce monitoring overhead when this information is not needed. The collection of
- statistics in all three of these files is activated by writing anything into any one of
- the files.</para>
+ statistics in all three of these files is activated by writing
+ anything, except for 0 (zero) and "disable", into any one of the
+ files.</para>
</note>
<section remap="h3">
<title>Client-Based I/O Extent Size Survey</title>
- <para>The <literal>extent_stats</literal> histogram in the <literal>llite</literal>
- directory shows the statistics for the sizes of the read/write I/O extents. This file does
- not maintain the per-process statistics.</para>
+ <para>The <literal>extents_stats</literal> histogram in the
+ <literal>llite</literal> directory shows the statistics for the sizes
+ of the read/write I/O extents. This file does not maintain the per
+ process statistics.</para>
<para><emphasis role="italic"><emphasis role="bold">Example:</emphasis></emphasis></para>
<screen># lctl get_param llite.testfs-*.extents_stats
snapshot_time: 1213828728.348516 (secs.usecs)
read | write
extents calls % cum% | calls % cum%
-
+
0K - 4K : 0 0 0 | 2 2 2
4K - 8K : 0 0 0 | 0 0 2
8K - 16K : 0 0 0 | 0 0 2
was read. The table shows cumulative extents organized according to size with statistics
provided separately for reads and writes. Each row in the table shows the number of RPCs
for reads and writes respectively (<literal>calls</literal>), the relative percentage of
- total calls (<literal>%</literal>), and the cumulative percentage to that point in the
- table of calls (<literal>cum %</literal>). </para>
- <para> The file can be cleared by issuing the following
- command:<screen># lctl set_param llite.testfs-*.extents_stats=0</screen></para>
+ total calls (<literal>%</literal>), and the cumulative percentage to
+ that point in the table of calls (<literal>cum %</literal>). </para>
+ <para> The file can be cleared by issuing the following command:
+ <screen># lctl set_param llite.testfs-*.extents_stats=1</screen></para>
</section>
<section>
<title>Per-Process Client I/O Statistics</title>
</section>
<section>
<title>Tuning Lustre File System I/O</title>
- <para>Each OSC has its own tree of tunables. For example:</para>
- <screen>$ ls -d /proc/fs/testfs/osc/OSC_client_ost1_MNT_client_2 /localhost
-/proc/fs/testfs/osc/OSC_uml0_ost1_MNT_localhost
-/proc/fs/testfs/osc/OSC_uml0_ost2_MNT_localhost
-/proc/fs/testfs/osc/OSC_uml0_ost3_MNT_localhost
-
-$ ls /proc/fs/testfs/osc/OSC_uml0_ost1_MNT_localhost
-blocksizefilesfree max_dirty_mb ost_server_uuid stats
-
-...</screen>
- <para>The following sections describe some of the parameters that can be tuned in a Lustre file
- system.</para>
+ <para>Each OSC has its own tree of tunables. For example:</para>
+ <screen>$ lctl lctl list_param osc.*.*
+osc.myth-OST0000-osc-ffff8804296c2800.active
+osc.myth-OST0000-osc-ffff8804296c2800.blocksize
+osc.myth-OST0000-osc-ffff8804296c2800.checksum_dump
+osc.myth-OST0000-osc-ffff8804296c2800.checksum_type
+osc.myth-OST0000-osc-ffff8804296c2800.checksums
+osc.myth-OST0000-osc-ffff8804296c2800.connect_flags
+:
+:
+osc.myth-OST0000-osc-ffff8804296c2800.state
+osc.myth-OST0000-osc-ffff8804296c2800.stats
+osc.myth-OST0000-osc-ffff8804296c2800.timeouts
+osc.myth-OST0000-osc-ffff8804296c2800.unstable_stats
+osc.myth-OST0000-osc-ffff8804296c2800.uuid
+osc.myth-OST0001-osc-ffff8804296c2800.active
+osc.myth-OST0001-osc-ffff8804296c2800.blocksize
+osc.myth-OST0001-osc-ffff8804296c2800.checksum_dump
+osc.myth-OST0001-osc-ffff8804296c2800.checksum_type
+:
+:
+</screen>
+ <para>The following sections describe some of the parameters that can
+ be tuned in a Lustre file system.</para>
<section remap="h3" xml:id="TuningClientIORPCStream">
<title><indexterm>
<primary>proc</primary>
<secondary>RPC tunables</secondary>
</indexterm>Tuning the Client I/O RPC Stream</title>
- <para>Ideally, an optimal amount of data is packed into each I/O RPC and a consistent number
- of issued RPCs are in progress at any time. To help optimize the client I/O RPC stream,
- several tuning variables are provided to adjust behavior according to network conditions and
- cluster size. For information about monitoring the client I/O RPC stream, see <xref
+ <para>Ideally, an optimal amount of data is packed into each I/O RPC
+ and a consistent number of issued RPCs are in progress at any time.
+ To help optimize the client I/O RPC stream, several tuning variables
+ are provided to adjust behavior according to network conditions and
+ cluster size. For information about monitoring the client I/O RPC
+ stream, see <xref
xmlns:xlink="http://www.w3.org/1999/xlink" linkend="MonitoringClientRCPStream"/>.</para>
<para>RPC stream tunables include:</para>
<para>
<itemizedlist>
<listitem>
- <para><literal>osc.<replaceable>osc_instance</replaceable>.max_dirty_mb</literal> -
- Controls how many MBs of dirty data can be written and queued up in the OSC. POSIX
- file writes that are cached contribute to this count. When the limit is reached,
- additional writes stall until previously-cached writes are written to the server. This
- may be changed by writing a single ASCII integer to the file. Only values between 0
- and 2048 or 1/4 of RAM are allowable. If 0 is specified, no writes are cached.
- Performance suffers noticeably unless you use large writes (1 MB or more).</para>
- <para>To maximize performance, the value for <literal>max_dirty_mb</literal> is
- recommended to be 4 * <literal>max_pages_per_rpc </literal>*
- <literal>max_rpcs_in_flight</literal>.</para>
+ <para><literal>osc.<replaceable>osc_instance</replaceable>.checksums</literal>
+ - Controls whether the client will calculate data integrity
+ checksums for the bulk data transferred to the OST. Data
+ integrity checksums are enabled by default. The algorithm used
+ can be set using the <literal>checksum_type</literal> parameter.
+ </para>
</listitem>
<listitem>
- <para><literal>osc.<replaceable>osc_instance</replaceable>.cur_dirty_bytes</literal> - A
- read-only value that returns the current number of bytes written and cached on this
- OSC.</para>
+ <para><literal>osc.<replaceable>osc_instance</replaceable>.checksum_type</literal>
+ - Controls the data integrity checksum algorithm used by the
+ client. The available algorithms are determined by the set of
+ algorihtms. The checksum algorithm used by default is determined
+ by first selecting the fastest algorithms available on the OST,
+ and then selecting the fastest of those algorithms on the client,
+ which depends on available optimizations in the CPU hardware and
+ kernel. The default algorithm can be overridden by writing the
+ algorithm name into the <literal>checksum_type</literal>
+ parameter. Available checksum types can be seen on the client by
+ reading the <literal>checksum_type</literal> parameter. Currently
+ supported checksum types are:
+ <literal>adler</literal>,
+ <literal>crc32</literal>,
+ <literal>crc32c</literal>
+ </para>
+ <para condition="l2C">
+ In Lustre release 2.12 additional checksum types were added to
+ allow end-to-end checksum integration with T10-PI capable
+ hardware. The client will compute the appropriate checksum
+ type, based on the checksum type used by the storage, for the
+ RPC checksum, which will be verified by the server and passed
+ on to the storage. The T10-PI checksum types are:
+ <literal>t10ip512</literal>,
+ <literal>t10ip4K</literal>,
+ <literal>t10crc512</literal>,
+ <literal>t10crc4K</literal>
+ </para>
</listitem>
<listitem>
- <para><literal>osc.<replaceable>osc_instance</replaceable>.max_pages_per_rpc</literal> -
- The maximum number of pages that will undergo I/O in a single RPC to the OST. The
- minimum setting is a single page and the maximum setting is 1024 (for systems with a
- <literal>PAGE_SIZE</literal> of 4 KB), with the default maximum of 1 MB in the RPC.
- It is also possible to specify a units suffix (e.g. <literal>4M</literal>), so that
- the RPC size can be specified independently of the client
- <literal>PAGE_SIZE</literal>.</para>
+ <para><literal>osc.<replaceable>osc_instance</replaceable>.max_dirty_mb</literal>
+ - Controls how many MiB of dirty data can be written into the
+ client pagecache for writes by <emphasis>each</emphasis> OSC.
+ When this limit is reached, additional writes block until
+ previously-cached data is written to the server. This may be
+ changed by the <literal>lctl set_param</literal> command. Only
+ values larger than 0 and smaller than the lesser of 2048 MiB or
+ 1/4 of client RAM are valid. Performance can suffers if the
+ client cannot aggregate enough data per OSC to form a full RPC
+ (as set by the <literal>max_pages_per_rpc</literal>) parameter,
+ unless the application is doing very large writes itself.
+ </para>
+ <para>To maximize performance, the value for
+ <literal>max_dirty_mb</literal> is recommended to be at least
+ 4 * <literal>max_pages_per_rpc</literal> *
+ <literal>max_rpcs_in_flight</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para><literal>osc.<replaceable>osc_instance</replaceable>.cur_dirty_bytes</literal>
+ - A read-only value that returns the current number of bytes
+ written and cached by this OSC.
+ </para>
+ </listitem>
+ <listitem>
+ <para><literal>osc.<replaceable>osc_instance</replaceable>.max_pages_per_rpc</literal>
+ - The maximum number of pages that will be sent in a single RPC
+ request to the OST. The minimum value is one page and the maximum
+ value is 16 MiB (4096 on systems with <literal>PAGE_SIZE</literal>
+ of 4 KiB), with the default value of 4 MiB in one RPC. The upper
+ limit may also be constrained by <literal>ofd.*.brw_size</literal>
+ setting on the OSS, and applies to all clients connected to that
+ OST. It is also possible to specify a units suffix (e.g.
+ <literal>max_pages_per_rpc=4M</literal>), so the RPC size can be
+ set independently of the client <literal>PAGE_SIZE</literal>.
+ </para>
</listitem>
<listitem>
<para><literal>osc.<replaceable>osc_instance</replaceable>.max_rpcs_in_flight</literal>
- - The maximum number of concurrent RPCs in flight from an OSC to its OST. If the OSC
- tries to initiate an RPC but finds that it already has the same number of RPCs
- outstanding, it will wait to issue further RPCs until some complete. The minimum
- setting is 1 and maximum setting is 256. </para>
+ - The maximum number of concurrent RPCs in flight from an OSC to
+ its OST. If the OSC tries to initiate an RPC but finds that it
+ already has the same number of RPCs outstanding, it will wait to
+ issue further RPCs until some complete. The minimum setting is 1
+ and maximum setting is 256. The default value is 8 RPCs.
+ </para>
<para>To improve small file I/O performance, increase the
- <literal>max_rpcs_in_flight</literal> value.</para>
+ <literal>max_rpcs_in_flight</literal> value.
+ </para>
</listitem>
<listitem>
- <para><literal>llite.<replaceable>fsname-instance</replaceable>/max_cache_mb</literal> -
- Maximum amount of inactive data cached by the client (default is 3/4 of RAM). For
- example:</para>
- <screen># lctl get_param llite.testfs-ce63ca00.max_cached_mb
-128</screen>
+ <para><literal>llite.<replaceable>fsname_instance</replaceable>.max_cache_mb</literal>
+ - Maximum amount of inactive data cached by the client. The
+ default value is 3/4 of the client RAM.
+ </para>
</listitem>
</itemizedlist>
</para>
<note>
- <para>The value for <literal><replaceable>osc_instance</replaceable></literal> is typically
- <literal><replaceable>fsname</replaceable>-OST<replaceable>ost_index</replaceable>-osc-<replaceable>mountpoint_instance</replaceable></literal>,
- where the value for <literal><replaceable>mountpoint_instance</replaceable></literal> is
- unique to each mount point to allow associating osc, mdc, lov, lmv, and llite parameters
- with the same mount point. For
- example:<screen>lctl get_param osc.testfs-OST0000-osc-ffff88107412f400.rpc_stats
+ <para>The value for <literal><replaceable>osc_instance</replaceable></literal>
+ and <literal><replaceable>fsname_instance</replaceable></literal>
+ are unique to each mount point to allow associating osc, mdc, lov,
+ lmv, and llite parameters with the same mount point. However, it is
+ common for scripts to use a wildcard <literal>*</literal> or a
+ filesystem-specific wildcard
+ <literal><replaceable>fsname-*</replaceable></literal> to specify
+ the parameter settings uniformly on all clients. For example:
+<screen>
+client$ lctl get_param osc.testfs-OST0000*.rpc_stats
osc.testfs-OST0000-osc-ffff88107412f400.rpc_stats=
snapshot_time: 1375743284.337839 (secs.usecs)
read RPCs in flight: 0
</screen></para>
</note>
</section>
- <section remap="h3">
+ <section remap="h3" xml:id="TuningClientReadahead">
<title><indexterm>
<primary>proc</primary>
<secondary>readahead</secondary>
</indexterm>Tuning File Readahead and Directory Statahead</title>
- <para>File readahead and directory statahead enable reading of data into memory before a
- process requests the data. File readahead reads file content data into memory and directory
- statahead reads metadata into memory. When readahead and statahead work well, a process that
- accesses data finds that the information it needs is available immediately when requested in
- memory without the delay of network I/O.</para>
- <para condition="l22">In Lustre release 2.2.0, the directory statahead feature was improved to
- enhance directory traversal performance. The improvements primarily addressed two
- issues:
- <orderedlist>
- <listitem>
- <para>A race condition existed between the statahead thread and other VFS operations while
- processing asynchronous <literal>getattr</literal> RPC replies, causing duplicate
- entries in dcache. This issue was resolved by using statahead local dcache. </para>
- </listitem>
- <listitem>
- <para>File size/block attributes pre-fetching was not supported, so the traversing thread
- had to send synchronous glimpse size RPCs to OST(s). This issue was resolved by using
- asynchronous glimpse lock (AGL) RPCs to pre-fetch file size/block attributes from
- OST(s).</para>
- </listitem>
- </orderedlist>
- </para>
+ <para>File readahead and directory statahead enable reading of data
+ into memory before a process requests the data. File readahead prefetches
+ file content data into memory for <literal>read()</literal> related
+ calls, while directory statahead fetches file metadata into memory for
+ <literal>readdir()</literal> and <literal>stat()</literal> related
+ calls. When readahead and statahead work well, a process that accesses
+ data finds that the information it needs is available immediately in
+ memory on the client when requested without the delay of network I/O.
+ </para>
<section remap="h4">
<title>Tuning File Readahead</title>
- <para>File readahead is triggered when two or more sequential reads by an application fail
- to be satisfied by data in the Linux buffer cache. The size of the initial readahead is 1
- MB. Additional readaheads grow linearly and increment until the readahead cache on the
- client is full at 40 MB.</para>
+ <para>File readahead is triggered when two or more sequential reads
+ by an application fail to be satisfied by data in the Linux buffer
+ cache. The size of the initial readahead is determined by the RPC
+ size and the file stripe size, but will typically be at least 1 MiB.
+ Additional readaheads grow linearly and increment until the per-file
+ or per-system readahead cache limit on the client is reached.</para>
<para>Readahead tunables include:</para>
<itemizedlist>
<listitem>
- <para><literal>llite.<replaceable>fsname-instance</replaceable>.max_read_ahead_mb</literal>
- - Controls the maximum amount of data readahead on a file. Files are read ahead in
- RPC-sized chunks (1 MB or the size of the <literal>read()</literal> call, if larger)
- after the second sequential read on a file descriptor. Random reads are done at the
- size of the <literal>read()</literal> call only (no readahead). Reads to
- non-contiguous regions of the file reset the readahead algorithm, and readahead is not
- triggered again until sequential reads take place again. </para>
- <para>To disable readahead, set this tunable to 0. The default value is 40 MB.</para>
+ <para><literal>llite.<replaceable>fsname_instance</replaceable>.max_read_ahead_mb</literal>
+ - Controls the maximum amount of data readahead on a file.
+ Files are read ahead in RPC-sized chunks (4 MiB, or the size of
+ the <literal>read()</literal> call, if larger) after the second
+ sequential read on a file descriptor. Random reads are done at
+ the size of the <literal>read()</literal> call only (no
+ readahead). Reads to non-contiguous regions of the file reset
+ the readahead algorithm, and readahead is not triggered until
+ sequential reads take place again.
+ </para>
+ <para>
+ This is the global limit for all files and cannot be larger than
+ 1/2 of the client RAM. To disable readahead, set
+ <literal>max_read_ahead_mb=0</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para><literal>llite.<replaceable>fsname_instance</replaceable>.max_read_ahead_per_file_mb</literal>
+ - Controls the maximum number of megabytes (MiB) of data that
+ should be prefetched by the client when sequential reads are
+ detected on a file. This is the per-file readahead limit and
+ cannot be larger than <literal>max_read_ahead_mb</literal>.
+ </para>
</listitem>
<listitem>
- <para><literal>llite.<replaceable>fsname-instance</replaceable>.max_read_ahead_whole_mb</literal>
- - Controls the maximum size of a file that is read in its entirety, regardless of the
- size of the <literal>read()</literal>.</para>
+ <para><literal>llite.<replaceable>fsname_instance</replaceable>.max_read_ahead_whole_mb</literal>
+ - Controls the maximum size of a file in MiB that is read in its
+ entirety upon access, regardless of the size of the
+ <literal>read()</literal> call. This avoids multiple small read
+ RPCs on relatively small files, when it is not possible to
+ efficiently detect a sequential read pattern before the whole
+ file has been read.
+ </para>
+ <para>The default value is the greater of 2 MiB or the size of one
+ RPC, as given by <literal>max_pages_per_rpc</literal>.
+ </para>
</listitem>
</itemizedlist>
</section>
<section>
<title>Tuning Directory Statahead and AGL</title>
- <para>Many system commands, such as <literal>ls –l</literal>, <literal>du</literal>, and
- <literal>find</literal>, traverse a directory sequentially. To make these commands run
- efficiently, the directory statahead and asynchronous glimpse lock (AGL) can be enabled to
- improve the performance of traversing.</para>
+ <para>Many system commands, such as <literal>ls –l</literal>,
+ <literal>du</literal>, and <literal>find</literal>, traverse a
+ directory sequentially. To make these commands run efficiently, the
+ directory statahead can be enabled to improve the performance of
+ directory traversal.</para>
<para>The statahead tunables are:</para>
<itemizedlist>
<listitem>
- <para><literal>statahead_max</literal> - Controls whether directory statahead is enabled
- and the maximum statahead window size (i.e., how many files can be pre-fetched by the
- statahead thread). By default, statahead is enabled and the value of
- <literal>statahead_max</literal> is 32.</para>
- <para>To disable statahead, run:</para>
+ <para><literal>statahead_max</literal> -
+ Controls the maximum number of file attributes that will be
+ prefetched by the statahead thread. By default, statahead is
+ enabled and <literal>statahead_max</literal> is 32 files.</para>
+ <para>To disable statahead, set <literal>statahead_max</literal>
+ to zero via the following command on the client:</para>
<screen>lctl set_param llite.*.statahead_max=0</screen>
- <para>To set the maximum statahead window size (<replaceable>n</replaceable>),
- run:</para>
+ <para>To change the maximum statahead window size on a client:</para>
<screen>lctl set_param llite.*.statahead_max=<replaceable>n</replaceable></screen>
- <para>The maximum value of <replaceable>n</replaceable> is 8192.</para>
- <para>The AGL can be controlled by entering:</para>
- <screen>lctl set_param llite.*.statahead_agl=<replaceable>n</replaceable></screen>
- <para>The default value for <replaceable>n</replaceable> is 1, which enables the AGL. If
- <replaceable>n</replaceable> is 0, the AGL is disabled.</para>
+ <para>The maximum <literal>statahead_max</literal> is 8192 files.
+ </para>
+ <para>The directory statahead thread will also prefetch the file
+ size/block attributes from the OSTs, so that all file attributes
+ are available on the client when requested by an application.
+ This is controlled by the asynchronous glimpse lock (AGL) setting.
+ The AGL behaviour can be disabled by setting:</para>
+ <screen>lctl set_param llite.*.statahead_agl=0</screen>
</listitem>
<listitem>
- <para><literal>statahead_stats</literal> - A read-only interface that indicates the
- current statahead and AGL statistics, such as how many times statahead/AGL has been
- triggered since the last mount, how many statahead/AGL failures have occurred due to
- an incorrect prediction or other causes.</para>
+ <para><literal>statahead_stats</literal> -
+ A read-only interface that provides current statahead and AGL
+ statistics, such as how many times statahead/AGL has been triggered
+ since the last mount, how many statahead/AGL failures have occurred
+ due to an incorrect prediction or other causes.</para>
<note>
- <para>The AGL is affected by statahead because the inodes processed by AGL are built
- by the statahead thread, which means the statahead thread is the input of the AGL
- pipeline. So if statahead is disabled, then the AGL is disabled by force.</para>
+ <para>AGL behaviour is affected by statahead since the inodes
+ processed by AGL are built by the statahead thread. If
+ statahead is disabled, then AGL is also disabled.</para>
</note>
</listitem>
</itemizedlist>
<para>To re-enable the writethrough cache on one OST, run:</para>
<screen>root@oss1# lctl set_param obdfilter.{OST_name}.writethrough_cache_enable=1</screen>
<para>To check if the writethrough cache is enabled, run:</para>
- <screen>root@oss1# lctl set_param obdfilter.*.writethrough_cache_enable=1</screen>
+ <screen>root@oss1# lctl get_param obdfilter.*.writethrough_cache_enable</screen>
</listitem>
<listitem>
<para><literal>readcache_max_filesize</literal> - Controls the maximum size of a file
<screen>$ lctl get_param obdfilter.*.sync_on_lock_cancel
obdfilter.lol-OST0001.sync_on_lock_cancel=never</screen>
</section>
+ <section xml:id="dbdoclet.TuningModRPCs" condition='l28'>
+ <title>
+ <indexterm>
+ <primary>proc</primary>
+ <secondary>client metadata performance</secondary>
+ </indexterm>
+ Tuning the Client Metadata RPC Stream
+ </title>
+ <para>The client metadata RPC stream represents the metadata RPCs issued
+ in parallel by a client to a MDT target. The metadata RPCs can be split
+ in two categories: the requests that do not modify the file system
+ (like getattr operation), and the requests that do modify the file system
+ (like create, unlink, setattr operations). To help optimize the client
+ metadata RPC stream, several tuning variables are provided to adjust
+ behavior according to network conditions and cluster size.</para>
+ <para>Note that increasing the number of metadata RPCs issued in parallel
+ might improve the performance metadata intensive parallel applications,
+ but as a consequence it will consume more memory on the client and on
+ the MDS.</para>
+ <section>
+ <title>Configuring the Client Metadata RPC Stream</title>
+ <para>The MDC <literal>max_rpcs_in_flight</literal> parameter defines
+ the maximum number of metadata RPCs, both modifying and
+ non-modifying RPCs, that can be sent in parallel by a client to a MDT
+ target. This includes every file system metadata operations, such as
+ file or directory stat, creation, unlink. The default setting is 8,
+ minimum setting is 1 and maximum setting is 256.</para>
+ <para>To set the <literal>max_rpcs_in_flight</literal> parameter, run
+ the following command on the Lustre client:</para>
+ <screen>client$ lctl set_param mdc.*.max_rpcs_in_flight=16</screen>
+ <para>The MDC <literal>max_mod_rpcs_in_flight</literal> parameter
+ defines the maximum number of file system modifying RPCs that can be
+ sent in parallel by a client to a MDT target. For example, the Lustre
+ client sends modify RPCs when it performs file or directory creation,
+ unlink, access permission modification or ownership modification. The
+ default setting is 7, minimum setting is 1 and maximum setting is
+ 256.</para>
+ <para>To set the <literal>max_mod_rpcs_in_flight</literal> parameter,
+ run the following command on the Lustre client:</para>
+ <screen>client$ lctl set_param mdc.*.max_mod_rpcs_in_flight=12</screen>
+ <para>The <literal>max_mod_rpcs_in_flight</literal> value must be
+ strictly less than the <literal>max_rpcs_in_flight</literal> value.
+ It must also be less or equal to the MDT
+ <literal>max_mod_rpcs_per_client</literal> value. If one of theses
+ conditions is not enforced, the setting fails and an explicit message
+ is written in the Lustre log.</para>
+ <para>The MDT <literal>max_mod_rpcs_per_client</literal> parameter is a
+ tunable of the kernel module <literal>mdt</literal> that defines the
+ maximum number of file system modifying RPCs in flight allowed per
+ client. The parameter can be updated at runtime, but the change is
+ effective to new client connections only. The default setting is 8.
+ </para>
+ <para>To set the <literal>max_mod_rpcs_per_client</literal> parameter,
+ run the following command on the MDS:</para>
+ <screen>mds$ echo 12 > /sys/module/mdt/parameters/max_mod_rpcs_per_client</screen>
+ </section>
+ <section>
+ <title>Monitoring the Client Metadata RPC Stream</title>
+ <para>The <literal>rpc_stats</literal> file contains histogram data
+ showing information about modify metadata RPCs. It can be helpful to
+ identify the level of parallelism achieved by an application doing
+ modify metadata operations.</para>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>client$ lctl get_param mdc.*.rpc_stats
+snapshot_time: 1441876896.567070 (secs.usecs)
+modify_RPCs_in_flight: 0
+
+ modify
+rpcs in flight rpcs % cum %
+0: 0 0 0
+1: 56 0 0
+2: 40 0 0
+3: 70 0 0
+4 41 0 0
+5: 51 0 1
+6: 88 0 1
+7: 366 1 2
+8: 1321 5 8
+9: 3624 15 23
+10: 6482 27 50
+11: 7321 30 81
+12: 4540 18 100</screen>
+ <para>The file information includes:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>snapshot_time</literal> - UNIX epoch instant the
+ file was read.</para>
+ </listitem>
+ <listitem>
+ <para><literal>modify_RPCs_in_flight</literal> - Number of modify
+ RPCs issued by the MDC, but not completed at the time of the
+ snapshot. This value should always be less than or equal to
+ <literal>max_mod_rpcs_in_flight</literal>.</para>
+ </listitem>
+ <listitem>
+ <para><literal>rpcs in flight</literal> - Number of modify RPCs
+ that are pending when a RPC is sent, the relative percentage
+ (<literal>%</literal>) of total modify RPCs, and the cumulative
+ percentage (<literal>cum %</literal>) to that point.</para>
+ </listitem>
+ </itemizedlist>
+ <para>If a large proportion of modify metadata RPCs are issued with a
+ number of pending metadata RPCs close to the
+ <literal>max_mod_rpcs_in_flight</literal> value, it means the
+ <literal>max_mod_rpcs_in_flight</literal> value could be increased to
+ improve the modify metadata performance.</para>
+ </section>
+ </section>
</section>
<section>
<title>Configuring Timeouts in a Lustre File System</title>
</informaltable>
<section>
<title>Interpreting Adaptive Timeout Information</title>
- <para>Adaptive timeout information can be obtained from the <literal>timeouts</literal>
- files in <literal>/proc/fs/lustre/*/</literal> on each server and client using the
- <literal>lctl</literal> command. To read information from a <literal>timeouts</literal>
- file, enter a command similar to:</para>
+ <para>Adaptive timeout information can be obtained via
+ <literal>lctl get_param {osc,mdc}.*.timeouts</literal> files on each
+ client and <literal>lctl get_param {ost,mds}.*.*.timeouts</literal>
+ on each server. To read information from a
+ <literal>timeouts</literal> file, enter a command similar to:</para>
<screen># lctl get_param -n ost.*.ost_io.timeouts
-service : cur 33 worst 34 (at 1193427052, 0d0h26m40s ago) 1 1 33 2</screen>
- <para>In this example, the <literal>ost_io</literal> service on this node is currently
- reporting an estimated RPC service time of 33 seconds. The worst RPC service time was 34
- seconds, which occurred 26 minutes ago.</para>
- <para>The output also provides a history of service times. Four "bins" of adaptive
- timeout history are shown, with the maximum RPC time in each bin reported. In both the
- 0-150s bin and the 150-300s bin, the maximum RPC time was 1. The 300-450s bin shows the
- worst (maximum) RPC time at 33 seconds, and the 450-600s bin shows a maximum of RPC time
- of 2 seconds. The estimated service time is the maximum value across the four bins (33
- seconds in this example).</para>
- <para>Service times (as reported by the servers) are also tracked in the client OBDs, as
- shown in this example:</para>
+service : cur 33 worst 34 (at 1193427052, 1600s ago) 1 1 33 2</screen>
+ <para>In this example, the <literal>ost_io</literal> service on this
+ node is currently reporting an estimated RPC service time of 33
+ seconds. The worst RPC service time was 34 seconds, which occurred
+ 26 minutes ago.</para>
+ <para>The output also provides a history of service times.
+ Four "bins" of adaptive timeout history are shown, with the
+ maximum RPC time in each bin reported. In both the 0-150s bin and the
+ 150-300s bin, the maximum RPC time was 1. The 300-450s bin shows the
+ worst (maximum) RPC time at 33 seconds, and the 450-600s bin shows a
+ maximum of RPC time of 2 seconds. The estimated service time is the
+ maximum value in the four bins (33 seconds in this example).</para>
+ <para>Service times (as reported by the servers) are also tracked in
+ the client OBDs, as shown in this example:</para>
<screen># lctl get_param osc.*.timeouts
last reply : 1193428639, 0d0h00m00s ago
network : cur 1 worst 2 (at 1193427053, 0d0h26m26s ago) 1 1 1 1
portal 7 : cur 1 worst 1 (at 1193426141, 0d0h41m38s ago) 1 0 1 1
portal 17 : cur 1 worst 1 (at 1193426177, 0d0h41m02s ago) 1 0 0 1
</screen>
- <para>In this example, portal 6, the <literal>ost_io</literal> service portal, shows the
- history of service estimates reported by the portal.</para>
- <para>Server statistic files also show the range of estimates including min, max, sum, and
- sumsq. For example:</para>
+ <para>In this example, portal 6, the <literal>ost_io</literal> service
+ portal, shows the history of service estimates reported by the portal.
+ </para>
+ <para>Server statistic files also show the range of estimates including
+ min, max, sum, and sum-squared. For example:</para>
<screen># lctl get_param mdt.*.mdt.stats
...
req_timeout 6 samples [sec] 1 10 15 105
messages or enable printing of <literal>D_NETERROR</literal> messages to the console
using:<screen>lctl set_param printk=+neterror</screen></para>
<para>Congested routers can be a source of spurious LND timeouts. To avoid this
- situation, increase the number of LNET router buffers to reduce back-pressure and/or
+ situation, increase the number of LNet router buffers to reduce back-pressure and/or
increase LND timeouts on all nodes on all connected networks. Also consider increasing
- the total number of LNET router nodes in the system so that the aggregate router
+ the total number of LNet router nodes in the system so that the aggregate router
bandwidth matches the aggregate server bandwidth.</para>
</listitem>
<listitem>
<section remap="h3">
<title><indexterm>
<primary>proc</primary>
- <secondary>LNET</secondary>
+ <secondary>LNet</secondary>
</indexterm><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>proc</secondary>
- </indexterm>Monitoring LNET</title>
- <para>LNET information is located in <literal>/proc/sys/lnet</literal> in these files:<itemizedlist>
+ </indexterm>Monitoring LNet</title>
+ <para>LNet information is located via <literal>lctl get_param</literal>
+ in these parameters:
+ <itemizedlist>
<listitem>
- <para><literal>peers</literal> - Shows all NIDs known to this node and provides
- information on the queue state.</para>
+ <para><literal>peers</literal> - Shows all NIDs known to this node
+ and provides information on the queue state.</para>
<para>Example:</para>
<screen># lctl get_param peers
nid refs state max rtr min tx min queue
</tgroup>
</informaltable>
<para>Credits are initialized to allow a certain number of operations (in the example
- above the table, eight as shown in the <literal>max</literal> column. LNET keeps track
+ above the table, eight as shown in the <literal>max</literal> column. LNet keeps track
of the minimum number of credits ever seen over time showing the peak congestion that
has occurred during the time monitored. Fewer available credits indicates a more
congested resource. </para>
credits (<literal>rtr/tx</literal>) that is less than <literal>max</literal> indicates
operations are in progress. If the ratio <literal>rtr/tx</literal> is greater than
<literal>max</literal>, operations are blocking.</para>
- <para>LNET also limits concurrent sends and number of router buffers allocated to a single
+ <para>LNet also limits concurrent sends and number of router buffers allocated to a single
peer so that no peer can occupy all these resources.</para>
</listitem>
<listitem>
</listitem>
</itemizedlist></para>
</section>
- <section remap="h3">
+ <section remap="h3" xml:id="dbdoclet.balancing_free_space">
<title><indexterm>
<primary>proc</primary>
<secondary>free space</secondary>
</indexterm>Allocating Free Space on OSTs</title>
- <para>Free space is allocated using either a round-robin or a weighted algorithm. The allocation
- method is determined by the maximum amount of free-space imbalance between the OSTs. When free
- space is relatively balanced across OSTs, the faster round-robin allocator is used, which
- maximizes network balancing. The weighted allocator is used when any two OSTs are out of
- balance by more than a specified threshold.</para>
- <para>Free space distribution can be tuned using these two <literal>/proc</literal>
- tunables:</para>
+ <para>Free space is allocated using either a round-robin or a weighted
+ algorithm. The allocation method is determined by the maximum amount of
+ free-space imbalance between the OSTs. When free space is relatively
+ balanced across OSTs, the faster round-robin allocator is used, which
+ maximizes network balancing. The weighted allocator is used when any two
+ OSTs are out of balance by more than a specified threshold.</para>
+ <para>Free space distribution can be tuned using these two
+ tunable parameters:</para>
<itemizedlist>
<listitem>
- <para><literal>qos_threshold_rr</literal> - The threshold at which the allocation method
- switches from round-robin to weighted is set in this file. The default is to switch to the
- weighted algorithm when any two OSTs are out of balance by more than 17 percent.</para>
+ <para><literal>lod.*.qos_threshold_rr</literal> - The threshold at which
+ the allocation method switches from round-robin to weighted is set
+ in this file. The default is to switch to the weighted algorithm when
+ any two OSTs are out of balance by more than 17 percent.</para>
+ </listitem>
+ <listitem>
+ <para><literal>lod.*.qos_prio_free</literal> - The weighting priority
+ used by the weighted allocator can be adjusted in this file. Increasing
+ the value of <literal>qos_prio_free</literal> puts more weighting on the
+ amount of free space available on each OST and less on how stripes are
+ distributed across OSTs. The default value is 91 percent weighting for
+ free space rebalancing and 9 percent for OST balancing. When the
+ free space priority is set to 100, weighting is based entirely on free
+ space and location is no longer used by the striping algorithm.</para>
</listitem>
<listitem>
- <para><literal>qos_prio_free</literal> - The weighting priority used by the weighted
- allocator can be adjusted in this file. Increasing the value of
- <literal>qos_prio_free</literal> puts more weighting on the amount of free space
- available on each OST and less on how stripes are distributed across OSTs. The default
- value is 91 percent. When the free space priority is set to 100, weighting is based
- entirely on free space and location is no longer used by the striping algorthm.</para>
+ <para condition="l29"><literal>osp.*.reserved_mb_low</literal>
+ - The low watermark used to stop object allocation if available space
+ is less than this. The default is 0.1% of total OST size.</para>
+ </listitem>
+ <listitem>
+ <para condition="l29"><literal>osp.*.reserved_mb_high</literal>
+ - The high watermark used to start object allocation if available
+ space is more than this. The default is 0.2% of total OST size.</para>
</listitem>
</itemizedlist>
<para>For more information about monitoring and managing free space, see <xref
<primary>proc</primary>
<secondary>locking</secondary>
</indexterm>Configuring Locking</title>
- <para>The <literal>lru_size</literal> parameter is used to control the number of client-side
- locks in an LRU cached locks queue. LRU size is dynamic, based on load to optimize the number
- of locks available to nodes that have different workloads (e.g., login/build nodes vs. compute
- nodes vs. backup nodes).</para>
- <para>The total number of locks available is a function of the server RAM. The default limit is
- 50 locks/1 MB of RAM. If memory pressure is too high, the LRU size is shrunk. The number of
- locks on the server is limited to <emphasis role="italic">the number of OSTs per
- server</emphasis> * <emphasis role="italic">the number of clients</emphasis> * <emphasis
- role="italic">the value of the</emphasis>
- <literal>lru_size</literal>
- <emphasis role="italic">setting on the client</emphasis> as follows: </para>
+ <para>The <literal>lru_size</literal> parameter is used to control the
+ number of client-side locks in the LRU cached locks queue. LRU size is
+ normally dynamic, based on load to optimize the number of locks cached
+ on nodes that have different workloads (e.g., login/build nodes vs.
+ compute nodes vs. backup nodes).</para>
+ <para>The total number of locks available is a function of the server RAM.
+ The default limit is 50 locks/1 MB of RAM. If memory pressure is too high,
+ the LRU size is shrunk. The number of locks on the server is limited to
+ <replaceable>num_osts_per_oss * num_clients * lru_size</replaceable>
+ as follows: </para>
<itemizedlist>
<listitem>
- <para>To enable automatic LRU sizing, set the <literal>lru_size</literal> parameter to 0. In
- this case, the <literal>lru_size</literal> parameter shows the current number of locks
- being used on the export. LRU sizing is enabled by default.</para>
+ <para>To enable automatic LRU sizing, set the
+ <literal>lru_size</literal> parameter to 0. In this case, the
+ <literal>lru_size</literal> parameter shows the current number of locks
+ being used on the client. Dynamic LRU resizing is enabled by default.
+ </para>
</listitem>
<listitem>
- <para>To specify a maximum number of locks, set the <literal>lru_size</literal> parameter to
- a value other than zero but, normally, less than 100 * <emphasis role="italic">number of
- CPUs in client</emphasis>. It is recommended that you only increase the LRU size on a
- few login nodes where users access the file system interactively.</para>
+ <para>To specify a maximum number of locks, set the
+ <literal>lru_size</literal> parameter to a value other than zero.
+ A good default value for compute nodes is around
+ <literal>100 * <replaceable>num_cpus</replaceable></literal>.
+ It is recommended that you only set <literal>lru_size</literal>
+ to be signifivantly larger on a few login nodes where multiple
+ users access the file system interactively.</para>
</listitem>
</itemizedlist>
- <para>To clear the LRU on a single client, and, as a result, flush client cache without changing
- the <literal>lru_size</literal> value, run:</para>
- <screen>$ lctl set_param ldlm.namespaces.<replaceable>osc_name|mdc_name</replaceable>.lru_size=clear</screen>
- <para>If the LRU size is set to be less than the number of existing unused locks, the unused
- locks are canceled immediately. Use <literal>echo clear</literal> to cancel all locks without
- changing the value.</para>
+ <para>To clear the LRU on a single client, and, as a result, flush client
+ cache without changing the <literal>lru_size</literal> value, run:</para>
+ <screen># lctl set_param ldlm.namespaces.<replaceable>osc_name|mdc_name</replaceable>.lru_size=clear</screen>
+ <para>If the LRU size is set lower than the number of existing locks,
+ <emphasis>unused</emphasis> locks are canceled immediately. Use
+ <literal>clear</literal> to cancel all locks without changing the value.
+ </para>
<note>
- <para>The <literal>lru_size</literal> parameter can only be set temporarily using
- <literal>lctl set_param</literal>; it cannot be set permanently.</para>
+ <para>The <literal>lru_size</literal> parameter can only be set
+ temporarily using <literal>lctl set_param</literal>, it cannot be set
+ permanently.</para>
</note>
- <para>To disable LRU sizing, on the Lustre clients, run:</para>
- <screen>$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((<replaceable>NR_CPU</replaceable>*100))</screen>
- <para>Replace <literal><replaceable>NR_CPU</replaceable></literal> with the number of CPUs on
- the node.</para>
- <para>To determine the number of locks being granted, run:</para>
+ <para>To disable dynamic LRU resizing on the clients, run for example:
+ </para>
+ <screen># lctl set_param ldlm.namespaces.*osc*.lru_size=5000</screen>
+ <para>To determine the number of locks being granted with dynamic LRU
+ resizing, run:</para>
<screen>$ lctl get_param ldlm.namespaces.*.pool.limit</screen>
+ <para>The <literal>lru_max_age</literal> parameter is used to control the
+ age of client-side locks in the LRU cached locks queue. This limits how
+ long unused locks are cached on the client, and avoids idle clients from
+ holding locks for an excessive time, which reduces memory usage on both
+ the client and server, as well as reducing work during server recovery.
+ </para>
+ <para>The <literal>lru_max_age</literal> is set and printed in milliseconds,
+ and by default is 3900000 ms (65 minutes).</para>
+ <para condition='l2B'>Since Lustre 2.11, in addition to setting the
+ maximum lock age in milliseconds, it can also be set using a suffix of
+ <literal>s</literal> or <literal>ms</literal> to indicate seconds or
+ milliseconds, respectively. For example to set the client's maximum
+ lock age to 15 minutes (900s) run:
+ </para>
+ <screen>
+# lctl set_param ldlm.namespaces.*MDT*.lru_max_age=900s
+# lctl get_param ldlm.namespaces.*MDT*.lru_max_age
+ldlm.namespaces.myth-MDT0000-mdc-ffff8804296c2800.lru_max_age=900000
+ </screen>
</section>
<section xml:id="dbdoclet.50438271_87260">
<title><indexterm>
</tbody>
</tgroup>
</informaltable>
- <para>For each service, an entry as shown below is
- created:<screen>/proc/fs/lustre/<replaceable>service</replaceable>/*/threads_<replaceable>min|max|started</replaceable></screen></para>
+ <para>For each service, tunable parameters as shown below are available.
+ </para>
<itemizedlist>
<listitem>
- <para>To temporarily set this tunable, run:</para>
- <screen># lctl <replaceable>get|set</replaceable>_param <replaceable>service</replaceable>.threads_<replaceable>min|max|started</replaceable> </screen>
- </listitem>
+ <para>To temporarily set these tunables, run:</para>
+ <screen># lctl set_param <replaceable>service</replaceable>.threads_<replaceable>min|max|started=num</replaceable> </screen>
+ </listitem>
<listitem>
<para>To permanently set this tunable, run:</para>
<screen># lctl conf_param <replaceable>obdname|fsname.obdtype</replaceable>.threads_<replaceable>min|max|started</replaceable> </screen>
<listitem>
<para>To set the maximum thread count to 256 instead of 512 permanently, run:</para>
<screen># lctl conf_param testfs.ost.ost_io.threads_max=256</screen>
- <para condition='l25'>For version 2.5 or later, run:
- <screen># lctl set_param -P ost.OSS.ost_io.threads_max=256
+
<para condition='l25'>For version 2.5 or later, run:
+ <screen># lctl set_param -P ost.OSS.ost_io.threads_max=256
ost.OSS.ost_io.threads_max=256 </screen> </para>
</listitem>
<listitem>
<primary>proc</primary>
<secondary>debug</secondary>
</indexterm>Enabling and Interpreting Debugging Logs</title>
- <para>By default, a detailed log of all operations is generated to aid in debugging. Flags that
- control debugging are found in <literal>/proc/sys/lnet/debug</literal>. </para>
- <para>The overhead of debugging can affect the performance of Lustre file system. Therefore, to
- minimize the impact on performance, the debug level can be lowered, which affects the amount
- of debugging information kept in the internal log buffer but does not alter the amount of
- information to goes into syslog. You can raise the debug level when you need to collect logs
- to debug problems. </para>
- <para>The debugging mask can be set using "symbolic names". The symbolic format is
- shown in the examples below.<itemizedlist>
+ <para>By default, a detailed log of all operations is generated to aid in
+ debugging. Flags that control debugging are found via
+ <literal>lctl get_param debug</literal>.</para>
+ <para>The overhead of debugging can affect the performance of Lustre file
+ system. Therefore, to minimize the impact on performance, the debug level
+ can be lowered, which affects the amount of debugging information kept in
+ the internal log buffer but does not alter the amount of information to
+ goes into syslog. You can raise the debug level when you need to collect
+ logs to debug problems. </para>
+ <para>The debugging mask can be set using "symbolic names". The
+ symbolic format is shown in the examples below.
+ <itemizedlist>
<listitem>
- <para>To verify the debug level used, examine the <literal>sysctl</literal> that controls
- debugging by running:</para>
- <screen># sysctl lnet.debug
-lnet.debug = ioctl neterror warning error emerg ha config console</screen>
+ <para>To verify the debug level used, examine the parameter that
+ controls debugging by running:</para>
+ <screen># lctl get_param debug
+debug=
+ioctl neterror warning error emerg ha config console</screen>
</listitem>
<listitem>
- <para>To turn off debugging (except for network error debugging), run the following
- command on all nodes concerned:</para>
+ <para>To turn off debugging except for network error debugging, run
+ the following command on all nodes concerned:</para>
<screen># sysctl -w lnet.debug="neterror"
-lnet.debug = neterror</screen>
+debug=neterror</screen>
</listitem>
- </itemizedlist><itemizedlist>
+ </itemizedlist>
+ <itemizedlist>
<listitem>
- <para>To turn off debugging completely, run the following command on all nodes
+ <para>To turn off debugging completely (except for the minimum error
+ reporting to the console), run the following command on all nodes
concerned:</para>
- <screen># sysctl -w lnet.debug=0
-lnet.debug = 0</screen>
- </listitem>
- <listitem>
- <para>To set an appropriate debug level for a production environment, run:</para>
- <screen># sysctl -w lnet.debug="warning dlmtrace error emerg ha rpctrace vfstrace"
-lnet.debug = warning dlmtrace error emerg ha rpctrace vfstrace</screen>
- <para>The flags shown in this example collect enough high-level information to aid
- debugging, but they do not cause any serious performance impact.</para>
+ <screen># lctl set_param debug=0
+debug=0</screen>
</listitem>
- </itemizedlist><itemizedlist>
<listitem>
- <para>To clear all flags and set new flags, run:</para>
- <screen># sysctl -w lnet.debug="warning"
-lnet.debug = warning</screen>
+ <para>To set an appropriate debug level for a production environment,
+ run:</para>
+ <screen># lctl set_param debug="warning dlmtrace error emerg ha rpctrace vfstrace"
+debug=warning dlmtrace error emerg ha rpctrace vfstrace</screen>
+ <para>The flags shown in this example collect enough high-level
+ information to aid debugging, but they do not cause any serious
+ performance impact.</para>
</listitem>
- </itemizedlist><itemizedlist>
+ </itemizedlist>
+ <itemizedlist>
<listitem>
- <para>To add new flags to flags that have already been set, precede each one with a
- "<literal>+</literal>":</para>
- <screen># sysctl -w lnet.debug="+neterror +ha"
-lnet.debug = +neterror +ha
-# sysctl lnet.debug
-lnet.debug = neterror warning ha</screen>
+ <para>To add new flags to flags that have already been set,
+ precede each one with a "<literal>+</literal>":</para>
+ <screen># lctl set_param debug="+neterror +ha"
+debug=+neterror +ha
+# lctl get_param debug
+debug=neterror warning error emerg ha console</screen>
</listitem>
<listitem>
<para>To remove individual flags, precede them with a
"<literal>-</literal>":</para>
- <screen># sysctl -w lnet.debug="-ha"
-lnet.debug = -ha
-# sysctl lnet.debug
-lnet.debug = neterror warning</screen>
+ <screen># lctl set_param debug="-ha"
+debug=-ha
+# lctl get_param debug
+debug=neterror warning error emerg console</screen>
</listitem>
- <listitem>
- <para>To verify or change the debug level, run commands such as the following: :</para>
- <screen># lctl get_param debug
-debug=
-neterror warning
-# lctl set_param debug=+ha
-# lctl get_param debug
-debug=
-neterror warning ha
-# lctl set_param debug=-warning
-# lctl get_param debug
-debug=
-neterror ha</screen>
- </listitem>
- </itemizedlist></para>
+ </itemizedlist>
+ </para>
<para>Debugging parameters include:</para>
<itemizedlist>
<listitem>
<literal>/tmp/lustre-log</literal>.</para>
</listitem>
</itemizedlist>
- <para>These parameters are also set using:<screen>sysctl -w lnet.debug={value}</screen></para>
+ <para>These parameters can also be set using:<screen>sysctl -w lnet.debug={value}</screen></para>
<para>Additional useful parameters: <itemizedlist>
<listitem>
<para><literal>panic_on_lbug</literal> - Causes ''panic'' to be called
obd_ping 212</screen>
<para>Use the <literal>llstat</literal> utility to monitor statistics over time.</para>
<para>To clear the statistics, use the <literal>-c</literal> option to
- <literal>llstat</literal>. To specify how frequently the statistics should be reported (in
- seconds), use the <literal>-i</literal> option. In the example below, the
- <literal>-c</literal> option clears the statistics and <literal>-i10</literal> option
- reports statistics every 10 seconds:</para>
- <screen role="smaller">$ llstat -c -i10 /proc/fs/lustre/ost/OSS/ost_io/stats
+ <literal>llstat</literal>. To specify how frequently the statistics
+ should be reported (in seconds), use the <literal>-i</literal> option.
+ In the example below, the <literal>-c</literal> option clears the
+ statistics and <literal>-i10</literal> option reports statistics every
+ 10 seconds:</para>
+<screen role="smaller">$ llstat -c -i10 ost_io
/usr/bin/llstat: STATS on 06/06/07
/proc/fs/lustre/ost/OSS/ost_io/ stats on 192.168.16.35@tcp
<para>See also <xref linkend="dbdoclet.50438219_84890"/> (<literal>llobdstat</literal>) and
<xref linkend="dbdoclet.50438273_80593"/> (<literal>collectl</literal>).</para>
</note>
- <para>MDT <literal>stats</literal> files can be used to track MDT statistics for the MDS. The
- example below shows sample output from an MDT <literal>stats</literal> file.</para>
+ <para>MDT <literal>stats</literal> files can be used to track MDT
+ statistics for the MDS. The example below shows sample output from an
+ MDT <literal>stats</literal> file.</para>
<screen># lctl get_param mds.*-MDT0000.stats
snapshot_time 1244832003.676892 secs.usecs
open 2 samples [reqs]
git://git.whamcloud.com / doc / manual.git / blobdiff