LUDOC-304 tuning: support multiple modify RPCs in parallel

Update the Lustre manual to reflect the support of multiple modify
metadata RPCs in parallel, LU-5319:
https://jira.hpdd.intel.com/browse/LU-5319.

The following sections are added or modified:
- Tuning the Client Metadata RPC Stream
- Reply Reconstruction
- lr_reader debugging utility

Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: I3b4199424b2ca43b94e26ee715ae7cc9d5770080
Reviewed-on: http://review.whamcloud.com/16429
Reviewed-by: Richard Henwood <richard.henwood@intel.com>
Tested-by: Jenkins

diff --git a/LustreProc.xml b/LustreProc.xml
index d4ea8a1..efa0ec4 100644 (file)
--- a/LustreProc.xml
+++ b/LustreProc.xml
@@ -1495,6 +1495,55 @@ obdfilter.lol-OST0001.sync_journal=0</screen>
       <screen>$ lctl get_param obdfilter.*.sync_on_lock_cancel
 obdfilter.lol-OST0001.sync_on_lock_cancel=never</screen>
     </section>
+    <section xml:id="dbdoclet.TuningModRPCs" condition='l28'>
+      <title><indexterm><primary>proc</primary><secondary>client metadata performance</secondary></indexterm>Tuning the Client Metadata RPC Stream</title>
+      <para>The client metadata RPC stream represents the metadata RPCs issued in parallel by a client to a MDT target. The metadata RPCs can be split in two categories: the requests that do not modify the file system (like getattr operation), and the requests that do modify the file system (like create, unlink, setattr operations). To help optimize the client metadata RPC stream, several tuning variables are provided to adjust behavior according to network conditions and cluster size.</para>
+      <para>Note that increasing the number of metadata RPCs issued in parallel might improve the performance metadata intensive parallel applications, but as a consequence it will consum more memory on the client and on the MDS.</para>
+      <section>
+        <title>Configuring the Client Metadata RPC Stream</title>
+        <para>The MDC <literal>max_rpcs_in_flight</literal> parameter defines the maximum number of metadata RPCs, both modifying and non-modifying RPCs, that can be sent in parallel by a client to a MDT target. This includes every file system metadata operations, such as file or directory stat, creation, unlink. The default setting is 8, minimum setting is 1 and maximum setting is 256.</para>
+        <para>To set the <literal>max_rpcs_in_flight</literal> parameter, run the following command on the Lustre client:</para>
+        <screen>$ lctl set_param mdc.*.max_rcps_in_flight=16</screen>
+        <para>The MDC <literal>max_mod_rpcs_in_flight</literal> parameter defines the maximum number of file system modifying RPCs that can be sent in parallel by a client to a MDT target. For example, the Lustre client sends modify RPCs when it performs file or directory creation, unlink, access permission modification or ownership modification. The default setting is 7, minimum setting is 1 and maximum setting is 256.</para>
+        <para>To set the <literal>max_mod_rpcs_in_flight</literal> parameter, run the following command on the Lustre client:</para>
+        <screen>$ lctl set_param mdc.*.max_mod_rcps_in_flight=12</screen>
+        <para>The <literal>max_mod_rpcs_in_flight</literal> value must be strictly less than the <literal>max_rpcs_in_flight</literal> value. It must also be less or equal to the MDT <literal>max_mod_rpcs_per_client</literal> value. If one of theses conditions is not enforced, the setting fails and an explicit message is written in the Lustre log.</para>
+        <para>The MDT <literal>max_mod_rpcs_per_client</literal> parameter is a tunable of the kernel module <literal>mdt</literal> that defines the maximum number of file system modifying RPCs in flight allowed per client. The parameter can be updated at runtime, but the change is effective to new client connections only. The default setting is 8.</para>
+        <para>To set the <literal>max_mod_rpcs_per_client</literal> parameter, run the following command on the MDS:</para>
+        <screen>$ echo 12 > /sys/module/mdt/parameters/max_mod_rpcs_per_client</screen>
+      </section>
+      <section>
+        <title>Monitoring the Client Metadata RPC Stream</title>
+        <para>The <literal>rpc_stats</literal> file contains histogram data showing information about modify metadata RPCs. It can be helpful to identify the level of parallelism achieved by an application doing modify metadata operations.</para>
+        <para><emphasis role="bold">Example:</emphasis></para>
+        <screen>$ lctl get_param mdc.testfs-MDT0000-mdt-ffff88077fb3a000.rpc_stats
+snapshot_time:         1441876896.567070 (secs.usecs)
+modify_RPCs_in_flight:  0
+
+                        modify
+rpcs in flight        rpcs   % cum %
+0:                       0   0   0
+1:                      56   0   0
+2:                      40   0   0
+3:                      70   0   0
+4                       41   0   0
+5:                      51   0   1
+6:                      88   0   1
+7:                     366   1   2
+8:                    1321   5   8
+9:                    3624  15  23
+10:                   6482  27  50
+11:                   7321  30  81
+12:                   4540  18 100</screen>
+        <para>The file information includes:</para>
+        <itemizedlist>
+          <listitem><para><literal>snapshot_time</literal> - UNIX epoch instant the file was read.</para></listitem>
+          <listitem><para><literal>modify_RPCs_in_flight</literal> - Number of modify RPCs issued by the MDC, but not completed at the time of the snapshot. This value should always be less than or equal to <literal>max_mod_rpcs_in_flight</literal>.</para></listitem>
+          <listitem><para><literal>rpcs in flight</literal> - Number of modify RPCs that are pending when a RPC is sent, the relative percentage (<literal>%</literal>) of total modify RPCs, and the cumulative percentage (<literal>cum %</literal>) to that point.</para></listitem>
+        </itemizedlist>
+        <para>If a large proportion of modify metadata RPCs are issued with a number of pending metadata RPCs close to the <literal>max_mod_rpcs_in_flight</literal> value, it means the <literal>max_mod_rpcs_in_flight</literal> value could be increased to improve the modify metadata performance.</para>
+      </section>
+    </section>
   </section>
   <section>
     <title>Configuring Timeouts in a Lustre File System</title>

diff --git a/LustreRecovery.xml b/LustreRecovery.xml
index 36205b9..67a1832 100644 (file)
--- a/LustreRecovery.xml
+++ b/LustreRecovery.xml
@@ -286,6 +286,10 @@
         <para>The lock handle can be found by walking the list of granted locks for the resource looking for one with the appropriate remote file handle (present in the re-sent request). Verify that the lock has the right mode (determined by performing the disposition/request/status analysis above) and is granted to the proper client.</para>
       </section>
     </section>
+    <section remap="h3" condition="l28">
+      <title>Multiple Reply Data per Client</title>
+      <para>Since Lustre 2.8, the MDS is able to save several reply data per client. The reply data are stored in the <literal>reply_data</literal> internal file of the MDT. Additionally to the XID of the request, the transaction number, the result code and the open "disposition", the reply data contains a generation number that identifies the client thanks to the content of the <literal>last_rcvd</literal> file.</para>
+    </section>
   </section>
   <section xml:id="versionbasedrecovery">
     <title><indexterm><primary>Version-based recovery (VBR)</primary></indexterm>Version-based Recovery</title>

diff --git a/SystemConfigurationUtilities.xml b/SystemConfigurationUtilities.xml
index bd11b10..f4a21bb 100644 (file)
--- a/SystemConfigurationUtilities.xml
+++ b/SystemConfigurationUtilities.xml
@@ -2723,7 +2723,7 @@ EOF
       <section remap="h5">
         <title><indexterm><primary>lr_reader</primary></indexterm>
 lr_reader</title>
-        <para>The lr_reader utility translates a last received (last_rcvd) file into human-readable form.</para>
+        <para>The lr_reader utility translates the content of the <literal>last_rcvd</literal> and <literal>reply_data</literal> files into human-readable form.</para>
         <para>The following utilities are part of the Lustre I/O kit. For more information, see <xref linkend="benchmarkingtests"/>.</para>
       </section>
       <section remap="h5">