From: Gregoire Pichon Date: Wed, 23 Mar 2016 07:38:18 +0000 (+0100) Subject: LUDOC-304 fix: improvements related to multiple modify RPCs X-Git-Tag: 2.9.0~30 X-Git-Url: https://git.whamcloud.com/?a=commitdiff_plain;h=024ff0c6ba844b29fcc2b391ddab2cbd772170d9;p=doc%2Fmanual.git LUDOC-304 fix: improvements related to multiple modify RPCs Address several minor improvements detected during review: wrap text at 80 columns, fix typo, ... Signed-off-by: Gregoire Pichon Change-Id: I1813ac6911a3309f2b99d4f60bacb25a0cba1dfc Reviewed-on: http://review.whamcloud.com/19079 Tested-by: Jenkins Reviewed-by: Richard Henwood --- diff --git a/LustreProc.xml b/LustreProc.xml index efa0ec4..48dd499 100644 --- a/LustreProc.xml +++ b/LustreProc.xml @@ -1496,27 +1496,69 @@ obdfilter.lol-OST0001.sync_journal=0 obdfilter.lol-OST0001.sync_on_lock_cancel=never
- <indexterm><primary>proc</primary><secondary>client metadata performance</secondary></indexterm>Tuning the Client Metadata RPC Stream - The client metadata RPC stream represents the metadata RPCs issued in parallel by a client to a MDT target. The metadata RPCs can be split in two categories: the requests that do not modify the file system (like getattr operation), and the requests that do modify the file system (like create, unlink, setattr operations). To help optimize the client metadata RPC stream, several tuning variables are provided to adjust behavior according to network conditions and cluster size. - Note that increasing the number of metadata RPCs issued in parallel might improve the performance metadata intensive parallel applications, but as a consequence it will consum more memory on the client and on the MDS. + + <indexterm> + <primary>proc</primary> + <secondary>client metadata performance</secondary> + </indexterm> + Tuning the Client Metadata RPC Stream + + The client metadata RPC stream represents the metadata RPCs issued + in parallel by a client to a MDT target. The metadata RPCs can be split + in two categories: the requests that do not modify the file system + (like getattr operation), and the requests that do modify the file system + (like create, unlink, setattr operations). To help optimize the client + metadata RPC stream, several tuning variables are provided to adjust + behavior according to network conditions and cluster size. + Note that increasing the number of metadata RPCs issued in parallel + might improve the performance metadata intensive parallel applications, + but as a consequence it will consume more memory on the client and on + the MDS.
Configuring the Client Metadata RPC Stream - The MDC max_rpcs_in_flight parameter defines the maximum number of metadata RPCs, both modifying and non-modifying RPCs, that can be sent in parallel by a client to a MDT target. This includes every file system metadata operations, such as file or directory stat, creation, unlink. The default setting is 8, minimum setting is 1 and maximum setting is 256. - To set the max_rpcs_in_flight parameter, run the following command on the Lustre client: - $ lctl set_param mdc.*.max_rcps_in_flight=16 - The MDC max_mod_rpcs_in_flight parameter defines the maximum number of file system modifying RPCs that can be sent in parallel by a client to a MDT target. For example, the Lustre client sends modify RPCs when it performs file or directory creation, unlink, access permission modification or ownership modification. The default setting is 7, minimum setting is 1 and maximum setting is 256. - To set the max_mod_rpcs_in_flight parameter, run the following command on the Lustre client: - $ lctl set_param mdc.*.max_mod_rcps_in_flight=12 - The max_mod_rpcs_in_flight value must be strictly less than the max_rpcs_in_flight value. It must also be less or equal to the MDT max_mod_rpcs_per_client value. If one of theses conditions is not enforced, the setting fails and an explicit message is written in the Lustre log. - The MDT max_mod_rpcs_per_client parameter is a tunable of the kernel module mdt that defines the maximum number of file system modifying RPCs in flight allowed per client. The parameter can be updated at runtime, but the change is effective to new client connections only. The default setting is 8. - To set the max_mod_rpcs_per_client parameter, run the following command on the MDS: - $ echo 12 > /sys/module/mdt/parameters/max_mod_rpcs_per_client + The MDC max_rpcs_in_flight parameter defines + the maximum number of metadata RPCs, both modifying and + non-modifying RPCs, that can be sent in parallel by a client to a MDT + target. This includes every file system metadata operations, such as + file or directory stat, creation, unlink. The default setting is 8, + minimum setting is 1 and maximum setting is 256. + To set the max_rpcs_in_flight parameter, run + the following command on the Lustre client: + client$ lctl set_param mdc.*.max_rcps_in_flight=16 + The MDC max_mod_rpcs_in_flight parameter + defines the maximum number of file system modifying RPCs that can be + sent in parallel by a client to a MDT target. For example, the Lustre + client sends modify RPCs when it performs file or directory creation, + unlink, access permission modification or ownership modification. The + default setting is 7, minimum setting is 1 and maximum setting is + 256. + To set the max_mod_rpcs_in_flight parameter, + run the following command on the Lustre client: + client$ lctl set_param mdc.*.max_mod_rcps_in_flight=12 + The max_mod_rpcs_in_flight value must be + strictly less than the max_rpcs_in_flight value. + It must also be less or equal to the MDT + max_mod_rpcs_per_client value. If one of theses + conditions is not enforced, the setting fails and an explicit message + is written in the Lustre log. + The MDT max_mod_rpcs_per_client parameter is a + tunable of the kernel module mdt that defines the + maximum number of file system modifying RPCs in flight allowed per + client. The parameter can be updated at runtime, but the change is + effective to new client connections only. The default setting is 8. + + To set the max_mod_rpcs_per_client parameter, + run the following command on the MDS: + mds$ echo 12 > /sys/module/mdt/parameters/max_mod_rpcs_per_client
Monitoring the Client Metadata RPC Stream - The rpc_stats file contains histogram data showing information about modify metadata RPCs. It can be helpful to identify the level of parallelism achieved by an application doing modify metadata operations. + The rpc_stats file contains histogram data + showing information about modify metadata RPCs. It can be helpful to + identify the level of parallelism achieved by an application doing + modify metadata operations. Example: - $ lctl get_param mdc.testfs-MDT0000-mdt-ffff88077fb3a000.rpc_stats + client$ lctl get_param mdc.*.rpc_stats snapshot_time: 1441876896.567070 (secs.usecs) modify_RPCs_in_flight: 0 @@ -1537,11 +1579,28 @@ rpcs in flight rpcs % cum % 12: 4540 18 100 The file information includes: - snapshot_time - UNIX epoch instant the file was read. - modify_RPCs_in_flight - Number of modify RPCs issued by the MDC, but not completed at the time of the snapshot. This value should always be less than or equal to max_mod_rpcs_in_flight. - rpcs in flight - Number of modify RPCs that are pending when a RPC is sent, the relative percentage (%) of total modify RPCs, and the cumulative percentage (cum %) to that point. + + snapshot_time - UNIX epoch instant the + file was read. + + + modify_RPCs_in_flight - Number of modify + RPCs issued by the MDC, but not completed at the time of the + snapshot. This value should always be less than or equal to + max_mod_rpcs_in_flight. + + + rpcs in flight - Number of modify RPCs + that are pending when a RPC is sent, the relative percentage + (%) of total modify RPCs, and the cumulative + percentage (cum %) to that point. + - If a large proportion of modify metadata RPCs are issued with a number of pending metadata RPCs close to the max_mod_rpcs_in_flight value, it means the max_mod_rpcs_in_flight value could be increased to improve the modify metadata performance. + If a large proportion of modify metadata RPCs are issued with a + number of pending metadata RPCs close to the + max_mod_rpcs_in_flight value, it means the + max_mod_rpcs_in_flight value could be increased to + improve the modify metadata performance.