From: Andreas Dilger Date: Wed, 5 Aug 2020 08:58:46 +0000 (-0600) Subject: LU-8066 misc: replace remaining /proc access with lctl X-Git-Url: https://git.whamcloud.com/?a=commitdiff_plain;h=69b68cf79fff38864cf0888c7bd1053f658c55c7;p=doc%2Fmanual.git LU-8066 misc: replace remaining /proc access with lctl Replace most remaining parameter access references with calls to "lctl get_param" or "lctl set_param", since these aprameters are moved as needed to meet upstream kernel requirements. The last few stragglers are in the output of specific tools that will need to be fixed before their output will be correct. Signed-off-by: Andreas Dilger Change-Id: I4bbd7bf1bcbe8dadf612daa4917af337d23ebbe5 Reviewed-on: https://review.whamcloud.com/39578 Reviewed-by: Ben Evans Tested-by: jenkins --- diff --git a/LustreOperations.xml b/LustreOperations.xml index 8fea523..6881b13 100644 --- a/LustreOperations.xml +++ b/LustreOperations.xml @@ -635,8 +635,12 @@ mds# tunefs.lustre --param mdt.identity_upcall=NONE /dev/sda1 Setting Temporary Parameters Use lctl set_param to set temporary parameters on the - node where it is run. These parameters map to items in - /proc/{fs,sys}/{lnet,lustre}. The + node where it is run. These parameters internally map to corresponding + items in the kernel /proc/{fs,sys}/{lnet,lustre} and + /sys/{fs,kernel/debug}/lustre virtual filesystems. + However, since the mapping between a particular parameter name and the + underlying virtual pathname may change, it is not + recommended to access the virtual pathname directly. The lctl set_param command uses this syntax: lctl set_param [-n] [-P] @@ -661,11 +665,9 @@ osc.myth-OST0004-osc.max_dirty_mb=32 lctl conf_param command to set permanent parameters. In general, the lctl conf_param command can be used to specify any - parameter settable in a - /proc/fs/lustre file, with its own OBD device. The - lctl conf_param command uses this syntax (same as the - - mkfs.lustre and + settable parameter with its own OBD device. The + lctl conf_param command uses the following syntax + (the same as the mkfs.lustre and tunefs.lustre commands): obdname|fsname. @@ -673,6 +675,9 @@ osc.myth-OST0004-osc.max_dirty_mb=32 proc_file_name= value) + The lctl conf_param and + lctl set_param syntax is not + the same. Here are a few examples of lctl conf_param commands: @@ -693,11 +698,12 @@ $ lctl conf_param testfs.sys.timeout=40
Setting Permanent Parameters with lctl set_param -P The lctl set_param -P command can also - set parameters permanently. This command must be issued on the MGS. + set parameters permanently using the same syntax as + lctl set_param and lctl + get_param commands. This command must be issued on the MGS. The given parameter is set on every host using - lctl upcall. Parameters map to items in - /proc/{fs,sys}/{lnet,lustre}. The - lctl set_param command uses this syntax: + lctl upcall. The lctl set_param + command uses the following syntax: lctl set_param -P obdtype. @@ -721,12 +727,19 @@ osc.myth-OST0004-osc.max_dirty_mb=32 lctl set_param -P -d obdtype. obdname. -proc_file_name +parameter_name For example: # lctl set_param -P -d osc.*.max_dirty_mb + Starting in Lustre 2.12, there is + lctl get_param command can provide + tab completion when using an interactive shell + with bash-completion installed. This simplifies + the use of get_param significantly, since it + provides an interactive list of available parameters. +
Listing Parameters @@ -764,6 +777,13 @@ lctl get_param [-n] obdname. proc_file_name + Starting in Lustre 2.12, there is + lctl get_param command can provide + tab completion when using an interactive shell + with bash-completion installed. This simplifies + the use of get_param significantly, since it + provides an interactive list of available parameters. + This example reports data on RPC service times. oss# lctl get_param -n ost.*.ost_io.timeouts diff --git a/LustreProc.xml b/LustreProc.xml index 7496e2c..3c37c8a 100644 --- a/LustreProc.xml +++ b/LustreProc.xml @@ -3,10 +3,11 @@ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustreproc"> Lustre Parameters - The /proc and /sys file systems - acts as an interface to internal data structures in the kernel. This chapter - describes parameters and tunables that are useful for optimizing and - monitoring aspects of a Lustre file system. It includes these sections: + There are many parameters for Lustre that can tune client and server + performance, change behavior of the system, and report statistics about + various subsystems. This chapter describes the various parameters and + tunables that are useful for optimizing and monitoring aspects of a Lustre + file system. It includes these sections: @@ -24,9 +25,11 @@ Typically, metrics are accessed via lctl get_param files and settings are changed by via lctl set_param. - While it is possible to access parameters in /proc + They allow getting and setting multiple parameters with a single command, + through the use of wildcards in one or more part of the parameter name. + While each of these parameters maps to files in /proc and /sys directly, the location of these parameters may - change between releases, so it is recommended to always use + change between Lustre releases, so it is recommended to always use lctl to access the parameters from userspace scripts. Some data is server-only, some data is client-only, and some data is exported from the client to the server and is thus duplicated in both @@ -35,24 +38,29 @@ In the examples in this chapter, # indicates a command is entered as root. Lustre servers are named according to the convention fsname-MDT|OSTnumber. - The standard UNIX wildcard designation (*) is used. + The standard UNIX wildcard designation (*) is used to represent any + part of a single component of the parameter name, excluding + "." and "/". + It is also possible to use brace {}expansion + to specify a list of parameter names efficiently. Some examples are shown below: - To obtain data from a Lustre client: - # lctl list_param osc.* -osc.testfs-OST0000-osc-ffff881071d5cc00 -osc.testfs-OST0001-osc-ffff881071d5cc00 -osc.testfs-OST0002-osc-ffff881071d5cc00 -osc.testfs-OST0003-osc-ffff881071d5cc00 -osc.testfs-OST0004-osc-ffff881071d5cc00 -osc.testfs-OST0005-osc-ffff881071d5cc00 -osc.testfs-OST0006-osc-ffff881071d5cc00 -osc.testfs-OST0007-osc-ffff881071d5cc00 -osc.testfs-OST0008-osc-ffff881071d5cc00 + To list available OST targets on a Lustre client: + # lctl list_param -F osc.* +osc.testfs-OST0000-osc-ffff881071d5cc00/ +osc.testfs-OST0001-osc-ffff881071d5cc00/ +osc.testfs-OST0002-osc-ffff881071d5cc00/ +osc.testfs-OST0003-osc-ffff881071d5cc00/ +osc.testfs-OST0004-osc-ffff881071d5cc00/ +osc.testfs-OST0005-osc-ffff881071d5cc00/ +osc.testfs-OST0006-osc-ffff881071d5cc00/ +osc.testfs-OST0007-osc-ffff881071d5cc00/ +osc.testfs-OST0008-osc-ffff881071d5cc00/ In this example, information about OST connections available - on a client is displayed (indicated by "osc"). + on a client is displayed (indicated by "osc"). Each of these + connections may have numerous sub-parameters as well. @@ -72,6 +80,16 @@ osc.testfs-OST0000-osc-ffff881071d5cc00.rpc_stats + To see a specific subset of parameters, use braces, like: +# lctl list_param osc.*.{checksum,connect}* +osc.testfs-OST0000-osc-ffff881071d5cc00.checksum_type +osc.testfs-OST0000-osc-ffff881071d5cc00.checksums +osc.testfs-OST0000-osc-ffff881071d5cc00.connect_flags + + + + + To view a specific file, use lctl get_param: # lctl get_param osc.lustre-OST0000*.rpc_stats @@ -90,31 +108,14 @@ osc.testfs-OST0000-osc-ffff881071d5cc00.rpc_stats version and the Lustre version being used. The lctl command insulates scripts from these changes and is preferred over direct file access, unless as part of a high-performance monitoring system. - In the cat command: - - - Replace the dots in the path with slashes. - - - Prepend the path with the appropriate directory component: - /{proc,sys}/{fs,sys}/{lustre,lnet} - - - For example, an lctl get_param command may look like - this:# lctl get_param osc.*.uuid -osc.testfs-OST0000-osc-ffff881071d5cc00.uuid=594db456-0685-bd16-f59b-e72ee90e9819 -osc.testfs-OST0001-osc-ffff881071d5cc00.uuid=594db456-0685-bd16-f59b-e72ee90e9819 -... - The equivalent cat command may look like this: - # cat /proc/fs/lustre/osc/*/uuid -594db456-0685-bd16-f59b-e72ee90e9819 -594db456-0685-bd16-f59b-e72ee90e9819 -... - or like this: - # cat /sys/fs/lustre/osc/*/uuid -594db456-0685-bd16-f59b-e72ee90e9819 -594db456-0685-bd16-f59b-e72ee90e9819 -... + + Starting in Lustre 2.12, there is + lctl get_param and lctl set_param + command can provide tab completion when using an + interactive shell with bash-completion installed. + This simplifies the use of get_param significantly, + since it provides an interactive list of available parameters. + The llstat utility can be used to monitor some Lustre file system I/O activity over a specified time period. For more details, see @@ -2310,19 +2311,19 @@ nid refs peer max tx min To enable automatic LRU sizing, set the - lru_size parameter to 0. In this case, the - lru_size parameter shows the current number of locks + lru_size parameter to 0. In this case, the + lru_size parameter shows the current number of locks being used on the client. Dynamic LRU resizing is enabled by default. - + To specify a maximum number of locks, set the - lru_size parameter to a value other than zero. - A good default value for compute nodes is around - 100 * num_cpus. + lru_size parameter to a value other than zero. + A good default value for compute nodes is around + 100 * num_cpus. It is recommended that you only set lru_size - to be signifivantly larger on a few login nodes where multiple - users access the file system interactively. + to be signifivantly larger on a few login nodes where multiple + users access the file system interactively. To clear the LRU on a single client, and, as a result, flush client @@ -2335,7 +2336,7 @@ nid refs peer max tx min The lru_size parameter can only be set temporarily using lctl set_param, it cannot be set - permanently. + permanently. To disable dynamic LRU resizing on the clients, run for example: @@ -2461,15 +2462,18 @@ ldlm.namespaces.myth-MDT0000-mdc-ffff8804296c2800.lru_max_age=900000 # lctl set_param service.threads_min|max|started=num - To permanently set this tunable, run: - # lctl conf_param obdname|fsname.obdtype.threads_min|max|started - For version 2.5 or later, run: - # lctl set_param -P service.threads_min|max|started + To permanently set this tunable, run the following command on + the MGS: + mgs# lctl set_param -P service.threads_min|max|started + For Lustre 2.5 or earlier, run: + mgs# lctl conf_param obdname|fsname.obdtype.threads_min|max|started + - The following examples show how to set thread counts and get the number of running threads - for the service ost_io using the tunable - service.threads_min|max|started. + The following examples show how to set thread counts and get the + number of running threads for the service ost_io + using the tunable + service.threads_min|max|started. To get the number of running threads, run: diff --git a/LustreRecovery.xml b/LustreRecovery.xml index 07f081b..a04a87f 100644 --- a/LustreRecovery.xml +++ b/LustreRecovery.xml @@ -703,20 +703,31 @@ $ lctl get_param osc.testfs-OST0001-osc-*.import |grep instance
<indexterm><primary>pings</primary><secondary>suppress_pings</secondary></indexterm>"suppress_pings" Kernel Module Parameter - The new option that controls whether pings are suppressed is implemented as the ptlrpc kernel module parameter "suppress_pings". Setting it to "1" on a server turns on ping suppressing for all targets on that server, while leaving it with the default value "0" gives previous pinging behavior. The parameter is ignored on clients and the MGS. While the parameter is recommended to be set persistently via the modprobe.conf(5) mechanism, it also accept online changes through sysfs. Note that an online change only affects connections established later; existing connections' pinging behaviors stay the same. + The new option that controls whether pings are suppressed is + implemented as the ptlrpc kernel module parameter "suppress_pings". + Setting it to "1" on a server turns on ping suppressing for all + targets on that server, while leaving it with the default value "0" + gives previous pinging behavior. The parameter is ignored on clients + and the MGS. While the parameter is recommended to be set persistently + via the modprobe.conf(5) mechanism, it also accept online changes + through sysfs. Note that an online change only affects connections + established later; existing connections' pinging behaviors stay the same. +
<indexterm><primary>pings</primary><secondary>evict_client</secondary></indexterm>Client Death Notification - The required external client death notification shall write UUIDs of dead clients into targets' "evict_client" procfs entries like - -/proc/fs/lustre/obdfilter/testfs-OST0000/evict_client -/proc/fs/lustre/obdfilter/testfs-OST0001/evict_client -/proc/fs/lustre/mdt/testfs-MDT0000/evict_client - - Clients' UUIDs can be obtained from their "uuid" procfs entries like - -/proc/fs/lustre/llite/testfs-ffff8800612bf800/uuid - + The required external client death notification shall write UUIDs + of dead clients into targets' evict_client procfs + entries in order to remove stale clients from recovery. + A client UUID can be obtained from their uuid + procfs entry and that UUID can be used to evict the client, like: + + +client$ lctl get_param llite.testfs-*.uuid +llite.testfs-ffff991ae1992000.uuid=dd599d28-0a85-a9e4-82cd-dc6357a42c77 +oss# lctl set_param obdfilter.testfs-*.evict_client=dd599d28-0a85-a9e4-82cd-dc6357a42c77 +mds# lctl set_param mdt.testfs-*.evict_client=dd599d28-0a85-a9e4-82cd-dc6357a42c77 +
diff --git a/LustreTroubleshooting.xml b/LustreTroubleshooting.xml index 178e97f..b82c558 100644 --- a/LustreTroubleshooting.xml +++ b/LustreTroubleshooting.xml @@ -204,7 +204,8 @@ Which server node it was communicating with, and so on. - Lustre logs are dumped to /proc/sys/lnet/debug_path. + Lustre logs are dumped to the pathname stored in the parameter + lnet.debug_path. Collect the first group of messages related to a problem, and any messages that precede "LBUG" or "assertion failure" errors. Messages that mention server nodes (OST or MDS) are specific to that server; you must collect similar messages from the relevant server console logs. Another Lustre debug log holds information for a short period of time for action by the Lustre software, which, in turn, depends on the processes on the Lustre node. Use the @@ -376,7 +377,7 @@ If the SCSI devices are inaccessible to the Lustre file system at the block device level, then ldiskfs remounts the device read-only to prevent file system corruption. This is a normal - behavior. The status in /proc/fs/lustre/health_check + behavior. The status in the parameter health_check also shows "not healthy" on the affected nodes. To determine what caused the "not healthy" condition: diff --git a/LustreTuning.xml b/LustreTuning.xml index 91e5ae8..f51e325 100644 --- a/LustreTuning.xml +++ b/LustreTuning.xml @@ -384,9 +384,8 @@ lnet large_router_buffers=8192 events across all CPTs. This may balance load better across the CPU but can incur a cross CPU overhead. The current policy can be changed by an administrator with - echo - value> - /proc/sys/lnet/portal_rotor. There are four options for + lctl set_param portal_rotor=value. + There are four options for value : @@ -2096,9 +2095,7 @@ ldlm.namespaces.filter-fsname-*. Client-side: - -/proc/fs/lustre/llite/lustre-* - + llite.fsname-* contention_seconds- llite inode remembers its contended state for the @@ -2109,8 +2106,8 @@ ldlm.namespaces.filter-fsname-*. Client-side statistics: The - /proc/fs/lustre/llite/lustre-*/stats file has new - rows for lockless I/O statistics. + llite.fsname-*.stats + parameter has several entries for lockless I/O statistics. lockless_read_bytes and lockless_write_bytes- To count the total bytes read diff --git a/ManagingFileSystemIO.xml b/ManagingFileSystemIO.xml index b4db9ce..3e75363 100644 --- a/ManagingFileSystemIO.xml +++ b/ManagingFileSystemIO.xml @@ -480,8 +480,8 @@ oss# mount -t lustre /dev/sda /mnt/testfs/ost12 If a Lustre file system administrator wants to explore this approach - further, per-OST disk-usage statistics can be found under - /proc/fs/lustre/osc/*/rpc_stats + further, per-OST disk-usage statistics can be found in the + osc.*.rpc_stats parameter file.
diff --git a/SystemConfigurationUtilities.xml b/SystemConfigurationUtilities.xml index 6a2ac57..a946de8 100644 --- a/SystemConfigurationUtilities.xml +++ b/SystemConfigurationUtilities.xml @@ -193,8 +193,10 @@ l_getidentity
Files - The l_getidentity files are located at: - /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall + The parameter to set the l_getidentity path is: + +mds# lctl set_param -P mdt.*-MDT*.identity_upcall=path +
@@ -220,15 +222,32 @@ quit
Setting Parameters with lctl - Lustre parameters are not always accessible using the procfs interface, as it is platform-specific. As a solution, lctl {get,set}_param has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to /proc/{fs,sys}/{lustre,lnet}. For future portability, use lctl {get,set}_param . - When the file system is running, use the lctl set_param command on the affected node(s) to temporarily set parameters (mapping to items in /proc/{fs,sys}/{lnet,lustre}). The lctl set_param command uses this syntax: + Lustre parameters are not always accessible using the procfs + interface, as it is platform-specific. As a solution, + lctl {get,set}_param provides a platform-independent + interface to the Lustre tunables. Avoid any direct references to + /proc and /sys files in scripts. + For future portability, instead use lctl {get,set}_param, which handles + these details internally. + When the file system is running, use the + lctl set_param command on the affected node(s) to + temporarily set parameters (mapping to items in). + The lctl set_param command uses this syntax: lctl set_param [-n] [-P] [-d] obdtype.obdname.property=value For example: - mds# lctl set_param mdt.testfs-MDT0000.identity_upcall=NONE - Use -P option to set parameters permanently. Option -d deletes permanent parameters. For example: - mgs# lctl set_param -P mdt.testfs-MDT0000.identity_upcall=NONE +mds# lctl set_param mdt.testfs-MDT0000.identity_upcall=NONE + Use -P option to set parameters + permanently. Option -d deletes permanent parameters. + For example: + +mgs# lctl set_param -P mdt.testfs-MDT0000.identity_upcall=NONE mgs# lctl set_param -P -d mdt.testfs-MDT0000.identity_upcall - Many permanent parameters can be set with lctl conf_param. In general, lctl conf_param can be used to specify any OBD device parameter settable in a /proc/fs/lustre file. The lctl conf_param command must be run on the MGS node, and uses this syntax: + Many permanent parameters can be set with the + lctl conf_param utility. In general, + lctl conf_param can be used to specify any OBD + device parameter settable in a /proc/fs/lustre file. The + lctl conf_param command must be run on the MGS node, + and uses this syntax: obd|fsname.obdtype.property=value) For example: mgs# lctl conf_param testfs-MDT0000.mdt.identity_upcall=NONE @@ -2724,21 +2743,20 @@ Application Profiling Utilities The lustre_req_history.sh utility (run from a client), assembles as much Lustre RPC request history as possible from the local node and from the servers that were contacted, providing a better picture of the coordinated network activity.
- More /proc Statistics for Application Profiling + More Statistics for Application Profiling The following utilities provide additional statistics. vfs_ops_stats - The client vfs_ops_stats utility tracks Linux VFS operation calls into Lustre for a single PID, PPID, GID or everything. - /proc/fs/lustre/llite/*/vfs_ops_stats -/proc/fs/lustre/llite/*/vfs_track_[pid|ppid|gid] - + The client vfs_ops_stats utility tracks Linux VFS operation calls + into Lustre for a single PID, PPID, GID or everything. + llite.*.vfs_ops_stats llite.*.vfs_track_[pid|ppid|gid] extents_stats - The client extents_stats utility shows the size distribution of I/O calls from the client (cumulative and by process). - /proc/fs/lustre/llite/*/extents_stats, extents_stats_per_process - + The client extents_stats utility shows the size distribution of + I/O calls from the client (cumulative and by process). + llite.*.{extents_stats,extents_stats_per_process} offset_stats - The client offset_stats utility shows the read/write seek activity of a client by offsets and ranges. - /proc/fs/lustre/llite/*/offset_stats - + The client offset_stats utility shows the read/write seek activity + of a client by offsets and ranges. + llite.*.offset_stats Lustre includes per-client and improved MDT statistics: @@ -2748,7 +2766,7 @@ Application Profiling Utilities Each MDS and OSS now tracks LDLM and operations statistics for every connected client, for comparisons and simpler collection of distributed job statistics. - /proc/fs/lustre/mds|obdfilter/*/exports/ + {mds,obdfilter}.*.exports @@ -2757,7 +2775,7 @@ Application Profiling Utilities More detailed MDT operations statistics are collected for better profiling. - /proc/fs/lustre/mdt/*/md_stats + mdt.*.md_stats
diff --git a/TroubleShootingRecovery.xml b/TroubleShootingRecovery.xml index 30e4e10..d61e22d 100644 --- a/TroubleShootingRecovery.xml +++ b/TroubleShootingRecovery.xml @@ -218,7 +218,7 @@ root# e2fsck -fp /dev/sda # fix errors with prudent answers (usually is enhanced to support verify and repair inconsistencies between multiple MDTs. Control and monitoring of LFSCK is through LFSCK and the - /proc file system interfaces. LFSCK supports three types + lctl get_param command. LFSCK supports three types of interface: switch interface, status interface, and adjustment interface. These interfaces are detailed below.
diff --git a/UserUtilities.xml b/UserUtilities.xml index 449d115..b490a83 100644 --- a/UserUtilities.xml +++ b/UserUtilities.xml @@ -272,17 +272,25 @@ lfs help guarantee that atime is kept coherent across the cluster.) - OSTs store a transient + OSTs by default only hold a transient atime that is updated when clients do read requests. Permanent - atime is written to the MDS when the file is + atime is written to the MDT when the file is closed. However, on-disk atime is only updated if it is more than 60 seconds old ( - /proc/fs/lustre/mds/*/max_atime_diff). The - Lustre software considers the latest - atime from all OSTs. If a + mdd.*.atime_diff). + + In Lustre 2.14, it is possible to set + the OSTs to persistently store atime with each object, in + order to get more accurate persistent atime updates for files + that are open for a long time via the similarly-named + obdfilter.*.atime_diff parameter. + + + The client considers the latest atime from + all OSTs and MDTs. If a setattr is set by user, then it is updated on - both the MDS and OST, allowing the + both the MDT and OST, allowing the atime to go backward.