From 649bcdd7fc96e5caa068df2aba15905be5d70e41 Mon Sep 17 00:00:00 2001 From: Andreas Dilger Date: Thu, 4 Aug 2016 16:51:53 -0600 Subject: [PATCH] LUDOC-338 proc: use lctl {get,set}_param Document the use of "lctl {get,set}_param" instead of "cat" for accessing /proc tunables and stats, since these may have been moved to /sys or /sys/kernel/debug in newer kernels, and scripts using "cat" will break. Fix some places where "mdt" should be used instead of "mds" since 2.x. Signed-off-by: Andreas Dilger Change-Id: I3c97d18f5397a38ff01b431cf765b3b40d8bea4b Reviewed-on: http://review.whamcloud.com/21730 Tested-by: Jenkins Reviewed-by: James Simmons --- LustreMaintenance.xml | 42 ++++++++-------- LustreOperations.xml | 2 +- LustreProc.xml | 105 +++++++++++++++++++++++---------------- SystemConfigurationUtilities.xml | 11 ++-- 4 files changed, 92 insertions(+), 68 deletions(-) diff --git a/LustreMaintenance.xml b/LustreMaintenance.xml index 9e5c812..b640810 100644 --- a/LustreMaintenance.xml +++ b/LustreMaintenance.xml @@ -65,41 +65,43 @@ To mount a client or an MDT with one or more inactive OSTs, run commands similar to this: client# mount -o exclude=testfs-OST0000 -t lustre \ uml1:/testfs /mnt/testfs - client# cat /proc/fs/lustre/lov/testfs-clilov-*/target_obd - To activate an inactive OST on a live client or MDT, use the lctl activate command on the OSC device. For example: + client# lctl get_param lov.testfs-clilov-*.target_obd + To activate an inactive OST on a live client or MDT, use the + lctl activate command on the OSC device. For example: lctl --device 7 activate - A colon-separated list can also be specified. For example, exclude=testfs-OST0000:testfs-OST0001. + A colon-separated list can also be specified. For example, + exclude=testfs-OST0000:testfs-OST0001.
<indexterm><primary>maintenance</primary><secondary>finding nodes</secondary></indexterm> Finding Nodes in the Lustre File System - There may be situations in which you need to find all nodes in your Lustre file system or get the names of all OSTs. + There may be situations in which you need to find all nodes in + your Lustre file system or get the names of all OSTs. To get a list of all Lustre nodes, run this command on the MGS: - # cat /proc/fs/lustre/mgs/MGS/live/* + # lctl get_param mgs.MGS.live.* - This command must be run on the MGS. - + This command must be run on the MGS. - In this example, file system testfs has three nodes, - testfs-MDT0000, testfs-OST0000, and - testfs-OST0001. - cfs21:/tmp# cat /proc/fs/lustre/mgs/MGS/live/* + In this example, file system testfs has three + nodes, testfs-MDT0000, + testfs-OST0000, and + testfs-OST0001. + mgs:/root# lctl get_param mgs.MGS.live.* fsname: testfs flags: 0x0 gen: 26 testfs-MDT0000 testfs-OST0000 testfs-OST0001 To get the names of all OSTs, run this command on the MDS: - # cat /proc/fs/lustre/lov/fsname-mdtlov/target_obd + mds:/root# lctl get_param lov.*-mdtlov.target_obd - This command must be run on the MDS. - + This command must be run on the MDS. - In this example, there are two OSTs, testfs-OST0000 and testfs-OST0001, which are both - active. - cfs21:/tmp# cat /proc/fs/lustre/lov/testfs-mdtlov/target_obd + In this example, there are two OSTs, testfs-OST0000 and + testfs-OST0001, which are both active. + mgs:/root# lctl get_param lov.testfs-mdtlov.target_obd 0: testfs-OST0000_UUID ACTIVE 1: testfs-OST0001_UUID ACTIVE
@@ -659,7 +661,7 @@ Separate a combined MGS/MDT Start the MGS. mgs# mount -t lustre /dev/mgs_device /mgs_mount_point Check to make sure it knows about all your file system - cat /proc/fs/lustre/mgs/MGS/filesystems + mgs:/root# lctl get_param mgs.MGS.filesystems Remove the MGS option from the MDT, and set the new MGS nid. @@ -668,8 +670,8 @@ Separate a combined MGS/MDT Start the MDT. mds# mount -t lustre /dev/mdt_device /mdt_mount_point - Check to make sure the MGS configuration look right - mds# cat /proc/fs/lustre/mgs/MGS/live/filesystem_name + Check to make sure the MGS configuration looks right: + mgs# lctl get_param mgs.MGS.live.filesystem_name diff --git a/LustreOperations.xml b/LustreOperations.xml index cbe0f63..a70364f 100644 --- a/LustreOperations.xml +++ b/LustreOperations.xml @@ -732,7 +732,7 @@ client# mount -t lustre 192.168.10.1@tcp0:192.168.10.2@tcp0:/testfs \ /mnt/testfs mds0# umount /mnt/mdt mds1# mount -t lustre /dev/sda1 /mnt/test/mdt -mds1# cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status +mds1# lctl get_param mdt.testfs-MDT0000.recovery_status Where multiple NIDs are specified separated by commas (for example, 10.67.73.200@tcp,192.168.10.1@tcp), the two NIDs refer diff --git a/LustreProc.xml b/LustreProc.xml index 48dd499..511b2b3 100644 --- a/LustreProc.xml +++ b/LustreProc.xml @@ -1,10 +1,11 @@ - LustreProc - The /proc file system acts as an interface to internal data structures in - the kernel. This chapter describes entries in /proc that are useful for - tuning and monitoring aspects of a Lustre file system. It includes these sections: + Lustre Parameters + The /proc and /sys file systems + acts as an interface to internal data structures in the kernel. This chapter + describes parameters and tunables that are useful for optimizing and + monitoring aspects of a Lustre file system. It includes these sections: @@ -12,26 +13,26 @@
- Introduction to <literal>/proc</literal> - The /proc directory provides an interface to internal data structures - in the kernel that enables monitoring and tuning of many aspects of Lustre file system and - application performance These data structures include settings and metrics for components such - as memory, networking, file systems, and kernel housekeeping routines, which are available - throughout the hierarchical file layout in /proc. + Introduction to Lustre Parameters + Lustre parameters and statistics files provide an interface to + internal data structures in the kernel that enables monitoring and + tuning of many aspects of Lustre file system and application performance. + These data structures include settings and metrics for components such + as memory, networking, file systems, and kernel housekeeping routines, + which are available throughout the hierarchical file layout. - Typically, metrics are accessed by reading from /proc files and - settings are changed by writing to /proc files. Some data is server-only, - some data is client-only, and some data is exported from the client to the server and is thus - duplicated in both locations. + Typically, metrics are accessed via lctl get_param + files and settings are changed by via lctl set_param. + Some data is server-only, some data is client-only, and some data is + exported from the client to the server and is thus duplicated in both + locations. - In the examples in this chapter, # indicates a command is entered as - root. Servers are named according to the convention - fsname-MDT|OSTnumber. + In the examples in this chapter, # indicates + a command is entered as root. Lustre servers are named according to the + convention fsname-MDT|OSTnumber. The standard UNIX wildcard designation (*) is used. - In most cases, information is accessed using the lctl get_param command - and settings are changed using the lctl set_param command. Some examples - are shown below: + Some examples are shown below: To obtain data from a Lustre client: @@ -45,8 +46,8 @@ osc.testfs-OST0005-osc-ffff881071d5cc00 osc.testfs-OST0006-osc-ffff881071d5cc00 osc.testfs-OST0007-osc-ffff881071d5cc00 osc.testfs-OST0008-osc-ffff881071d5cc00 - In this example, information about OST connections available on a client is displayed - (indicated by "osc"). + In this example, information about OST connections available + on a client is displayed (indicated by "osc"). @@ -66,22 +67,32 @@ osc.testfs-OST0000-osc-ffff881071d5cc00.rpc_stats - To view a specific file, use lctl get_param - :# lctl get_param osc.lustre-OST0000-osc-ffff881071d5cc00.rpc_stats + To view a specific file, use lctl get_param: + # lctl get_param osc.lustre-OST0000*.rpc_stats For more information about using lctl, see . - Data can also be viewed using the cat command with the full path to the - file. The form of the cat command is similar to that of the lctl - get_param command with these differences. In the cat command: + Data can also be viewed using the cat command + with the full path to the file. The form of the cat + command is similar to that of the lctl get_param + command with some differences. Unfortunately, as the Linux kernel has + changed over the years, the location of statistics and parameter files + has also changed, which means that the Lustre parameter files may be + located in either the /proc directory, in the + /sys directory, and/or in the + /sys/kernel/debug directory, depending on the kernel + version and the Lustre version being used. The lctl + command insulates scripts from these changes and is preferred over direct + file access, unless as part of a high-performance monitoring system. + In the cat command: - Replace the dots in the path with slashes. + Replace the dots in the path with slashes. - Prepend the path with the following as - appropriate:/proc/{fs,sys}/{lustre,lnet} + Prepend the path with the following as appropriate: + /{proc,sys}/{fs,sys}/{lustre,lnet} For example, an lctl get_param command may look like @@ -89,23 +100,30 @@ osc.testfs-OST0000-osc-ffff881071d5cc00.rpc_stats osc.testfs-OST0000-osc-ffff881071d5cc00.uuid=594db456-0685-bd16-f59b-e72ee90e9819 osc.testfs-OST0001-osc-ffff881071d5cc00.uuid=594db456-0685-bd16-f59b-e72ee90e9819 ... - The equivalent cat command looks like - this:# cat /proc/fs/lustre/osc/*/uuid + The equivalent cat command may look like this: + # cat /proc/fs/lustre/osc/*/uuid 594db456-0685-bd16-f59b-e72ee90e9819 594db456-0685-bd16-f59b-e72ee90e9819 ... - The llstat utility can be used to monitor some Lustre file system I/O - activity over a specified time period. For more details, see - Some data is imported from attached clients and is available in a directory called - exports located in the corresponding per-service directory on a Lustre - server. For - example:# ls /proc/fs/lustre/obdfilter/testfs-OST0000/exports/192.168.124.9\@o2ib1/ + or like this: + # cat /sys/fs/lustre/osc/*/uuid +594db456-0685-bd16-f59b-e72ee90e9819 +594db456-0685-bd16-f59b-e72ee90e9819 +... + The llstat utility can be used to monitor some + Lustre file system I/O activity over a specified time period. For more + details, see + + Some data is imported from attached clients and is available in a + directory called exports located in the corresponding + per-service directory on a Lustre server. For example: + oss:/root# lctl list_param obdfilter.testfs-OST0000.exports.* # hash ldlm_stats stats uuid
Identifying Lustre File Systems and Servers - Several /proc files on the MGS list existing Lustre file systems and - file system servers. The examples below are for a Lustre file system called + Several /proc files on the MGS list existing + Lustre file systems and file system servers. The examples below are for + a Lustre file system called testfs with one MDT and three OSTs. @@ -2724,8 +2742,9 @@ ost_write 21 2 59 [bytes] 7648424 15019 332725.08 910694 180397.87 See also (llobdstat) and (collectl). - MDT stats files can be used to track MDT statistics for the MDS. The - example below shows sample output from an MDT stats file. + MDT stats files can be used to track MDT + statistics for the MDS. The example below shows sample output from an + MDT stats file. # lctl get_param mds.*-MDT0000.stats snapshot_time 1244832003.676892 secs.usecs open 2 samples [reqs] diff --git a/SystemConfigurationUtilities.xml b/SystemConfigurationUtilities.xml index f4a21bb..c4d65c2 100644 --- a/SystemConfigurationUtilities.xml +++ b/SystemConfigurationUtilities.xml @@ -1020,7 +1020,7 @@ llstat Files The llstat files are located at: /proc/fs/lustre/mdt/MDS/*/stats -/proc/fs/lustre/mds/*/exports/*/stats +/proc/fs/lustre/mdt/*/exports/*/stats /proc/fs/lustre/mdc/*/stats /proc/fs/lustre/ldlm/services/*/stats /proc/fs/lustre/ldlm/namespaces/*/pool/stats @@ -2594,7 +2594,9 @@ Application Profiling Utilities Per-client statistics tracked on the servers - Each MDT and OST now tracks LDLM and operations statistics for every connected client, for comparisons and simpler collection of distributed job statistics. + Each MDS and OSS now tracks LDLM and operations statistics for + every connected client, for comparisons and simpler collection of + distributed job statistics. /proc/fs/lustre/mds|obdfilter/*/exports/ @@ -2602,8 +2604,9 @@ Application Profiling Utilities Improved MDT statistics - More detailed MDT operations statistics are collected for better profiling. - /proc/fs/lustre/mds/*/stats + More detailed MDT operations statistics are collected for better + profiling. + /proc/fs/lustre/mdt/*/md_stats
-- 1.8.3.1