X-Git-Url: https://git.whamcloud.com/?a=blobdiff_plain;f=LustreDebugging.xml;h=089dd54063d45bbff5fe89c9a9d5347203b39c96;hb=4354e542d77e8ee5e08fc43106f0bd09eb7f4464;hp=c9ec6065c30fdbc6fab5bb933b85b497141de815;hpb=2d711c221201e1bcc90fb9f66133f5f972ce3ba2;p=doc%2Fmanual.git diff --git a/LustreDebugging.xml b/LustreDebugging.xml index c9ec606..089dd54 100644 --- a/LustreDebugging.xml +++ b/LustreDebugging.xml @@ -1,7 +1,10 @@ - - Lustre Debugging - This chapter describes tips and information to debug Lustre, and includes the following sections: + + Debugging a Lustre File System + This chapter describes tips and information to debug a Lustre file system, and includes the + following sections: @@ -16,22 +19,53 @@
<indexterm><primary>debugging</primary></indexterm> Diagnostic and Debugging Tools - A variety of diagnostic and analysis tools are available to debug issues with the Lustre software. Some of these are provided in Linux distributions, while others have been developed and are made available by the Lustre project. -
- <indexterm><primary>debugging</primary><secondary>tools</secondary></indexterm> -Lustre Debugging Tools - The following in-kernel debug mechanisms are incorporated into the Lustre software: + A variety of diagnostic and analysis tools are available to debug + issues with the Lustre software. Some of these are provided in Linux + distributions, while others have been developed and are made available + by the Lustre project. +
+ <indexterm> + <primary>debugging</primary> + <secondary>tools</secondary> + </indexterm> Lustre Debugging Tools + The following in-kernel debug mechanisms are incorporated into + the Lustre software: - Debug logs - A circular debug buffer to which Lustre internal debug messages are written (in contrast to error messages, which are printed to the syslog or console). Entries to the Lustre debug log are controlled by the mask set by /proc/sys/lnet/debug. The log size defaults to 5 MB per CPU but can be increased as a busy system will quickly overwrite 5 MB. When the buffer fills, the oldest information is discarded. + Debug logs + - A circular debug buffer to which Lustre internal debug messages + are written (in contrast to error messages, which are printed to the + syslog or console). Entries in the Lustre debug log are controlled + by a mask set by lctl set_param debug=mask. + The log size defaults to 5 MB per CPU but can be increased as a + busy system will quickly overwrite 5 MB. When the buffer fills, + the oldest log records are discarded. + + + + lctl get_param debug + - This shows the current debug mask used to delimit + the debugging information written out to the kernel debug logs. + - Debug daemon - The debug daemon controls logging of debug messages. + + lctl debug_kernel file + - Dump the Lustre kernel debug log to the specified + file as ASCII text for further debugging and analysis. + - /proc/sys/lnet/debug - - This file contains a mask that can be used to delimit the debugging information written out to the kernel debug logs. + lctl set_param debug_mb=size + - This sets the maximum size of the in-kernel Lustre + debug buffer, in units of MiB. + + + + Debug daemon + - The debug daemon controls the continuous logging of debug + messages to a log file in userspace. The following tools are also provided with the Lustre software: @@ -39,17 +73,27 @@ Lustre Debugging Tools lctl - - This tool is used with the debug_kernel option to manually dump the Lustre debugging log or post-process debugging logs that are dumped automatically. For more information about the lctl tool, see and . + - This tool is used with the debug_kernel option to + manually dump the Lustre debugging log or post-process debugging + logs that are dumped automatically. For more information about the + lctl tool, see and . - Lustre subsystem asserts - A panic-style assertion (LBUG) in the kernel causes Lustre to dump the debug log to the file /tmp/lustre-log.timestamp where it can be retrieved after a reboot. For more information, see . + Lustre subsystem asserts - A panic-style assertion (LBUG) in the kernel causes the + Lustre file system to dump the debug log to the file + /tmp/lustre-log.timestamp where it can + be retrieved after a reboot. For more information, see . - + lfs - - - This utility provides access to the extended attributes (EAs) of a Lustre file (along with other information). For more information about lfs, see . + - This utility provides access to the layout of + of a Lustre file, along with other information relevant to users. + For more information about lfs, see .
@@ -58,7 +102,8 @@ Lustre Debugging Tools The tools described in this section are provided in the Linux kernel or are available at an external website. For information about using some of these tools for Lustre debugging, see and .
<indexterm><primary>debugging</primary><secondary>admin tools</secondary></indexterm>Tools for Administrators and Developers - Some general debugging tools provided as a part of the standard Linux distro are: + Some general debugging tools provided as a part of the standard Linux distribution + are: @@ -71,7 +116,9 @@ Lustre Debugging Tools . syslogd prints fatal or serious messages at this log. - Crash dumps . On crash-dump enabled kernels, sysrq c produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64 KB of the log) to the console. + Crash dumps . On crash-dump enabled kernels, + sysrq c produces a crash dump. The Lustre software enhances this crash dump with a log + dump (the last 64 KB of the log) to the console. @@ -82,56 +129,72 @@ Lustre Debugging Tools The following logging and data collection tools can be used to collect information for debugging Lustre kernel issues: - - kdump - . A Linux kernel crash utility useful for debugging a system running Red Hat Enterprise Linux. For more information about kdump, see the Red Hat knowledge base article How do I configure kexec/kdump on Red Hat Enterprise Linux 5?. To download kdump, go to the Fedora Project Download site. + kdump. + A Linux kernel crash utility useful for debugging a system running + Red Hat Enterprise Linux. For more information about + kdump, see the Red Hat knowledge base article + How to + troubleshoot kernel crashes, hangs, or reboots with kdump on + Red Hat Enterprise Linux. To download + kdump, install the RPM package via + yum install kexec-tools. + - - netconsole - . Enables kernel-level network logging over UDP. A system requires (SysRq) allows users to collect relevant data through netconsole. + netconsole. + Enables kernel-level network logging over UDP. A system requires + (SysRq) allows users to collect relevant data through + netconsole. - - netdump - . A crash dump utility from Red Hat that allows memory images to be dumped over a network to a central server for analysis. The netdump utility was replaced by kdump in RHEL 5. For more information about netdump, see Red Hat, Inc.'s Network Console and Crash Dump Facility. + wireshark . A network + packet inspection tool that allows debugging of information that was + sent between the various Lustre nodes. This tool is built on top of + tcpdump and can read packet dumps generated by + it. There are plug-ins available to dissassemble the LNet and + Lustre protocols. They are included with + wireshark since version 2.6.0. See also the + Wireshark Website + for more details. - - wireshark . A network - packet inspection tool that allows debugging of information that was - sent between the various Lustre nodes. This tool is built on top of - tcpdump and can read packet dumps generated by - it. There are plug-ins available to dissassemble the LNET and - Lustre protocols. They are located within the Lustre git repository - under lustre/contrib/wireshark/. Installation - instruction are included in that directory. See also Wireshark Website for - more details. -
<indexterm><primary>debugging</primary><secondary>developer tools</secondary></indexterm>Tools for Developers - The tools described in this section may be useful for debugging Lustre in a development environment. + The tools described in this section may be useful for debugging a Lustre file system + in a development environment. Of general interest is: leak_finder.pl - . This program provided with Lustre is useful for finding memory leaks in the code. + . This program provided with the Lustre software is useful for finding + memory leaks in the code. A virtual machine is often used to create an isolated development and test environment. Some commonly-used virtual machines are: - VirtualBox Open Source Edition . Provides enterprise-class virtualization capability for all major platforms and is available free at Get Sun VirtualBox. + + VirtualBox Open Source Edition. + Provides enterprise-class virtualization capability for all major + platforms and is available free at + + https://www.virtualbox.org/wiki/Downloads. - VMware Server . Virtualization platform available as free introductory software at Download VMware Server. + VMware Server. + Virtualization platform available as free introductory software at + + https://my.vmware.com/web/vmware/downloads/. - Xen . A para-virtualized environment with virtualization capabilities similar to VMware Server and Virtual Box. However, Xen allows the use of modified kernels to provide near-native performance and the ability to emulate shared storage. For more information, go to xen.org. + Xen. A para-virtualized + environment with virtualization capabilities similar to VMware + Server and Virtual Box. However, Xen allows the use of modified + kernels to provide near-native performance and the ability to + emulate shared storage. For more information, go to + https://xen.org/. A variety of debuggers and analysis tools are available including: @@ -139,7 +202,14 @@ Lustre Debugging Tools kgdb - . The Linux Kernel Source Level Debugger kgdb is used in conjunction with the GNU Debugger gdb for debugging the Linux kernel. For more information about using kgdb with gdb, see Chapter 6. Running Programs Under gdb in the Red Hat Linux 4 Debugging with GDB guide. + . The Linux Kernel Source Level Debugger kgdb is + used in conjunction with the GNU Debugger gdb + for debugging the Linux kernel. For more information about using + kgdb with gdb, see + + Chapter 6. Running Programs Under gdb in the + Red Hat Linux 4 Debugging with GDB guide. + @@ -147,16 +217,10 @@ Lustre Debugging Tools . Used to analyze saved crash dump data when a system had panicked or locked up or appears unresponsive. For more information about using crash to analyze a crash dump, see: - Red Hat Magazine article: A quick overview of Linux kernel crash dump analysis - - - Crash Usage: A Case Study from the white paper Red Hat Crash Utility by David Anderson - - - Kernel Trap forum entry: Linux: Kernel Crash Dumps - - - White paper: A Quick Overview of Linux Kernel Crash Dump Analysis + Overview on how to use crash by the author: + + White Paper: Red Hat Crash Utility + @@ -171,12 +235,16 @@ Lustre Debugging Tools <indexterm><primary>debugging</primary><secondary>message format</secondary></indexterm>Understanding the Lustre Debug Messaging Format Lustre debug messages are categorized by originating subsystem, message type, and location in the source code. For a list of subsystems and message types, see . - For a current list of subsystems and debug message types, see libcfs/include/libcfs/libcfs_debug.h in the Lustre tree + For a current list of subsystems and debug message types, see + libcfs/include/libcfs/libcfs_debug.h in the Lustre software + tree The elements of a Lustre debug message are described in Format of Lustre Debug Messages.
Lustre Debug Messages - Each Lustre debug message has the tag of the subsystem it originated in, the message type, and the location in the source code. The subsystems and debug types used in Lustre are as follows: + Each Lustre debug message has the tag of the subsystem it originated in, the message + type, and the location in the source code. The subsystems and debug types used are as + follows: Standard Subsystems: @@ -208,7 +276,7 @@ Lustre Debugging Tools trace - Entry/Exit markers + Function entry/exit markers @@ -216,7 +284,7 @@ Lustre Debugging Tools dlmtrace - Locking-related information + Distributed locking-related information @@ -237,50 +305,58 @@ Lustre Debugging Tools - ext2 + malloc - Anything from the ext2_debug + Memory allocation or free information - malloc + cache - Print malloc or free information + Cache-related information - cache + info - Cache-related information + Non-critical general information - info + dentry + + + kernel namespace cache handling + + + + + mmap - General information + Memory-mapped IO interface - ioctl + page - IOCTL-related information + Page cache and bulk data transfers - blocks + info - Ext2 block allocation information + Miscellaneous informational messages @@ -288,7 +364,15 @@ Lustre Debugging Tools net - Networking + LNet network related debugging + + + + + console + + + Significant system events, printed to console @@ -296,63 +380,106 @@ Lustre Debugging Tools warning -   + Significant but non-fatal exceptions, printed + to console - buffs + error -   + Critical error messages, printed to console - other + neterror -   + Significant LNet error messages - dentry + emerg -   + Fatal system errors, printed to console - portals + config - Entry/Exit markers + Configuration and setup, enabled by default - page + ha - Bulk page handling + Failover and recovery-related information, + enabled by default - error + hsm - Error messages + Hierarchical space management/tiering - emerg + ioctl -   + IOCTL-related information, enabled by default + + + + + layout + + + File layout handling (PFL, FLR, DoM) + + + + + lfsck + + + Filesystem consistency checking, enabled by + default + + + + + other + + + Miscellaneious other debug messages + + + + + quota + + + Space accounting and management + + + + + reada + + + Client readahead management @@ -360,15 +487,31 @@ Lustre Debugging Tools rpctrace - For distributed debugging + Remote request/reply tracing and debugging - ha + sec - Failover and recovery-related information + Security, Kerberos, Shared Secret Key handling + + + + + snapshot + + + Filesystem snapshot management + + + + + vfstrace + + + Kernel VFS interface operations @@ -380,7 +523,11 @@ Lustre Debugging Tools
Format of Lustre Debug Messages - Lustre uses the CDEBUG() and CERROR() macros to print the debug or error messages. To print the message, the CDEBUG() macro uses the function libcfs_debug_msg() (libcfs/libcfs/tracefile.c). The message format is described below, along with an example. + The Lustre software uses the CDEBUG() and + CERROR() macros to print the debug or error messages. To print the + message, the CDEBUG() macro uses the function + libcfs_debug_msg() (libcfs/libcfs/tracefile.c). + The message format is described below, along with an example. @@ -574,11 +721,13 @@ Debug log: 324 lines, 258 kept, 66 dropped.
<indexterm><primary>debugging</primary><secondary>kernel debug log</secondary></indexterm>Controlling Information Written to the Kernel Debug Log - The lctl set_param subsystem_debug=subsystem_mask and lctl set_param debug=debug_mask are used to determine which information is written to the debug log. The subsystem_debug mask determines the information written to the log based on the functional area of the code (such as lnet, osc, or ldlm). The debug mask controls information based on the message type (such as info, error, trace, or malloc). + The lctl set_param subsystem_debug=subsystem_mask and lctl set_param debug=debug_mask are used to determine which information is written to the debug log. The subsystem_debug mask determines the information written to the log based on the functional area of the code (such as lnet, osc, or ldlm). The debug mask controls information based on the message type (such as info, error, trace, or malloc). For a complete list of possible debug masks use the lctl debug_list types command. To turn off Lustre debugging completely: lctl set_param debug=0 To turn on full Lustre debugging: lctl set_param debug=-1 + To list all possible debug masks: + lctl debug_list types To log only messages related to network communications: lctl set_param debug=net To turn on logging of messages related to network communications and existing debug flags: @@ -600,18 +749,37 @@ Debug log: 324 lines, 258 kept, 66 dropped.
<indexterm><primary>debugging</primary><secondary>disk contents</secondary></indexterm>Looking at Disk Content - In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The lfs tool can be used to obtain this information for a given file using the getstripe subcommand. Use a corresponding lfs setstripe command to specify striping attributes for a new file or directory. - The lfs getstripe command takes a Lustre filename as input and lists all the objects that form a part of this file. To obtain this information for the file /mnt/lustre/frog in Lustre file system, run: - $ lfs getstripe /mnt/lustre/frog + In a Lustre file system, the inodes on the metadata server contain extended attributes + (EAs) that store information about file striping. EAs contain a list of all object IDs and + their locations (that is, the OST that stores them). The lfs tool can be + used to obtain this information for a given file using the getstripe + subcommand. Use a corresponding lfs setstripe command to specify striping + attributes for a new file or directory. + The lfs getstripe command takes a Lustre filename as input and lists + all the objects that form a part of this file. To obtain this information for the file + /mnt/testfs/frog in a Lustre file system, run: + $ lfs getstripe /mnt/testfs/frog lmm_stripe_count: 2 lmm_stripe_size: 1048576 +lmm_pattern: 1 +lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 818855 0xc7ea7 0 0 873123 0xd52a3 0 - The debugfs tool is provided in the e2fsprogs package. It can be used for interactive debugging of an ldiskfs file system. The debugfs tool can either be used to check status or modify information in the file system. In Lustre, all objects that belong to a file are stored in an underlying ldiskfs file system on the OSTs. The file system uses the object IDs as the file names. Once the object IDs are known, use the debugfs tool to obtain the attributes of all objects from different OSTs. - A sample run for the /mnt/lustre/frog file used in the above example is shown here: + The debugfs tool is provided in the + e2fsprogs package. It can be used for interactive + debugging of an ldiskfs file system. The + debugfs tool can either be used to check status or + modify information in the file system. In a Lustre file system, all + objects that belong to a file are stored in an underlying + ldiskfs file system on the OSTs. The file system + uses the object IDs as the file names. Once the object IDs are known, + use the debugfs tool to obtain the attributes of + all objects from different OSTs. + A sample run for the /mnt/testfs/frog file used + in the above example is shown here: $ debugfs -c -R "stat O/0/d$((818855 % 32))/818855" /dev/vgmyth/lvmythost2 debugfs 1.41.90.wc3 (28-May-2011) @@ -642,7 +810,8 @@ EXTENTS: strings /tmp/last_rcvd | head -1 myth-OST0004_UUID - It is also possible (and easier) to extract this from the filesystem label using the dumpe2fs command: + It is also possible (and easier) to extract this from the file system label using the + dumpe2fs command: dumpe2fs -h /dev/sdc | grep volume dumpe2fs 1.41.90.wc3 (28-May-2011) Filesystem volume name: myth-OST0004 @@ -658,15 +827,24 @@ Filesystem volume name: myth-OST0004
Tracing Lock Traffic - Lustre has a specific debug type category for tracing lock traffic. Use: + The Lustre software provides a specific debug type category for tracing lock traffic. + Use: lctl> filter all_types lctl> show dlmtrace lctl> debug_kernel [filename]
+
+ Controlling Console Message Rate Limiting + Some console messages which are printed by Lustre are rate limited. When such messages are printed, they may be followed by a message saying "Skipped N previous similar message(s)," where N is the number of messages skipped. This rate limiting can be completely disabled by a libcfs module parameter called libcfs_console_ratelimit. To disable console message rate limiting, add this line to /etc/modprobe.d/lustre.conf and then reload Lustre modules. + options libcfs libcfs_console_ratelimit=0 + It is also possible to set the minimum and maximum delays between rate-limited console messages using the module parameters libcfs_console_max_delay and libcfs_console_min_delay. Set these in /etc/modprobe.d/lustre.conf and then reload Lustre modules. Additional information on libcfs module parameters is available via modinfo: + modinfo libcfs +
<indexterm><primary>debugging</primary><secondary>developers tools</secondary></indexterm>Lustre Debugging for Developers - The procedures in this section may be useful to developers debugging Lustre code. + The procedures in this section may be useful to developers debugging Lustre source + code.
Adding Debugging to the Lustre Source Code The debugging infrastructure provides a number of macros that can be used in Lustre source code to aid in debugging or reporting serious errors. @@ -695,7 +873,11 @@ lctl> debug_kernel [filename] - A panic-style assertion in the kernel which causes Lustre to dump its circular log to the /tmp/lustre-log file. This file can be retrieved after a reboot. LBUG() freezes the thread to allow capture of the panic stack. A system reboot is needed to clear the thread. + A panic-style assertion in the kernel which causes the Lustre file system to + dump its circular log to the /tmp/lustre-log file. This file + can be retrieved after a reboot. LBUG() freezes the thread to + allow capture of the panic stack. A system reboot is needed to clear the + thread. @@ -768,7 +950,7 @@ lctl> debug_kernel [filename] - Behaves similarly to CERROR(), but prints error messages for LNET if D_NETERR is set in the debug mask. This is appropriate for serious networking errors. Messages printed to the console are rate-limited. + Behaves similarly to CERROR(), but prints error messages for LNet if D_NETERR is set in the debug mask. This is appropriate for serious networking errors. Messages printed to the console are rate-limited. @@ -842,7 +1024,12 @@ lctl> debug_kernel [filename] - Allows insertion of failure points into the Lustre code. This is useful to generate regression tests that can hit a very specific sequence of events. This works in conjunction with "lctl set_param fail_loc=fail_loc" to set a specific failure point for which a given OBD_FAIL_CHECK() will test. + Allows insertion of failure points into the Lustre source code. This is useful + to generate regression tests that can hit a very specific sequence of events. This + works in conjunction with "lctl set_param + fail_loc=fail_loc" to set a specific + failure point for which a given OBD_FAIL_CHECK() will + test. @@ -912,7 +1099,7 @@ lctl> debug_kernel [filename]
Accessing the <literal>ptlrpc</literal> Request History Each service maintains a request history, which can be useful for first occurrence troubleshooting. - ptlrpc is an RPC protocol layered on LNET that deals with stateful servers and has semantics and built-in support for recovery. + ptlrpc is an RPC protocol layered on LNet that deals with stateful servers and has semantics and built-in support for recovery. The ptlrpc request history works as follows: @@ -925,7 +1112,7 @@ lctl> debug_kernel [filename] Buffers are culled from the service request buffer history if it has grown above req_buffer_history_max and its reqs are removed from the service request history. - Request history is accessed and controlled using the following /proc files under the service directory: + Request history is accessed and controlled using the following parameters for each service: req_buffer_history_len @@ -1073,3 +1260,6 @@ freed 8bytes at a3116744 (called pathcopy)
+