-<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustredebugging">
+<?xml version='1.0' encoding='UTF-8'?>
+<chapter xmlns="http://docbook.org/ns/docbook"
+ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
+ xml:id="lustredebugging">
<title xml:id="lustredebugging.title">Debugging a Lustre File System</title>
<para>This chapter describes tips and information to debug a Lustre file system, and includes the
following sections:</para>
<section xml:id="dbdoclet.50438274_15874">
<title><indexterm><primary>debugging</primary></indexterm>
Diagnostic and Debugging Tools</title>
- <para>A variety of diagnostic and analysis tools are available to debug issues with the Lustre software. Some of these are provided in Linux distributions, while others have been developed and are made available by the Lustre project.</para>
+ <para>A variety of diagnostic and analysis tools are available to debug
+ issues with the Lustre software. Some of these are provided in Linux
+ distributions, while others have been developed and are made available
+ by the Lustre project.</para>
<section remap="h3" xml:id="section_dms_q21_kk">
<title><indexterm>
<primary>debugging</primary>
<secondary>tools</secondary>
</indexterm> Lustre Debugging Tools</title>
- <para>The following in-kernel debug mechanisms are incorporated into the Lustre
- software:</para>
+ <para>The following in-kernel debug mechanisms are incorporated into
+ the Lustre software:</para>
<itemizedlist>
<listitem>
- <para xml:id="para_fkj_rld_hk"><emphasis role="bold">Debug logs</emphasis> - A circular
- debug buffer to which Lustre internal debug messages are written (in contrast to error
- messages, which are printed to the syslog or console). Entries to the Lustre debug log
- are controlled by the mask set by <literal>/proc/sys/lnet/debug</literal>. The log size
- defaults to 5 MB per CPU but can be increased as a busy system will quickly overwrite 5
- MB. When the buffer fills, the oldest information is discarded.</para>
+ <para xml:id="para_fkj_rld_hk"><emphasis role="bold">Debug logs</emphasis>
+ - A circular debug buffer to which Lustre internal debug messages
+ are written (in contrast to error messages, which are printed to the
+ syslog or console). Entries in the Lustre debug log are controlled
+ by a mask set by <literal>lctl set_param debug=<replaceable>mask</replaceable></literal>.
+ The log size defaults to 5 MB per CPU but can be increased as a
+ busy system will quickly overwrite 5 MB. When the buffer fills,
+ the oldest log records are discarded.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Debug daemon</emphasis> - The debug daemon controls logging of
- debug messages.</para>
+ <para><emphasis role="bold">
+ <literal>lctl get_param debug</literal>
+ </emphasis> - This shows the current debug mask used to delimit
+ the debugging information written out to the kernel debug logs.
+ </para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">
+ <literal>lctl debug_kernel <replaceable>file</replaceable></literal>
+ </emphasis> - Dump the Lustre kernel debug log to the specified
+ file as ASCII text for further debugging and analysis.
+ </para>
</listitem>
<listitem>
<para><emphasis role="bold">
- <literal>/proc/sys/lnet/debug</literal>
- </emphasis> - This file contains a mask that can be used to delimit the debugging
- information written out to the kernel debug logs.</para>
+ <literal>lctl set_param debug_mb=<replaceable>size</replaceable></literal>
+ </emphasis> - This sets the maximum size of the in-kernel Lustre
+ debug buffer, in units of MiB.
+ </para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Debug daemon</emphasis>
+ - The debug daemon controls the continuous logging of debug
+ messages to a log file in userspace.</para>
</listitem>
</itemizedlist>
<para>The following tools are also provided with the Lustre software:</para>
<listitem>
<para><emphasis role="bold">
<literal>lctl</literal>
- </emphasis> - This tool is used with the debug_kernel option to manually dump the Lustre
- debugging log or post-process debugging logs that are dumped automatically. For more
- information about the lctl tool, see <xref linkend="dbdoclet.50438274_62472"/> and <xref
+ </emphasis> - This tool is used with the debug_kernel option to
+ manually dump the Lustre debugging log or post-process debugging
+ logs that are dumped automatically. For more information about the
+ lctl tool, see <xref linkend="dbdoclet.50438274_62472"/> and <xref
linkend="dbdoclet.50438219_38274"/>.</para>
</listitem>
<listitem>
<para>
<literal>
<replaceable>lfs</replaceable>
- </literal> - This utility provides access to the extended attributes (EAs) of a Lustre
- file (along with other information). For more information about lfs, see <xref
- linkend="dbdoclet.50438206_94597"/>.</para>
+ </literal> - This utility provides access to the layout of
+ of a Lustre file, along with other information relevant to users.
+ For more information about lfs, see <xref
+ linkend="userutilities.lfs"/>.</para>
</listitem>
</itemizedlist>
</section>
<para>The following logging and data collection tools can be used to collect information for debugging Lustre kernel issues:</para>
<itemizedlist>
<listitem>
- <para><emphasis role="bold">
- <literal>kdump</literal>
- </emphasis> . A Linux kernel crash utility useful for debugging a system running Red Hat Enterprise Linux. For more information about <literal>kdump</literal>, see the Red Hat knowledge base article <link xl:href="http://kbase.redhat.com/faq/docs/DOC-6039">How do I configure kexec/kdump on Red Hat Enterprise Linux 5?</link>. To download <literal>kdump</literal>, go to the <link xl:href="http://fedoraproject.org/wiki/SystemConfig/kdump#Download">Fedora Project Download</link> site.</para>
+ <para><emphasis role="bold"><literal>kdump</literal></emphasis>.
+ A Linux kernel crash utility useful for debugging a system running
+ Red Hat Enterprise Linux. For more information about
+ <literal>kdump</literal>, see the Red Hat knowledge base article
+ <link xl:href="https://access.redhat.com/solutions/6038">How to
+ troubleshoot kernel crashes, hangs, or reboots with kdump on
+ Red Hat Enterprise Linux</link>. To download
+ <literal>kdump</literal>, install the RPM package via
+ <literal>yum install kexec-tools</literal>.
+ </para>
</listitem>
<listitem>
- <para><emphasis role="bold">
- <literal>netconsole</literal>
- </emphasis>. Enables kernel-level network logging over UDP. A system requires (SysRq) allows users to collect relevant data through <literal>netconsole</literal>.</para>
+ <para><emphasis role="bold"><literal>netconsole</literal></emphasis>.
+ Enables kernel-level network logging over UDP. A system requires
+ (SysRq) allows users to collect relevant data through
+ <literal>netconsole</literal>.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">
- <literal>netdump</literal>
- </emphasis>. A crash dump utility from Red Hat that allows memory images to be dumped
- over a network to a central server for analysis. The <literal>netdump</literal>
- utility was replaced by <literal>kdump</literal> in Red Hat Enterprise Linux 5. For
- more information about <literal>netdump</literal>, see <link
- xl:href="http://www.redhat.com/support/wpapers/redhat/netdump/">Red Hat, Inc.'s
- Network Console and Crash Dump Facility</link>.</para>
+ <para><emphasis><literal>wireshark</literal> </emphasis>. A network
+ packet inspection tool that allows debugging of information that was
+ sent between the various Lustre nodes. This tool is built on top of
+ <literal>tcpdump</literal> and can read packet dumps generated by
+ it. There are plug-ins available to dissassemble the LNet and
+ Lustre protocols. They are included with
+ <literal>wireshark</literal> since version 2.6.0. See also the
+ <link xl:href="https://www.wireshark.org/">Wireshark Website</link>
+ for more details.</para>
</listitem>
- <listitem>
- <para><emphasis><literal>wireshark</literal> </emphasis> . A network
- packet inspection tool that allows debugging of information that was
- sent between the various Lustre nodes. This tool is built on top of
- <literal>tcpdump</literal> and can read packet dumps generated by
- it. There are plug-ins available to dissassemble the LNET and
- Lustre protocols. They are located within the <link
- xl:href="http://git.hpdd.intel.com/">Lustre git repository</link>
- under <literal>lustre/contrib/wireshark/</literal>. Installation
- instruction are included in that directory. See also <link
- xl:href="http://www.wireshark.org/">Wireshark Website</link> for
- more details.</para>
- </listitem>
</itemizedlist>
</section>
<section remap="h4">
<para>A virtual machine is often used to create an isolated development and test environment. Some commonly-used virtual machines are:</para>
<itemizedlist>
<listitem>
- <para><emphasis role="bold">VirtualBox Open Source Edition</emphasis> . Provides enterprise-class virtualization capability for all major platforms and is available free at <link xl:href="http://www.sun.com/software/products/virtualbox/get.jsp?intcmp=2945">Get Sun VirtualBox</link>.</para>
+ <para>
+ <emphasis role="bold">VirtualBox Open Source Edition</emphasis>.
+ Provides enterprise-class virtualization capability for all major
+ platforms and is available free at
+ <link xl:href="https://www.virtualbox.org/wiki/Downloads">
+ https://www.virtualbox.org/wiki/Downloads</link>.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">VMware Server</emphasis> . Virtualization platform available as free introductory software at <link xl:href="http://downloads.vmware.com/d/info/datacenter_downloads/vmware_server/2_0">Download VMware Server</link>.</para>
+ <para><emphasis role="bold">VMware Server</emphasis>.
+ Virtualization platform available as free introductory software at
+ <link xl:href="https://my.vmware.com/web/vmware/downloads/">
+ https://my.vmware.com/web/vmware/downloads/</link>.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Xen</emphasis> . A para-virtualized environment with virtualization capabilities similar to VMware Server and Virtual Box. However, Xen allows the use of modified kernels to provide near-native performance and the ability to emulate shared storage. For more information, go to <link xl:href="http://xen.org/">xen.org</link>.</para>
+ <para><emphasis role="bold">Xen</emphasis>. A para-virtualized
+ environment with virtualization capabilities similar to VMware
+ Server and Virtual Box. However, Xen allows the use of modified
+ kernels to provide near-native performance and the ability to
+ emulate shared storage. For more information, go to
+ <link xl:href="https://xen.org/">https://xen.org/</link>.</para>
</listitem>
</itemizedlist>
<para>A variety of debuggers and analysis tools are available including:</para>
<listitem>
<para><emphasis role="bold">
<literal>kgdb</literal>
- </emphasis> . The Linux Kernel Source Level Debugger kgdb is used in conjunction with the GNU Debugger <literal>gdb</literal> for debugging the Linux kernel. For more information about using <literal>kgdb</literal> with <literal>gdb</literal>, see <link xl:href="http://www.linuxtopia.org/online_books/redhat_linux_debugging_with_gdb/running.html">Chapter 6. Running Programs Under gdb</link> in the <emphasis>Red Hat Linux 4 Debugging with GDB</emphasis> guide.</para>
+ </emphasis> . The Linux Kernel Source Level Debugger kgdb is
+ used in conjunction with the GNU Debugger <literal>gdb</literal>
+ for debugging the Linux kernel. For more information about using
+ <literal>kgdb</literal> with <literal>gdb</literal>, see
+ <link xl:href="https://www.linuxtopia.org/online_books/redhat_linux_debugging_with_gdb/running.html">
+ Chapter 6. Running Programs Under gdb</link> in the
+ <emphasis>Red Hat Linux 4 Debugging with GDB</emphasis> guide.
+ </para>
</listitem>
<listitem>
<para><emphasis role="bold">
</emphasis> . Used to analyze saved crash dump data when a system had panicked or locked up or appears unresponsive. For more information about using crash to analyze a crash dump, see:</para>
<itemizedlist>
<listitem>
- <para> Red Hat Magazine article: <link xl:href="http://magazine.redhat.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/">A quick overview of Linux kernel crash dump analysis</link></para>
- </listitem>
- <listitem>
- <para><link xl:href="http://people.redhat.com/anderson/crash_whitepaper/#EXAMPLES">Crash Usage: A Case Study</link> from the white paper <emphasis>Red Hat Crash Utility</emphasis> by David Anderson</para>
- </listitem>
- <listitem>
- <para> Kernel Trap forum entry: <link xl:href="http://kerneltrap.org/node/5758">Linux: Kernel Crash Dumps</link></para>
- </listitem>
- <listitem>
- <para> White paper: <link xl:href="http://www.google.com/url?sa=t&source=web&ct=res&cd=8&ved=0CCUQFjAH&url=http%3A%2F%2Fwww.kernel.sg%2Fpapers%2Fcrash-dump-analysis.pdf&rct=j&q=redhat+crash+dump&ei=6aQBS-ifK4T8tAPcjdiHCw&usg=AFQjCNEk03E3GDtAsawG3gfpwc1gGNELAg">A Quick Overview of Linux Kernel Crash Dump Analysis</link></para>
+ <para> Overview on how to use crash by the author:
+ <link xl:href="https://crash-utility.github.io/crash_whitepaper.html">
+ White Paper: Red Hat Crash Utility</link>
+ </para>
</listitem>
</itemizedlist>
</listitem>
<para> <emphasis role="bold">trace</emphasis></para>
</entry>
<entry>
- <para> Entry/Exit markers</para>
+ <para> Function entry/exit markers</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">dlmtrace</emphasis></para>
</entry>
<entry>
- <para> Locking-related information</para>
+ <para> Distributed locking-related information</para>
</entry>
</row>
<row>
</row>
<row>
<entry>
- <para> <emphasis role="bold">ext2</emphasis></para>
+ <para> <emphasis role="bold">malloc</emphasis></para>
</entry>
<entry>
- <para> Anything from the ext2_debug</para>
+ <para> Memory allocation or free information</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">malloc</emphasis></para>
+ <para> <emphasis role="bold">cache</emphasis></para>
</entry>
<entry>
- <para> Print malloc or free information</para>
+ <para> Cache-related information</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">cache</emphasis></para>
+ <para> <emphasis role="bold">info</emphasis></para>
</entry>
<entry>
- <para> Cache-related information</para>
+ <para> Non-critical general information</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">info</emphasis></para>
+ <para> <emphasis role="bold">dentry</emphasis></para>
+ </entry>
+ <entry>
+ <para> kernel namespace cache handling</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">mmap</emphasis></para>
</entry>
<entry>
- <para> General information</para>
+ <para> Memory-mapped IO interface</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">ioctl</emphasis></para>
+ <para> <emphasis role="bold">page</emphasis></para>
</entry>
<entry>
- <para> IOCTL-related information</para>
+ <para> Page cache and bulk data transfers</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">blocks</emphasis></para>
+ <para> <emphasis role="bold">info</emphasis></para>
</entry>
<entry>
- <para> Ext2 block allocation information</para>
+ <para> Miscellaneous informational messages</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">net</emphasis></para>
</entry>
<entry>
- <para> Networking</para>
+ <para> LNet network related debugging</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">console</emphasis></para>
+ </entry>
+ <entry>
+ <para>Significant system events, printed to console</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">warning</emphasis></para>
</entry>
<entry>
- <para>  </para>
+ <para>Significant but non-fatal exceptions, printed
+ to console</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">buffs</emphasis></para>
+ <para> <emphasis role="bold">error</emphasis></para>
</entry>
<entry>
- <para>  </para>
+ <para>Critical error messages, printed to console</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">other</emphasis></para>
+ <para> <emphasis role="bold">neterror</emphasis></para>
</entry>
<entry>
- <para>  </para>
+ <para>Significant LNet error messages</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">dentry</emphasis></para>
+ <para> <emphasis role="bold">emerg</emphasis></para>
</entry>
<entry>
- <para>  </para>
+ <para>Fatal system errors, printed to console</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">portals</emphasis></para>
+ <para><emphasis role="bold">config</emphasis></para>
</entry>
<entry>
- <para> Entry/Exit markers</para>
+ <para>Configuration and setup, enabled by default</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">page</emphasis></para>
+ <para><emphasis role="bold">ha</emphasis></para>
</entry>
<entry>
- <para> Bulk page handling</para>
+ <para>Failover and recovery-related information,
+ enabled by default</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">error</emphasis></para>
+ <para><emphasis role="bold">hsm</emphasis></para>
</entry>
<entry>
- <para> Error messages</para>
+ <para>Hierarchical space management/tiering</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">emerg</emphasis></para>
+ <para><emphasis role="bold">ioctl</emphasis></para>
</entry>
<entry>
- <para>  </para>
+ <para>IOCTL-related information, enabled by default</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">layout</emphasis></para>
+ </entry>
+ <entry>
+ <para>File layout handling (PFL, FLR, DoM)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">lfsck</emphasis></para>
+ </entry>
+ <entry>
+ <para>Filesystem consistency checking, enabled by
+ default</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">other</emphasis></para>
+ </entry>
+ <entry>
+ <para>Miscellaneious other debug messages</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">quota</emphasis></para>
+ </entry>
+ <entry>
+ <para>Space accounting and management</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">reada</emphasis></para>
+ </entry>
+ <entry>
+ <para>Client readahead management</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">rpctrace</emphasis></para>
</entry>
<entry>
- <para> For distributed debugging</para>
+ <para>Remote request/reply tracing and debugging</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">sec</emphasis></para>
+ </entry>
+ <entry>
+ <para>Security, Kerberos, Shared Secret Key handling</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">snapshot</emphasis></para>
+ </entry>
+ <entry>
+ <para>Filesystem snapshot management</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">ha</emphasis></para>
+ <para><emphasis role="bold">vfstrace</emphasis></para>
</entry>
<entry>
- <para> Failover and recovery-related information</para>
+ <para>Kernel VFS interface operations</para>
</entry>
</row>
</tbody>
</emphasis></para>
</entry>
<entry>
- <para> Behaves similarly to <literal>CERROR()</literal>, but prints error messages for LNET if <literal>D_NETERR</literal> is set in the <literal>debug</literal> mask. This is appropriate for serious networking errors. Messages printed to the console are rate-limited.</para>
+ <para> Behaves similarly to <literal>CERROR()</literal>, but prints error messages for LNet if <literal>D_NETERR</literal> is set in the <literal>debug</literal> mask. This is appropriate for serious networking errors. Messages printed to the console are rate-limited.</para>
</entry>
</row>
<row>
<section remap="h3">
<title>Accessing the <literal>ptlrpc</literal> Request History</title>
<para>Each service maintains a request history, which can be useful for first occurrence troubleshooting.</para>
- <para><literal>ptlrpc</literal> is an RPC protocol layered on LNET that deals with stateful servers and has semantics and built-in support for recovery.</para>
+ <para><literal>ptlrpc</literal> is an RPC protocol layered on LNet that deals with stateful servers and has semantics and built-in support for recovery.</para>
<para>The ptlrpc request history works as follows:</para>
<orderedlist>
<listitem>
<para>Buffers are culled from the service request buffer history if it has grown above <literal>req_buffer_history_max</literal> and its reqs are removed from the service request history.</para>
</listitem>
</orderedlist>
- <para>Request history is accessed and controlled using the following /proc files under the service directory:</para>
+ <para>Request history is accessed and controlled using the following parameters for each service:</para>
<itemizedlist>
<listitem>
<para><literal>req_buffer_history_len </literal></para>
</section>
</section>
</chapter>
+<!--
+ vim:expandtab:shiftwidth=2:tabstop=8:
+ -->
git://git.whamcloud.com / doc / manual.git / blobdiff