following sections:</para>
<itemizedlist>
<listitem>
- <para><xref linkend="dbdoclet.50438274_15874"/></para>
+ <para><xref linkend="dbdoclet.debugging_tools"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438274_23607"/></para>
+ <para><xref linkend="dbdoclet.debugging_procedures"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438274_80443"/></para>
+ <para><xref linkend="dbdoclet.debugging_for_devs"/></para>
</listitem>
</itemizedlist>
- <section xml:id="dbdoclet.50438274_15874">
+ <section xml:id="dbdoclet.debugging_tools">
<title><indexterm><primary>debugging</primary></indexterm>
Diagnostic and Debugging Tools</title>
<para>A variety of diagnostic and analysis tools are available to debug
</emphasis> - This tool is used with the debug_kernel option to
manually dump the Lustre debugging log or post-process debugging
logs that are dumped automatically. For more information about the
- lctl tool, see <xref linkend="dbdoclet.50438274_62472"/> and <xref
- linkend="dbdoclet.50438219_38274"/>.</para>
+ lctl tool, see <xref linkend="dbdoclet.using_lctl_tool"/> and <xref
+ linkend="dbdoclet.lctl"/>.</para>
</listitem>
<listitem>
<para>Lustre subsystem asserts - A panic-style assertion (LBUG) in the kernel causes the
Lustre file system to dump the debug log to the file
<literal>/tmp/lustre-log.<replaceable>timestamp</replaceable></literal> where it can
be retrieved after a reboot. For more information, see <xref
- linkend="dbdoclet.50438198_40669"/>.</para>
+ linkend="dbdoclet.troubleshooting"/>.</para>
</listitem>
<listitem>
<para>
<literal>
<replaceable>lfs</replaceable>
- </literal> - This utility provides access to the extended attributes (EAs) of a Lustre
- file (along with other information). For more information about lfs, see <xref
- linkend="dbdoclet.50438206_94597"/>.</para>
+ </literal> - This utility provides access to the layout of
+ of a Lustre file, along with other information relevant to users.
+ For more information about lfs, see <xref
+ linkend="userutilities.lfs"/>.</para>
</listitem>
</itemizedlist>
</section>
<section remap="h3">
<title><indexterm><primary>debugging</primary><secondary>external tools</secondary></indexterm>External Debugging Tools</title>
- <para>The tools described in this section are provided in the Linux kernel or are available at an external website. For information about using some of these tools for Lustre debugging, see <xref linkend="dbdoclet.50438274_23607"/> and <xref linkend="dbdoclet.50438274_80443"/>.</para>
+ <para>The tools described in this section are provided in the Linux
+ kernel or are available at an external website. For information about
+ using some of these tools for Lustre debugging, see
+ <xref linkend="dbdoclet.debugging_procedures"/> and
+ <xref linkend="dbdoclet.debugging_for_devs"/>.
+ </para>
<section remap="h4">
<title><indexterm><primary>debugging</primary><secondary>admin tools</secondary></indexterm>Tools for Administrators and Developers</title>
<para>Some general debugging tools provided as a part of the standard Linux distribution
<para>The following logging and data collection tools can be used to collect information for debugging Lustre kernel issues:</para>
<itemizedlist>
<listitem>
- <para><emphasis role="bold">
- <literal>kdump</literal>
- </emphasis> . A Linux kernel crash utility useful for debugging a system running Red Hat Enterprise Linux. For more information about <literal>kdump</literal>, see the Red Hat knowledge base article <link xl:href="http://kbase.redhat.com/faq/docs/DOC-6039">How do I configure kexec/kdump on Red Hat Enterprise Linux 5?</link>. To download <literal>kdump</literal>, go to the <link xl:href="http://fedoraproject.org/wiki/SystemConfig/kdump#Download">Fedora Project Download</link> site.</para>
+ <para><emphasis role="bold"><literal>kdump</literal></emphasis>.
+ A Linux kernel crash utility useful for debugging a system running
+ Red Hat Enterprise Linux. For more information about
+ <literal>kdump</literal>, see the Red Hat knowledge base article
+ <link xl:href="https://access.redhat.com/solutions/6038">How to
+ troubleshoot kernel crashes, hangs, or reboots with kdump on
+ Red Hat Enterprise Linux</link>. To download
+ <literal>kdump</literal>, install the RPM package via
+ <literal>yum install kexec-tools</literal>.
+ </para>
</listitem>
<listitem>
- <para><emphasis role="bold">
- <literal>netconsole</literal>
- </emphasis>. Enables kernel-level network logging over UDP. A system requires (SysRq) allows users to collect relevant data through <literal>netconsole</literal>.</para>
+ <para><emphasis role="bold"><literal>netconsole</literal></emphasis>.
+ Enables kernel-level network logging over UDP. A system requires
+ (SysRq) allows users to collect relevant data through
+ <literal>netconsole</literal>.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">
- <literal>netdump</literal>
- </emphasis>. A crash dump utility from Red Hat that allows memory images to be dumped
- over a network to a central server for analysis. The <literal>netdump</literal>
- utility was replaced by <literal>kdump</literal> in Red Hat Enterprise Linux 5. For
- more information about <literal>netdump</literal>, see <link
- xl:href="http://www.redhat.com/support/wpapers/redhat/netdump/">Red Hat, Inc.'s
- Network Console and Crash Dump Facility</link>.</para>
+ <para><emphasis><literal>wireshark</literal> </emphasis>. A network
+ packet inspection tool that allows debugging of information that was
+ sent between the various Lustre nodes. This tool is built on top of
+ <literal>tcpdump</literal> and can read packet dumps generated by
+ it. There are plug-ins available to dissassemble the LNet and
+ Lustre protocols. They are included with
+ <literal>wireshark</literal> since version 2.6.0. See also the
+ <link xl:href="https://www.wireshark.org/">Wireshark Website</link>
+ for more details.</para>
</listitem>
- <listitem>
- <para><emphasis><literal>wireshark</literal> </emphasis> . A network
- packet inspection tool that allows debugging of information that was
- sent between the various Lustre nodes. This tool is built on top of
- <literal>tcpdump</literal> and can read packet dumps generated by
- it. There are plug-ins available to dissassemble the LNet and
- Lustre protocols. They are located within the <link
- xl:href="http://git.whamcloud.com/">Lustre git repository</link>
- under <literal>lustre/contrib/wireshark/</literal>. Installation
- instruction are included in that directory. See also <link
- xl:href="http://www.wireshark.org/">Wireshark Website</link> for
- more details.</para>
- </listitem>
</itemizedlist>
</section>
<section remap="h4">
<para>A virtual machine is often used to create an isolated development and test environment. Some commonly-used virtual machines are:</para>
<itemizedlist>
<listitem>
- <para><emphasis role="bold">VirtualBox Open Source Edition</emphasis> . Provides enterprise-class virtualization capability for all major platforms and is available free at <link xl:href="http://www.sun.com/software/products/virtualbox/get.jsp?intcmp=2945">Get Sun VirtualBox</link>.</para>
+ <para>
+ <emphasis role="bold">VirtualBox Open Source Edition</emphasis>.
+ Provides enterprise-class virtualization capability for all major
+ platforms and is available free at
+ <link xl:href="https://www.virtualbox.org/wiki/Downloads">
+ https://www.virtualbox.org/wiki/Downloads</link>.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">VMware Server</emphasis> . Virtualization platform available as free introductory software at <link xl:href="http://downloads.vmware.com/d/info/datacenter_downloads/vmware_server/2_0">Download VMware Server</link>.</para>
+ <para><emphasis role="bold">VMware Server</emphasis>.
+ Virtualization platform available as free introductory software at
+ <link xl:href="https://my.vmware.com/web/vmware/downloads/">
+ https://my.vmware.com/web/vmware/downloads/</link>.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Xen</emphasis> . A para-virtualized environment with virtualization capabilities similar to VMware Server and Virtual Box. However, Xen allows the use of modified kernels to provide near-native performance and the ability to emulate shared storage. For more information, go to <link xl:href="http://xen.org/">xen.org</link>.</para>
+ <para><emphasis role="bold">Xen</emphasis>. A para-virtualized
+ environment with virtualization capabilities similar to VMware
+ Server and Virtual Box. However, Xen allows the use of modified
+ kernels to provide near-native performance and the ability to
+ emulate shared storage. For more information, go to
+ <link xl:href="https://xen.org/">https://xen.org/</link>.</para>
</listitem>
</itemizedlist>
<para>A variety of debuggers and analysis tools are available including:</para>
<listitem>
<para><emphasis role="bold">
<literal>kgdb</literal>
- </emphasis> . The Linux Kernel Source Level Debugger kgdb is used in conjunction with the GNU Debugger <literal>gdb</literal> for debugging the Linux kernel. For more information about using <literal>kgdb</literal> with <literal>gdb</literal>, see <link xl:href="http://www.linuxtopia.org/online_books/redhat_linux_debugging_with_gdb/running.html">Chapter 6. Running Programs Under gdb</link> in the <emphasis>Red Hat Linux 4 Debugging with GDB</emphasis> guide.</para>
+ </emphasis> . The Linux Kernel Source Level Debugger kgdb is
+ used in conjunction with the GNU Debugger <literal>gdb</literal>
+ for debugging the Linux kernel. For more information about using
+ <literal>kgdb</literal> with <literal>gdb</literal>, see
+ <link xl:href="https://www.linuxtopia.org/online_books/redhat_linux_debugging_with_gdb/running.html">
+ Chapter 6. Running Programs Under gdb</link> in the
+ <emphasis>Red Hat Linux 4 Debugging with GDB</emphasis> guide.
+ </para>
</listitem>
<listitem>
<para><emphasis role="bold">
</emphasis> . Used to analyze saved crash dump data when a system had panicked or locked up or appears unresponsive. For more information about using crash to analyze a crash dump, see:</para>
<itemizedlist>
<listitem>
- <para> Red Hat Magazine article: <link xl:href="http://magazine.redhat.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/">A quick overview of Linux kernel crash dump analysis</link></para>
- </listitem>
- <listitem>
- <para><link xl:href="http://people.redhat.com/anderson/crash_whitepaper/#EXAMPLES">Crash Usage: A Case Study</link> from the white paper <emphasis>Red Hat Crash Utility</emphasis> by David Anderson</para>
- </listitem>
- <listitem>
- <para> Kernel Trap forum entry: <link xl:href="http://kerneltrap.org/node/5758">Linux: Kernel Crash Dumps</link></para>
- </listitem>
- <listitem>
- <para> White paper: <link xl:href="http://www.google.com/url?sa=t&source=web&ct=res&cd=8&ved=0CCUQFjAH&url=http%3A%2F%2Fwww.kernel.sg%2Fpapers%2Fcrash-dump-analysis.pdf&rct=j&q=redhat+crash+dump&ei=6aQBS-ifK4T8tAPcjdiHCw&usg=AFQjCNEk03E3GDtAsawG3gfpwc1gGNELAg">A Quick Overview of Linux Kernel Crash Dump Analysis</link></para>
+ <para> Overview on how to use crash by the author:
+ <link xl:href="https://crash-utility.github.io/crash_whitepaper.html">
+ White Paper: Red Hat Crash Utility</link>
+ </para>
</listitem>
</itemizedlist>
</listitem>
</section>
</section>
</section>
- <section xml:id="dbdoclet.50438274_23607">
+ <section xml:id="dbdoclet.debugging_procedures">
<title><indexterm><primary>debugging</primary><secondary>procedure</secondary></indexterm>Lustre Debugging Procedures</title>
- <para>The procedures below may be useful to administrators or developers debugging a Lustre files system.</para>
+ <para>The procedures below may be useful to administrators or developers
+ debugging a Lustre files system.</para>
<section remap="h3">
<title><indexterm><primary>debugging</primary><secondary>message format</secondary></indexterm>Understanding the Lustre Debug Messaging Format</title>
- <para>Lustre debug messages are categorized by originating subsystem, message type, and location in the source code. For a list of subsystems and message types, see <xref linkend="dbdoclet.50438274_57603"/>.</para>
+ <para>Lustre debug messages are categorized by originating subsystem,
+ message type, and location in the source code. For a list of subsystems
+ and message types, see <xref linkend="dbdoclet.lctl"/>.
+ </para>
<note>
<para>For a current list of subsystems and debug message types, see
<literal>libcfs/include/libcfs/libcfs_debug.h</literal> in the Lustre software
tree</para>
</note>
- <para>The elements of a Lustre debug message are described in <xref linkend="dbdoclet.50438274_57177"/> Format of Lustre Debug Messages.</para>
- <section xml:id="dbdoclet.50438274_57603">
+ <para>The elements of a Lustre debug message are described in <xref linkend="dbdoclet.lustre_msg_format"/> Format of Lustre Debug Messages.</para>
+ <section xml:id="dbdoclet.lustre_debug_msg">
<title>Lustre Debug Messages</title>
<para>Each Lustre debug message has the tag of the subsystem it originated in, the message
type, and the location in the source code. The subsystems and debug types used are as
</listitem>
</itemizedlist>
</section>
- <section xml:id="dbdoclet.50438274_57177">
+ <section xml:id="dbdoclet.lustre_msg_format">
<title>Format of Lustre Debug Messages</title>
<para>The Lustre software uses the <literal>CDEBUG()</literal> and
<literal>CERROR()</literal> macros to print the debug or error messages. To print the
<para>Lustre debug messages are maintained in a buffer, with the maximum buffer size specified (in MBs) by the <literal>debug_mb</literal> parameter (<literal>lctl get_param debug_mb</literal>). The buffer is circular, so debug messages are kept until the allocated buffer limit is reached, and then the first messages are overwritten.</para>
</section>
</section>
- <section xml:id="dbdoclet.50438274_62472">
+ <section xml:id="dbdoclet.using_lctl_tool">
<title><indexterm><primary>debugging</primary><secondary>using lctl</secondary></indexterm>Using the lctl Tool to View Debug Messages</title>
- <para>The <literal>lctl</literal> tool allows debug messages to be filtered based on subsystems and message types to extract information useful for troubleshooting from a kernel debug log. For a command reference, see <xref linkend="dbdoclet.50438219_38274"/>.</para>
+ <para>The <literal>lctl</literal> tool allows debug messages to be
+ filtered based on subsystems and message types to extract information
+ useful for troubleshooting from a kernel debug log. For a command
+ reference, see <xref linkend="dbdoclet.lctl"/>.</para>
<para>You can use <literal>lctl</literal> to:</para>
<itemizedlist>
<listitem>
<screen>modinfo libcfs</screen>
</section>
</section>
- <section xml:id="dbdoclet.50438274_80443">
+ <section xml:id="dbdoclet.debugging_for_devs">
<title><indexterm><primary>debugging</primary><secondary>developers tools</secondary></indexterm>Lustre Debugging for Developers</title>
<para>The procedures in this section may be useful to developers debugging Lustre source
code.</para>
<para>Then complete the following steps:</para>
<orderedlist>
<listitem>
- <para>Dump the log into a user-specified log file using lctl (see <xref linkend="dbdoclet.50438274_62472"/>).</para>
+ <para>Dump the log into a user-specified log file using lctl
+ (see <xref linkend="dbdoclet.using_lctl_tool"/>).
+ </para>
</listitem>
<listitem>
<para>Run the leak finder on the newly-created log dump:</para>
git://git.whamcloud.com / doc / manual.git / blobdiff