<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustredebugging">
- <title xml:id="lustredebugging.title">Lustre Debugging</title>
- <para>This chapter describes tips and information to debug Lustre, and includes the following sections:</para>
+ <title xml:id="lustredebugging.title">Debugging a Lustre File System</title>
+ <para>This chapter describes tips and information to debug a Lustre file system, and includes the
+ following sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438274_15874"/></para>
linkend="dbdoclet.50438219_38274"/>.</para>
</listitem>
<listitem>
- <para>Lustre subsystem asserts - A panic-style assertion (LBUG) in the kernel causes
- Lustre to dump the debug log to the file
+ <para>Lustre subsystem asserts - A panic-style assertion (LBUG) in the kernel causes the
+ Lustre file system to dump the debug log to the file
<literal>/tmp/lustre-log.<replaceable>timestamp</replaceable></literal> where it can
be retrieved after a reboot. For more information, see <xref
linkend="dbdoclet.50438198_40669"/>.</para>
</emphasis> . <literal>syslogd</literal> prints fatal or serious messages at this log.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Crash dumps</emphasis> . On crash-dump enabled kernels, sysrq c produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64 KB of the log) to the console.</para>
+ <para><emphasis role="bold">Crash dumps</emphasis> . On crash-dump enabled kernels,
+ sysrq c produces a crash dump. The Lustre software enhances this crash dump with a log
+ dump (the last 64 KB of the log) to the console.</para>
</listitem>
<listitem>
<para><emphasis role="bold">
packet inspection tool that allows debugging of information that was
sent between the various Lustre nodes. This tool is built on top of
<literal>tcpdump</literal> and can read packet dumps generated by
- it. There are plug-ins available to dissassemble the LNET and
+ it. There are plug-ins available to dissassemble the LNet and
Lustre protocols. They are located within the <link
- xl:href="http://git.hpdd.intel.com/">Lustre git repository</link>
+ xl:href="http://git.whamcloud.com/">Lustre git repository</link>
under <literal>lustre/contrib/wireshark/</literal>. Installation
instruction are included in that directory. See also <link
xl:href="http://www.wireshark.org/">Wireshark Website</link> for
</section>
<section remap="h4">
<title><indexterm><primary>debugging</primary><secondary>developer tools</secondary></indexterm>Tools for Developers</title>
- <para>The tools described in this section may be useful for debugging Lustre in a development environment.</para>
+ <para>The tools described in this section may be useful for debugging a Lustre file system
+ in a development environment.</para>
<para>Of general interest is:</para>
<itemizedlist>
<listitem>
<para><literal>
<replaceable>leak_finder.pl</replaceable>
- </literal> . This program provided with Lustre is useful for finding memory leaks in the code.</para>
+ </literal> . This program provided with the Lustre software is useful for finding
+ memory leaks in the code.</para>
</listitem>
</itemizedlist>
<para>A virtual machine is often used to create an isolated development and test environment. Some commonly-used virtual machines are:</para>
<title><indexterm><primary>debugging</primary><secondary>message format</secondary></indexterm>Understanding the Lustre Debug Messaging Format</title>
<para>Lustre debug messages are categorized by originating subsystem, message type, and location in the source code. For a list of subsystems and message types, see <xref linkend="dbdoclet.50438274_57603"/>.</para>
<note>
- <para>For a current list of subsystems and debug message types, see <literal>libcfs/include/libcfs/libcfs_debug.h</literal> in the Lustre tree</para>
+ <para>For a current list of subsystems and debug message types, see
+ <literal>libcfs/include/libcfs/libcfs_debug.h</literal> in the Lustre software
+ tree</para>
</note>
<para>The elements of a Lustre debug message are described in <xref linkend="dbdoclet.50438274_57177"/> Format of Lustre Debug Messages.</para>
<section xml:id="dbdoclet.50438274_57603">
<title>Lustre Debug Messages</title>
- <para>Each Lustre debug message has the tag of the subsystem it originated in, the message type, and the location in the source code. The subsystems and debug types used in Lustre are as follows:</para>
+ <para>Each Lustre debug message has the tag of the subsystem it originated in, the message
+ type, and the location in the source code. The subsystems and debug types used are as
+ follows:</para>
<itemizedlist>
<listitem>
<para> Standard Subsystems:</para>
</section>
<section xml:id="dbdoclet.50438274_57177">
<title>Format of Lustre Debug Messages</title>
- <para>Lustre uses the <literal>CDEBUG()</literal> and <literal>CERROR()</literal> macros to print the debug or error messages. To print the message, the <literal>CDEBUG()</literal> macro uses the function <literal>libcfs_debug_msg()</literal> (<literal>libcfs/libcfs/tracefile.c</literal>). The message format is described below, along with an example.</para>
+ <para>The Lustre software uses the <literal>CDEBUG()</literal> and
+ <literal>CERROR()</literal> macros to print the debug or error messages. To print the
+ message, the <literal>CDEBUG()</literal> macro uses the function
+ <literal>libcfs_debug_msg()</literal> (<literal>libcfs/libcfs/tracefile.c</literal>).
+ The message format is described below, along with an example.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
</section>
<section remap="h3">
<title><indexterm><primary>debugging</primary><secondary>disk contents</secondary></indexterm>Looking at Disk Content</title>
- <para>In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The <literal>lfs</literal> tool can be used to obtain this information for a given file using the <literal>getstripe</literal> subcommand. Use a corresponding <literal>lfs setstripe</literal> command to specify striping attributes for a new file or directory.</para>
- <para>The <literal>lfs getstripe</literal> command takes a Lustre filename as input and lists all the objects that form a part of this file. To obtain this information for the file <literal>/mnt/lustre/frog</literal> in Lustre file system, run:</para>
- <screen>$ lfs getstripe /mnt/lustre/frog
+ <para>In a Lustre file system, the inodes on the metadata server contain extended attributes
+ (EAs) that store information about file striping. EAs contain a list of all object IDs and
+ their locations (that is, the OST that stores them). The <literal>lfs</literal> tool can be
+ used to obtain this information for a given file using the <literal>getstripe</literal>
+ subcommand. Use a corresponding <literal>lfs setstripe</literal> command to specify striping
+ attributes for a new file or directory.</para>
+ <para>The <literal>lfs getstripe</literal> command takes a Lustre filename as input and lists
+ all the objects that form a part of this file. To obtain this information for the file
+ <literal>/mnt/testfs/frog</literal> in a Lustre file system, run:</para>
+ <screen>$ lfs getstripe /mnt/testfs/frog
lmm_stripe_count: 2
lmm_stripe_size: 1048576
+lmm_pattern: 1
+lmm_layout_gen: 0
lmm_stripe_offset: 2
obdidx objid objid group
2 818855 0xc7ea7 0
0 873123 0xd52a3 0
</screen>
- <para>The <literal>debugfs</literal> tool is provided in the <literal>e2fsprogs</literal> package. It can be used for interactive debugging of an <literal>ldiskfs</literal> file system. The <literal>debugfs</literal> tool can either be used to check status or modify information in the file system. In Lustre, all objects that belong to a file are stored in an underlying <literal>ldiskfs</literal> file system on the OSTs. The file system uses the object IDs as the file names. Once the object IDs are known, use the <literal>debugfs</literal> tool to obtain the attributes of all objects from different OSTs.</para>
- <para>A sample run for the <literal>/mnt/lustre/frog</literal> file used in the above example is shown here:</para>
+ <para>The <literal>debugfs</literal> tool is provided in the
+ <literal>e2fsprogs</literal> package. It can be used for interactive
+ debugging of an <literal>ldiskfs</literal> file system. The
+ <literal>debugfs</literal> tool can either be used to check status or
+ modify information in the file system. In a Lustre file system, all
+ objects that belong to a file are stored in an underlying
+ <literal>ldiskfs</literal> file system on the OSTs. The file system
+ uses the object IDs as the file names. Once the object IDs are known,
+ use the <literal>debugfs</literal> tool to obtain the attributes of
+ all objects from different OSTs.</para>
+ <para>A sample run for the <literal>/mnt/testfs/frog</literal> file used
+ in the above example is shown here:</para>
<screen>$ debugfs -c -R "stat O/0/d$((818855 % 32))/818855" /dev/vgmyth/lvmythost2
debugfs 1.41.90.wc3 (28-May-2011)
strings /tmp/last_rcvd | head -1
myth-OST0004_UUID
</screen>
- <para>It is also possible (and easier) to extract this from the filesystem label using the <literal>dumpe2fs</literal> command:</para>
+ <para>It is also possible (and easier) to extract this from the file system label using the
+ <literal>dumpe2fs</literal> command:</para>
<screen>dumpe2fs -h /dev/sdc | grep volume
dumpe2fs 1.41.90.wc3 (28-May-2011)
Filesystem volume name: myth-OST0004
</section>
<section remap="h3">
<title>Tracing Lock Traffic</title>
- <para>Lustre has a specific debug type category for tracing lock traffic. Use:</para>
+ <para>The Lustre software provides a specific debug type category for tracing lock traffic.
+ Use:</para>
<screen>lctl> filter all_types
lctl> show dlmtrace
lctl> debug_kernel [<replaceable>filename</replaceable>] </screen>
</section>
+ <section remap="h3">
+ <title>Controlling Console Message Rate Limiting</title>
+ <para>Some console messages which are printed by Lustre are rate limited. When such messages are printed, they may be followed by a message saying "Skipped N previous similar message(s)," where N is the number of messages skipped. This rate limiting can be completely disabled by a libcfs module parameter called <literal>libcfs_console_ratelimit</literal>. To disable console message rate limiting, add this line to <literal>/etc/modprobe.d/lustre.conf</literal> and then reload Lustre modules.</para>
+ <screen>options libcfs libcfs_console_ratelimit=0</screen>
+ <para>It is also possible to set the minimum and maximum delays between rate-limited console messages using the module parameters <literal>libcfs_console_max_delay</literal> and <literal>libcfs_console_min_delay</literal>. Set these in <literal>/etc/modprobe.d/lustre.conf</literal> and then reload Lustre modules. Additional information on libcfs module parameters is available via <literal>modinfo</literal>:</para>
+ <screen>modinfo libcfs</screen>
+ </section>
</section>
<section xml:id="dbdoclet.50438274_80443">
<title><indexterm><primary>debugging</primary><secondary>developers tools</secondary></indexterm>Lustre Debugging for Developers</title>
- <para>The procedures in this section may be useful to developers debugging Lustre code.</para>
+ <para>The procedures in this section may be useful to developers debugging Lustre source
+ code.</para>
<section remap="h3">
<title>Adding Debugging to the Lustre Source Code</title>
<para>The debugging infrastructure provides a number of macros that can be used in Lustre source code to aid in debugging or reporting serious errors.</para>
</emphasis></para>
</entry>
<entry>
- <para>A panic-style assertion in the kernel which causes Lustre to dump its circular log to the <literal>/tmp/lustre-log</literal> file. This file can be retrieved after a reboot. <literal>LBUG()</literal> freezes the thread to allow capture of the panic stack. A system reboot is needed to clear the thread.</para>
+ <para>A panic-style assertion in the kernel which causes the Lustre file system to
+ dump its circular log to the <literal>/tmp/lustre-log</literal> file. This file
+ can be retrieved after a reboot. <literal>LBUG()</literal> freezes the thread to
+ allow capture of the panic stack. A system reboot is needed to clear the
+ thread.</para>
</entry>
</row>
<row>
</emphasis></para>
</entry>
<entry>
- <para> Behaves similarly to <literal>CERROR()</literal>, but prints error messages for LNET if <literal>D_NETERR</literal> is set in the <literal>debug</literal> mask. This is appropriate for serious networking errors. Messages printed to the console are rate-limited.</para>
+ <para> Behaves similarly to <literal>CERROR()</literal>, but prints error messages for LNet if <literal>D_NETERR</literal> is set in the <literal>debug</literal> mask. This is appropriate for serious networking errors. Messages printed to the console are rate-limited.</para>
</entry>
</row>
<row>
</emphasis></para>
</entry>
<entry>
- <para>Allows insertion of failure points into the Lustre code. This is useful to generate regression tests that can hit a very specific sequence of events. This works in conjunction with "<literal>lctl set_param fail_loc=<replaceable>fail_loc</replaceable></literal>" to set a specific failure point for which a given <literal>OBD_FAIL_CHECK()</literal> will test.</para>
+ <para>Allows insertion of failure points into the Lustre source code. This is useful
+ to generate regression tests that can hit a very specific sequence of events. This
+ works in conjunction with "<literal>lctl set_param
+ fail_loc=<replaceable>fail_loc</replaceable></literal>" to set a specific
+ failure point for which a given <literal>OBD_FAIL_CHECK()</literal> will
+ test.</para>
</entry>
</row>
<row>
<section remap="h3">
<title>Accessing the <literal>ptlrpc</literal> Request History</title>
<para>Each service maintains a request history, which can be useful for first occurrence troubleshooting.</para>
- <para><literal>ptlrpc</literal> is an RPC protocol layered on LNET that deals with stateful servers and has semantics and built-in support for recovery.</para>
+ <para><literal>ptlrpc</literal> is an RPC protocol layered on LNet that deals with stateful servers and has semantics and built-in support for recovery.</para>
<para>The ptlrpc request history works as follows:</para>
<orderedlist>
<listitem>
git://git.whamcloud.com / doc / manual.git / blobdiff