-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretroubleshooting">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretroubleshooting">
<title xml:id="lustretroubleshooting.title">Lustre Troubleshooting</title>
<para>This chapter provides information to troubleshoot Lustre, submit a Lustre bug, and Lustre performance tips. It includes the following sections:</para>
<itemizedlist>
<para>Several resources are available to help troubleshoot Lustre. This section describes error numbers, error messages and logs.</para>
<section remap="h3">
<title><indexterm><primary>troubleshooting</primary><secondary>error numbers</secondary></indexterm>Error Numbers</title>
- <para>Error numbers for Lustre come from the Linux errno.h, and are located in <literal>/usr/include/asm-generic/errno.h</literal>. Lustre does not use all of the available Linux error numbers. The exact meaning of an error number depends on where it is used. Here is a summary of the basic errors that Lustre users may encounter.</para>
+ <para>Error numbers are generated by the Linux operating system and are located in
+ <literal>/usr/include/asm-generic/errno.h</literal>. The Lustre software does not use all
+ of the available Linux error numbers. The exact meaning of an error number depends on where
+ it is used. Here is a summary of the basic errors that Lustre users may encounter.</para>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
<para>Which server node it was communicating with, and so on.</para>
</listitem>
</itemizedlist>
- <para>Lustre logs are dumped to <literal>/proc/sys/lnet/debug_pat</literal>h.</para>
+ <para>Lustre logs are dumped to <literal>/proc/sys/lnet/debug_path</literal>.</para>
<para>Collect the first group of messages related to a problem, and any messages that precede "LBUG" or "assertion failure" errors. Messages that mention server nodes (OST or MDS) are specific to that server; you must collect similar messages from the relevant server console logs.</para>
<para>Another Lustre debug log holds information for Lustre action for a short period of time which, in turn, depends on the processes on the node to use Lustre. Use the following command to extract debug logs on each of the nodes, run</para>
- <screen>$ lctl dk <filename></screen>
+ <screen>$ lctl dk <replaceable>filename</replaceable></screen>
<note>
<para>LBUG freezes the thread to allow capture of the panic stack. A system reboot is needed to clear the thread.</para>
</note>
<section xml:id="dbdoclet.50438198_30989">
<title><indexterm><primary>troubleshooting</primary><secondary>reporting bugs</secondary></indexterm><indexterm><primary>reporting bugs</primary><see>troubleshooting</see></indexterm>
Reporting a Lustre Bug</title>
- <para>If, after troubleshooting your Lustre system, you cannot resolve the problem, consider reporting a Lustre bug. The process for reporting a bug is described in the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</link>.</para>
- <para>You can also post a question to the <link xl:href="http://wiki.whamcloud.com/display/PUB/Community+Resources">lustre-discuss mailing list</link> or search the <link xl:href="http://groups.google.com/group/lustre-discuss-list">lustre-discuss Archives</link> for information about your issue.</para>
- <para>A Lustre diagnostics tool is available for downloading at: <link xl:href="http://downloads.whamcloud.com/public/tools/">http://downloads.whamcloud.com/public/tools/</link></para>
- <para>You can run this tool to capture diagnostics output to include in the reported bug. To run this tool, enter one of these commands:</para>
- <screen># lustre-diagnostics -t <bugzilla bug #>
-# lustre-diagnostics.
-</screen>
- <para>Output is sent directly to the terminal. Use normal file redirection to send the output to a file, and then manually attach the file to the bug you are submitting.</para>
+ <para>If you cannot resolve a problem by troubleshooting your Lustre file system, other options are:<itemizedlist>
+ <listitem>
+ <para>Post a question to the <link xmlns:xlink="http://www.w3.org/1999/xlink"
+ xlink:href="https://lists.01.org/mailman/listinfo/hpdd-discuss">hppd-discuss</link>
+ email list or search the archives for information about your issue.</para>
+ </listitem>
+ <listitem>
+ <para>Submit a ticket to the <link xmlns:xlink="http://www.w3.org/1999/xlink"
+ xlink:href="https://jira.hpdd.intel.com/secure/Dashboard.jspa"
+ >Jira</link><abbrev><superscript>*</superscript></abbrev> bug tracking and project
+ management tool used for the Lustre project. If you are a first-time user, you'll need
+ to open an account by clicking on <emphasis role="bold">Sign up</emphasis> on the
+ Welcome page.</para>
+ </listitem>
+ </itemizedlist> To submit a Jira ticket, follow these steps:<orderedlist>
+ <listitem>
+ <para>To avoid filing a duplicate ticket, search for existing tickets for your issue.
+ <emphasis role="italic">For search tips, see <xref
+ xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_jj2_4b1_kk"
+ />.</emphasis></para>
+ </listitem>
+ <listitem>
+ <para>To create a ticket, click <emphasis role="bold">+Create Issue</emphasis> in the
+ upper right corner. <emphasis role="italic">Create a separate ticket for each issue you
+ wish to submit.</emphasis></para>
+ </listitem>
+ <listitem>
+ <para>In the form displayed, enter the following information:<itemizedlist>
+ <listitem>
+ <para><emphasis role="italic">Project</emphasis> - Select <emphasis role="bold"
+ >Lustre</emphasis> or <emphasis role="bold">Lustre Documentation</emphasis> or
+ an appropriate project.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="italic">Issue type</emphasis> - Select <emphasis role="bold"
+ >Bug</emphasis>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="italic">Summary</emphasis> - Enter a short description of the
+ issue. Use terms that would be useful for someone searching for a similar issue. A
+ LustreError or ASSERT/panic message often makes a good summary.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="italic">Affects version(s)</emphasis> - Select your Lustre
+ version.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="italic">Environment</emphasis> - Enter your kernel with
+ version number.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="italic">Description</emphasis> - Include a detailed
+ description of <emphasis role="italic">visible symptoms</emphasis> and, if
+ possible, <emphasis role="italic">how the problem is produced</emphasis>. Other
+ useful information may include <emphasis role="italic">the behavior you expect to
+ see</emphasis> and <emphasis role="italic">what you have tried so far to
+ diagnose the problem</emphasis>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="italic">Attachments</emphasis> - Attach log sources such as
+ Lustre debug log dumps (see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="dbdoclet.50438274_15874"/>), syslogs, or console logs. <emphasis
+ role="italic"><emphasis role="bold">Note:</emphasis></emphasis> Lustre debug
+ logs must be processed using <code>lctl df</code> prior to attaching to a Jira
+ ticket. For more information, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="dbdoclet.50438274_62472"/>. </para>
+ </listitem>
+ </itemizedlist>Other fields in the form are used for project tracking and are irrelevant
+ to reporting an issue. You can leave these in their default state.</para>
+ </listitem>
+ </orderedlist></para>
+ <section xml:id="section_jj2_4b1_kk">
+ <title>Searching the Jira<superscript>*</superscript> Bug Tracker for Duplicate
+ Tickets</title>
+ <para>Before submitting a ticket, always search the Jira bug tracker for an existing ticket
+ for your issue.This avoids duplicating effort and may immediately provide you with a
+ solution to your problem. </para>
+ <para>To do a search in the Jira bug tracker, select the <emphasis role="bold"
+ >Issues</emphasis> tab and click on <emphasis role="bold">New filter</emphasis>. Use the
+ filters provided to select criteria for your search. To search for specific text, enter the
+ text in the "Contains text" field and click the magnifying glass icon.</para>
+ <para>When searching for text such as an ASSERTION or LustreError message, you can remove NIDS
+ and other installation-specific text from your search string by following the example
+ below.</para>
+ <para><emphasis role="italic">Original error message:</emphasis></para>
+ <para><code>"(filter_io_26.c:</code><emphasis role="bold"
+ >791</emphasis><code>:filter_commitrw_write()) ASSERTION(oti->oti_transno
+ <=obd->obd_last_committed) failed: oti_transno </code><emphasis role="bold"
+ >752</emphasis>
+ <code>last_committed </code><emphasis role="bold">750</emphasis><code>"</code></para>
+ <para><emphasis role="italic">Optimized search string :</emphasis></para>
+ <para><code>"(filter_io_26.c:" </code><emphasis role="bold">AND</emphasis>
+ <code>":filter_commitrw_write()) ASSERTION(oti->oti_transno
+ <=obd->obd_last_committed) failed:</code></para>
+ </section>
</section>
<section xml:id="dbdoclet.50438198_93109">
<title><indexterm><primary>troubleshooting</primary><secondary>common problems</secondary></indexterm>Common Lustre Problems</title>
</listitem>
<listitem>
<para>Deactivate the OST (on the OSS at the MDS). Run:</para>
- <screen>$ lctl --device <OST device name or number> deactivate</screen>
+ <screen>$ lctl --device <replaceable>lustre_device_number</replaceable> deactivate</screen>
<para>The OST device number or device name is generated by the lctl dl command.</para>
<para>The <literal>deactivate</literal> command prevents clients from creating new objects on the specified OST, although you can still access the OST for reading.</para>
<note>
<para>If the OST later becomes available it needs to be reactivated, run:</para>
- <screen># lctl --device <OST device name or number> activate</screen>
+ <screen># lctl --device <replaceable>lustre_device_number</replaceable> activate</screen>
</note>
</listitem>
<listitem>
<note>
<para>The file system must be stopped on all servers before performing this procedure.</para>
</note>
- <para>For hex < -> decimal translations:</para>
+ <para>For hex-to-decimal translations:</para>
<para>Use GDB:</para>
<screen>(gdb) p /x 15028
$2 = 0x3ab4</screen>
<orderedlist>
<listitem>
<para>Determine a reasonable value for the LAST_ID file. Check on the MDS:</para>
- <screen># mount -t ldiskfs /dev/<mdsdev> /mnt/mds
+ <screen># mount -t ldiskfs <replaceable>/dev/mdt_device</replaceable> /mnt/mds
# od -Ax -td8 /mnt/mds/lov_objid
</screen>
<para>There is one entry for each OST, in OST index order. This is what the MDS thinks is the last in-use object.</para>
<section remap="h3">
<title>Setting SCSI I/O Sizes</title>
<para>Some SCSI drivers default to a maximum I/O size that is too small for good Lustre performance. we have fixed quite a few drivers, but you may still find that some drivers give unsatisfactory performance with Lustre. As the default value is hard-coded, you need to recompile the drivers to change their default. On the other hand, some drivers may have a wrong default set.</para>
- <para>If you suspect bad I/O performance and an analysis of Lustre statistics indicates that I/O is not 1 MB, check <literal>/sys/block/<device>/queue/max_sectors_kb</literal>. If the <literal>max_sectors_kb</literal> value is less than 1024, set it to at least 1024 to improve performance. If changing <literal>max_sectors_kb</literal> does not change the I/O size as reported by Lustre, you may want to examine the SCSI driver code.</para>
+ <para>If you suspect bad I/O performance and an analysis of Lustre statistics indicates that I/O is not 1 MB, check <literal>/sys/block/<replaceable>device</replaceable>/queue/max_sectors_kb</literal>. If the <literal>max_sectors_kb</literal> value is less than 1024, set it to at least 1024 to improve performance. If changing <literal>max_sectors_kb</literal> does not change the I/O size as reported by Lustre, you may want to examine the SCSI driver code.</para>
</section>
</section>
</chapter>
git://git.whamcloud.com / doc / manual.git / blobdiff