-<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretroubleshooting">
+<?xml version='1.0' encoding='UTF-8'?>
+<chapter xmlns="http://docbook.org/ns/docbook"
+ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
+ xml:id="lustretroubleshooting">
<title xml:id="lustretroubleshooting.title">Lustre File System Troubleshooting</title>
<para>This chapter provides information about troubleshooting a Lustre file system, submitting a
bug to the Jira bug tracking system, and Lustre file system performance tips. It includes the
</tgroup>
</informaltable>
</section>
- <section xml:id="dbdoclet.50438198_40669">
+ <section xml:id="dbdoclet.troubleshooting">
<title><indexterm><primary>troubleshooting</primary><secondary>error messages</secondary></indexterm>Viewing Error Messages</title>
<para>As Lustre software code runs on the kernel, single-digit error codes display to the
application; these error codes are an indication of the problem. Refer to the kernel console
<para>Which server node it was communicating with, and so on.</para>
</listitem>
</itemizedlist>
- <para>Lustre logs are dumped to <literal>/proc/sys/lnet/debug_path</literal>.</para>
+ <para>Lustre logs are dumped to the pathname stored in the parameter
+ <literal>lnet.debug_path</literal>.</para>
<para>Collect the first group of messages related to a problem, and any messages that precede "LBUG" or "assertion failure" errors. Messages that mention server nodes (OST or MDS) are specific to that server; you must collect similar messages from the relevant server console logs.</para>
<para>Another Lustre debug log holds information for a short period of time for action by the
Lustre software, which, in turn, depends on the processes on the Lustre node. Use the
</listitem>
<listitem>
<para>Submit a ticket to the <link xmlns:xlink="http://www.w3.org/1999/xlink"
- xlink:href="https://jira.whamcloud.com/secure/Dashboard.jspa">Jira</link><abbrev><superscript>*</superscript></abbrev>
+ xlink:href="https://jira.whamcloud.com/">Jira</link>
+ <abbrev><superscript>*</superscript></abbrev>
bug tracking and project management tool used for the Lustre project.
If you are a first-time user, you'll need to open an account by
clicking on <emphasis role="bold">Sign up</emphasis> on the
<listitem>
<para><emphasis role="italic">Attachments</emphasis> - Attach log sources such as
Lustre debug log dumps (see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
- linkend="dbdoclet.50438274_15874"/>), syslogs, or console logs. <emphasis
+ linkend="dbdoclet.debugging_tools"/>), syslogs, or console logs. <emphasis
role="italic"><emphasis role="bold">Note:</emphasis></emphasis> Lustre debug
logs must be processed using <code>lctl df</code> prior to attaching to a Jira
ticket. For more information, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
- linkend="dbdoclet.50438274_62472"/>. </para>
+ linkend="dbdoclet.using_lctl_tool"/>. </para>
</listitem>
</itemizedlist>Other fields in the form are used for project tracking and are irrelevant
to reporting an issue. You can leave these in their default state.</para>
<para>If the SCSI devices are inaccessible to the Lustre file system
at the block device level, then <literal>ldiskfs</literal> remounts
the device read-only to prevent file system corruption. This is a normal
- behavior. The status in <literal>/proc/fs/lustre/health_check</literal>
+ behavior. The status in the parameter <literal>health_check</literal>
also shows "not healthy" on the affected nodes.</para>
<para>To determine what caused the "not healthy" condition:</para>
<itemizedlist>
<para>To address this issue, you can expand the disk space on the OST,
or use the <literal>lfs_migrate</literal> command to migrate (move)
files to a less full OST. For details on both of these options
- see <xref linkend="dbdoclet.adding_new_ost" /></para>
+ see <xref linkend="lustremaint.adding_new_ost" /></para>
<para condition='l26'>In some cases, there may be processes holding
files open that are consuming a significant amount of space (e.g.
runaway process writing lots of data to an open file that has been
such as <literal>No space left on device</literal>. The numeric
error message may also appear in the system log.</para>
<para>For more information about the <literal>lfs df</literal> command,
- see <xref linkend="dbdoclet.checking_free_space"/>.</para>
+ see <xref linkend="file_striping.checking_free_space"/>.</para>
<para>You can also use the <literal>lctl get_param</literal> command to
monitor the space and object usage on the OSTs and MDTs from any
client:</para>
0xe74021a4b41b954e from nid 0x7f000001 (0:127.0.0.1)
</screen>
</section>
- <section remap="h3">
+ <section remap="h3" xml:id="went_back_in_time">
<title>Handling/Debugging "LustreError: xxx went back in time"</title>
- <para>Each time the Lustre software changes the state of the disk file system, it records a
- unique transaction number. Occasionally, when committing these transactions to the disk, the
- last committed transaction number displays to other nodes in the cluster to assist the
- recovery. Therefore, the promised transactions remain absolutely safe on the disappeared
- disk.</para>
+ <para>Each time the MDS or OSS modifies the state of the MDT or OST disk
+ filesystem for a client, it records a per-target increasing transaction
+ number for the operation and returns it to the client along with the
+ reply to that operation. Periodically, when the server commits these
+ transactions to disk, the <literal>last_committed</literal> transaction
+ number is returned to the client to allow it to discard pending operations
+ from memory, as they will no longer be needed for recovery in case of
+ server failure.</para>
+ <para>In some cases error messages similar to the following have
+ been observed after a server was restarted or failed over:</para>
+ <screen>
+LustreError: 3769:0:(import.c:517:ptlrpc_connect_interpret())
+testfs-ost12_UUID went back in time (transno 831 was previously committed,
+server now claims 791)!
+ </screen>
<para>This situation arises when:</para>
<itemizedlist>
<listitem>
- <para>You are using a disk device that claims to have data written to disk before it
- actually does, as in case of a device with a large cache. If that disk device crashes or
- loses power in a way that causes the loss of the cache, there can be a loss of
- transactions that you believe are committed. This is a very serious event, and you
- should run e2fsck against that storage before restarting the Lustre file system.</para>
+ <para>You are using a disk device that claims to have data written
+ to disk before it actually does, as in case of a device with a large
+ cache. If that disk device crashes or loses power in a way that
+ causes the loss of the cache, there can be a loss of transactions
+ that you believe are committed. This is a very serious event, and
+ you should run e2fsck against that storage before restarting the
+ Lustre file system.</para>
</listitem>
<listitem>
- <para>As required by the Lustre software, the shared storage used for failover is
- completely cache-coherent. This ensures that if one server takes over for another, it
- sees the most up-to-date and accurate copy of the data. In case of the failover of the
- server, if the shared storage does not provide cache coherency between all of its ports,
- then the Lustre software can produce an error.</para>
+ <para>As required by the Lustre software, the shared storage used
+ for failover is completely cache-coherent. This ensures that if one
+ server takes over for another, it sees the most up-to-date and
+ accurate copy of the data. In case of the failover of the server,
+ if the shared storage does not provide cache coherency between all
+ of its ports, then the Lustre software can produce an error.</para>
</listitem>
</itemizedlist>
- <para>If you know the exact reason for the error, then it is safe to proceed with no further action. If you do not know the reason, then this is a serious issue and you should explore it with your disk vendor.</para>
- <para>If the error occurs during failover, examine your disk cache settings. If it occurs after a restart without failover, try to determine how the disk can report that a write succeeded, then lose the Data Device corruption or Disk Errors.</para>
+ <para>If you know the exact reason for the error, then it is safe to
+ proceed with no further action. If you do not know the reason, then this
+ is a serious issue and you should explore it with your disk vendor.</para>
+ <para>If the error occurs during failover, examine your disk cache
+ settings. If it occurs after a restart without failover, try to
+ determine how the disk can report that a write succeeded, then lose the
+ Data Device corruption or Disk Errors.</para>
</section>
<section remap="h3">
<title>Lustre Error: "<literal>Slow Start_Page_Write</literal>"</title>
<para> Lustre or kernel stack traces showing processes stuck in "<literal>try_to_free_pages</literal>"</para>
</listitem>
</itemizedlist>
- <para>For information on determining the MDS memory and OSS memory requirements, see <xref linkend="dbdoclet.50438256_26456"/>.</para>
+ <para>For information on determining the MDS memory and OSS memory
+ requirements, see <xref linkend="dbdoclet.mds_oss_memory"/>.</para>
</section>
<section remap="h3">
<title>Setting SCSI I/O Sizes</title>
</section>
</section>
</chapter>
+<!--
+ vim:expandtab:shiftwidth=2:tabstop=8:
+ -->
git://git.whamcloud.com / doc / manual.git / blobdiff