-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretroubleshooting">
- <title xml:id="lustretroubleshooting.title">Lustre Troubleshooting</title>
- <para>This chapter provides information to troubleshoot Lustre, submit a Lustre bug, and Lustre performance tips. It includes the following sections:</para>
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretroubleshooting">
+ <title xml:id="lustretroubleshooting.title">Lustre File System Troubleshooting</title>
+ <para>This chapter provides information about troubleshooting a Lustre file system, submitting a
+ bug to the Jira bug tracking system, and Lustre file system performance tips. It includes the
+ following sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438198_11171"/></para>
<indexterm><primary>lustre</primary><secondary>errors</secondary><see>troubleshooting</see></indexterm>
<indexterm><primary>errors</primary><see>troubleshooting</see></indexterm>
Lustre Error Messages</title>
- <para>Several resources are available to help troubleshoot Lustre. This section describes error numbers, error messages and logs.</para>
+ <para>Several resources are available to help troubleshoot an issue in a Lustre file system.
+ This section describes error numbers, error messages and logs.</para>
<section remap="h3">
<title><indexterm><primary>troubleshooting</primary><secondary>error numbers</secondary></indexterm>Error Numbers</title>
- <para>Error numbers for Lustre come from the Linux errno.h, and are located in <literal>/usr/include/asm/errno.h</literal>. Lustre does not use all of the available Linux error numbers. The exact meaning of an error number depends on where it is used. Here is a summary of the basic errors that Lustre users may encounter.</para>
+ <para>Error numbers are generated by the Linux operating system and are located in
+ <literal>/usr/include/asm-generic/errno.h</literal>. The Lustre software does not use all
+ of the available Linux error numbers. The exact meaning of an error number depends on where
+ it is used. Here is a summary of the basic errors that Lustre file system users may
+ encounter.</para>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
<para> The operation took too long and timed out.</para>
</entry>
</row>
+ <row>
+ <entry>
+ <para> -122</para>
+ </entry>
+ <entry>
+ <literal> -EDQUOT </literal>
+ </entry>
+ <entry>
+ <para> The operation exceeded the user disk quota and was aborted.</para>
+ </entry>
+ </row>
</tbody>
</tgroup>
</informaltable>
</section>
<section xml:id="dbdoclet.50438198_40669">
<title><indexterm><primary>troubleshooting</primary><secondary>error messages</secondary></indexterm>Viewing Error Messages</title>
- <para>As Lustre code runs on the kernel, single-digit error codes display to the application; these error codes are an indication of the problem. Refer to the kernel console log (dmesg) for all recent kernel messages from that node. On the node, <literal>/var/log/messages</literal> holds a log of all messages for at least the past day.</para>
+ <para>As Lustre software code runs on the kernel, single-digit error codes display to the
+ application; these error codes are an indication of the problem. Refer to the kernel console
+ log (dmesg) for all recent kernel messages from that node. On the node,
+ <literal>/var/log/messages</literal> holds a log of all messages for at least the past
+ day.</para>
<para>The error message initiates with "LustreError" in the console log and provides a short description of:</para>
<itemizedlist>
<listitem>
<para>Which server node it was communicating with, and so on.</para>
</listitem>
</itemizedlist>
- <para>Lustre logs are dumped to <literal>/proc/sys/lnet/debug_pat</literal>h.</para>
+ <para>Lustre logs are dumped to <literal>/proc/sys/lnet/debug_path</literal>.</para>
<para>Collect the first group of messages related to a problem, and any messages that precede "LBUG" or "assertion failure" errors. Messages that mention server nodes (OST or MDS) are specific to that server; you must collect similar messages from the relevant server console logs.</para>
- <para>Another Lustre debug log holds information for Lustre action for a short period of time which, in turn, depends on the processes on the node to use Lustre. Use the following command to extract debug logs on each of the nodes, run</para>
- <screen>$ lctl dk <filename></screen>
+ <para>Another Lustre debug log holds information for a short period of time for action by the
+ Lustre software, which, in turn, depends on the processes on the Lustre node. Use the
+ following command to extract debug logs on each of the nodes, run</para>
+ <screen>$ lctl dk <replaceable>filename</replaceable></screen>
<note>
<para>LBUG freezes the thread to allow capture of the panic stack. A system reboot is needed to clear the thread.</para>
</note>
</section>
</section>
<section xml:id="dbdoclet.50438198_30989">
- <title><indexterm><primary>troubleshooting</primary><secondary>reporting bugs</secondary></indexterm><indexterm><primary>reporting bugs</primary><see>troubleshooing</see></indexterm>
- Reporting a Lustre Bug</title>
- <para>If, after troubleshooting your Lustre system, you cannot resolve the problem, consider reporting a Lustre bug. The process for reporting a bug is described in the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</link>.</para>
- <para>You can also post a question to the <link xl:href="http://wiki.lustre.org/index.php/Lustre_Mailing_Lists">lustre-discuss mailing list</link> or search the <link xl:href="http://groups.google.com/group/lustre-discuss-list">lustre-discuss Archives</link> for information about your issue.</para>
- <para>A Lustre diagnostics tool is available for downloading at: <link xl:href="http://downloads.lustre.org/public/tools/lustre-diagnostics/">http://downloads.lustre.org/public/tools/lustre-diagnostics/</link></para>
- <para>You can run this tool to capture diagnostics output to include in the reported bug. To run this tool, enter one of these commands:</para>
- <screen># lustre-diagnostics -t <bugzilla bug #>
-# lustre-diagnostics.
-</screen>
- <para>Output is sent directly to the terminal. Use normal file redirection to send the output to a file, and then manually attach the file to the bug you are submitting.</para>
+ <title><indexterm>
+ <primary>troubleshooting</primary>
+ <secondary>reporting bugs</secondary>
+ </indexterm><indexterm>
+ <primary>reporting bugs</primary>
+ <see>troubleshooting</see>
+ </indexterm> Reporting a Lustre File System Bug</title>
+ <para>If you cannot resolve a problem by troubleshooting your Lustre file system, other options are:<itemizedlist>
+ <listitem>
+ <para>Post a question to the <link xmlns:xlink="http://www.w3.org/1999/xlink"
+ xlink:href="https://lists.01.org/mailman/listinfo/hpdd-discuss">hppd-discuss</link>
+ email list or search the archives for information about your issue.</para>
+ </listitem>
+ <listitem>
+ <para>Submit a ticket to the <link xmlns:xlink="http://www.w3.org/1999/xlink"
+ xlink:href="https://jira.hpdd.intel.com/secure/Dashboard.jspa"
+ >Jira</link><abbrev><superscript>*</superscript></abbrev> bug tracking and project
+ management tool used for the Lustre software project. If you are a first-time user,
+ you'll need to open an account by clicking on <emphasis role="bold">Sign up</emphasis>
+ on the Welcome page.</para>
+ </listitem>
+ </itemizedlist> To submit a Jira ticket, follow these steps:<orderedlist>
+ <listitem>
+ <para>To avoid filing a duplicate ticket, search for existing tickets for your issue.
+ <emphasis role="italic">For search tips, see <xref
+ xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_jj2_4b1_kk"
+ />.</emphasis></para>
+ </listitem>
+ <listitem>
+ <para>To create a ticket, click <emphasis role="bold">+Create Issue</emphasis> in the
+ upper right corner. <emphasis role="italic">Create a separate ticket for each issue you
+ wish to submit.</emphasis></para>
+ </listitem>
+ <listitem>
+ <para>In the form displayed, enter the following information:<itemizedlist>
+ <listitem>
+ <para><emphasis role="italic">Project</emphasis> - Select <emphasis role="bold"
+ >Lustre</emphasis> or <emphasis role="bold">Lustre Documentation</emphasis> or
+ an appropriate project.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="italic">Issue type</emphasis> - Select <emphasis role="bold"
+ >Bug</emphasis>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="italic">Summary</emphasis> - Enter a short description of the
+ issue. Use terms that would be useful for someone searching for a similar issue. A
+ LustreError or ASSERT/panic message often makes a good summary.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="italic">Affects version(s)</emphasis> - Select your Lustre
+ release.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="italic">Environment</emphasis> - Enter your kernel with
+ version number.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="italic">Description</emphasis> - Include a detailed
+ description of <emphasis role="italic">visible symptoms</emphasis> and, if
+ possible, <emphasis role="italic">how the problem is produced</emphasis>. Other
+ useful information may include <emphasis role="italic">the behavior you expect to
+ see</emphasis> and <emphasis role="italic">what you have tried so far to
+ diagnose the problem</emphasis>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="italic">Attachments</emphasis> - Attach log sources such as
+ Lustre debug log dumps (see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="dbdoclet.50438274_15874"/>), syslogs, or console logs. <emphasis
+ role="italic"><emphasis role="bold">Note:</emphasis></emphasis> Lustre debug
+ logs must be processed using <code>lctl df</code> prior to attaching to a Jira
+ ticket. For more information, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="dbdoclet.50438274_62472"/>. </para>
+ </listitem>
+ </itemizedlist>Other fields in the form are used for project tracking and are irrelevant
+ to reporting an issue. You can leave these in their default state.</para>
+ </listitem>
+ </orderedlist></para>
+ <section xml:id="section_jj2_4b1_kk">
+ <title>Searching the Jira<superscript>*</superscript> Bug Tracker for Duplicate
+ Tickets</title>
+ <para>Before submitting a ticket, always search the Jira bug tracker for an existing ticket
+ for your issue.This avoids duplicating effort and may immediately provide you with a
+ solution to your problem. </para>
+ <para>To do a search in the Jira bug tracker, select the <emphasis role="bold"
+ >Issues</emphasis> tab and click on <emphasis role="bold">New filter</emphasis>. Use the
+ filters provided to select criteria for your search. To search for specific text, enter the
+ text in the "Contains text" field and click the magnifying glass icon.</para>
+ <para>When searching for text such as an ASSERTION or LustreError message, you can remove NIDS
+ and other installation-specific text from your search string by following the example
+ below.</para>
+ <para><emphasis role="italic">Original error message:</emphasis></para>
+ <para><code>"(filter_io_26.c:</code><emphasis role="bold"
+ >791</emphasis><code>:filter_commitrw_write()) ASSERTION(oti->oti_transno
+ <=obd->obd_last_committed) failed: oti_transno </code><emphasis role="bold"
+ >752</emphasis>
+ <code>last_committed </code><emphasis role="bold">750</emphasis><code>"</code></para>
+ <para><emphasis role="italic">Optimized search string :</emphasis></para>
+ <para><code>"(filter_io_26.c:" </code><emphasis role="bold">AND</emphasis>
+ <code>":filter_commitrw_write()) ASSERTION(oti->oti_transno
+ <=obd->obd_last_committed) failed:</code></para>
+ </section>
</section>
<section xml:id="dbdoclet.50438198_93109">
- <title><indexterm><primary>troubleshooting</primary><secondary>common problems</secondary></indexterm>Common Lustre Problems</title>
- <para>This section describes how to address common issues encountered with Lustre.</para>
+ <title><indexterm>
+ <primary>troubleshooting</primary>
+ <secondary>common problems</secondary>
+ </indexterm>Common Lustre File System Problems</title>
+ <para>This section describes how to address common issues encountered with a Lustre file
+ system.</para>
<section remap="h3">
<title>OST Object is Missing or Damaged</title>
<para>If the OSS fails to find an object or finds a damaged object, this message appears:</para>
<para>If the reported error is -2 (<literal>-ENOENT</literal>, or "No such file or directory"), then the object is missing. This can occur either because the MDS and OST are out of sync, or because an OST object was corrupted and deleted.</para>
<para>If you have recovered the file system from a disk failure by using e2fsck, then unrecoverable objects may have been deleted or moved to /lost+found on the raw OST partition. Because files on the MDS still reference these objects, attempts to access them produce this error.</para>
<para>If you have recovered a backup of the raw MDS or OST partition, then the restored partition is very likely to be out of sync with the rest of your cluster. No matter which server partition you restored from backup, files on the MDS may reference objects which no longer exist (or did not exist when the backup was taken); accessing those files produces this error.</para>
- <para>If neither of those descriptions is applicable to your situation, then it is possible that you have discovered a programming error that allowed the servers to get out of sync. Please report this condition to the Lustre group, and we will investigate.</para>
+ <para>If neither of those descriptions is applicable to your situation, then it is possible
+ that you have discovered a programming error that allowed the servers to get out of sync.
+ Please submit a Jira ticket (see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="dbdoclet.50438198_30989"/>).</para>
<para>If the reported error is anything else (such as -5, "<literal>I/O error</literal>"), it likely indicates a storage failure. The low-level file system returns this error if it is unable to read from the storage device.</para>
<para><emphasis role="bold">Suggested Action</emphasis></para>
<para>If the reported error is -2, you can consider checking in <literal>/lost+found</literal> on your raw OST device, to see if the missing object is there. However, it is likely that this object is lost forever, and that the file that references the object is now partially or completely lost. Restore this file from backup, or salvage what you can and delete it.</para>
</section>
<section remap="h3">
<title>OSTs Become Read-Only</title>
- <para>If the SCSI devices are inaccessible to Lustre at the block device level, then <literal>ldiskfs</literal> remounts the device read-only to prevent file system corruption. This is a normal behavior. The status in <literal>/proc/fs/lustre/health_check</literal> also shows "not healthy" on the affected nodes.</para>
+ <para>If the SCSI devices are inaccessible to the Lustre file system at the block device
+ level, then <literal>ldiskfs</literal> remounts the device read-only to prevent file system
+ corruption. This is a normal behavior. The status in
+ <literal>/proc/fs/lustre/health_check</literal> also shows "not healthy" on the
+ affected nodes.</para>
<para>To determine what caused the "not healthy" condition:</para>
<itemizedlist>
<listitem>
</listitem>
<listitem>
<para>Deactivate the OST (on the OSS at the MDS). Run:</para>
- <screen>$ lctl --device <OST device name or number> deactivate</screen>
+ <screen>$ lctl --device <replaceable>lustre_device_number</replaceable> deactivate</screen>
<para>The OST device number or device name is generated by the lctl dl command.</para>
<para>The <literal>deactivate</literal> command prevents clients from creating new objects on the specified OST, although you can still access the OST for reading.</para>
<note>
<para>If the OST later becomes available it needs to be reactivated, run:</para>
- <screen># lctl --device <OST device name or number> activate</screen>
+ <screen># lctl --device <replaceable>lustre_device_number</replaceable> activate</screen>
</note>
</listitem>
<listitem>
<para>For example, for a 2 stripe file, stripe size = 1M, the bad OST is at index 0, and you have holes in the file at: [(2*N + 0)*1M, (2*N + 0)*1M + 1M - 1], N = { 0, 1, 2, ...}</para>
<para>If the file system cannot be mounted, currently there is no way that parses metadata directly from an MDS. If the bad OST does not start, options to mount the file system are to provide a loop device OST in its place or replace it with a newly-formatted OST. In that case, the missing objects are created and are read as zero-filled.</para>
</section>
- <section xml:id="dbdoclet.50438198_69657">
+ <section xml:id="dbdoclet.repair_ost_lastid">
<title>Fixing a Bad LAST_ID on an OST</title>
<para>Each OST contains a LAST_ID file, which holds the last object (pre-)created by the MDS <footnote>
<para>The contents of the LAST_ID file must be accurate regarding the actual objects that exist on the OST.</para>
<para>However, in the case where lov_objid < LAST_ID, bad things can happen as the MDS is not aware of objects that have already been allocated on the OST, and it reallocates them to new files, overwriting their existing contents.</para>
<para>Here is the rule to avoid this scenario:</para>
<para>LAST_ID >= lov_objid and LAST_ID == last_physical_object and lov_objid >= last_used_object</para>
- <para>Although the lov_objid value should be equal to the last_used_object value, the above rule suffices to keep Lustre happy at the expense of a few leaked objects.</para>
+ <para>Although the lov_objid value should be equal to the last_used_object value, the above
+ rule suffices to keep the Lustre file system happy at the expense of a few leaked
+ objects.</para>
<para>In situations where there is on-disk corruption of the OST, for example caused by running with write cache enabled on the disks, the LAST_ID value may become inconsistent and result in a message similar to:</para>
<screen>"filter_precreate()) HOME-OST0003: Serious error:
objid 3478673 already exists; is this filesystem corrupt?"</screen>
<note>
<para>The file system must be stopped on all servers before performing this procedure.</para>
</note>
- <para>For hex < -> decimal translations:</para>
+ <para>For hex-to-decimal translations:</para>
<para>Use GDB:</para>
<screen>(gdb) p /x 15028
$2 = 0x3ab4</screen>
<orderedlist>
<listitem>
<para>Determine a reasonable value for the LAST_ID file. Check on the MDS:</para>
- <screen># mount -t ldiskfs /dev/<mdsdev> /mnt/mds
+ <screen># mount -t ldiskfs <replaceable>/dev/mdt_device</replaceable> /mnt/mds
# od -Ax -td8 /mnt/mds/lov_objid
</screen>
<para>There is one entry for each OST, in OST index order. This is what the MDS thinks is the last in-use object.</para>
</section>
<section remap="h3">
<title><indexterm><primary>troubleshooting</primary><secondary>'Address already in use'</secondary></indexterm>Handling/Debugging "<literal>Bind: Address already in use</literal>" Error</title>
- <para>During startup, Lustre may report a <literal>bind: Address already in use</literal> error and reject to start the operation. This is caused by a portmap service (often NFS locking) which starts before Lustre and binds to the default port 988. You must have port 988 open from firewall or IP tables for incoming connections on the client, OSS, and MDS nodes. LNET will create three outgoing connections on available, reserved ports to each client-server pair, starting with 1023, 1022 and 1021.</para>
+ <para>During startup, the Lustre software may report a <literal>bind: Address already in
+ use</literal> error and reject to start the operation. This is caused by a portmap service
+ (often NFS locking) that starts before the Lustre file system and binds to the default port
+ 988. You must have port 988 open from firewall or IP tables for incoming connections on the
+ client, OSS, and MDS nodes. LNet will create three outgoing connections on available,
+ reserved ports to each client-server pair, starting with 1023, 1022 and 1021.</para>
<para>Unfortunately, you cannot set sunprc to avoid port 988. If you receive this error, do the following:</para>
<itemizedlist>
<listitem>
- <para>Start Lustre before starting any service that uses sunrpc.</para>
+ <para>Start the Lustre file system before starting any service that uses sunrpc.</para>
</listitem>
<listitem>
- <para>Use a port other than 988 for Lustre. This is configured in /etc/modprobe.conf as an option to the LNET module. For example:</para>
+ <para>Use a port other than 988 for the Lustre file system. This is configured in
+ <literal>/etc/modprobe.d/lustre.conf</literal> as an option to the LNet module. For
+ example:</para>
<screen>options lnet accept_port=988</screen>
</listitem>
</itemizedlist>
<itemizedlist>
<listitem>
- <para>Add modprobe ptlrpc to your system startup scripts before the service that uses sunrpc. This causes Lustre to bind to port 988 and sunrpc to select a different port.</para>
+ <para>Add modprobe ptlrpc to your system startup scripts before the service that uses
+ sunrpc. This causes the Lustre file system to bind to port 988 and sunrpc to select a
+ different port.</para>
</listitem>
</itemizedlist>
<note>
</itemizedlist>
<para>A Linux error -28 (<literal>ENOSPC</literal>) that occurs when a new file is being created may indicate that the MDS has run out of inodes and needs to be made larger. Newly created files do not written to full OSTs, while existing files continue to reside on the OST where they were initially created. To view inode information on the MDS, enter:</para>
<screen>lfs df -i</screen>
- <para>Typically, Lustre reports this error to your application. If the application is checking the return code from its function calls, then it decodes it into a textual error message such as <literal>No space left on device</literal>. Both versions of the error message also appear in the system log.</para>
+ <para>Typically, the Lustre software reports this error to your application. If the
+ application is checking the return code from its function calls, then it decodes it into a
+ textual error message such as <literal>No space left on device</literal>. Both versions of
+ the error message also appear in the system log.</para>
<para>For more information about the <literal>lfs df</literal> command, see <xref linkend="dbdoclet.50438209_35838"/>.</para>
<para>Although it is less efficient, you can also use the grep command to determine which OST or MDS is running out of space. To check the free space and inodes on a client, enter:</para>
<screen>grep '[0-9]' /proc/fs/lustre/osc/*/kbytes{free,avail,total}
</screen>
</section>
<section remap="h3">
- <title><indexterm><primary>troubleshooting</primary><secondary>timeouts on setup</secondary></indexterm>Handling Timeouts on Initial Lustre Setup</title>
- <para>If you come across timeouts or hangs on the initial setup of your Lustre system, verify that name resolution for servers and clients is working correctly. Some distributions configure <literal>/etc/hosts sts</literal> so the name of the local machine (as reported by the 'hostname' command) is mapped to local host (127.0.0.1) instead of a proper IP address.</para>
+ <title><indexterm>
+ <primary>troubleshooting</primary>
+ <secondary>timeouts on setup</secondary>
+ </indexterm>Handling Timeouts on Initial Lustre File System Setup</title>
+ <para>If you come across timeouts or hangs on the initial setup of your Lustre file system,
+ verify that name resolution for servers and clients is working correctly. Some distributions
+ configure <literal>/etc/hosts</literal> so the name of the local machine (as reported by the
+ 'hostname' command) is mapped to local host (127.0.0.1) instead of a proper IP
+ address.</para>
<para>This might produce this error:</para>
<screen>LustreError:(ldlm_handle_cancel()) received cancel for unknown lock cookie
0xe74021a4b41b954e from nid 0x7f000001 (0:127.0.0.1)
</section>
<section remap="h3">
<title>Handling/Debugging "LustreError: xxx went back in time"</title>
- <para>Each time Lustre changes the state of the disk file system, it records a unique transaction number. Occasionally, when committing these transactions to the disk, the last committed transaction number displays to other nodes in the cluster to assist the recovery. Therefore, the promised transactions remain absolutely safe on the disappeared disk.</para>
+ <para>Each time the Lustre software changes the state of the disk file system, it records a
+ unique transaction number. Occasionally, when committing these transactions to the disk, the
+ last committed transaction number displays to other nodes in the cluster to assist the
+ recovery. Therefore, the promised transactions remain absolutely safe on the disappeared
+ disk.</para>
<para>This situation arises when:</para>
<itemizedlist>
<listitem>
- <para>You are using a disk device that claims to have data written to disk before it actually does, as in case of a device with a large cache. If that disk device crashes or loses power in a way that causes the loss of the cache, there can be a loss of transactions that you believe are committed. This is a very serious event, and you should run e2fsck against that storage before restarting Lustre.</para>
+ <para>You are using a disk device that claims to have data written to disk before it
+ actually does, as in case of a device with a large cache. If that disk device crashes or
+ loses power in a way that causes the loss of the cache, there can be a loss of
+ transactions that you believe are committed. This is a very serious event, and you
+ should run e2fsck against that storage before restarting the Lustre file system.</para>
</listitem>
<listitem>
- <para>As per the Lustre requirement, the shared storage used for failover is completely cache-coherent. This ensures that if one server takes over for another, it sees the most up-to-date and accurate copy of the data. In case of the failover of the server, if the shared storage does not provide cache coherency between all of its ports, then Lustre can produce an error.</para>
+ <para>As required by the Lustre software, the shared storage used for failover is
+ completely cache-coherent. This ensures that if one server takes over for another, it
+ sees the most up-to-date and accurate copy of the data. In case of the failover of the
+ server, if the shared storage does not provide cache coherency between all of its ports,
+ then the Lustre software can produce an error.</para>
</listitem>
</itemizedlist>
<para>If you know the exact reason for the error, then it is safe to proceed with no further action. If you do not know the reason, then this is a serious issue and you should explore it with your disk vendor.</para>
</itemizedlist>
</section>
<section remap="h3">
- <title><indexterm><primary>troubleshooting</primary><secondary>slowdown during startup</secondary></indexterm>Slowdown Occurs During Lustre Startup</title>
- <para>When Lustre starts, the Lustre file system needs to read in data from the disk. For the very first mdsrate run after the reboot, the MDS needs to wait on all the OSTs for object pre-creation. This causes a slowdown to occur when Lustre starts up.</para>
+ <title><indexterm>
+ <primary>troubleshooting</primary>
+ <secondary>slowdown during startup</secondary>
+ </indexterm>Slowdown Occurs During Lustre File System Startup</title>
+ <para>When a Lustre file system starts, it needs to read in data from the disk. For the very
+ first mdsrate run after the reboot, the MDS needs to wait on all the OSTs for object
+ pre-creation. This causes a slowdown to occur when the file system starts up.</para>
<para>After the file system has been running for some time, it contains more data in cache and hence, the variability caused by reading critical metadata from disk is mostly eliminated. The file system now reads data from the cache.</para>
</section>
<section remap="h3">
<title><indexterm><primary>troubleshooting</primary><secondary>OST out of memory</secondary></indexterm>Log Message <literal>'Out of Memory</literal>' on OST</title>
- <para>When planning the hardware for an OSS node, consider the memory usage of several components in the Lustre system. If insufficient memory is available, an 'out of memory' message can be logged.</para>
+ <para>When planning the hardware for an OSS node, consider the memory usage of several
+ components in the Lustre file system. If insufficient memory is available, an 'out of
+ memory' message can be logged.</para>
<para>During normal operation, several conditions indicate insufficient RAM on a server node:</para>
<itemizedlist>
<listitem>
</section>
<section remap="h3">
<title>Setting SCSI I/O Sizes</title>
- <para>Some SCSI drivers default to a maximum I/O size that is too small for good Lustre performance. we have fixed quite a few drivers, but you may still find that some drivers give unsatisfactory performance with Lustre. As the default value is hard-coded, you need to recompile the drivers to change their default. On the other hand, some drivers may have a wrong default set.</para>
- <para>If you suspect bad I/O performance and an analysis of Lustre statistics indicates that I/O is not 1 MB, check <literal>/sys/block/<device>/queue/max_sectors_kb</literal>. If the <literal>max_sectors_kb</literal> value is less than 1024, set it to at least 1024 to improve performance. If changing <literal>max_sectors_kb</literal> does not change the I/O size as reported by Lustre, you may want to examine the SCSI driver code.</para>
+ <para>Some SCSI drivers default to a maximum I/O size that is too small for good Lustre file
+ system performance. we have fixed quite a few drivers, but you may still find that some
+ drivers give unsatisfactory performance with the Lustre file system. As the default value is
+ hard-coded, you need to recompile the drivers to change their default. On the other hand,
+ some drivers may have a wrong default set.</para>
+ <para>If you suspect bad I/O performance and an analysis of Lustre file system statistics
+ indicates that I/O is not 1 MB, check
+ <literal>/sys/block/<replaceable>device</replaceable>/queue/max_sectors_kb</literal>. If
+ the <literal>max_sectors_kb</literal> value is less than 1024, set it to at least 1024 to
+ improve performance. If changing <literal>max_sectors_kb</literal> does not change the I/O
+ size as reported by the Lustre software, you may want to examine the SCSI driver
+ code.</para>
</section>
</section>
</chapter>
git://git.whamcloud.com / doc / manual.git / blobdiff