-<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustrerecovery">
+<?xml version='1.0' encoding='UTF-8'?>
+<chapter xmlns="http://docbook.org/ns/docbook"
+ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
+ xml:id="lustrerecovery">
<title xml:id="lustrerecovery.title">Lustre File System Recovery</title>
<para>This chapter describes how recovery is implemented in a Lustre file system and includes the
following sections:</para>
<para> Transient network partition</para>
</listitem>
</itemizedlist>
- <para>For Lustre software release 2.1.x and all earlier releases, all Lustre file system failure
- and recovery operations are based on the concept of connection failure; all imports or exports
- associated with a given connection are considered to fail if any of them fail. Lustre software
- release 2.2.x adds the <xref linkend="imperativerecovery"/> feature which enables the MGS to
- actively inform clients when a target restarts after a failure, failover or other
- interruption.</para>
- <para>For information on Lustre file system recovery, see <xref linkend="metadatereplay"/>. For
- information on recovering from a corrupt file system, see <xref linkend="commitonshare"/>. For
- information on resolving orphaned objects, a common issue after recovery, see <xref
- linkend="dbdoclet.50438225_13916"/>. For information on imperative recovery see <xref
- linkend="imperativerecovery"/>
+ <para>For Lustre, all Lustre file system failure and recovery operations
+ are based on the concept of connection failure; all imports or exports
+ associated with a given connection are considered to fail if any of
+ them fail. The <xref linkend="imperativerecovery"/> feature allows
+ the MGS to actively inform clients when a target restarts after a
+ failure, failover, or other interruption to speed up recovery.</para>
+ <para>For information on Lustre file system recovery, see
+ <xref linkend="metadatereplay"/>. For information on recovering from a
+ corrupt file system, see <xref linkend="commitonshare"/>. For
+ information on resolving orphaned objects, a common issue after recovery,
+ see <xref linkend="dbdoclet.50438225_13916"/>. For information on
+ imperative recovery see <xref linkend="imperativerecovery"/>
</para>
<section remap="h3">
<title><indexterm><primary>recovery</primary><secondary>client failure</secondary></indexterm>Client Failure</title>
recovery will take as long as is needed for the single MDS to be restarted.</para>
<para>When <xref linkend="imperativerecovery"/> is enabled, clients are notified of an MDS restart (either the backup or a restored primary). Clients always may detect an MDS failure either by timeouts of in-flight requests or idle-time ping messages. In either case the clients then connect to the new backup MDS and use the Metadata Replay protocol. Metadata Replay is responsible for ensuring that the backup MDS re-acquires state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk.</para>
<para>The reconnection to a new (or restarted) MDS is managed by the file system configuration loaded by the client when the file system is first mounted. If a failover MDS has been configured (using the <literal>--failnode=</literal> option to <literal>mkfs.lustre</literal> or <literal>tunefs.lustre</literal>), the client tries to reconnect to both the primary and backup MDS until one of them responds that the failed MDT is again available. At that point, the client begins recovery. For more information, see <xref linkend="metadatereplay"/>.</para>
- <para>Transaction numbers are used to ensure that operations are replayed in the order they
- were originally performed, so that they are guaranteed to succeed and present the same file
- system state as before the failure. In addition, clients inform the new server of their
- existing lock state (including locks that have not yet been granted). All metadata and lock
- replay must complete before new, non-recovery operations are permitted. In addition, only
- clients that were connected at the time of MDS failure are permitted to reconnect during the
- recovery window, to avoid the introduction of state changes that might conflict with what is
- being replayed by previously-connected clients.</para>
- <para condition="l24">Lustre software release 2.4 introduces multiple metadata targets. If
- multiple metadata targets are in use, active-active failover is possible. See <xref
- linkend="dbdoclet.mdtactiveactive"/> for more information.</para>
+ <para>Transaction numbers are used to ensure that operations are
+ replayed in the order they were originally performed, so that they
+ are guaranteed to succeed and present the same file system state as
+ before the failure. In addition, clients inform the new server of their
+ existing lock state (including locks that have not yet been granted).
+ All metadata and lock replay must complete before new, non-recovery
+ operations are permitted. In addition, only clients that were connected
+ at the time of MDS failure are permitted to reconnect during the recovery
+ window, to avoid the introduction of state changes that might conflict
+ with what is being replayed by previously-connected clients.</para>
+ <para>If multiple MDTs are in use, active-active failover
+ is possible (e.g. two MDS nodes, each actively serving one or more
+ different MDTs for the same filesystem). See
+ <xref linkend="dbdoclet.mdtactiveactive"/> for more information.</para>
</section>
<section remap="h3">
<title><indexterm><primary>recovery</primary><secondary>OST failure</secondary></indexterm>OST Failure (Failover)</title>
<para>The lock handle can be found by walking the list of granted locks for the resource looking for one with the appropriate remote file handle (present in the re-sent request). Verify that the lock has the right mode (determined by performing the disposition/request/status analysis above) and is granted to the proper client.</para>
</section>
</section>
+ <section remap="h3" condition="l28">
+ <title>Multiple Reply Data per Client</title>
+ <para>Since Lustre 2.8, the MDS is able to save several reply data per client. The reply data are stored in the <literal>reply_data</literal> internal file of the MDT. Additionally to the XID of the request, the transaction number, the result code and the open "disposition", the reply data contains a generation number that identifies the client thanks to the content of the <literal>last_rcvd</literal> file.</para>
+ </section>
</section>
<section xml:id="versionbasedrecovery">
<title><indexterm><primary>Version-based recovery (VBR)</primary></indexterm>Version-based Recovery</title>
</section>
<section xml:id="imperativerecovery">
<title><indexterm><primary>imperative recovery</primary></indexterm>Imperative Recovery</title>
- <para>Imperative Recovery (IR) was first introduced in Lustre software release 2.2.0.</para>
<para>Large-scale Lustre file system implementations have historically experienced problems
recovering in a timely manner after a server failure. This is due to the way that clients
detect the server failure and how the servers perform their recovery. Many of the processes
$ lctl get_param osc.testfs-OST0001-osc-*.import |grep instance
instance: 5
</screen>
- </section>
- </section>
- <section remap="h3" xml:id="imperativerecoveryrecomendations">
- <title><indexterm><primary>imperative recovery</primary><secondary>Configuration Suggestions</secondary></indexterm>Configuration Suggestions for Imperative Recovery</title>
-<para>We used to build the MGS and MDT0 on the same target to save a server node. However, to make
- IR work efficiently, we strongly recommend running the MGS node on a separate node for any
- significant Lustre file system installation. There are three main advantages of doing this: </para>
-<orderedlist>
-<listitem><para>Be able to notify clients if the MDT0 is dead</para></listitem>
-<listitem><para>Load balance. The load on the MDS may be very high which may make the MGS unable to notify the clients in time</para></listitem>
-<listitem><para>Safety. The MGS code is simpler and much smaller compared to the code of MDT. This means the chance of MGS down time due to a software bug is very low.</para></listitem>
-</orderedlist>
- </section>
+ </section>
+ </section>
+ <section remap="h3" xml:id="imperativerecoveryrecomendations">
+ <title><indexterm><primary>imperative recovery</primary><secondary>Configuration Suggestions</secondary></indexterm>Configuration Suggestions for Imperative Recovery</title>
+ <para>We used to build the MGS and MDT0000 on the same target to save
+ a server node. However, to make IR work efficiently, we strongly
+ recommend running the MGS node on a separate node for any
+ significant Lustre file system installation. There are three main
+ advantages of doing this: </para>
+ <orderedlist>
+ <listitem><para>Be able to notify clients when MDT0000 recovered.
+ </para></listitem>
+ <listitem><para>Improved load balance. The load on the MDS may be
+ very high which may make the MGS unable to notify the clients in
+ time.</para></listitem>
+ <listitem><para>Robustness. The MGS code is simpler and much smaller
+ compared to the MDS code. This means the chance of an MGS downtime
+ due to a software bug is very low.
+ </para></listitem>
+ </orderedlist>
+ </section>
</section>
<section xml:id="suppressingpings">
<title><indexterm><primary>suppressing pings</primary></indexterm>Suppressing Pings</title>
- <para>On clusters with large numbers of clients and OSTs, OBD_PING messages may impose
- significant performance overheads. As an intermediate solution before a more self-contained
- one is built, Lustre software release 2.4 introduces an option to suppress pings, allowing
- ping overheads to be considerably reduced. Before turning on this option, administrators
- should consider the following requirements and understand the trade-offs involved:</para>
+ <para>On clusters with large numbers of clients and OSTs,
+ <literal>OBD_PING</literal> messages may impose significant performance
+ overheads. There is an option to suppress pings, allowing ping overheads
+ to be considerably reduced. Before turning on this option, administrators
+ should consider the following requirements and understand the trade-offs
+ involved:</para>
<itemizedlist>
<listitem>
- <para>When suppressing pings, a target can not detect client deaths, since clients do not
- send pings that are only to keep their connections alive. Therefore, a mechanism external
- to the Lustre file system shall be set up to notify Lustre targets of client deaths in a
- timely manner, so that stale connections do not exist for too long and lock callbacks to
+ <para>When suppressing pings, a server cannot detect client deaths,
+ since clients do not send pings that are only to keep their
+ connections alive. Therefore, a mechanism external to the Lustre
+ file system shall be set up to notify Lustre targets of client
+ deaths in a timely manner, so that stale connections do not exist
+ for too long and lock callbacks to
dead clients do not always have to wait for timeouts.</para>
</listitem>
<listitem>
</section>
</chapter>
+<!--vim:expandtab:shiftwidth=2:tabstop=8:-->
git://git.whamcloud.com / doc / manual.git / blobdiff