-<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustrerecovery">
+<?xml version='1.0' encoding='UTF-8'?>
+<chapter xmlns="http://docbook.org/ns/docbook"
+ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
+ xml:id="lustrerecovery">
<title xml:id="lustrerecovery.title">Lustre File System Recovery</title>
<para>This chapter describes how recovery is implemented in a Lustre file system and includes the
following sections:</para>
<para> Transient network partition</para>
</listitem>
</itemizedlist>
- <para>For Lustre software release 2.1.x and all earlier releases, all Lustre file system failure
- and recovery operations are based on the concept of connection failure; all imports or exports
- associated with a given connection are considered to fail if any of them fail. Lustre software
- release 2.2.x adds the <xref linkend="imperativerecovery"/> feature which enables the MGS to
- actively inform clients when a target restarts after a failure, failover or other
- interruption.</para>
- <para>For information on Lustre file system recovery, see <xref linkend="metadatereplay"/>. For
- information on recovering from a corrupt file system, see <xref linkend="commitonshare"/>. For
- information on resolving orphaned objects, a common issue after recovery, see <xref
- linkend="dbdoclet.50438225_13916"/>. For information on imperative recovery see <xref
- linkend="imperativerecovery"/>
+ <para>For Lustre, all Lustre file system failure and recovery operations
+ are based on the concept of connection failure; all imports or exports
+ associated with a given connection are considered to fail if any of
+ them fail. The <xref linkend="imperativerecovery"/> feature allows
+ the MGS to actively inform clients when a target restarts after a
+ failure, failover, or other interruption to speed up recovery.</para>
+ <para>For information on Lustre file system recovery, see
+ <xref linkend="metadatereplay"/>. For information on recovering from a
+ corrupt file system, see <xref linkend="commitonshare"/>. For
+ information on resolving orphaned objects, a common issue after recovery,
+ see <xref linkend="orphan_objects"/>. For information on
+ imperative recovery see <xref linkend="imperativerecovery"/>
</para>
<section remap="h3">
<title><indexterm><primary>recovery</primary><secondary>client failure</secondary></indexterm>Client Failure</title>
recovery will take as long as is needed for the single MDS to be restarted.</para>
<para>When <xref linkend="imperativerecovery"/> is enabled, clients are notified of an MDS restart (either the backup or a restored primary). Clients always may detect an MDS failure either by timeouts of in-flight requests or idle-time ping messages. In either case the clients then connect to the new backup MDS and use the Metadata Replay protocol. Metadata Replay is responsible for ensuring that the backup MDS re-acquires state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk.</para>
<para>The reconnection to a new (or restarted) MDS is managed by the file system configuration loaded by the client when the file system is first mounted. If a failover MDS has been configured (using the <literal>--failnode=</literal> option to <literal>mkfs.lustre</literal> or <literal>tunefs.lustre</literal>), the client tries to reconnect to both the primary and backup MDS until one of them responds that the failed MDT is again available. At that point, the client begins recovery. For more information, see <xref linkend="metadatereplay"/>.</para>
- <para>Transaction numbers are used to ensure that operations are replayed in the order they
- were originally performed, so that they are guaranteed to succeed and present the same file
- system state as before the failure. In addition, clients inform the new server of their
- existing lock state (including locks that have not yet been granted). All metadata and lock
- replay must complete before new, non-recovery operations are permitted. In addition, only
- clients that were connected at the time of MDS failure are permitted to reconnect during the
- recovery window, to avoid the introduction of state changes that might conflict with what is
- being replayed by previously-connected clients.</para>
- <para condition="l24">Lustre software release 2.4 introduces multiple metadata targets. If
- multiple metadata targets are in use, active-active failover is possible. See <xref
- linkend="dbdoclet.mdtactiveactive"/> for more information.</para>
+ <para>Transaction numbers are used to ensure that operations are
+ replayed in the order they were originally performed, so that they
+ are guaranteed to succeed and present the same file system state as
+ before the failure. In addition, clients inform the new server of their
+ existing lock state (including locks that have not yet been granted).
+ All metadata and lock replay must complete before new, non-recovery
+ operations are permitted. In addition, only clients that were connected
+ at the time of MDS failure are permitted to reconnect during the recovery
+ window, to avoid the introduction of state changes that might conflict
+ with what is being replayed by previously-connected clients.</para>
+ <para>If multiple MDTs are in use, active-active failover
+ is possible (e.g. two MDS nodes, each actively serving one or more
+ different MDTs for the same filesystem). See
+ <xref linkend="dbdoclet.mdtactiveactive"/> for more information.</para>
</section>
<section remap="h3">
<title><indexterm><primary>recovery</primary><secondary>OST failure</secondary></indexterm>OST Failure (Failover)</title>
</section>
<section xml:id="imperativerecovery">
<title><indexterm><primary>imperative recovery</primary></indexterm>Imperative Recovery</title>
- <para>Imperative Recovery (IR) was first introduced in Lustre software release 2.2.0.</para>
- <para>Large-scale Lustre file system implementations have historically experienced problems
- recovering in a timely manner after a server failure. This is due to the way that clients
- detect the server failure and how the servers perform their recovery. Many of the processes
- are driven by the RPC timeout, which must be scaled with system size to prevent false
- diagnosis of server death. The purpose of imperative recovery is to reduce the recovery window
- by actively informing clients of server failure. The resulting reduction in the recovery
- window will minimize target downtime and therefore increase overall system availability.
+ <para>Large-scale Lustre filesystems will experience server hardware
+ failures over their lifetime, and it is important that servers can
+ recover in a timely manner after such failures. High Availability
+ software can move storage targets over to a backup server automatically.
+ Clients can detect the server failure by RPC timeouts, which must be
+ scaled with system size to prevent false diagnosis of server death in
+ cases of heavy load. The purpose of imperative recovery is to reduce
+ the recovery window by actively informing clients of server failure.
+ The resulting reduction in the recovery window will minimize target
+ downtime and therefore increase overall system availability.</para>
+ <para>
Imperative Recovery does not remove previous recovery mechanisms, and client timeout-based
recovery actions can occur in a cluster when IR is enabled as each client can still
independently disconnect and reconnect from a target. In case of a mix of IR and non-IR
$ lctl get_param osc.testfs-OST0001-osc-*.import |grep instance
instance: 5
</screen>
- </section>
- </section>
- <section remap="h3" xml:id="imperativerecoveryrecomendations">
- <title><indexterm><primary>imperative recovery</primary><secondary>Configuration Suggestions</secondary></indexterm>Configuration Suggestions for Imperative Recovery</title>
-<para>We used to build the MGS and MDT0 on the same target to save a server node. However, to make
- IR work efficiently, we strongly recommend running the MGS node on a separate node for any
- significant Lustre file system installation. There are three main advantages of doing this: </para>
-<orderedlist>
-<listitem><para>Be able to notify clients if the MDT0 is dead</para></listitem>
-<listitem><para>Load balance. The load on the MDS may be very high which may make the MGS unable to notify the clients in time</para></listitem>
-<listitem><para>Safety. The MGS code is simpler and much smaller compared to the code of MDT. This means the chance of MGS down time due to a software bug is very low.</para></listitem>
-</orderedlist>
- </section>
+ </section>
+ </section>
+ <section remap="h3" xml:id="imperativerecoveryrecomendations">
+ <title><indexterm><primary>imperative recovery</primary><secondary>Configuration Suggestions</secondary></indexterm>Configuration Suggestions for Imperative Recovery</title>
+ <para>We used to build the MGS and MDT0000 on the same target to save
+ a server node. However, to make IR work efficiently, we strongly
+ recommend running the MGS node on a separate node for any
+ significant Lustre file system installation. There are three main
+ advantages of doing this: </para>
+ <orderedlist>
+ <listitem><para>Be able to notify clients when MDT0000 recovered.
+ </para></listitem>
+ <listitem><para>Improved load balance. The load on the MDS may be
+ very high which may make the MGS unable to notify the clients in
+ time.</para></listitem>
+ <listitem><para>Robustness. The MGS code is simpler and much smaller
+ compared to the MDS code. This means the chance of an MGS downtime
+ due to a software bug is very low.
+ </para></listitem>
+ </orderedlist>
+ </section>
</section>
<section xml:id="suppressingpings">
<title><indexterm><primary>suppressing pings</primary></indexterm>Suppressing Pings</title>
- <para>On clusters with large numbers of clients and OSTs, OBD_PING messages may impose
- significant performance overheads. As an intermediate solution before a more self-contained
- one is built, Lustre software release 2.4 introduces an option to suppress pings, allowing
- ping overheads to be considerably reduced. Before turning on this option, administrators
- should consider the following requirements and understand the trade-offs involved:</para>
+ <para>On clusters with large numbers of clients and OSTs,
+ <literal>OBD_PING</literal> messages may impose significant performance
+ overheads. There is an option to suppress pings, allowing ping overheads
+ to be considerably reduced. Before turning on this option, administrators
+ should consider the following requirements and understand the trade-offs
+ involved:</para>
<itemizedlist>
<listitem>
- <para>When suppressing pings, a target can not detect client deaths, since clients do not
- send pings that are only to keep their connections alive. Therefore, a mechanism external
- to the Lustre file system shall be set up to notify Lustre targets of client deaths in a
- timely manner, so that stale connections do not exist for too long and lock callbacks to
+ <para>When suppressing pings, a server cannot detect client deaths,
+ since clients do not send pings that are only to keep their
+ connections alive. Therefore, a mechanism external to the Lustre
+ file system shall be set up to notify Lustre targets of client
+ deaths in a timely manner, so that stale connections do not exist
+ for too long and lock callbacks to
dead clients do not always have to wait for timeouts.</para>
</listitem>
<listitem>
</itemizedlist>
<section remap="h3">
<title><indexterm><primary>pings</primary><secondary>suppress_pings</secondary></indexterm>"suppress_pings" Kernel Module Parameter</title>
- <para>The new option that controls whether pings are suppressed is implemented as the ptlrpc kernel module parameter "suppress_pings". Setting it to "1" on a server turns on ping suppressing for all targets on that server, while leaving it with the default value "0" gives previous pinging behavior. The parameter is ignored on clients and the MGS. While the parameter is recommended to be set persistently via the modprobe.conf(5) mechanism, it also accept online changes through sysfs. Note that an online change only affects connections established later; existing connections' pinging behaviors stay the same.</para>
+ <para>The new option that controls whether pings are suppressed is
+ implemented as the ptlrpc kernel module parameter "suppress_pings".
+ Setting it to "1" on a server turns on ping suppressing for all
+ targets on that server, while leaving it with the default value "0"
+ gives previous pinging behavior. The parameter is ignored on clients
+ and the MGS. While the parameter is recommended to be set persistently
+ via the modprobe.conf(5) mechanism, it also accept online changes
+ through sysfs. Note that an online change only affects connections
+ established later; existing connections' pinging behaviors stay the same.
+ </para>
</section>
<section remap="h3">
<title><indexterm><primary>pings</primary><secondary>evict_client</secondary></indexterm>Client Death Notification</title>
- <para>The required external client death notification shall write UUIDs of dead clients into targets' "evict_client" procfs entries like</para>
- <screen>
-/proc/fs/lustre/obdfilter/testfs-OST0000/evict_client
-/proc/fs/lustre/obdfilter/testfs-OST0001/evict_client
-/proc/fs/lustre/mdt/testfs-MDT0000/evict_client
- </screen>
- <para>Clients' UUIDs can be obtained from their "uuid" procfs entries like</para>
- <screen>
-/proc/fs/lustre/llite/testfs-ffff8800612bf800/uuid
- </screen>
+ <para>The required external client death notification shall write UUIDs
+ of dead clients into targets' <literal>evict_client</literal> procfs
+ entries in order to remove stale clients from recovery.</para>
+ <para>A client UUID can be obtained from their <literal>uuid</literal>
+ procfs entry and that UUID can be used to evict the client, like:
+ </para>
+<screen>
+client$ lctl get_param llite.testfs-*.uuid
+llite.testfs-ffff991ae1992000.uuid=dd599d28-0a85-a9e4-82cd-dc6357a42c77
+oss# lctl set_param obdfilter.testfs-*.evict_client=dd599d28-0a85-a9e4-82cd-dc6357a42c77
+mds# lctl set_param mdt.testfs-*.evict_client=dd599d28-0a85-a9e4-82cd-dc6357a42c77
+</screen>
</section>
</section>
</chapter>
+<!--
+ vim:expandtab:shiftwidth=2:tabstop=8:
+ -->
git://git.whamcloud.com / doc / manual.git / blobdiff