<para>When <xref linkend="imperativerecovery"/> is enabled, clients are notified of an MDS restart (either the backup or a restored primary). Clients always may detect an MDS failure either by timeouts of in-flight requests or idle-time ping messages. In either case the clients then connect to the new backup MDS and use the Metadata Replay protocol. Metadata Replay is responsible for ensuring that the backup MDS re-acquires state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk.</para>
<para>The reconnection to a new (or restarted) MDS is managed by the file system configuration loaded by the client when the file system is first mounted. If a failover MDS has been configured (using the <literal>--failnode=</literal> option to <literal>mkfs.lustre</literal> or <literal>tunefs.lustre</literal>), the client tries to reconnect to both the primary and backup MDS until one of them responds that the failed MDT is again available. At that point, the client begins recovery. For more information, see <xref linkend="metadatereplay"/>.</para>
<para>Transaction numbers are used to ensure that operations are replayed in the order they were originally performed, so that they are guaranteed to succeed and present the same filesystem state as before the failure. In addition, clients inform the new server of their existing lock state (including locks that have not yet been granted). All metadata and lock replay must complete before new, non-recovery operations are permitted. In addition, only clients that were connected at the time of MDS failure are permitted to reconnect during the recovery window, to avoid the introduction of state changes that might conflict with what is being replayed by previously-connected clients.</para>
+ <para condition='l24'>Lustre 2.4 introduces multiple metadata targets. If multiple metadata targets are in use, active-active failover is possible. See <xref linkend='dbdoclet.mdtactiveactive'/> for more information.</para>
</section>
<section remap="h3">
<title><indexterm><primary>recovery</primary><secondary>OST failure</secondary></indexterm>OST Failure (Failover)</title>
<para>To force an OST recovery, unmount the OST and then mount it again. If the OST was connected to clients before it failed, then a recovery process starts after the remount, enabling clients to reconnect to the OST and replay transactions in their queue. When the OST is in recovery mode, all new client connections are refused until the recovery finishes. The recovery is complete when either all previously-connected clients reconnect and their transactions are replayed or a client connection attempt times out. If a connection attempt times out, then all clients waiting to reconnect (and their transactions) are lost.</para>
<note>
<para>If you know an OST will not recover a previously-connected client (if, for example, the client has crashed), you can manually abort the recovery using this command:</para>
- <para><screen>lctl --device <OST device number> abort_recovery</screen></para>
+ <para><screen>oss# lctl --device <replaceable>lustre_device_number</replaceable> abort_recovery</screen></para>
<para>To determine an OST's device number and device name, run the <literal>lctl dl</literal> command. Sample <literal>lctl dl</literal> command output is shown below:</para>
<screen>7 UP obdfilter ddn_data-OST0009 ddn_data-OST0009_UUID 1159 </screen>
<para>In this example, 7 is the OST device number. The device name is <literal>ddn_data-OST0009</literal>. In most instances, the device name can be used in place of the device number.</para>
</section>
</section>
+ <section xml:id="suppressingpings">
+ <title><indexterm><primary>suppressing pings</primary></indexterm>Suppressing Pings</title>
+ <para>On clusters with large numbers of clients and OSTs, OBD_PING messages may impose significant performance overheads. As an intermediate solution before a more self-contained one is built, Lustre 2.4 introduces an option to suppress pings, allowing ping overheads to be considerably reduced. Before turning on this option, administrators should consider the following requirements and understand the trade-offs involved:</para>
+ <itemizedlist>
+ <listitem>
+ <para>When suppressing pings, a target can not detect client deaths, since clients do not send pings that are only to keep their connections alive. Therefore, a mechanism external to Lustre shall be set up to notify Lustre targets of client deaths in a timely manner, so that stale connections do not exist for too long and lock callbacks to dead clients do not always have to wait for timeouts.</para>
+ </listitem>
+ <listitem>
+ <para>Without pings, a client has to rely on Imperative Recovery to notify it of target failures, in order to join recoveries in time. This dictates that the client shall eargerly keep its MGS connection alive. Thus, a highly available standalone MGS is recommended and, on the other hand, MGS pings are always sent regardless of how the option is set.</para>
+ </listitem>
+ <listitem>
+ <para>If a client has uncommitted requests to a target and it is not sending any new requests on the connection, it will still ping that target even when pings should be suppressed. This is because the client needs to query the target's last committed transaction numbers in order to free up local uncommitted requests (and possibly other resources associated). However, these pings shall stop as soon as all the uncommitted requests have been freed or new requests need to be sent, rendering the pings unnecessary.</para>
+ </listitem>
+ </itemizedlist>
+ <section remap="h3">
+ <title><indexterm><primary>pings</primary><secondary>suppress_pings</secondary></indexterm>"suppress_pings" Kernel Module Parameter</title>
+ <para>The new option that controls whether pings are suppressed is implemented as the ptlrpc kernel module parameter "suppress_pings". Setting it to "1" on a server turns on ping suppressing for all targets on that server, while leaving it with the default value "0" gives previous pinging behavior. The parameter is ignored on clients and the MGS. While the parameter is recommended to be set persistently via the modprobe.conf(5) mechanism, it also accept online changes through sysfs. Note that an online change only affects connections established later; existing connections' pinging behaviors stay the same.</para>
+ </section>
+ <section remap="h3">
+ <title><indexterm><primary>pings</primary><secondary>evict_client</secondary></indexterm>Client Death Notification</title>
+ <para>The required external client death notification shall write UUIDs of dead clients into targets' "evict_client" procfs entries like</para>
+ <screen>
+/proc/fs/lustre/obdfilter/testfs-OST0000/evict_client
+/proc/fs/lustre/obdfilter/testfs-OST0001/evict_client
+/proc/fs/lustre/mdt/testfs-MDT0000/evict_client
+ </screen>
+ <para>Clients' UUIDs can be obtained from their "uuid" procfs entries like</para>
+ <screen>
+/proc/fs/lustre/llite/testfs-ffff8800612bf800/uuid
+ </screen>
+ </section>
+ </section>
+
</chapter>
git://git.whamcloud.com / doc / manual.git / blobdiff