From 0c9dbf5250723806a09535f56bdbdd313a3e2612 Mon Sep 17 00:00:00 2001 From: Linda Bebernes Date: Wed, 30 Oct 2013 20:04:02 -0700 Subject: [PATCH] LUDOC-69 failover: Updated failover documentation globally across 6 chapters. Ch 3 Intro to Failover - edits for clarity, fixed missing figure Ch 11 Configuring Lustre Failover - major rewrite to update and clarify content Ch 13 Lustre Operations - edited failover-related entries for clarity, updated example from Elan to Ethernet, added cross-ref to Ch 11 Ch 14 Lustre Maintenance - edited failover-related entries for clarity, added crossref to Ch 11 Ch 20 MMP - changed chapter name from "Managing Failover" to "Lustre Failover and Multi-Mount Protection", minor edits, added xref to Ch 11 Ch 36 - updated --servicenode and --failnode descriptons for mkfs.lustre and tunefs.lustre Changes to address Intel trademark compliance have also been made in these chapters. Signed-off-by: Linda Bebernes Change-Id: I6f9b9f0b84d01e4d590c290dbc30738f5a0688cd Reviewed-on: http://review.whamcloud.com/8058 Tested-by: Hudson Reviewed-by: Richard Henwood --- ConfiguringFailover.xml | 214 ++++++++++++++++++++++++++++++++------- LustreMaintenance.xml | 14 ++- LustreOperations.xml | 94 +++++++++++------ ManagingFailover.xml | 76 +++++++------- SystemConfigurationUtilities.xml | 101 +++++++++++------- UnderstandingFailover.xml | 85 ++++++++++------ figures/MDTs_Failover.png | Bin 0 -> 18788 bytes figures/MDTs_Failover.svg | 209 ++++++++++++++++++++++++++++++++++++++ 8 files changed, 613 insertions(+), 180 deletions(-) create mode 100644 figures/MDTs_Failover.png create mode 100644 figures/MDTs_Failover.svg diff --git a/ConfiguringFailover.xml b/ConfiguringFailover.xml index 3b89a30..dc3511d 100644 --- a/ConfiguringFailover.xml +++ b/ConfiguringFailover.xml @@ -1,54 +1,192 @@ - Configuring Lustre Failover - This chapter describes how to configure Lustre failover using the Heartbeat cluster infrastructure daemon. It includes: + Configuring Failover in a Lustre File System + This chapter describes how to configure failover in a Lustre file system. It + includes: - + + - + + + + - - Using Lustre Failover is optional - - + For an overview of failover functionality in a Lustre file system, see .
- <indexterm><primary>High availability</primary><see>failover</see></indexterm><indexterm><primary>failover</primary></indexterm>Creating a Failover Environment - Lustre provides failover mechanisms only at the file system level. No failover functionality is provided for system-level components, such as node failure detection or power control, as would typically be provided in a complete failover solution. Additional tools are also needed to provide resource fencing, control and monitoring. + <indexterm> + <primary>High availability</primary> + <see>failover</see> + </indexterm><indexterm> + <primary>failover</primary> + </indexterm>Setting Up a Failover Environment + The Lustre software provides failover mechanisms only at the layer of the Lustre file + system. No failover functionality is provided for system-level components such as failing + hardware or applications, or even for the entire failure of a node, as would typically be + provided in a complete failover solution. Failover functionality such as node monitoring, + failure detection, and resource fencing must be provided by external HA software, such as + PowerMan or the open source Corosync and Pacemaker packages provided by Linux operating system + vendors. Corosync provides support for detecting failures, and Pacemaker provides the actions + to take once a failure has been detected.
- <indexterm><primary>failover</primary><secondary>power management</secondary></indexterm>Power Management Software - Lustre failover requires power control and management capability to verify that a failed node is shut down before I/O is directed to the failover node. This avoids double-mounting the two nodes, and the risk of unrecoverable data corruption. A variety of power management tools will work, but two packages that are commonly used with Lustre are STONITH and PowerMan. - Shoot The Other Node In The HEAD (STONITH), is a set of power management tools provided with the Linux-HA package. STONITH has native support for many power control devices and is extensible. It uses expect scripts to automate control. - PowerMan, available from the Lawrence Livermore National Laboratory (LLNL), is used to control remote power control (RPC) devices from a central location. PowerMan provides native support for several RPC varieties and expect-like configuration simplifies the addition of new devices. - The latest versions of PowerMan are available at: - http://sourceforge.net/projects/powerman - For more information about PowerMan, go to: - https://computing.llnl.gov/linux/powerman.html + <indexterm> + <primary>failover</primary> + <secondary>power control device</secondary> + </indexterm>Selecting Power Equipment + Failover in a Lustre file system requires the use of a remote power control (RPC) + mechanism, which comes in different configurations. For example, Lustre server nodes may be + equipped with IPMI/BMC devices that allow remote power control. In the past, software or + even “sneakerware” has been used, but these are not recommended. For recommended devices, + refer to the list of supported RPC devices on the website for the PowerMan cluster power + management utility: + http://code.google.com/p/powerman/wiki/SupportedDevs
- <indexterm><primary>failover</primary><secondary>power equipment</secondary></indexterm>Power Equipment - Lustre failover also requires the use of RPC devices, which come in different configurations. Lustre server nodes may be equipped with some kind of service processor that allows remote power control. If a Lustre server node is not equipped with a service processor, then a multi-port, Ethernet-addressable RPC may be used as an alternative. For recommended products, refer to the list of supported RPC devices on the PowerMan website. - https://computing.llnl.gov/linux/powerman.html + <indexterm> + <primary>failover</primary> + <secondary>power management software</secondary> + </indexterm>Selecting Power Management Software + Lustre failover requires RPC and management capability to verify that a failed node is + shut down before I/O is directed to the failover node. This avoids double-mounting the two + nodes and the risk of unrecoverable data corruption. A variety of power management tools + will work. Two packages that have been commonly used with the Lustre software are PowerMan + and Linux-HA (aka. STONITH ). + The PowerMan cluster power management utility is used to control RPC devices from a + central location. PowerMan provides native support for several RPC varieties and Expect-like + configuration simplifies the addition of new devices. The latest versions of PowerMan are + available at: + http://code.google.com/p/powerman/ + STONITH, or “Shoot The Other Node In The Head”, is a set of power management tools + provided with the Linux-HA package prior to Red Hat Enterprise Linux 6. Linux-HA has native + support for many power control devices, is extensible (uses Expect scripts to automate + control), and provides the software to detect and respond to failures. With Red Hat + Enterprise Linux 6, Linux-HA is being replaced in the open source community by the + combination of Corosync and Pacemaker. For Red Hat Enterprise Linux subscribers, cluster + management using CMAN is available from Red Hat. +
+
+ <indexterm> + <primary>failover</primary> + <secondary>high-availability (HA) software</secondary> + </indexterm>Selecting High-Availability (HA) Software + The Lustre file system must be set up with high-availability (HA) software to enable a + complete Lustre failover solution. Except for PowerMan, the HA software packages mentioned + above provide both power management and cluster management. For information about setting + up failover with Pacemaker, see: + + + Pacemaker Project website: http://clusterlabs.org/ + + + Article Using Pacemaker with a Lustre File + System: https://wiki.hpdd.intel.com/display/PUB/Using+Pacemaker+with+a+Lustre+File+System + +
- <indexterm><primary>failover</primary><secondary>setup</secondary></indexterm>Setting up High-Availability (HA) Software with Lustre - Lustre must be combined with high-availability (HA) software to enable a complete Lustre failover solution. Lustre can be used with several HA packages including: - - - Red Hat*Cluster Manager - For more - information about Red Hat Cluster Manager, see http://www.redhat.com/software/rha/cluster/manager/. - - - Pacemaker - For more information about Pacemaker, see http://clusterlabs.org/doc/en-US/Pacemaker/1.0/html/Pacemaker_Explained/index.html. - - + <indexterm> + <primary>failover</primary> + <secondary>setup</secondary> + </indexterm>Preparing a Lustre File System for Failover + To prepare a Lustre file system to be configured and managed as an HA system by a + third-party HA application, each storage target (MGT, MGS, OST) must be associated with a + second node to create a failover pair. This configuration information is then communicated by + the MGS to a client when the client mounts the file system. + The per-target configuration is relayed to the MGS at mount time. Some rules related to + this are: + + When a target is initially mounted, the MGS reads the configuration + information from the target (such as mgt vs. ost, failnode, fsname) to configure the + target into a Lustre file system. If the MGS is reading the initial mount configuration, + the mounting node becomes that target's “primary” node. + + + When a target is subsequently mounted, the MGS reads the current configuration + from the target and, as needed, will reconfigure the MGS database target + information + + + When the target is formatted using the mkfs.lustrecommand, the failover + service node(s) for the target are designated using the --servicenode + option. In the example below, an OST with index 0 in the file system + testfs is formatted with two service nodes designated to serve as a + failover + pair:mkfs.lustre --reformat --ost --fsname testfs --mgsnode=192.168.10.1@o3ib \ + --index=0 --servicenode=192.168.10.7@o2ib \ + --servicenode=192.168.10.8@o2ib \ + /dev/sdb + More than two potential service nodes caan be designated for a target. The target can then + be mounted on any of the designated service nodes. + When HA is configured on a storage target, the Lustre software enables multi-mount + protection (MMP) on that storage target. MMP prevents multiple nodes from simultaneously + mounting and thus corrupting the data on the target. For more about MMP, see . + If the MGT has been formatted with multiple service nodes designated, this information + must be conveyed to the Lustre client in the mount command used to mount the file system. In + the example below, NIDs for two MGSs that have been designated as service nodes for the MGT + are specified in the mount command executed on the + client:mount -t lustre 10.10.120.1@tcp1:10.10.120.2@tcp1:/testfs /lustre/testfs + When a client mounts the file system, the MGS provides configuration information to the + client for the MDT(s) and OST(s) in the file system along with the NIDs for all service nodes + associated with each target and the service node on which the target is mounted. Later, when + the client attempts to access data on a target, it will try the NID for each specified service + node until it connects to the target. + Previous to Lustre software release 2.0, the --failnode option to + mkfs.lustre was used to designate a failover service node for a primary + server for a target. When the --failnode option is used, certain + restrictions apply: + + The target must be initially mounted on the primary service node, not the failover + node designated by the --failnode option. + + + If the tunefs.lustre –-writeconf option is used to erase and + regenerate the configuration log for the file system, a target cannot be initially + mounted on a designated failnode. + + + If a --failnode option is added to a target to designate a + failover server for the target, the target must be re-mounted on the primary node before + the --failnode option takes effect + + +
+
+ Administering Failover in a Lustre File System + For additional information about administering failover features in a Lustre file system, see: + + + + + + + + + + + + +
diff --git a/LustreMaintenance.xml b/LustreMaintenance.xml index 5533df0..841bd5f 100644 --- a/LustreMaintenance.xml +++ b/LustreMaintenance.xml @@ -599,11 +599,15 @@ osc.testfs-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
<indexterm><primary>maintenance</primary><secondary>changing failover node address</secondary></indexterm> Changing the Address of a Failover Node - To change the address of a failover node (e.g, to use node X instead of node Y), run this command on the OSS/OST partition: - oss# tunefs.lustre --erase-params --failnode=NID /dev/ost_device - or - oss# tunefs.lustre --erase-params --servicenode=NID /dev/ost_device - + To change the address of a failover node (e.g, to use node X instead of node Y), run + this command on the OSS/OST partition (depending on which option was used to originally + identify the NID): + oss# tunefs.lustre --erase-params --servicenode=NID /dev/ost_device + or + oss# tunefs.lustre --erase-params --failnode=NID /dev/ost_device + For more information about the --servicenode and + --failnode options, see .
<indexterm><primary>maintenance</primary><secondary>separate a combined MGS/MDT</secondary></indexterm> diff --git a/LustreOperations.xml b/LustreOperations.xml index bd80559..27f2c95 100644 --- a/LustreOperations.xml +++ b/LustreOperations.xml @@ -119,24 +119,37 @@ LABEL=testfs-OST0000 /mnt/test/ost0 lustre defaults,_netdev,noauto 0 0</screen> </section> <section xml:id="dbdoclet.50438194_57420"> <title><indexterm><primary>operations</primary><secondary>failover</secondary></indexterm>Specifying Failout/Failover Mode for OSTs - Lustre uses two modes, failout and failover, to handle an OST that has become unreachable because it fails, is taken off the network, is unmounted, etc. + In a Lustre file system, an OST that has become unreachable because it fails, is taken off + the network, or is unmounted can be handled in one of two ways: - In failout mode, Lustre clients immediately receive errors (EIOs) after a timeout, instead of waiting for the OST to recover. + In failout mode, Lustre clients immediately receive errors (EIOs) + after a timeout, instead of waiting for the OST to recover. - In failover mode, Lustre clients wait for the OST to recover. + In failover mode, Lustre clients wait for the OST to + recover. - By default, the Lustre file system uses failover mode for OSTs. To specify failout mode instead, use the --param="failover.mode=failout" option: - oss# mkfs.lustre --fsname=fsname --mgsnode=mgs_NID --param=failover.mode=failout --ost --index=ost_index /dev/ost_block_device - In this example, failout mode is specified for the OSTs on MGS mds0, file system testfs. - oss# mkfs.lustre --fsname=testfs --mgsnode=mds0 --param=failover.mode=failout --ost --index=3 /dev/sdb + By default, the Lustre file system uses failover mode for OSTs. To + specify failout mode instead, use the + --param="failover.mode=failout" option as shown below (entered + on one line): + oss# mkfs.lustre --fsname=fsname --mgsnode=mgs_NID --param=failover.mode=failout + --ost --index=ost_index /dev/ost_block_device + In the example below, failout mode is specified for the OSTs on the MGS + mds0 in the file system testfs (entered on one + line). + oss# mkfs.lustre --fsname=testfs --mgsnode=mds0 --param=failover.mode=failout + --ost --index=3 /dev/sdb - Before running this command, unmount all OSTs that will be affected by the change in the failover/failout mode. + Before running this command, unmount all OSTs that will be affected by a change in + failover / failout mode. - After initial file system configuration, use the tunefs.lustre utility to change the failover/failout mode. For example, to set the failout mode, run: + After initial file system configuration, use the tunefs.lustre + utility to change the mode. For example, to set the failout mode, + run: $ tunefs.lustre --param failover.mode=failout /dev/ost_device
@@ -311,35 +324,52 @@ osc.myth-OST0004-osc-ffff8800376bdc00.cur_grant_bytes=33808384
<indexterm><primary>operations</primary><secondary>failover</secondary></indexterm>Specifying NIDs and Failover - If a node has multiple network interfaces, it may have multiple NIDs. When a node is specified, all of its NIDs must be listed, delimited by commas (,) so other nodes can choose the NID that is appropriate for their network interfaces. When failover nodes are specified, they are delimited by a colon (:) or by repeating a keyword (--mgsnode= or --failnode= or --servicenode=). To obtain all NIDs from a node (while LNET is running), run: + If a node has multiple network interfaces, it may have multiple NIDs, which must all be + identified so other nodes can choose the NID that is appropriate for their network interfaces. + Typically, NIDs are specified in a list delimited by commas (,). However, + when failover nodes are specified, the NIDs are delimited by a colon (:) or + by repeating a keyword such as --mgsnode= or + --servicenode=). + To display the NIDs of all servers in networks configured to work with the Lustre file + system, run (while LNET is running): lctl list_nids - This displays the server's NIDs (networks configured to work with Lustre). - This example has a combined MGS/MDT failover pair on mds0 and mds1, and a OST failover pair on oss0 and oss1. There are corresponding Elan addresses on mds0 and mds1. - mds0# mkfs.lustre --fsname=testfs --mdt --mgs --failnode=mds1,2@elan /dev/sda1 + In the example below, mds0 and mds1 are configured + as a combined MGS/MDT failover pair and oss0 and oss1 + are configured as an OST failover pair. The Ethernet address for mds0 is + 192.168.10.1, and for mds1 is 192.168.10.2. The Ethernet addresses for + oss0 and oss1 are 192.168.10.20 and 192.168.10.21 + respectively. + mds0# mkfs.lustre --fsname=testfs --mdt --mgs \ + --servicenode=192.168.10.2@tcp0 \ + -–servicenode=192.168.10.1@tcp0 /dev/sda1 mds0# mount -t lustre /dev/sda1 /mnt/test/mdt -oss0# mkfs.lustre --fsname=testfs --failnode=oss1 --ost --index=0 \ - --mgsnode=mds0,1@elan --mgsnode=mds1,2@elan /dev/sdb +oss0# mkfs.lustre --fsname=testfs --servicenode=192.168.10.20@tcp0 \ + --servicenode=192.168.10.21 --ost --index=0 \ + --mgsnode=192.168.10.1@tcp0 --mgsnode=192.168.10.2@tcp0 \ + /dev/sdb oss0# mount -t lustre /dev/sdb /mnt/test/ost0 -client# mount -t lustre mds0,1@elan:mds1,2@elan:/testfs /mnt/testfs +client# mount -t lustre 192.168.10.1@tcp0:192.168.10.2@tcp0:/testfs \ + /mnt/testfs mds0# umount /mnt/mdt mds1# mount -t lustre /dev/sda1 /mnt/test/mdt mds1# cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status - Where multiple NIDs are specified, comma-separation (for example, mds1,2@elan) means that the two NIDs refer to the same host, and that Lustre needs to choose the best one for communication. Colon-separation (for example, mds0:mds1) means that the two NIDs refer to two different hosts, and should be treated as failover locations (Lustre tries the first one, and if that fails, it tries the second one.) - Two options exist to specify failover nodes. --failnode and --servicenode. --failnode specifies the NIDs of failover nodes. --servicenode specifies all service NIDs, including those of the primary node and of failover nodes. Option --servicenode makes the MDT or OST treat all its service nodes equally. The first service node to load the target device becomes the primary service node. Other node NIDs will become failover locations for the target device. - - If you have an MGS or MDT configured for failover, perform these steps: - - - On the oss0 node, list the NIDs of all MGS nodes at mkfs time. - oss0# mkfs.lustre --fsname sunfs --mgsnode=10.0.0.1 \ - --mgsnode=10.0.0.2 --ost --index=0 /dev/sdb - - - On the client, mount the file system. - client# mount -t lustre 10.0.0.1:10.0.0.2:/sunfs /cfs/client/ - - - + Where multiple NIDs are specified separated by commas (for example, + 10.67.73.200@tcp,192.168.10.1@tcp), the two NIDs refer to the same host, + and the Lustre software chooses the best one for communication. When a + pair of NIDs is separated by a colon (for example, + 10.67.73.200@tcp:10.67.73.201@tcp), the two NIDs refer to two different + hosts and are treated as a failover pair (the Lustre software tries the first one, and if that + fails, it tries the second one.) + Two options to mkfs.lustre can be used to specify failover nodes. + Introduced in Lustre software release 2.0, the --servicenode option is used + to specify all service NIDs, including those for primary nodes and failover nodes. When the + --servicenodeoption is used, the first service node to load the target + device becomes the primary service node, while nodes corresponding to the other specified NIDs + become failover locations for the target device. An older option, + --failnode, specifies just the NIDS of failover nodes. For more + information about the --servicenode and --failnode + options, see .
<indexterm><primary>operations</primary><secondary>erasing a file system</secondary></indexterm>Erasing a File System diff --git a/ManagingFailover.xml b/ManagingFailover.xml index 653f7cb..058b6c4 100644 --- a/ManagingFailover.xml +++ b/ManagingFailover.xml @@ -1,34 +1,33 @@ - Managing Failover - This chapter describes failover in a Lustre system and includes the following sections: + Lustre File System Failover and Multiple-Mount Protection + This chapter describes the multiple-mount protection (MMP) feature, which protects the file + system from being mounted simultaneously to more than one node. It includes the following + sections: + + + - For information about high availability(HA) management software, see the documentation for: - - Red Hat Cluster Manager at http://www.redhat.com/software/rha/cluster/manager/ - - - Pacemaker at http://clusterlabs.org/doc/en-US/Pacemaker/1.0/html/Pacemaker_Explained/index.html - - + For information about configuring a Lustre file system for failover, see
- <indexterm><primary>multiple-mount protection</primary><see>failover</see></indexterm> - <indexterm><primary>failover</primary></indexterm> - Lustre Failover and Multiple-Mount Protection - The failover functionality in Lustre is implemented with the multiple-mount protection (MMP) feature, which protects the file system from being mounted simultaneously to more than one node. This feature is important in a shared storage environment (for example, when a failover pair of OSTs share a partition). - Lustre's backend file system, ldiskfs, supports the MMP mechanism. A block in the file system is updated by a kmmpd daemon at one second intervals, and a sequence number is written in this block. If the file system is cleanly unmounted, then a special "clean" sequence is written to this block. When mounting the file system, ldiskfs checks if the MMP block has a clean sequence or not. + + multiple-mount protection + Overview of Multiple-Mount Protection + The multiple-mount protection (MMP) feature protects the Lustre file system from being + mounted simultaneously to more than one node. This feature is important in a shared storage + environment (for example, when a failover pair of OSSs share a LUN). + The backend file system, ldiskfs, supports the MMP mechanism. A block + in the file system is updated by a kmmpd daemon at one second intervals, + and a sequence number is written in this block. If the file system is cleanly unmounted, then + a special "clean" sequence is written to this block. When mounting the file system, + ldiskfs checks if the MMP block has a clean sequence or not. Even if the MMP block has a clean sequence, ldiskfs waits for some interval to guard against the following situations: @@ -42,21 +41,28 @@ The MMP feature is only supported on Linux kernel versions newer than 2.6.9. -
- <indexterm><primary>failover</primary><secondary>multiple-mount protection</secondary></indexterm>Working with Multiple-Mount Protection - On a new Lustre file system, MMP is automatically enabled by mkfs.lustre at format time if failover is being used and the kernel and e2fsprogs version support it. On an existing file system, a Lustre administrator can manually enable MMP when the file system is unmounted. - Use the following commands to determine whether MMP is running in Lustre and to enable or disable the MMP feature. - To determine if MMP is enabled, run: - dumpe2fs -h /dev/block_device | grep mmp - Here is a sample command: - dumpe2fs -h /dev/sdc | grep mmp +
+
+ Working with Multiple-Mount Protection + On a new Lustre file system, MMP is automatically enabled by + mkfs.lustre at format time if failover is being used and the kernel and + e2fsprogs version support it. On an existing file system, a Lustre file + system administrator can manually enable MMP when the file system is unmounted. + Use the following commands to determine whether MMP is running in the Lustre file system + and to enable or disable the MMP feature. + To determine if MMP is enabled, run: + dumpe2fs -h /dev/block_device | grep mmp + Here is a sample command: + dumpe2fs -h /dev/sdc | grep mmp Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent mmp sparse_super large_file uninit_bg - To manually disable MMP, run: - tune2fs -O ^mmp /dev/block_device - To manually enable MMP, run: - tune2fs -O mmp /dev/block_device - When MMP is enabled, if ldiskfs detects multiple mount attempts after the file system is mounted, it blocks these later mount attempts and reports the time when the MMP block was last updated, the node name, and the device name of the node where the file system is currently mounted. -
+ To manually disable MMP, run: + tune2fs -O ^mmp /dev/block_device + To manually enable MMP, run: + tune2fs -O mmp /dev/block_device + When MMP is enabled, if ldiskfs detects multiple mount attempts after + the file system is mounted, it blocks these later mount attempts and reports the time when the + MMP block was last updated, the node name, and the device name of the node where the file + system is currently mounted.
diff --git a/SystemConfigurationUtilities.xml b/SystemConfigurationUtilities.xml index ae8f762..da51d84 100644 --- a/SystemConfigurationUtilities.xml +++ b/SystemConfigurationUtilities.xml @@ -1472,7 +1472,7 @@ Changelog records consumed: 42
<indexterm><primary>mkfs.lustre</primary></indexterm> mkfs.lustre - The mkfs.lustre utility formats a disk for a Lustre service. + The mkfs.lustre utility formats a disk for a Lustre service.
Synopsis mkfs.lustre target_type [options] device @@ -1497,7 +1497,7 @@ mkfs.lustre --ost - Object Storage Target (OST) + Object storage target (OST) @@ -1505,7 +1505,7 @@ mkfs.lustre --mdt - Metadata Storage Target (MDT) + Metadata storage target (MDT) @@ -1521,7 +1521,9 @@ mkfs.lustre --mgs - Configuration Management Service (MGS), one per site. This service can be combined with one --mdt service by specifying both types. + Configuration management service (MGS), one per site. This service can be + combined with one --mdt service by specifying both + types. @@ -1530,13 +1532,17 @@ mkfs.lustre
Description - mkfs.lustre is used to format a disk device for use as part of a Lustre file system. After formatting, a disk can be mounted to start the Lustre service defined by this command. - When the file system is created, parameters can simply be added as a --param option to the mkfs.lustre command. See . + mkfs.lustre is used to format a disk device for use as part of a + Lustre file system. After formatting, a disk can be mounted to start the Lustre service + defined by this command. + When the file system is created, parameters can simply be added as a + --param option to the mkfs.lustre command. See . - - - + + + @@ -1561,7 +1567,7 @@ mkfs.lustre --comment=comment - Sets a user comment about this disk, ignored by Lustre. + Sets a user comment about this disk, ignored by the Lustre software. @@ -1569,7 +1575,7 @@ mkfs.lustre --device-size=#>KB - Sets the device size for loop devices. + Sets the device size for loop devices. @@ -1577,26 +1583,32 @@ mkfs.lustre --dryrun - Only prints what would be done; it does not affect the disk. + Only prints what would be done; it does not affect the disk. - - --failnode=nid,... - - - Sets the NID(s) of a failover partner. This option can be repeated as needed. - This cannot be used with --servicenode. - + --servicenode=nid,... + Sets the NID(s) of all service nodes, including primary and failover partner + service nodes. The --servicenode option cannot be used with + --failnode option. See for + more details. - --servicenode=nid,... + --failnode=nid,... - Sets the NID(s) of all service node, including failover partner as well as - primary node service nids. This option can be repeated as needed. - This cannot be used with --failnode. + Sets the NID(s) of a failover service node for a primary server for a target. + The --failnode option cannot be used with + --servicenode option. See + for more details. + When the --failnode option is used, certain + restrictions apply (see ). + @@ -1604,9 +1616,12 @@ mkfs.lustre --fsname=filesystem_name - The name of the Lustre file system to which this node belongs. The file system - name is limited to 8 characters. The default file system name is - lustre. + The Lustre file system of which this service/node will be a part. The default + file system name is lustre. +   + + The file system name is limited to 8 characters. + @@ -1633,11 +1648,13 @@ mkfs.lustre Sets the mount options used when the backing file system is mounted. - Unlike earlier versions of mkfs.lustre, this version completely replaces the default mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted. + Unlike earlier versions of mkfs.lustre, this version completely replaces + the default mount options with those specified on the command line, and issues a + warning on stderr if any default mount options are omitted. The defaults for ldiskfs are: OST: errors=remount-ro; MGS/MDT: errors=remount-ro,iopen_nopriv,user_xattr - Do not alter the default mount options unless you know what you are doing. + Use care when altering the default mount options. @@ -1749,7 +1766,8 @@ mkfs.lustre Examples Creates a combined MGS and MDT for file system testfs on, e.g., node cfs21: mkfs.lustre --fsname=testfs --mdt --mgs /dev/sda1 - Creates an OST for file system testfs on any node (using the above MGS): + Creates an OST for file system testfs on any node (using the above + MGS): mkfs.lustre --fsname=testfs --mgsnode=cfs21@tcp0 --ost --index=0 /dev/sdb Creates a standalone MGS on, e.g., node cfs22: mkfs.lustre --mgs /dev/sda1 @@ -2277,20 +2295,27 @@ tunefs.lustre - --failnode=nid,... - - - Sets the NID(s) of a failover partner. This option can be repeated as needed. - Cannot be used with --servicenode. - + --servicenode=nid,... + Sets the NID(s) of all service nodes, including primary and failover partner + service nodes. The --servicenode option cannot be used with + --failnode option. See for + more details. - --servicenode=nid,... + --failnode=nid,... - Sets the NID(s) of all service node, including failover partner as well as local service nids. This option can be repeated as needed. - : Cannot be used with --failnode. + Sets the NID(s) of a failover service node for a primary server for a target. + The --failnode option cannot be used with + --servicenode option. See + for more details. + When the --failnode option is used, certain + restrictions apply (see ). + diff --git a/UnderstandingFailover.xml b/UnderstandingFailover.xml index fafbbb2..acde085 100644 --- a/UnderstandingFailover.xml +++ b/UnderstandingFailover.xml @@ -16,13 +16,27 @@
<indexterm><primary>failover</primary></indexterm> What is Failover? - A computer system is ''highly available'' when the services it provides are available with minimal downtime. In a highly-available system, if a failure condition occurs, such as the loss of a server or a network or software fault, the system's services continue without interruption. Generally, we measure availability by the percentage of time the system is required to be available. - Availability is accomplished by replicating hardware and/or software so that when a primary server fails or is unavailable, a standby server can be switched into its place to run applications and associated resources. This process, called failover, should be automatic and, in most cases, completely application-transparent. - A failover hardware setup requires a pair of servers with a shared resource (typically a physical storage device, which may be based on SAN, NAS, hardware RAID, SCSI or FC technology). The method of sharing storage should be essentially transparent at the device level; the same physical logical unit number (LUN) should be visible from both servers. To ensure high availability at the physical storage level, we encourage the use of RAID arrays to protect against drive-level failures. + In a high-availability (HA) system, unscheduled downtime is minimized by using redundant + hardware and software components and software components that automate recovery when a failure + occurs. If a failure condition occurs, such as the loss of a server or storage device or a + network or software fault, the system's services continue with minimal interruption. + Generally, availability is specified as the percentage of time the system is required to be + available. + Availability is accomplished by replicating hardware and/or software so that when a + primary server fails or is unavailable, a standby server can be switched into its place to run + applications and associated resources. This process, called failover, is automatic in an HA system and, in most cases, completely + application-transparent. + A failover hardware setup requires a pair of servers with a shared resource (typically a + physical storage device, which may be based on SAN, NAS, hardware RAID, SCSI or Fibre Channel + (FC) technology). The method of sharing storage should be essentially transparent at the + device level; the same physical logical unit number (LUN) should be visible from both servers. + To ensure high availability at the physical storage level, we encourage the use of RAID arrays + to protect against drive-level failures. The Lustre software does not provide redundancy for data; it depends exclusively on redundancy of backing storage devices. The backing OST storage should be RAID 5 or, - preferably, RAID 6 storage. MDT storage should be RAID 1 or RAID 0+1. + preferably, RAID 6 storage. MDT storage should be RAID 1 or RAID 10.
<indexterm><primary>failover</primary><secondary>capabilities</secondary></indexterm>Failover Capabilities @@ -42,9 +56,9 @@ These capabilities can be provided by a variety of software and/or hardware solutions. For more information about using power management software or hardware and high availability - (HA) software with the Lustre software, see . + (HA) software with a Lustre file system, see . HA software is responsible for detecting failure of the primary Lustre server node and - controlling the failover. The Lustre software works with any HA software that includes + controlling the failover.The Lustre software works with any HA software that includes resource (I/O) fencing. For proper resource fencing, the HA software must be able to completely power off the failed server or disconnect it from the shared storage device. If two active nodes have access to the same storage device, data may be severely @@ -61,10 +75,11 @@ Active/active pair - In this configuration, both nodes are active, each providing a subset of resources. In case of a failure, the second node takes over resources from the failed node. - Before Lustre software release 2.4, MDSs are configured as an active/passive pair, while - OSSs are deployed in an active/active configuration that provides redundancy without extra - overhead. Often the standby MDS is the active MDS for another Lustre file system or the MGS, - so no nodes are idle in the cluster. + In Lustre software releases previous to Lustre software release 2.4, MDSs can be + configured as an active/passive pair, while OSSs can be deployed in an active/active + configuration that provides redundancy without extra overhead. Often the standby MDS is the + active MDS for another Lustre file system or the MGS, so no nodes are idle in the + cluster. Lustre software release 2.4 introduces metadata targets for individual sub-directories. Active-active failover configurations are available for MDSs that serve MDTs on shared storage. @@ -75,46 +90,50 @@ failover and Lustre Failover Functionality in a Lustre File System - The failover functionality provided in the Lustre software can be used for the following + The failover functionality provided by the Lustre software can be used for the following failover scenario. When a client attempts to do I/O to a failed Lustre target, it continues to try until it receives an answer from any of the configured failover nodes for the Lustre target. A user-space application does not detect anything unusual, except that the I/O may take longer to complete. - Lustre failover requires two nodes configured as a failover pair, which must share one or - more storage devices. A Lustre file system can be configured to provide MDT or OST - failover. + Failover in a Lustre file system requires that two nodes be configured as a failover pair, + which must share one or more storage devices. A Lustre file system can be configured to + provide MDT or OST failover. - For MDT failover, two MDSs are configured to serve the same MDT. Only one MDS node can serve an MDT at a time. - Lustre software release 2.4 allows multiple MDTs. By placing two or + For MDT failover, two MDSs can be configured to serve the same MDT. Only one MDS node + can serve an MDT at a time. + Lustresoftware release 2.4 allows multiple MDTs. By placing two or more MDT partitions on storage shared by two MDSs, one MDS can fail and the remaining MDS can begin serving the unserved MDT. This is described as an active/active failover pair. - For OST failover, multiple OSS nodes are configured to be able to serve the same OST. However, only one OSS node can serve the OST at a time. An OST can be moved between OSS nodes that have access to the same storage device using umount/mount commands. + For OST failover, multiple OSS nodes can be configured to be able to serve the same + OST. However, only one OSS node can serve the OST at a time. An OST can be moved between + OSS nodes that have access to the same storage device using + umount/mount commands. - To add a failover partner to a Lustre file system configuration, the - --failnode or --servicenode option is used. This can - be done at creation time (using mkfs.lustre) or later when the Lustre file - system is active (using tunefs.lustre). For explanations of these - utilities, see and The --servicenode option is used to set up nodes in a Lustre file + system for failover at creation time (using mkfs.lustre) or later when the + Lustre file system is active (using tunefs.lustre). For explanations of + these utilities, see and . - Lustre failover capability can be used to upgrade the Lustre software between successive minor versions without cluster downtime. For more information, see . + Failover capability in a Lustre file system can be used to upgrade the Lustre software + between successive minor versions without cluster downtime. For more information, see . For information about configuring failover, see . - Failover functionality is provided only at the file system level in the Lustre - software. In a complete failover solution, failover functionality for system-level - components, such as node failure detection or power control, must be provided by a - third-party tool. + The Lustre software provides failover functionality only at the file system level. In a + complete failover solution, failover functionality for system-level components, such as node + failure detection or power control, must be provided by a third-party tool. OST failover functionality does not protect against corruption caused by a disk failure. - If the storage media (i.e., physical disk) used for an OST fails, the Lustre software does - not provide a means to recover it. We strongly recommend that some form of RAID be used for - OSTs. Failover functionality provided in the Lustre software assumes that the storage is - reliable, so no extra reliability features are included. + If the storage media (i.e., physical disk) used for an OST fails, it cannot be recovered by + functionality provided in the Lustre software. We strongly recommend that some form of RAID + be used for OSTs. Lustre functionality assumes that the storage is reliable, so it adds no + extra reliability features.
<indexterm><primary>failover</primary><secondary>MDT</secondary></indexterm>MDT Failover Configuration (Active/Passive) @@ -143,7 +162,7 @@ Lustre failover configuration for a active/active MDTs - + Lustre failover configuration for two MDTs @@ -170,6 +189,8 @@ In an active configuration, 50% of the available OSTs are assigned to one OSS and the remaining OSTs are assigned to the other OSS. Each OSS serves as the primary node for half the OSTs and as a failover node for the remaining OSTs. In this mode, if one OSS fails, the other OSS takes over all of the failed OSTs. The clients attempt to connect to each OSS serving the OST, until one of them responds. Data on the OST is written synchronously, and the clients replay transactions that were in progress and uncommitted to disk before the OST failure. + For more information about configuring failover, see .
diff --git a/figures/MDTs_Failover.png b/figures/MDTs_Failover.png new file mode 100644 index 0000000000000000000000000000000000000000..36894aff4fdd8ddbb62e6b3cabd5141b78f1ee55 GIT binary patch literal 18788 zcmbSzbx@a47be}^9V#u|(%k}rfOLm+cb7DXw9?(3(%s$NT}qdG`M%km-I@JkXGcfy z_j}{sd(L^zbDsB}H%#fH6ef&xNDTtwCF=jktZFO|;^|Ae^Zrk&Pry$viFkf0KAen?EzGr@i_u_q26 zi}Z^QXWvpJC2bFk7E}Hi7ySqaRgGANgz(1t3l4_tm!+vhrdlV(hLhV{DQW4cS_PZL zwA}q?k89YbrrV$UYy9pGiThq?64M;0Q2xSpXn6Ju^?t$>7$g2Lj8Og?sq2vhLjzB9)}fGG(0|2&C(a9{vJLV2Om zAaL>)85vnb8HbIHt+rpSR0A?RJMsk=5AR4vXu`eOx9@2B>9eY;#=m^)$Q%I`R9swK z=r$>RbgWu}YUg9MoPO8SC6lX0>32{H(;mLNySoEJLtWvx47K}&ZRil-{~rCEICyw( z7#QRpeMCeczsJY_VzYyczLa;C`S`K^A{wFI4~7wLXKydNv2nrNZeV;|%EqQ#X7?Qx zRaRyu#Qpt!B-ALe?Qj3zzxlyukPvL_>~U#n5e@4;%O}+feah){b~PqacJk5_hSD}R zOgE>?1+Jx)l^Ad021*^k!IW5GBWN68+_X$gecv$OUPTw==28d=t(R7lz}9CJ<+qiV zzTes11$XN>IYz7xg<%ZY+1a@|9y?j8L;kxz&fQX4Tg&kyF_EphyAs)mpywx|lv{pE zN=l{$QHfd^+oNWHH?Ruj;NT#*#T;K^Mn=rZiB-+8tc=W|`kaz^pDA1yVSn9%Pd>~( zm~04`hR8cC3$tR5Rf5qwtWK{1cB*73L`^&v-xsgb5xi*|6o?-Q2`K_L>@y>YH9nJz zUjIE4d0=d84$UJ84CScLyj(#}&d90Efcx^#?r`LgJr2oc3Kj`5R_%M?>0gVL2GHRV z5rlO%NXW>_+S-x(8NTVtt05xArKKY~R}fM`qh_*pXl_=-8z;GWc?Q=3D_2!gf26nz(9cMH^Q{ASi}qt zmn;!StCA06N`b=4LxP~#vESV83yvX1hdkrA8^pqjFpsC5pi;?{D-Y7hhk&@?C2?!9 z-yGfVQtSw%$r!FqfXR=|G!Y~6G`?%iBR11&R-UHnyw>IYA;w|VgcGcU&Moea$TGDK zodeAud%yq_BLrGahR=8mB*MVcNRkv?fSQL}d5X@>G>GCrAw5z3OTQ#F6pB2f`%1{byZOsVo(%T=(5{6BC4QdnP%OD9=@ zd(t`JMp^I9SCv$o4A|3ai7I!@Nc+AgCPgC0CyBoYD@gv+y5m}qu*5~&6Zs-^1|b&~ ztzxbWS~LI;`)d!@)1wpPQF9r)sR=67>b=|9jd|q7U;SmA5(@RnyMrfAlEx*|Tf7rt zC7enMUKq!CQDGvTQjvhfrl+Y1|9|P$eyY4zC<(2b_O$%&axnQ4_chkx5*)XKmp8WR z%8QHr^>mUcU@^@4B*ldLnioKrI4E^rws(Ly-Ji&ff*BP{2$WXrzqJnk=< zwAl7{k(0T?-Ll0t+@!HXL@KBc34yAwb?@wDUlnJqGP3u{lUbKn zZTA8aS!o=PFQTj*4pEc(Vm6QZq;0CJp3oGm3#=BCp4l==eACooDEG&i1#g!LtfS^U zw^KLH=3fJ*>V7a~lndT@uzd8Xi(G-2%R?1L|7Kn4M+v@%kkRQ1DF^BX|MkAN?7=F1 zye|HXrHz;`5->+WVu`aIfjl-pIz`o93rUmOW0l5AA>csuVqe|2H<(?YTUrKBJT|k$ zQIu?lja3VmS)?BFFTKn)`Oqc5(42_aw?fK@xyAisUnbm zS+Kk}fr!NPf?fXDR#iBJD_M5y3g-4UnjSY3zneCKJ4xeT9=rJzCiN*ljePJ|yg=3e zV+z!~r%B;?Vz$m#cjR2D@}72|mT+zfM6|khyB)JGB$l59mxxCLcLedLCWGaH?(G-+hDJPGP5!AHN@>)dY3 zF){v-?Du!hQn+ivX`d$;?2hk7!mH_}5=7W{t}V@5W5H^^V{;n(!To}xM8NOh$^s$5m`jBsA-+Kb%J`$;g5xVW1-!AVy2@v~7&~QRm#y$=V|s zJ-uB(@=>!`byG72MTZUc18VIM#|14CxDx-pw%ea31gHJC8Ux&^6q;0t8-J0bc$TL| z2ii^=g+uS{?L?#G)h;^hc6koI;QY zrHEV-vURJ5mP2o;B?rU5(<<{1bN!?dl0k-|h1@?lqLSo_37n1oQOb7E`nt@~PH|K| zNH;AWbHQpy>m+`3zidjzJB^`wE7rNKbeENndY^}Cn#3-!MD9z3Z2^rOD)8>DZqj-2 zNXGNMm?lqKqZvnHB+@DOg{3vXlao@_{YZPd@T&AV*Y{%%zrLQa@#$0DBVJy&(%t#G z_E0YNvGK1uP~GTSbk-5Gry2$PCP?@z=M6uY*nnwF@dC%_?eNoz%{P*2X_yp>3BCq9 zkd=WN0(Q2HM|<7i+QN%6CXny#GmFrd*8EDb;*PinG)a?ZmsVt~V5GD6D0uj44XNG= zwMordUY4==uf~|qXTniy$gZvHFJ{yZhmw5sG_-MG$!?w1X%Rqy{;e&sEuKTbZUj|o zN-lzmsL9Q$WR#7ag?Hp zCI;FUl00q=O5ClPa!Jn5SO~rBR-LWDNY`e*>*xBbq15eBQ)#;!hLQ9e$xLMK}Z#3B9^RO`le zpTxHbR7z29d6d6Z;obn9K5|{_|C{CdX?1M)>`8%j|SHI*9pnd(`GhqoMO zbV_xDk!%9G23W}~^9fZ9TqQvqFLb!GcEb>`qRL>SG7M9B(abEqzaV4!hOD7^> zM!rqRW~7ZNSE`m+#G*hs!EXdXTl%#dEY}^P8v*uCu<{ETNV9L|wDcXeO0^|&bm*m8 z#Qk5v$-}ttIB`Q%tKu`h5Wt?sIR=R!niPpz>UZXl<_C#v=_zb0OA-0A;jro`Fee;3 zLd=2B)wspZ6CD#N5rSJw zM6L#duVb}(t+lmpe8!nILtKBG_n67$wqb);j&@_~O(`n}rBkPen2 z)gV0N4CZI6{M>1X78~TjnQ-b;Ih1?0(*Di`;nN%0AN1B*5m9)4dDZVh;K+@#Ml=!+ zmF!Iw)R4a+9$);Bdt z?m-dFLZBPDDU?t1SLB*tBT-HBm5LT|Q?enL+fK;~Z1ij2 z{S?R4V z?CLI~ljN>>+`M8yWiRSz@P^;~+ze$!yLsAO3^oDLO29}x6pYc17AeKv-&9Y1X({K~ zH=FbZo1~~FVr&=9v@5s)&a@G#ikOzLIE3@X zmZQC|7e;+T8q+KdpsnYN9{jGJC|KUXcNb^MwAQDR-VHcWt!%Y2I0q`UlW*0t)U(7Z zL`}Xo2lx$(`0=6ppLz8>HX4Z*qJM&()jUALjZ)YrA{hO;ig(hxlpad7M;*LjE>`g6 z#$!wUfd_lUaVy((O+v{;j#9Q_mS_y(l8$zGK!qPNNDa!m zP?1-+9ksMVrvXk}E*s6oF&!hP|&t6to)#ujb8bo<&U z)1-)myK1}aqQjxpNSyOK2c0~ly|2)ccE|qWoG<<)J;NkYaj4wDHo<*43>NX;jh|{| zrz)GQ$zkF;E1kc5((0D132T`?wW^{xK{>k9?YTZz9~1bUjm! z0n$ynjvWNO6}*$+cS2d8n|BOTuuQB4Xq)E3;uUlvZ&3&~_x*jpZJmCv;g$y_B>{)u ze2n_GYe-AK`D=ceRQ!gZ$Sv-HgmFkEoLoTs`)0&oXR9j5E@C|Js@EME*q32{9I8$`~<_F6IKXV z!09L0uPyHikJW^>>Fvpnnsf0=TmWPK&B+^QWo)1&Jwfy>QII$NIj<|X3X>Xq<96A! zp@ieHGjctxKshpvYHRYYYZ}U(KNE1izxK?J-3k{Z85Z^`{4b+6qyRs40%kY6v z7hR{UYRmiIStud~PBtN65brh8O&9aw0w)WUvYHg5^L0}cpcDCG0y!J{d5k-t{B|uc zxwMY5*VdK%0p|;&kzET3ck931^kVuI>Aam$ON} z{vK5|?YrAXx4`^p(NZHCFoyvOxUW_?->s7;%&k=&2z|WwE8JWR&;*|@#Og;`!$3e* zx*5|hnwA9N3!BNDGSeBCh{hwWpy05sZ zXl)Ggo}PR!kNv$oY3zVe|HS73gHI<2pNivW9o163Xm~a8TtrMw#P^Xz9iP?UkQXz(m2P{)v zhKS$)=u^w4_WhG3uKf3vZ#n?-q{H;_koy!(E)@NE-cCL`I_j}cLrt9xl&iOrk(D)h&tDpBNOe69sb~`( z=*XuF6#MI~mnXV!dpz6cv#R4)@v{AfJ6A!OpGH#p*_Wacl%4PnXwe(@@UhQwE~E=gKS+rja8efPfnj5X;<-6Bpp0!COy+p>udc3ME}CT+ zygc34v$zTq5fihTjgkEL@#BDW|L^@mqy1Y4N5?4Lc}DduC2KyP#-$L;53>1lK~Q$j zD>5_qr&3~p1?9#EdR!@sUVptA{ z545!35(kZ~(tzZwu)n)INrEOdE(mdTvRHGtwx(M$U!Y2V$Z0t@fg^EU!cIdIdUkYF z?eQy7|8DXA@p1Z7(GTky1nw`=0g0#@mHPha+y5?Oe2tgi3Amgrh#&`d`G~v|5ct*R z^V~dpba---TT@f>I4h=%^E|{*F+ra&^ws$ba*DhpaMYv47Ta~f`q4KlLBQ72Yil1{ zT3U{Iy=7z&mrbv?7pjb$?Ck8?{k|tBOY1c|4Q~A@DK0)>HR`)!Snuu*T&p&Tz&-15 zd9T-|J-(MGmsD+y^@#*hgNHLi$- zqFq~-L1Y>k8Xl+=eEf2WX4Typ2!DNYb8}NKGoRA&E!dbaCgC0!=}K4V>7x7n7dDQt zu&`IfcxN#FI=Z30e(S8|{EU0q9O=CMpP=d3&#*Z)@8{bvY?|!cTo|MN=(+(@PZsAR zRdMl^mAHbE5@c9d*!P;^6M?6dhq9`lKOds2XQ% z@e5oP^}H@&mrXGNNRYL^Jd~sdBEvv6mz70X*AN(dmF-Cv@HDrzJ@?tCdiQQ(f9#jh zC0c2BDyQYu83P|*nsqz^uv5(E6179#yPv#{3tWS5LS)14_J#h*I5w^LoGWGs{<5*Q zHd*#6u{)ZfYxUgzd+#JChrDcx>iUXHY$>Jd_+H?;R%s@z(0StE;Rs}Z+l~Y8vZ+?W zj~_U#o+IPqX3M#oXfp1Va6&J4Icp!fGczfS&CF`dLOgcl~IqMCU>#d)!8abSt zoedsuPSM)!;Z!7}Ny3(GS`;2N^FFRinh+Ld;LX~OT>Tx8h>9(;%RcA#``{KbbdeQ<2nDiIPDx- zBxJnzeoBSy=sRP-_*lIku6$Cj$~&?pB7_eSSQ3By$ZT#-6$lLB1?CWTcINbXxi>qt z49~8#T4cjxtVWxiolRylLHf@#l$Gvo=rnHIzC=bXyMu`=pOyy_J{OdDYK5Z$ne&sL zo}Pocbc5ZoBK_Cx zx6i#gHqFP}Zsc0Ljm+8jDs!iUaA9g9to6rxF4XUZMn5J%hbTIap z!UuIbb>mPqsTM8Ag6B4h=hWPsWJ^=i;p#lN!XlStBX}BzP2%1_s#29F8raG;rVrg6 z^M>L|HYQuwgkb09nJ|qtTNPAQ zVX~W!bO4XNj$Zzh)rm)JRvFdMz+GKkeH9(|laa66WxWe<$?OPT+ee`4h46@qqyOKz8l4 zxlzIi23|I zHJjX)=5-K$?C+NivL`kWcxq&OUXy?oNCGYod9oj0ZyWHIdAUk8s}s|1?gv8NCmAVm z@q@+R-j6`$cW{=zoxmY+Ui=7nY;{ddTt-HP>(A~@3NYmi?;AoGC`h@vlxSg!4iGUH zUC7J_r0Vnu8chyVYGs@|s0}n3Jm&R#$cBoOl9mz9;hiVh75+Sup}92gy~Yf-p) z$HIaEk4Dn*bpQ7{TJ9YdKS-W#=dHII)$$qcfAX_7${U-OQ!art z0NWkK0!&DVntl3?i_tj@96V6bsaTiezAq}{WO33xsfFvI~3jMY^BNE+WNmhrqkkLaI#RvRkpr3Q7CR~Vsf}^ZK3+c<$NtFG7_a_ z2?v1P)s+*8kUNUtq|HTyWA}sD_#3Yu%ty^blqxctgSY+m&FPng@?rc1Me>i*dl z^UMFFvky9Z>-Ch^T#%BMKjuhkhuD1CkDYJ&%OAIu5&fjzkaZ(Y^Nv&&LRdy7Oo6+? z?ceilmB;m0S{$U-Z2*ux0Pry}F%4>(fBpKu@bv36=UUh(f!W5H2V$nW=a6k&p{&ql zD6}M^Kh1U(ZYyM5(A%PB6odcNE_hBV%$%X{n&c;rzSrdxp7H(s`r%td8BLbbAR|*v zi!$2aXYNwHXF^H%k1(NInw?W=1G^4S_v5~24I6|I!1+U-p1cY)E=IUM72$yFdYshI zyX)}EdHJ8OrqwEHvY%1Hk8ih}rVTi*XEe0QGTsXR)+4e(PxF_FTSUP6PF@UY_mcT} z(H`%SUa-qH3Z6oJ&`M1)lZAxc5Z}<_1nLd}ijk?JwUrcPyN;LV$LnZE9+IcW^?!ZE zpG)4C8}W!tbLe_qO8mZ_(|TUh0wFk8p|`G>wx+#9Pw3OuO!`>TCTg8@nuw*GX(wj~ zXWGZy**Ia5N&Ed{QQ%*=GNMZftJVv&kKCm7maS0b7`N})n9Hi8W?XPtame)MWby8(q=EtmX_QGthQYqjl))w8 zAH219)EwLFyKHYs!CPhgWnLk8ujH;E*2|SDAs51VSK@3 zbG`8Fea%4rk4z_i&KpJ;r0!6DSC9tX4)c;SRgd78LHaZTIR^m&p>hDPOW}X7mHV-= zv6YpSCV*)$nqt69!jpC|F?jiwC!ZiB(rq~)w8xk zC;PIhn%a$A7{tsmA-r5zqq`z03z{6gt$Cp?AwX@hIaxR4l@}>6~B@9wPfuq z2Y%UY1F^TKn(L(PNspxPWhwh|9xSd}^g0AHw@{x64{J1eEtrWF|24`-&KHiIrok5@ zRt6tLYRMI3!G2V~W87TqPX%>;bD0!pqwFqO#ls9*1i!FVA~{J|jGm z@CXwNOK@V?+GSs#=G}-KDeVf2;T4lqQ*udvcl3dIr&m?5hx{$(U6D*!LRx|VS z$G@<#umVn(8;pj2&_y0mB!?E!VdgRR-pya)NJh=efk$tk0qGMthZ(>_b@Y~3To>9a;+)skdI+*N4r!Su1|>43Eqd=xz&m3cbK{=xPaNL+sdT+h z?dZ$QHo``VoH-vrq-GbwU+Djd|90^S{>Rjwp_j)QCx??B>n)s}p%FVh`hnTzKG1R^ znA`l4lJMUWBPJTXr|RMhh&lNUqXf0BQwQR?l&l?7*X9T(iRlYpz21snw|WD+O&mN@ zx4qsJISZqy;7a2ex8^({)svGUdLV@4;KYcX+iIq6QoGYGr~|z$ZZx*Q?dsf zfx?r!)vGRA3d7&}+g7GGG_fOmFGv5It|tfpYcJ9!yvR$~G9zU`B_Kw%BvysI>&9H~J>j?#jsWj8mvYGwO@%Ed5eb>3M6D%fA&?sU-j>&<<|5_4EmI3lkM0nLrR3vWrF4 z=}A!EbxJjNNcRU9{-*G?=@ngb91%aFO3uEa-ma8s!Ze+AXI{Mu7lS>hd3wpn?I~Urr1Z)D^*DNuUEOhIl8CXQ?yQhbsUC7Uj=T zUjICJ@KBXCeX&BCHX`fh!bVc!sbRN^{bS~w{obpp6jw;^84qw8${BVz|1P^AH%TMS z6R)28{Oq<8s9#ff5vNCS5q0jIXZ+|vo>ZAqa|WNrlCh9DZf5jludE_YDKyZ7Wup~L z#&fi6Jn}tE`p;io1>jZBLvBaldDNsoHp^QZN*)~e*H9Mf02}m)}DC z7{oKO`*V+pJ&Q5-`G_!e!|)b2J9s|Im1xa{&B7O?%%nCkAEIN(E!#C8#;(EhQyU}g zMQew>`?m~HXoelbiKG9R=gKHc2J5Hnk8S6=(w&SUhzw@#W6AFa(tydI>wnBf?2@q2 zN4Jt;5Jg{ zzq2;%?|_HIl=k#Tx}2v!7XzK=-_)yEfS!>3wxg zbvh%BOCi8yLG$y7ds>(%N+v5~=Q?<}I87Ia1 z&;7@q1~k@DyFW`l@$r8H_-D4vf!%@YASll~W7q{J6ZoK02aWbrg@~RsK9L=@(A(!9 zvW#(%_LWh(8jcMJyEU|TL0g*6E@AhbC`;;Qv_-c$6Z5aW6nXBXyey)^$D5gVhCl1ulnUQ!WU+U_}6Iqn^p>Mh6VV zTpGVWf_(YXYFpZ&xpNzSt-m)oc zx8sBP_5de*@_9z^Y4?F?fHBhti(+Kl;UiO*g8RMqYYYG~9)8l(4=AKxa85?Ici6D9+b%W;zLoe& z9FZ<*Yk|+Cd3S9`jwB`P1r5oX*{UU?$t^sZ<=?k@>c~MhTbrQaBP@k4v-v_Zq#2#& z)s|ne?r0=p*-(n@$sp8&{yAg2Jh{n@f)^iL%~)K(4=f*T@p51p{(Mf|Infv={-(Ag z=BGgF&8#>@Bh?KUP_l_toUa2$0%CvbV)k@t!}yzg5t;F9DJ6SC>KL;5Nzq)n^SgzN z_<0|eV=ve8zV#V8)C5wq%a@||S5&(M^dK240b1x*|Gk_i;Vsp!;r#CHrOK!p$S_qc zPwjVEvZTMOZD1z*^^v?{0vEjCn*DK{r{$%DScNr5+d4ic2n<|6{{T z4sopE2L;su8lbplxiEP-APwGp_K5@|Cz&FH9o1Ib0o5Xao}oLB0Fx)1EtH>duIYX2 zSAQ#0J-){J?o}+)YSg?GK*Aj-~ae0!5Bb;~!p7 z>-Bku`TF_o`vW-0x)`Dn8~dvD#=tRamj~pOQ>D<74vz^UbCD}y^PlFK=qg2@Ho)TY zd}_hY9(ZK&?8(3YM7Zg#`eyk-)7{b2zs;T~z2)k}ZGiOA!UjA!y~Q|zguCk%Bb$ zgmFRZlJZ8~8&*(Vzxq2}-mx(bY%=R2v-+T2Ta&TyFK)BoYiFuG`p~a)YDFQ^TPl;X zui%D{X0jDHZ8-rPLO({jG`$j&IZRNh_Euq6))6|9>YiEs!x&pzqCR2W*056Xh}_Ez z!X0YewGuAgalUhjZ$ICp(6w59DgT^@{6JhH zA}7P3%)n)Z|Jpv(RW)znPwL#TS&+Ant;*&Z*j@+Aga-VGHuK zd$d``V-8-00DYaWJ$rJNmQSNIl#xJ+@p8H`;eVW#2Sg#*kD+HJuH>TPaNS7$+x>$@ zvrw$O&wI-r{1CyxL48j1I=NIhTy1u^{l1g*ruJl{1QPh-_CCkZv(Cxp)s0Wuz&FCV z&z|s<*9OIO$9FGkKgi<5`(&ZX?>ZtRP7dh%isrw}fu*_i@q{~$7VtlAtwX?)G*ozy z()j+M9#3t@!N@LGvqaod!q~iqX~1dD8pkzfl|1AWm87v0{_m#;+Ogsb;wKYwP`YoO ztJGW8?4uqN5r4q-O}T><9O92^$j1#B-n1YMD-v^-Q5cqatAWq;bb5O%Rk^3cC3|!m z02K^U@1@tnzsQGZP@Ue=F!pOrBVAbB@EQ|Z>38-l%a8gxQEQNXJCEf;_x=va{p(v( zA|}Mo-#2exe4H@nOuY~Rs{lTz45}*}+Mc`UuAM@7%JARhFK3iXmY`&3{b2XzJ*iox z_-#Z7iRp_=PM%4psCiK}Iu2GC$v5o#mV@D z6tWhG{j>q5EbmVM>}L#CXiO}jKL@Qzr)ZH#KmC+q2(-`3dJprvd1kL(MG`Q#&hNGD20(>EXcX195(sl->~%B$VOl4^ z9_mnMEMz89%wjtwg>P9GEhKPDv=`$6N3R(cep7Q-XD`Im@Cyo+Xj`=4E#D<#BoYZa z5l6kwiatxeNn5+cpAl_DwLrY_5fher;p~InmiF?gwtQN&Pk~eRicg;+k$;str%o$` z0{Au)nWm4>l1*I=M7Rm=->k8Q0?b~ynEm7K?p}l^cA|mW)2FhZHS%{f&?-WzL zAvkb^{Q141v-nH1$uHXSe`vsII8v3Z2KykP%I$d0kWMg?ONI=P(+U25pq4N_lxm~!o)FAGAAM;aN9afie;quBBpkz}d{* zw+Q0N>OXrQPUkY^*8Bm7x0q^5jEp)s%aE#MehWpizeryALIfC)?ZDsn7PBguS1!d4 zxEdMAt(-d_>lGj%21caYD|7RlwFXrC=CR%bI&$j4>txr~^g5H1 z(M52X8iIxSIlw}0cz6_j5YmWHN3N0tQXSAgZ>u!Ojw=!gIZaI?9r^E)(8nSSOP)dL zl3ZHJ@draQanw=*nL17i`k{xc$O~MMeupo8P88`-!F0;Jk>$=#8t9r37y8u}Ml$t1 zKQntd+O0fJL?!1jOQl}5RKzj;Wt3?xO_Y|gvBY9&E*%4Hk z^6i>bV8^|kiwubFSTNd@X9{jG=kotB0pZH0p)^8EC1F+Zy>G;sKY=Z@5P5SNk^$4N ze>RXnUAD^rBtMXt#ocMPi|lQ;M~ggTF`b3BwfUmj7ab@BY3}JF@HP3^(6DQ1!vj~e z)=8!and;MfQoC&t1Z5LKgrqF7-@&k8}Q>1Twwx z`rK4UtWqEcie>{iPzb-RXWOn$kmVEV3I_`=KO}bqY5W`@u`saQuTnv^GLWAdlH@?3J6a+xzbi2HG zcDtSb3}gs+FoIu`ju&&aqWrurqyYliaExN*BuimHx?{^G>mYVL>sY7uhJyeESs=IA z7&s9b(S5DVt!SlCTL8uTkKU6ze|x9Yyu5u#)ku?HMF5My`J9H|0oUwk`jRVMKwSJo zO#FAqi?YF* zEhw9(@$HlDPNbXcBXXY7p#9yE%L)Cz80Dyokszrmj#DbDrVHj1y7k=K^rSR$(;lXt zkvl2`J9JnG)!K~C8@cnX6O1&U=D+-xF?_DgC$%*URfi)HclSpPrh5Af2%;`i*o}NQ z2B$thM_Zi%YVkej;WOAM^h9dzN@k=GbolKWZ$w3RFb zxeQ*5Sf_xpe_Y;e{lwEw6pz<<0?LoPq^+l5U)RWq5f*8?EB`IceT=5S8vu=~yX5+c z_tdTWZmuhD_yzaQfdqMcmv!U_AsnO*1zaBo)NQo^-U+8>m#4e8cKL@FvDc2g{QN?! z8Iy_1FpqClWG{{;VNK6Fi#h>rq#(v@rQhXgUWYW7J9yna4mYmcq^&jCDliD=x?&}I z73k)U=QSo{_@IN~Pg8#x{>(;Fd=KY2#GR!u#bNx6=+m&V8(9$&(U=Q#PY9q-ID?YL zF=D8ZiU*qa?Ted{7*ZYk^ZAI}(yx&7$n-#({&KOGl=iH;3dq-c&rP({)-Z#@p=ukf zoV92r)C?s1ZT&xXi`b+Mp4uzUjqB{tx({md7=M1V012B7n>;W^3j+8cD5k6OPV^2` zoS-4r8Xe)Yf5%_;le4o3xCjCkAOZZ^?;-RW&})!@{0I5qgFX7TW(T!47!VAt8saw> zL?Ym#6vx`ap=b59wAHHrX{@AGqF{HQ!$Oqu-N49dg|l)w(Wkd>`yM2?b_Sl8mEVTQ zD~Rm&H%@qT?R)@uK-2d`I&DO1RD{sUpXzyh;W23#!99%&DsQY`uQ~_j?ytzBF}6|x zHh7SwISetfb{`t;ti+rPNt(&#z~LrzPfoMwg{k2#{vX;5M^a0BdoQi3m6 z9B&JGp4I!3$8XE+8;L*=`saon9E-W17QlVJgettM@LDy1GMZJ3DHN2B=nZZDd^!6X zy&s|YIIgvO)2L!Hn;ORGcd71Hq0rkJ)C_sSs&>yLH$lws3E*nq6M)Rg<~hA(evYZZ zerR;6am>a>Rag=0hoB0rGpDS_&Q)=U_j*YgkwIBFhH?K@7cO=0GRv%cpvzVLyw z3F@}wxJp9D?Zv>DOsU>WWM`N>k~LF|k=Jru4e;5QwqZyeyKFlw+v3HDxi#Gzig%n6 zVDl3iiRM#`u#bHR!O{)@-U6zv;H`&NAaiJ`;=v+et|3Yz%J(P7(n+JH<|>3zTy;AK zBr+L(Auj$PtGwz?hPwLM{In$mR6I_DOOz@%7Fx#XUV3g9t(yRvzDeN#Fvi$S@B@+~ z#vU&BPk25=>g?1mR!6mv(famU7Tt7O{j`H=(i2({69LzyXXScBqiZ!!@ELIi6h(e zG&B&xvt#4z9W4uS8Pxo;X>)vogM*>xOirFUk7hvYcV(KFbR}B!fh7?=J$(qj8P5ie zh+OwvnN}3pdYxs`CbgDeZK_u5PB z?#^qTD*4|QaG4DUN|N4rCz+4bS`&~E<}||ZuwHk+aX>Wx{{4IX5#nHFMOzc}NXHPi z&E;igX6k(b-#XdYoMm1gw~Y-CPxF~o9wUwg=G>>+5T>>pjENR}wb%ulAlKWxJ^De* z^J~{PJq0`N0%WJnbX!4em zn|4gEvz&h;B-E~_;zNOwb`&BKxv|0%USVQt8jx>Ms#!O5;-Q0EfTa1K7V)`tNE#s_ z^2NnP)1-v&-?P=#OiihWhKIq15}|_*b9eXq%-!$yjPo6sUQzla;4B##8H_OTB|<_4 zM*%vz!LPg1AE0yI#C;nGpn%n&! z6G;~g5`3vk&H%xW__Z9PH>kv1nP5+n@^C(At?%vab$hwrzaAo?g_mlu8EyJcTR93a zk?FFio%X^FXMjE2diS!3jGSEe(QFAJY*=_W(kV`LxlyT33*MJ6UuYQ_CBVL~o5r&r zHJeRfDWqj(MGXun|NQ+MN|H;%LvHm*lqK|$u=c-g<`vem=;HSFRg+&*0s^3nO-+wT zr{x)m(+267ndcpwLDxBG;6IEKAV?RpLj|`N_5GIq`sxJg|Ndb5e|&>NxQ~cjj;lbd zB-qUa-hfb6LL^JGOk}lCDUw#$KXc}R%UNZ|_LvzL}_pCTg2j5OKw7aS^#Svkq*H2H&7HVd*p+ z$|>W37d&LmbBqiPO>rr3DvbNT0(ZJ@Z6cY1>IWEq^=%xmupYb zOM_Y!&&)J4r$3@^L>}2ho_J1hnAlHC zEM0IWXIJ4eKL>Gmm5IECG#Ya26JDiNAZ$)m-&+YS zn(H;X;pQ-0wCRaYnua6zq`l!7YFHbRmzSrV3f25FSn@^De8`%ODs?|}&HpWNz=1T} zHDy}z3uz6_aG<|#xH)wX2W@)dljfJrhEK8uXQ1Y=22JGFG}rU;^00dKYQ0~HjEoGu z-ExO=pxoTt0dq<6^74?Cm8Dm{CX?6HU#7Kd*JAbR)p}istgI~U)VU>Cq8SbhtQ&3$ zn)}i`2aPtpGdP+#5T7&;Hhq#UhRhZ=%tqGCGHCvEr)>?hagCL4Z2DIh!}2l2jKpAd zX#R}NIGK_qZ8#-v%=Uq1Y7Fl|^EXhvrp(kBJ{!YxjZNV@8^k9E$I*P!(2k+(wUC8G zBi_ZRsHni(Z@-OAn>Mj+3V{H?>OM&{;$4mM@^Yl4q_AxUfuP0OK1qU>i?M6hE{kT6 z2n1_mZJ#9FLjr-|Opxv&fk1FBNYEk>2u=kFS_A^YsUSg%Kp;33Rts8&I^4EISEOwq zAva60rN%eFOy}EWG`{Tw@k#S=B%d5?F9fRvt*oppYZc`XtX{nud3kxDX<92^Vhgrz z-D>psKnk5Ak*MFI-SYBsWMyS(m#rAS2lQX?YYb~B7dR5cCz-*vPioXqHSZS`6d*Y{ zSsS=W_izwwsqqb1Pv_g&G`L>>QxK zKDV*Oua_@h9#DUK#flYri)S@u8XnWVPtE-d)ogVXoo}bo__h+nC)K!;PnxtBf+;H$ zHH#(()4Vo(KQ}j5Z;|9cI^Md+jGbe+FjjZbX?c0M);8Vi3^lxsE#wr5M1w9)HM}2V z_pCY3UOL|{qVa7Nh)){7XU%!e>XRnyg|N^h4>iS^jXgNsij6s$?uTgX9Anqe^i8MH zbr8c9^M-oY#>yS+i)^f3wspQ;MC02k5T8`zYCbvCUI^cQ`|WCzx`!GeG;_v1Xqwjc zNec@L2OQ+pZ3Y@@`Dh004AmiXa&iVONB6a+KdmdNyKmnn*31N$i#ckPh7M@>T31rHtgOr;Ep1&KN$1;HG`_6`@k!%2vri5+ zgFR=?oJO-|4~vS5w9}Khxw-w+95qx_RABY$)hH+^7<7t#D9rhFFmyn}*P35O%`Pw2 zT6OM7I^WKs@oi0rPxi+Zd~&e85E!RCXav2C{aP7oLNobQ8)%!Rfo5t9UmG@QMIsR% zdE^mBOtR~UVoaS{>Qc^#C z?!v-Cy?yUA7;2}lUcI{Ct}65A&qr}_G2^DLqw0J+qsF(T5T8`ziat5yUI;@@DHx32 z>;4&=2G-0Z7}GM>onfJAKgH4@pwluyS72=b| zpMxv-0f`}SFFhK4|}2E-@LhXW#fumDqeqX98b5x# + + + + + + + + + + + image/svg+xml + + + + + + + + + + + MDT0 + MDT1 + MDS0 + MDS1 + Active for MDT0, standby for MDT1 + Active for MDT1, standby for MDT0 + + + + + + + + -- 1.8.3.1