From 553468d46d1e48c2c033f65c9e25b778b422d21e Mon Sep 17 00:00:00 2001 From: Richard Henwood Date: Fri, 20 May 2011 12:35:21 -0500 Subject: [PATCH] FIX: validation --- ConfiguringFailover.xml | 10 +++++----- ConfiguringStorage.xml | 29 ++++++++++++++-------------- InstallingLustre.xml | 12 ++++++------ LustreMaintenance.xml | 50 +++++++++++++++++++++++++++++++++---------------- LustreMonitoring.xml | 8 ++++---- LustreOperations.xml | 5 +++-- SettingUpBonding.xml | 8 ++------ 7 files changed, 69 insertions(+), 53 deletions(-) diff --git a/ConfiguringFailover.xml b/ConfiguringFailover.xml index 9db63cc..32dfae3 100644 --- a/ConfiguringFailover.xml +++ b/ConfiguringFailover.xml @@ -27,14 +27,14 @@ Shoot The Other Node In The HEAD (STONITH), is a set of power management tools provided with the Linux-HA package. STONITH has native support for many power control devices and is extensible. It uses expect scripts to automate control. PowerMan, available from the Lawrence Livermore National Laboratory (LLNL), is used to control remote power control (RPC) devices from a central location. PowerMan provides native support for several RPC varieties and expect-like configuration simplifies the addition of new devices. The latest versions of PowerMan are available at: - http://sourceforge.net/projects/powerman + http://sourceforge.net/projects/powerman For more information about PowerMan, go to: - https://computing.llnl.gov/linux/powerman.html + https://computing.llnl.gov/linux/powerman.html
11.1.2 Power Equipment Lustre failover also requires the use of RPC devices, which come in different configurations. Lustre server nodes may be equipped with some kind of service processor that allows remote power control. If a Lustre server node is not equipped with a service processor, then a multi-port, Ethernet-addressable RPC may be used as an alternative. For recommended products, refer to the list of supported RPC devices on the PowerMan website. - https://computing.llnl.gov/linux/powerman.html + https://computing.llnl.gov/linux/powerman.html
@@ -42,10 +42,10 @@ Lustre must be combined with high-availability (HA) software to enable a complete Lustre failover solution. Lustre can be used with several HA packages including: - Red Hat Cluster Manager - For more information about setting up Lustre failover with Red Hat Cluster Manager, see the Lustre wiki topic Using Red Hat Cluster Manager with Lustre. + Red Hat Cluster Manager - For more information about setting up Lustre failover with Red Hat Cluster Manager, see the Lustre wiki topic Using Red Hat Cluster Manager with Lustre. - Pacemaker - For more information about setting up Lustre failover with Pacemaker, see the Lustre wiki topic Using Pacemaker with Lustre. + Pacemaker - For more information about setting up Lustre failover with Pacemaker, see the Lustre wiki topic Using Pacemaker with Lustre.
diff --git a/ConfiguringStorage.xml b/ConfiguringStorage.xml index ce7f522..4e3e6a0 100644 --- a/ConfiguringStorage.xml +++ b/ConfiguringStorage.xml @@ -80,26 +80,27 @@ , where <number_of_data_disks> does not include the RAID parity disks (1 for RAID 5 and 2 for RAID 6): - <stripe_width_blocks> = <chunk_blocks> * <number_of_data_disks> = 1 MB - + <stripe_width_blocks> = <chunk_blocks> * <number_of_data_disks> = 1 MB If the RAID configuration does not allow <chunk_blocks> - to fit evenly into 1 MB, select - <chunkblocks> - , such that - <stripe_width_blocks> - is close to 1 MB, but not larger. + to fit evenly into 1 MB, select + <chunkblocks> + + <stripe_width_blocks> + , such that is close to 1 MB, but not larger. The <stripe_width_blocks> - value must equal <chunk_blocks>*<number_of_data_disks>. Specifying the + value must equal + <chunk_blocks>*<number_of_data_disks> + . Specifying the <stripe_width_blocks> parameter is only relevant for RAID 5 or RAID 6, and is not needed for RAID 1 plus 0. Run --reformat on the file system device (/dev/sdc), specifying the RAID geometry to the underlying ldiskfs file system, where: --mkfsoptions "<other options> -E stride=<chunk_blocks>, stripe_width=<stripe_width_blocks>" - A RAID 6 configuration with 6 disks has 4 data and 2 parity disks. The - <chunk_blocks> - <= 1024KB/4 = 256KB. +A RAID 6 configuration with 6 disks has 4 data and 2 parity disks. The + <chunk_blocks> + <= 1024KB/4 = 256KB. Because the number of data disks is equal to the power of 2, the stripe width is equal to 1 MB. --mkfsoptions "<other options> -E stride=<chunk_blocks>, stripe_width=<stripe_width_blocks>"... @@ -120,9 +121,9 @@ Create a journal device on the partition. Run: [oss#] mke2fs -b 4096 -O journal_dev /dev/sdb <journal_size> - The value of - <journal_size> - is specified in units of 4096-byte blocks. For example, 262144 for a 1 GB journal size. + The value of + <journal_size> + is specified in units of 4096-byte blocks. For example, 262144 for a 1 GB journal size. Create the OST. diff --git a/InstallingLustre.xml b/InstallingLustre.xml index 96eac51..7cae283 100644 --- a/InstallingLustre.xml +++ b/InstallingLustre.xml @@ -267,7 +267,7 @@ e2fsprogs : Lustre requires a recent version of e2fsprogs that understands extents. Use e2fsprogs-1.41-90-wc1 or later, available at: - http://build.whamcloud.com/ + http://build.whamcloud.com/ A quilt patchset of all changes to the vanilla e2fsprogs is available in e2fsprogs-{version}-patches.tgz. The Lustre-patched e2fsprogs utility is only required on machines that mount backend (ldiskfs) file systems, such as the OSS, MDS and MGS nodes. It does not need to be loaded on clients. @@ -285,10 +285,10 @@
8.1.1.4 (Optional) Debugging Tools and Other Optional Packages A variety of optional packages are provided on the - Whamcloud download site + Whamcloud download site . These include debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages. For more information about debugging tools, see the topic - Debugging Lustre + Debugging Lustre on the Lustre wiki.
@@ -322,7 +322,7 @@ (Recommended) - Ensure client clocks are synchronized . Lustre uses client clocks for timestamps. If clocks are out of sync between clients, files will appear with different time stamps when accessed by different clients. Drifting clocks can also cause problems, for example, by making it difficult to debug multi-node issues or correlate logs, which depend on timestamps. We recommend that you use Network Time Protocol (NTP) to keep client and server clocks in sync with each other. For more information about NTP, see: http://www.ntp.org. + Ensure client clocks are synchronized . Lustre uses client clocks for timestamps. If clocks are out of sync between clients, files will appear with different time stamps when accessed by different clients. Drifting clocks can also cause problems, for example, by making it difficult to debug multi-node issues or correlate logs, which depend on timestamps. We recommend that you use Network Time Protocol (NTP) to keep client and server clocks in sync with each other. For more information about NTP, see: http://www.ntp.org. @@ -350,7 +350,7 @@ Download the Lustre RPMs. - On the Lustre download site, select your platform. + On the Lustre download site, select your platform. The files required to install Lustre (kernels, modules and utilities RPMs) are listed for the selected platform. @@ -412,7 +412,7 @@ lustre-modules-<ver> lustre-ldiskfs-<ver> Note -The rpm command options --force or --nodeps should not be used to install or update the Lustre-specific e2fsprogs package. If errors are reported, file a bug (for instructions see the topic - Reporting Bugs + Reporting Bugs on the Lustre wiki). diff --git a/LustreMaintenance.xml b/LustreMaintenance.xml index d66c258..3e4046f 100644 --- a/LustreMaintenance.xml +++ b/LustreMaintenance.xml @@ -159,10 +159,12 @@ <mdt node>$ tunefs.lustre --writeconf <device> + - On each OST, run: + On each OST, run: - <ost node>$ tunefs.lustre --writeconf <device> + <ost node>$ tunefs.lustre --writeconf <device> + @@ -359,34 +361,40 @@ If the OST is still online and available, find all files with objects on the dea [client]# lfs find --obd <OST UUID> <mount_point> | lfs_migrate -y - If the OST is no longer available, delete the files on that OST and restore them from backup: [client]# lfs find --obd <OST UUID> -print0 <mount_point> | \ -tee /tmp/files_to_restore | xargs -0 -n 1 unlink The list of files that need to be restored from backup is stored in /tmp/files_to_restore. Restoring these files is beyond the scope of this document. + If the OST is no longer available, delete the files on that OST and restore them from backup: [client]# lfs find --obd <OST UUID> -print0 <mount_point> | \ + tee /tmp/files_to_restore | xargs -0 -n 1 unlink The list of files that need to be restored from backup is stored in /tmp/files_to_restore. Restoring these files is beyond the scope of this document. - Deactivate the OST. - To temporarily disable the deactivated OST, enter: [client]# lctl set_param osc.<fsname>-<OST name>-*.active=0If there is expected to be a replacement OST in some short time (a few days), the OST can temporarily be deactivated on the clients: + Deactivate the OST. + To temporarily disable the deactivated OST, enter: [client]# lctl set_param osc.<fsname>-<OST name>-*.active=0If there is expected to be a replacement OST in some short time (a few days), the OST can temporarily be deactivated on the clients: This setting is only temporary and will be reset if the clients or MDS are rebooted. It needs to be run on all clients. - - b. To permanently disable the deactivated OST, enter: [mgs]# lctl conf_param {OST name}.osc.active=0 - If there is not expected to be a replacement for this OST in the near future, permanently deactivate the OST on all clients and the MDS: + + To permanently disable the deactivated OST, enter: [mgs]# lctl conf_param {OST name}.osc.active=0 + If there is not expected to be a replacement for this OST in the near future, permanently deactivate the OST on all clients and the MDS: A removed OST still appears in the file system; do not create a new OST with the same name. -
14.7.2 Backing Up OST Configuration FilesIf the OST device is still accessible, then the Lustre configuration files on the OST should be backed up and saved for future use in order to avoid difficulties when a replacement OST is returned to service. These files rarely change, so they can and should be backed up while the OST is functional and accessible. If the deactivated OST is still available to mount (i.e. has not permanently failed or is unmountable due to severe corruption), an effort should be made to preserve these files. +
14.7.2 Backing Up OST Configuration FilesIf the OST device is still accessible, then the Lustre configuration files on the OST should be backed up and saved for future use in order to avoid difficulties when a replacement OST is returned to service. These files rarely change, so they can and should be backed up while the OST is functional and accessible. If the deactivated OST is still available to mount (i.e. has not permanently failed or is unmountable due to severe corruption), an effort should be made to preserve these files. + Mount the OST filesystem. [oss]# mkdir -p /mnt/ost [oss]# mount -t ldiskfs {ostdev} /mnt/ost + + Back up the OST configuration files. [oss]# tar cvf {ostname}.tar -C /mnt/ost last_rcvd \ CONFIGS/ O/0/LAST_ID + + Unmount the OST filesystem. [oss]# umount /mnt/ost +
14.7.3 Restoring OST Configuration Files @@ -401,21 +409,29 @@ CONFIGS/ O/0/LAST_ID If the OST configuration files were not backed up, due to the OST file system being completely inaccessible, it is still possible to replace the failed OST with a new one at the same OST index. + Format the OST file system. [oss]# mkfs.lustre --ost --index {OST index} {other options} \ {newdev} + + Mount the OST filesystem. [oss]# mkdir /mnt/ost [oss]# mount -t ldiskfs {newdev} /mnt/ost + + Restore the OST configuration files, if available. [oss]# tar xvf {ostname}.tar -C /mnt/ost + + Recreate the OST configuration files, if unavailable. + Follow the procedure in to recreate the LAST_ID file for this OST index. The last_rcvd file will be recreated when the OST is first mounted using the default parameters, which are normally correct for all file systems. The CONFIGS/mountdata file is created by mkfs.lustre at format time, but has flags set that request it to register itself with the MGS. It is possible to copy these flags from another working OST (which should be the same): @@ -427,21 +443,23 @@ The CONFIGS/mountdata file is created by mkfs.lustre seek=5 skip=5 + Unmount the OST filesystem. [oss]# umount /mnt/ost +
-
14.7.4 Returning a Deactivated OST to Service If the OST was permanently deactivated, it needs to be reactivated in the MGS configuration. [mgs]# lctl conf_param {OST name}.osc.active=1 If the OST was temporarily deactivated, it needs to be reactivated on the MDS and clients. [mds]# lctl --device <devno> activate -[client]# lctl set_param osc.<fsname>-<OST name>-*.active=0
+
14.7.4 Returning a Deactivated OST to Service If the OST was permanently deactivated, it needs to be reactivated in the MGS configuration. [mgs]# lctl conf_param {OST name}.osc.active=1 If the OST was temporarily deactivated, it needs to be reactivated on the MDS and clients. [mds]# lctl --device <devno> activate + [client]# lctl set_param osc.<fsname>-<OST name>-*.active=0
-
14.8 Aborting RecoveryYou can abort recovery with either the lctl utility or by mounting the target with the abort_recov option (mount -o abort_recov). When starting a target, run: $ mount -t lustre -L <MDT name> -o abort_recov <mount_point> Note - The recovery process is blocked until all OSTs are available.
-
14.9 Determining Which Machine is Serving an OST In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the NID that is serving a specific OST, run one of the following commands on a client (you do not need to be a root user): client$ lctl get_param osc.${fsname}-${OSTname}*.ost_conn_uuid For example: client$ lctl get_param osc.*-OST0000*.ost_conn_uuid +
14.8 Aborting RecoveryYou can abort recovery with either the lctl utility or by mounting the target with the abort_recov option (mount -o abort_recov). When starting a target, run: $ mount -t lustre -L <MDT name> -o abort_recov <mount_point> Note - The recovery process is blocked until all OSTs are available.
+
14.9 Determining Which Machine is Serving an OST In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the NID that is serving a specific OST, run one of the following commands on a client (you do not need to be a root user): client$ lctl get_param osc.${fsname}-${OSTname}*.ost_conn_uuid For example: client$ lctl get_param osc.*-OST0000*.ost_conn_uuid osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp - OR - client$ lctl get_param osc.*.ost_conn_uuid osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp osc.lustre-OST0001-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp osc.lustre-OST0002-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp osc.lustre-OST0003-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp -osc.lustre-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-
14.10 Changing the Address of a Failover NodeTo change the address of a failover node (e.g, to use node X instead of node Y), run this command on the OSS/OST partition: tunefs.lustre --erase-params --failnode=<NID> <device>
+osc.lustre-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
+
14.10 Changing the Address of a Failover NodeTo change the address of a failover node (e.g, to use node X instead of node Y), run this command on the OSS/OST partition: tunefs.lustre --erase-params --failnode=<NID> <device>
diff --git a/LustreMonitoring.xml b/LustreMonitoring.xml index c258cfa..437da39 100644 --- a/LustreMonitoring.xml +++ b/LustreMonitoring.xml @@ -335,17 +335,17 @@ $ ln /mnt/lustre/mydir/foo/file /mnt/lustre/mydir/myhardlink
12.2 Lustre <anchor xml:id="dbdoclet.50438273_marker-1297386" xreflabel=""/>Monitoring Tool The Lustre Monitoring Tool (LMT) is a Python-based, distributed system developed and maintained by Lawrence Livermore National Lab (LLNL)). It provides a ''top'' like display of activity on server-side nodes (MDS, OSS and portals routers) on one or more Lustre file systems. It does not provide support for monitoring clients. For more information on LMT, including the setup procedure, see: - http://code.google.com/p/lmt/ + http://code.google.com/p/lmt/ LMT questions can be directed to: - lmt-discuss@googlegroups.com + lmt-discuss@googlegroups.com
12.3 Collect<anchor xml:id="dbdoclet.50438273_marker-1297391" xreflabel=""/>L CollectL is another tool that can be used to monitor Lustre. You can run CollectL on a Lustre system that has any combination of MDSs, OSTs and clients. The collected data can be written to a file for continuous logging and played back at a later time. It can also be converted to a format suitable for plotting. For more information about CollectL, see: - http://collectl.sourceforge.net + http://collectl.sourceforge.net Lustre-specific documentation is also available. See: - http://collectl.sourceforge.net/Tutorial-Lustre.html + http://collectl.sourceforge.net/Tutorial-Lustre.html
12.4 Other Monitoring Options diff --git a/LustreOperations.xml b/LustreOperations.xml index 56459f4..9a7d382 100644 --- a/LustreOperations.xml +++ b/LustreOperations.xml @@ -337,7 +337,8 @@ uml2> cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status On the OST, list the NIDs of all MGS nodes at mkfs time. - OST# mkfs.lustre --fsname sunfs --ost --mgsnode=10.0.0.1 --mgsnode=10.0.0.2 /dev/{device} + OST# mkfs.lustre --fsname sunfs --ost --mgsnode=10.0.0.1 \ + --mgsnode=10.0.0.2 /dev/{device} On the client, mount the file system. @@ -473,7 +474,7 @@ Inode Pathname Debugfs' ''ncheck'' is a brute-force search that may take a long time to complete. - To find the Lustre file from a disk LBA, follow the steps listed in the document at this URL: http://smartmontools.sourceforge.net/badblockhowto.html. Then, follow the steps above to resolve the Lustre filename. + To find the Lustre file from a disk LBA, follow the steps listed in the document at this URL: http://smartmontools.sourceforge.net/badblockhowto.html. Then, follow the steps above to resolve the Lustre filename.
diff --git a/SettingUpBonding.xml b/SettingUpBonding.xml index 1b8658b..7a06f59 100644 --- a/SettingUpBonding.xml +++ b/SettingUpBonding.xml @@ -122,9 +122,7 @@ bond0 - - Append the following lines to the file. - + Append the following lines to the file. DEVICE=bond0 IPADDR=192.168.10.79 # Use the free IP Address of your network NETWORK=192.168.10.0 @@ -190,9 +188,7 @@ options bond0 mode=balance-alb miimon=100 - - Start/restart the slave interfaces (using your normal network method). - + Start/restart the slave interfaces (using your normal network method). You must modprobe the bonding module for each bonded interface. If you wish to create bond0 and bond1, two entries in modprobe.conf are required. -- 1.8.3.1