From feb018cdf25683c6ebbb0982f6b5c12040c0b9ec Mon Sep 17 00:00:00 2001 From: Andreas Dilger Date: Sat, 13 Jan 2018 16:53:18 -0700 Subject: [PATCH] LUDOC-305 maintenance: handling full/failed OSTs Improve documentation related to handling full or failing OSTs. Describe the "osp.*.max_create_count" parameter, and using it to disable object creation on an OST, rather than deactivating the OST completely. Differentiate between disabling an OST for space balancing reasons and a failing OST. Add references to QOS space balancing and also degraded mode for OSTs. Use "lctl set_param osp.*.active=0" to deactivate the OST rather than "lctl deactivate", since that is easier for users and they are functionally equivalent. Describe the use of "mkfs.lustre --replace" to replace failed OSTs, rather than the need to backup/restore/binary edit OST config files. Use proper names for affected sections and cross references, so that the auto-generated URLs for these sections are useful. Line-wrap modified sections of text. Signed-off-by: Andreas Dilger Change-Id: I4f253966f463ca1a83d0c7c4efcb22dfc02b6e3f Reviewed-on: https://review.whamcloud.com/30864 Tested-by: Jenkins Reviewed-by: Joseph Gmitter --- ConfiguringLustre.xml | 22 +-- LustreMaintenance.xml | 393 +++++++++++++++++++++++++++------------------- LustreOperations.xml | 8 +- LustreProc.xml | 44 +++--- LustreTroubleshooting.xml | 2 +- ManagingFileSystemIO.xml | 73 +-------- SettingUpLustreSystem.xml | 4 +- UserUtilities.xml | 62 ++++---- 8 files changed, 317 insertions(+), 291 deletions(-) diff --git a/ConfiguringLustre.xml b/ConfiguringLustre.xml index c40a5dd..c1aae97 100644 --- a/ConfiguringLustre.xml +++ b/ConfiguringLustre.xml @@ -152,7 +152,7 @@ mount -t lustre devices, mount them both. - + Create the OST. On the OSS node, run: mkfs.lustre --fsname= @@ -191,7 +191,7 @@ mkfs.lustre --fsname= instead format the whole disk for the file system. - + Mount the OST. On the OSS node where the OST was created, run: @@ -201,12 +201,12 @@ mount -t lustre To create additional OSTs, repeat Step - and Step - , specifying the + and Step + , specifying the next higher OST index number. - + Mount the Lustre file system on the client. On the client node, run: @@ -216,8 +216,8 @@ mount -t lustre /mount_point - To create additional clients, repeat Step - . + To mount the filesystem on additional clients, repeat Step + . If you have a problem mounting the file system, check the @@ -705,7 +705,7 @@ Lustre: temp-MDT0000.mdt: set parameter identity_upcall=/usr/sbin/l_getidentity Lustre: Server temp-MDT0000 on device /dev/sdb has started - + Create and mount ost0. In this example, the OSTs ( @@ -938,10 +938,10 @@ total 8.0M Scaling the Lustre File System A Lustre file system can be scaled by adding OSTs or clients. For instructions on creating additional OSTs repeat Step - and Step - above. For mounting + and Step + above. For mounting additional clients, repeat Step - for each client. + for each client.
diff --git a/LustreMaintenance.xml b/LustreMaintenance.xml index b9600c3..9b220ed 100644 --- a/LustreMaintenance.xml +++ b/LustreMaintenance.xml @@ -18,28 +18,28 @@ <para><xref linkend="dbdoclet.50438199_31353"/></para> </listitem> <listitem> - <para><xref linkend="dbdoclet.addingamdt"/></para> + <para><xref linkend="dbdoclet.adding_new_mdt"/></para> </listitem> <listitem> - <para><xref linkend="dbdoclet.50438199_22527"/></para> + <para><xref linkend="dbdoclet.adding_new_ost"/></para> </listitem> <listitem> - <para><xref linkend="dbdoclet.50438199_14978"/></para> + <para><xref linkend="dbdoclet.deactivating_mdt_ost"/></para> </listitem> <listitem> <para><xref linkend="dbdoclet.rmremotedir"/></para> </listitem> <listitem> - <para><xref linkend="dbdoclet.inactivemdt"/>\</para> + <para><xref linkend="dbdoclet.inactivemdt"/></para> </listitem> <listitem> - <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_k3l_4gt_tl"/></para> + <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_remove_ost"/></para> </listitem> <listitem> <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_ydg_pgt_tl"/></para> </listitem> <listitem> - <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_kzs_pgt_tl"/></para> + <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_restore_ost"/></para> </listitem> <listitem> <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_ucf_qgt_tl"/></para> @@ -281,7 +281,7 @@ Changing a Server NID The previous configuration log is backed up on the MGS disk with the suffix '.bak'.
-
+
<indexterm> <primary>maintenance</primary> <secondary>adding an MDT</secondary> @@ -334,144 +334,200 @@ client# lfs mkdir -c 4 /mnt/testfs/new_directory_striped_across_4_mdts </listitem> </orderedlist> </section> - <section xml:id="dbdoclet.50438199_22527"> + <section xml:id="dbdoclet.adding_new_ost"> <title><indexterm><primary>maintenance</primary><secondary>adding a OST</secondary></indexterm> Adding a New OST to a Lustre File System - To add an OST to existing Lustre file system: + A new OST can be added to existing Lustre file system on either + an existing OSS node or on a new OSS node. In order to keep client IO + load balanced across OSS nodes for maximum aggregate performance, it is + not recommended to configure different numbers of OSTs to each OSS node. + - Add a new OST by passing on the following commands, run: - oss# mkfs.lustre --fsname=spfs --mgsnode=mds16@tcp0 --ost --index=12 /dev/sda -oss# mkdir -p /mnt/test/ost12 -oss# mount -t lustre /dev/sda /mnt/test/ost12 - - - Migrate the data (possibly). - The file system is quite unbalanced when new empty OSTs are added. New file creations are automatically balanced. If this is a scratch file system or files are pruned at a regular interval, then no further work may be needed. - New files being created will preferentially be placed on the empty OST. As old files are deleted, they will release space on the old OST. - Files existing prior to the expansion can optionally be rebalanced with an in-place copy, which can be done with a simple script. The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs. - For example, to rebalance all files within /mnt/lustre/dir, enter: + Add a new OST by using mkfs.lustre as when + the filesystem was first formatted, see + for details. Each new OST + must have a unique index number, use lctl dl to + see a list of all OSTs. For example, to add a new OST at index 12 + to the testfs filesystem run following commands + should be run on the OSS: + oss# mkfs.lustre --fsname=testfs --mgsnode=mds16@tcp0 --ost --index=12 /dev/sda +oss# mkdir -p /mnt/testfs/ost12 +oss# mount -t lustre /dev/sda /mnt/testfs/ost12 + + + Balance OST space usage (possibly). + The file system can be quite unbalanced when new empty OSTs + are added to a relatively full filesystem. New file creations are + automatically balanced to favour the new OSTs. If this is a scratch + file system or files are pruned at regular intervals, then no further + work may be needed to balance the OST space usage as new files being + created will preferentially be placed on the less full OST(s). As old + files are deleted, they will release space on the old OST(s). + Files existing prior to the expansion can optionally be + rebalanced using the lfs_migrate utility. + This redistributes file data over the entire set of OSTs. + For example, to rebalance all files within the directory + /mnt/lustre/dir, enter: client# lfs_migrate /mnt/lustre/file - To migrate files within the /test file system on - OST0004 that are larger than 4GB in size, enter: - client# lfs find /test -obd test-OST0004 -size +4G | lfs_migrate -y - See for more details. + To migrate files within the /test file + system on OST0004 that are larger than 4GB in + size to other OSTs, enter: + client# lfs find /test --ost test-OST0004 -size +4G | lfs_migrate -y + See for details.
-
- <indexterm><primary>maintenance</primary><secondary>restoring a OST</secondary></indexterm> - <indexterm><primary>maintenance</primary><secondary>removing a OST</secondary></indexterm> -Removing and Restoring OSTs - OSTs can be removed from and restored to a Lustre file system. Removing a OST means the - OST is deactivated in the file system, not permanently - removed. - A removed OST still appears in the file system; do not create a new OST with the same name. - You may want to remove (deactivate) an OST and prevent new files from being written to it in several situations: +
+ <indexterm><primary>maintenance</primary><secondary>restoring an OST</secondary></indexterm> + <indexterm><primary>maintenance</primary><secondary>removing an OST</secondary></indexterm> +Removing and Restoring MDTs and OSTs + OSTs and DNE MDTs can be removed from and restored to a Lustre + filesystem. Deactivating an OST means that it is temporarily or + permanently marked unavailable. Deactivating an OST on the MDS means + it will not try to allocate new objects there or perform OST recovery, + while deactivating an OST the client means it will not wait for OST + recovery if it cannot contact the OST and will instead return an IO + error to the application immediately if files on the OST are accessed. + An OST may be permanently deactivated from the file system, + depending on the situation and commands used. + A permanently deactivated MDT or OST still appears in the + filesystem configuration until the configuration is regenerated with + writeconf or it is replaced with a new MDT or OST + at the same index and permanently reactivated. A deactivated OST + will not be listed by lfs df. + + You may want to temporarily deactivate an OST on the MDS to + prevent new files from being written to it in several situations: - Hard drive has failed and a RAID resync/rebuild is underway + A hard drive has failed and a RAID resync/rebuild is underway, + though the OST can also be marked degraded by + the RAID system to avoid allocating new files on the slow OST which + can reduce performance, see + for more details. + - OST is nearing its space capacity + OST is nearing its space capacity, though the MDS will already + try to avoid allocating new files on overly-full OSTs if possible, + see for details. + - OST storage has failed permanently - + MDT/OST storage or MDS/OSS node has failed, and will not + be available for some time (or forever), but there is still a + desire to continue using the filesystem before it is repaired. +
- <indexterm><primary>maintenance</primary><secondary>removing a MDT</secondary></indexterm>Removing a MDT from the File System - If the MDT is permanently inaccessible, + <indexterm><primary>maintenance</primary><secondary>removing an MDT</secondary></indexterm>Removing an MDT from the File System + If the MDT is permanently inaccessible, lfs rm_entry {directory} can be used to delete the - directory entry on the unavailable MDT. A normal rmdir - will report an IO error due to the remote MDT being inactive. Please note - that if the MDT is available, a standard rm -r should - be used to delete the remote directory. After the remote directory has been - removed, the administrator should mark the MDT as permanently inactive - with: -lctl conf_param {MDT name}.mdc.active=0 - + directory entry for the unavailable MDT. Using rmdir + would otherwise report an IO error due to the remote MDT being inactive. + Please note that if the MDT is available, standard + rm -r should be used to delete the remote directory. + After the remote directory has been removed, the administrator should + mark the MDT as permanently inactive with: +lctl conf_param {MDT name}.mdc.active=0 A user can identify which MDT holds a remote sub-directory using the lfs utility. For example: -client$ lfs getstripe -M /mnt/lustre/remote_dir1 +client$ lfs getstripe --mdt-index /mnt/lustre/remote_dir1 1 client$ mkdir /mnt/lustre/local_dir0 -client$ lfs getstripe -M /mnt/lustre/local_dir0 +client$ lfs getstripe --mdt-index /mnt/lustre/local_dir0 0 - The getstripe [--mdt-index|-M] parameters return - the index of the MDT that is serving the given directory. -
-
+ The lfs getstripe --mdt-index command + returns the index of the MDT that is serving the given directory. +
+
<indexterm><primary>maintenance</primary></indexterm> <indexterm><primary>maintenance</primary><secondary>inactive MDTs</secondary></indexterm>Working with Inactive MDTs - Files located on or below an inactive MDT are inaccessible until the MDT is activated again. Clients accessing an inactive MDT will receive an EIO error.
-
+ Files located on or below an inactive MDT are inaccessible until + the MDT is activated again. Clients accessing an inactive MDT will receive + an EIO error. +
+
<indexterm> <primary>maintenance</primary> - <secondary>removing a OST</secondary> - </indexterm> Removing an OST from the File System - When removing an OST, remember that the MDT does not communicate directly with OSTs. - Rather, each OST has a corresponding OSC which communicates with the MDT. It is necessary to - determine the device number of the OSC that corresponds to the OST. Then, you use this - device number to deactivate the OSC on the MDT. - To remove an OST from the file system: + removing an OST + Removing an OST from the File System + When deactivating an OST, note that the client and MDS each have + an OSC device that handles communication with the corresponding OST. + To remove an OST from the file system: - For the OST to be removed, determine the device number of the corresponding OSC on - the MDT. + If the OST is functional, and there are files located on + the OST that need to be migrated off of the OST, the file creation + for that OST should be temporarily deactivated on the MDS (each MDS + if running with multiple MDS nodes in DNE mode). + - List all OSCs on the node, along with their device numbers. Run: - lctl dl | grep osc - For example: lctl dl | grep - 11 UP osc testfs-OST-0000-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5 -12 UP osc testfs-OST-0001-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5 -13 IN osc testfs-OST-0000-osc testfs-MDT0000-mdtlov_UUID 5 -14 UP osc testfs-OST-0001-osc testfs-MDT0000-mdtlov_UUID 5 + With Lustre 2.9 and later, the MDS should be + set to only disable file creation on that OST by setting + max_create_count to zero: + mds# lctl set_param osp.osc_name.max_create_count=0 + This ensures that files deleted or migrated off of the OST + will have their corresponding OST objects destroyed, and the space + will be freed. For example, to disable OST0000 + in the filesystem testfs, run: + mds# lctl set_param osp.testfs-OST0000-osc-MDT*.max_create_count=0 + on each MDS in the testfs filesystem. - Determine the device number of the OSC that corresponds to the OST to be - removed. + With older versions of Lustre, to deactivate the OSC on the + MDS node(s) use: + mds# lctl set_param osp.osc_name.active=0 + This will prevent the MDS from attempting any communication with + that OST, including destroying objects located thereon. This is + fine if the OST will be removed permanently, if the OST is not + stable in operation, or if it is in a read-only state. Otherwise, + the free space and objects on the OST will not decrease when + files are deleted, and object destruction will be deferred until + the MDS reconnects to the OST. + For example, to deactivate OST0000 in + the filesystem testfs, run: + mds# lctl set_param osp.testfs-OST0000-osc-MDT*.active=0 + Deactivating the OST on the MDS does not + prevent use of existing objects for read/write by a client. + + If migrating files from a working OST, do not deactivate + the OST on clients. This causes IO errors when accessing files + located there, and migrating files on the OST would fail. + + + Do not use lctl conf_param to + deactivate the OST if it is still working, as this immediately + and permanently deactivates it in the file system configuration + on both the MDS and all clients. + - Temporarily deactivate the OSC on the MDT. On the MDT, run: - mds# lctl --device lustre_devno deactivate - For example, based on the command output in Step 1, to deactivate device 13 (the - MDT’s OSC for OST-0000), the command would be: - mds# lctl --device 13 deactivate - This marks the OST as inactive on the MDS, so no new objects are assigned to the - OST. This does not prevent use of existing objects for reads or writes. - - Do not deactivate the OST on the clients. Do so causes errors (EIOs), and the copy - out to fail. - - - Do not use lctl conf_param to deactivate the OST. It - permanently sets a parameter in the file system configuration. - - - - Discover all files that have objects residing on the deactivated OST. - Depending on whether the deactivated OST is available or not, the data from that OST - may be migrated to other OSTs, or may need to be restored from backup. + Discover all files that have objects residing on the + deactivated OST. Depending on whether the deactivated OST is + available or not, the data from that OST may be migrated to + other OSTs, or may need to be restored from backup. - If the OST is still online and available, find all files with objects on the - deactivated OST, and copy them to other OSTs in the file system to: - client# lfs find --obd ost_name /mount/point | lfs_migrate -y + If the OST is still online and available, find all + files with objects on the deactivated OST, and copy them + to other OSTs in the file system to: + client# lfs find --ost ost_name /mount/point | lfs_migrate -y - If the OST is no longer available, delete the files on that OST and restore them - from backup: - client# lfs find --obd ost_uuid -print0 /mount/point | \ - tee /tmp/files_to_restore | xargs -0 -n 1 unlink - The list of files that need to be restored from backup is stored in - /tmp/files_to_restore. Restoring these files is beyond the - scope of this document. + If the OST is no longer available, delete the files + on that OST and restore them from backup: + client# lfs find --ost ost_uuid -print0 /mount/point | + tee /tmp/files_to_restore | xargs -0 -n 1 unlink + The list of files that need to be restored from backup is + stored in /tmp/files_to_restore. Restoring + these files is beyond the scope of this document. @@ -480,26 +536,28 @@ client$ lfs getstripe -M /mnt/lustre/local_dir0 - If there is expected to be a replacement OST in some short - time (a few days), the OST can temporarily be deactivated on - the clients using: + If there is expected to be a replacement OST in some short + time (a few days), the OST can temporarily be deactivated on + the clients using: client# lctl set_param osc.fsname-OSTnumber-*.active=0 - This setting is only temporary and will be reset - if the clients are remounted or rebooted. It needs to be run - on all clients. - + This setting is only temporary and will be reset + if the clients are remounted or rebooted. It needs to be run + on all clients. + + + + + If there is not expected to be a replacement for this OST in + the near future, permanently deactivate it on all clients and + the MDS by running the following command on the MGS: + mgs# lctl conf_param ost_name.osc.active=0 + A deactivated OST still appears in the file system + configuration, though a replacement OST can be created using the + mkfs.lustre --replace option, see + . + - If there is not expected to be a replacement for this OST in - the near future, permanently deactivate it on all clients and the MDS - by running the following command on the MGS: - mgs# lctl conf_param ost_name.osc.active=0 - - A deactivated OST still appears in the file system - configuration, though a new OST with the same name can be - created using the --replace option for - mkfs.lustre. -
@@ -512,17 +570,19 @@ client$ lfs getstripe -M /mnt/lustre/local_dir0 backup OST config Backing Up OST Configuration Files - If the OST device is still accessible, then the Lustre configuration files on the OST - should be backed up and saved for future use in order to avoid difficulties when a - replacement OST is returned to service. These files rarely change, so they can and should be - backed up while the OST is functional and accessible. If the deactivated OST is still - available to mount (i.e. has not permanently failed or is unmountable due to severe - corruption), an effort should be made to preserve these files. + If the OST device is still accessible, then the Lustre + configuration files on the OST should be backed up and saved for + future use in order to avoid difficulties when a replacement OST is + returned to service. These files rarely change, so they can and + should be backed up while the OST is functional and accessible. If + the deactivated OST is still available to mount (i.e. has not + permanently failed or is unmountable due to severe corruption), an + effort should be made to preserve these files. Mount the OST file system. oss# mkdir -p /mnt/ost -[oss]# mount -t ldiskfs /dev/ost_device /mnt/ost +oss# mount -t ldiskfs /dev/ost_device /mnt/ost @@ -537,7 +597,7 @@ client$ lfs getstripe -M /mnt/lustre/local_dir0
-
+
<indexterm> <primary>maintenance</primary> <secondary>restoring OST config</secondary> @@ -546,22 +606,36 @@ client$ lfs getstripe -M /mnt/lustre/local_dir0 <primary>backup</primary> <secondary>restoring OST config</secondary> </indexterm> Restoring OST Configuration Files - If the original OST is still available, it is best to follow the OST backup and restore - procedure given in either , or - and - . - To replace an OST that was removed from service due to corruption or hardware failure, - the file system needs to be formatted using mkfs.lustre, and the Lustre - file system configuration should be restored, if available. - If the OST configuration files were not backed up, due to the OST file system being - completely inaccessible, it is still possible to replace the failed OST with a new one at - the same OST index. + If the original OST is still available, it is best to follow the + OST backup and restore procedure given in either + , or + and + . + To replace an OST that was removed from service due to corruption + or hardware failure, the replacement OST needs to be formatted using + mkfs.lustre, and the Lustre file system configuration + should be restored, if available. Any objects stored on the OST will + be permanently lost, and files using the OST should be deleted and/or + restored from backup. + With Lustre 2.5 and later, it is possible to + replace an OST to the same index without restoring the configuration + files, using the --replace option at format time. + oss# mkfs.lustre --ost --reformat --replace --index=old_ost_index \ + other_options /dev/new_ost_dev + The MDS and OSS will negotiate the LAST_ID value + for the replacement OST. + + If the OST configuration files were not backed up, due to the + OST file system being completely inaccessible, it is still possible to + replace the failed OST with a new one at the same OST index. - Format the OST file system. - oss# mkfs.lustre --ost --index=old_ost_index other_options \ - /dev/new_ost_dev - + For older versions, format the OST file system without the + --replace option and restore the saved + configuration: + oss# mkfs.lustre --ost --reformat --index=old_ost_index \ + other_options /dev/new_ost_dev + Mount the OST file system. @@ -575,19 +649,22 @@ oss# mount -t ldiskfs /dev/new_ost_dev / Recreate the OST configuration files, if unavailable. - Follow the procedure in to recreate the - LAST_ID file for this OST index. The last_rcvd file will be recreated - when the OST is first mounted using the default parameters, which are normally correct - for all file systems. The CONFIGS/mountdata file is created by - mkfs.lustre at format time, but has flags set that request it to - register itself with the MGS. It is possible to copy these flags from another working - OST (which should be the same): - oss1# debugfs -c -R "dump CONFIGS/mountdata /tmp/ldd" /dev/other_osdev -oss1# scp /tmp/ldd oss0:/tmp/ldd -oss0# dd if=/tmp/ldd of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 skip=5 conv=notrunc - - - Unmount the OST file system. oss# umount /mnt/ost + Follow the procedure in + to recreate the LAST_ID + file for this OST index. The last_rcvd file + will be recreated when the OST is first mounted using the default + parameters, which are normally correct for all file systems. The + CONFIGS/mountdata file is created by + mkfs.lustre at format time, but has flags set + that request it to register itself with the MGS. It is possible to + copy the flags from another working OST (which should be the same): + oss1# debugfs -c -R "dump CONFIGS/mountdata /tmp" /dev/other_osdev +oss1# scp /tmp/mountdata oss0:/tmp/mountdata +oss0# dd if=/tmp/mountdata of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 skip=5 conv=notrunc + + + Unmount the OST file system. + oss# umount /mnt/ost @@ -600,9 +677,9 @@ oss0# dd if=/tmp/ldd of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 skip=5 co If the OST was permanently deactivated, it needs to be reactivated in the MGS configuration. mgs# lctl conf_param ost_name.osc.active=1 - If the OST was temporarily deactivated, it needs to be reactivated on - the MDS and clients. - mds# lctl --device lustre_devno activate + If the OST was temporarily deactivated, it needs to be reactivated on + the MDS and clients. + mds# lctl set_param osp.fsname-OSTnumber-*.active=1 client# lctl set_param osc.fsname-OSTnumber-*.active=1
diff --git a/LustreOperations.xml b/LustreOperations.xml index 18acb13..d5f2e31 100644 --- a/LustreOperations.xml +++ b/LustreOperations.xml @@ -292,7 +292,7 @@ $ tunefs.lustre --param failover.mode=failout
-
+
<indexterm> <primary>operations</primary> @@ -324,7 +324,9 @@ lctl get_param obdfilter.*.degraded resets to <literal>0</literal>.</para> <para>It is recommended that this be implemented by an automated script - that monitors the status of individual RAID devices.</para> + that monitors the status of individual RAID devices, such as MD-RAID's + <literal>mdadm(8)</literal> command with the <literal>--monitor</literal> + option to mark an affected device degraded or restored.</para> </section> <section xml:id="dbdoclet.50438194_88063"> <title> @@ -497,7 +499,7 @@ client# lfs mkdir –i <para>The Lustre 2.8 DNE feature enables individual files in a given directory to store their metadata on separate MDTs (a <emphasis>striped directory</emphasis>) once additional MDTs have been added to the - filesystem, see <xref linkend="dbdoclet.addingamdt"/>. + filesystem, see <xref linkend="dbdoclet.adding_new_mdt"/>. The result of this is that metadata requests for files in a striped directory are serviced by multiple MDTs and metadata service load is distributed over all the MDTs that service a given diff --git a/LustreProc.xml b/LustreProc.xml index 1d33d99..413bf0b 100644 --- a/LustreProc.xml +++ b/LustreProc.xml @@ -2153,36 +2153,40 @@ nid refs peer max tx min </listitem> </itemizedlist></para> </section> - <section remap="h3"> + <section remap="h3" xml:id="dbdoclet.balancing_free_space"> <title><indexterm> <primary>proc</primary> <secondary>free space</secondary> </indexterm>Allocating Free Space on OSTs - Free space is allocated using either a round-robin or a weighted algorithm. The allocation - method is determined by the maximum amount of free-space imbalance between the OSTs. When free - space is relatively balanced across OSTs, the faster round-robin allocator is used, which - maximizes network balancing. The weighted allocator is used when any two OSTs are out of - balance by more than a specified threshold. - Free space distribution can be tuned using these two /proc - tunables: + Free space is allocated using either a round-robin or a weighted + algorithm. The allocation method is determined by the maximum amount of + free-space imbalance between the OSTs. When free space is relatively + balanced across OSTs, the faster round-robin allocator is used, which + maximizes network balancing. The weighted allocator is used when any two + OSTs are out of balance by more than a specified threshold. + Free space distribution can be tuned using these two + /proc tunables: - qos_threshold_rr - The threshold at which the allocation method - switches from round-robin to weighted is set in this file. The default is to switch to the - weighted algorithm when any two OSTs are out of balance by more than 17 percent. + qos_threshold_rr - The threshold at which + the allocation method switches from round-robin to weighted is set + in this file. The default is to switch to the weighted algorithm when + any two OSTs are out of balance by more than 17 percent. - qos_prio_free - The weighting priority used by the weighted - allocator can be adjusted in this file. Increasing the value of - qos_prio_free puts more weighting on the amount of free space - available on each OST and less on how stripes are distributed across OSTs. The default - value is 91 percent. When the free space priority is set to 100, weighting is based - entirely on free space and location is no longer used by the striping algorithm. + qos_prio_free - The weighting priority used + by the weighted allocator can be adjusted in this file. Increasing the + value of qos_prio_free puts more weighting on the + amount of free space available on each OST and less on how stripes are + distributed across OSTs. The default value is 91 percent weighting for + free space rebalancing and 9 percent for OST balancing. When the + free space priority is set to 100, weighting is based entirely on free + space and location is no longer used by the striping algorithm. - reserved_mb_low - The low watermark used to stop - object allocation if available space is less than it. The default is 0.1 percent of total - OST size. + reserved_mb_low - The low + watermark used to stop object allocation if available space is less + than it. The default is 0.1 percent of total OST size. reserved_mb_high - The high watermark used to start diff --git a/LustreTroubleshooting.xml b/LustreTroubleshooting.xml index d349dd7..65d03b9 100644 --- a/LustreTroubleshooting.xml +++ b/LustreTroubleshooting.xml @@ -414,7 +414,7 @@ For example, for a 2 stripe file, stripe size = 1M, the bad OST is at index 0, and you have holes in the file at: [(2*N + 0)*1M, (2*N + 0)*1M + 1M - 1], N = { 0, 1, 2, ...} If the file system cannot be mounted, currently there is no way that parses metadata directly from an MDS. If the bad OST does not start, options to mount the file system are to provide a loop device OST in its place or replace it with a newly-formatted OST. In that case, the missing objects are created and are read as zero-filled.
-
+
Fixing a Bad LAST_ID on an OST Each OST contains a LAST_ID file, which holds the last object (pre-)created by the MDS The contents of the LAST_ID file must be accurate regarding the actual objects that exist on the OST. diff --git a/ManagingFileSystemIO.xml b/ManagingFileSystemIO.xml index 6318d99..51e2cb1 100644 --- a/ManagingFileSystemIO.xml +++ b/ManagingFileSystemIO.xml @@ -169,79 +169,18 @@ $ for i in $(seq 3); do lfs getstripe -M ./testremote/${i}.txt; done - For more information, see man lfs + For more information, see man lfs-migrate Currently, only whole directories can be migrated between MDTs. During migration each file receives a new identifier (FID). As a consequence, the file receives a new inode number. Some system tools (for example, backup and archiving tools) may consider the migrated files to be new, even though the contents are unchanged. - If there is a need to migrate the file data from the current - OST(s) to new OSTs, the data must be migrated (copied) to the new - location. The simplest way to do this is to use the - lfs_migrate command (see - ). However, the steps for - migrating a file by hand are also shown here for reference. - - - Identify the file(s) to be moved. - In the example below, the object information portion of the output from the - lfs getstripe command below shows that the - test_2file is located entirely on OST0002: - -client# lfs getstripe /mnt/testfs/test_2 -/mnt/testfs/test_2 -obdidx objid objid group - 2 8 0x8 0 - - - - To move the data, create a copy and remove the original: - -client# cp -a /mnt/testfs/test_2 /mnt/testfs/test_2.tmp -client# mv /mnt/testfs/test_2.tmp /mnt/testfs/test_2 - - - - If the space usage of OSTs is severely imbalanced, it is - possible to find and migrate large files from their current location - onto OSTs that have more space, one could run: - -client# lfs find --ost -ost_name -size +1G | lfs_migrate -y - - - - Check the file system balance. - The - lfs df output in the example below shows a more - balanced system compared to the - lfs df output in the example in - . - -client# lfs df -h -UUID bytes Used Available Use% \ - Mounted on -testfs-MDT0000_UUID 4.4G 214.5M 3.9G 4% \ - /mnt/testfs[MDT:0] -testfs-OST0000_UUID 2.0G 1.3G 598.1M 65% \ - /mnt/testfs[OST:0] -testfs-OST0001_UUID 2.0G 1.3G 594.1M 65% \ - /mnt/testfs[OST:1] -testfs-OST0002_UUID 2.0G 913.4M 1000.0M 45% \ - /mnt/testfs[OST:2] -testfs-OST0003_UUID 2.0G 1.3G 602.1M 65% \ - /mnt/testfs[OST:3] -testfs-OST0004_UUID 2.0G 1.3G 606.1M 64% \ - /mnt/testfs[OST:4] -testfs-OST0005_UUID 2.0G 1.3G 610.1M 64% \ - /mnt/testfs[OST:5] - -filesystem summary: 11.8G 7.3G 3.9G 61% \ -/mnt/testfs - - - + If there is a need to migrate the file data + from the current OST(s) to new OST(s), the data must be migrated (copied) + to the new location. The simplest way to do this is to use the + lfs_migrate command, see + .
diff --git a/SettingUpLustreSystem.xml b/SettingUpLustreSystem.xml index 898f7e5..472e433 100644 --- a/SettingUpLustreSystem.xml +++ b/SettingUpLustreSystem.xml @@ -89,7 +89,7 @@ <para condition='l24'>If multiple MDTs are going to be present in the system, each MDT should be specified for the anticipated usage and load. For details on how to add additional MDTs to the filesystem, see - <xref linkend="dbdoclet.addingamdt"/>.</para> + <xref linkend="dbdoclet.adding_new_mdt"/>.</para> <warning condition='l24'><para>MDT0 contains the root of the Lustre file system. If MDT0 is unavailable for any reason, the file system cannot be used.</para></warning> @@ -271,7 +271,7 @@ it is possible to increase the total number of inodes of a Lustre filesystem, as well as increasing the aggregate metadata performance, by configuring additional MDTs into the filesystem, see - <xref linkend="dbdoclet.addingamdt"/> for details. + <xref linkend="dbdoclet.adding_new_mdt"/> for details. </para> </note> </section> diff --git a/UserUtilities.xml b/UserUtilities.xml index a53139a..d1f5c89 100644 --- a/UserUtilities.xml +++ b/UserUtilities.xml @@ -1071,7 +1071,7 @@ $ lfs setstripe --pool my_pool /mnt/lustre/dir </para> </section> </section> - <section xml:id="dbdoclet.50438206_42260"> + <section xml:id="dbdoclet.lfs_migrate"> <title> <indexterm> <primary>lfs_migrate</primary> @@ -1079,50 +1079,54 @@ $ lfs setstripe --pool my_pool /mnt/lustre/dir <literal>lfs_migrate</literal> The - lfs_migrate utility is a simple tool to migrate files - between Lustre OSTs. + lfs_migrate utility is a simple to migrate file + data between OSTs.
Synopsis -lfs_migrate [-c stripecount] [-h] [-l] [-n] [-q] [-R] [-s] [-y] -[file|directory ...] +lfs_migrate [lfs_setstripe_options] + [-h] [-n] [-q] [-R] [-s] [-y] [file|directory ...]
Description The - lfs_migrate utility is a simple tool to assist - migration of files between Lustre OSTs. The utility copies each specified - file to a new file, verifies the file contents have not changed, and then - renames the new file to the original filename. This allows balanced space - usage between OSTs, moving files off OSTs that are starting to show - hardware problems (though are still functional) or OSTs that will be - discontinued. + lfs_migrate utility is a tool to assist migration + of file data between Lustre OSTs. The utility copies each specified + file to a temporary file using supplied lfs setstripe + options, if any, optionally verifies the file contents have not changed, + and then swaps the layout (OST objects) from the temporary file and the + original file (for Lustre 2.5 and later), or renames the temporary file + to the original filename. This allows the user/administrator to balance + space usage between OSTs, or move files off OSTs that are starting to show + hardware problems (though are still functional) or will be removed. For versions of Lustre before 2.5, - lfs_migrate is not closely integrated with the MDS, - it cannot determine whether a file is currently open and/or in-use by - other applications or nodes. This makes it UNSAFE for use on files that - might be modified by other applications, since the migrated file is - only a copy of the current file. This results in the old file becoming - an open-unlinked file and any modifications to that file are - lost. + lfs_migrate was not integrated with the MDS at all. + That made it UNSAFE for use on files that were being modified by other + applications, since the file was migrated through a copy and rename of + the file. With Lustre 2.5 and later, the new file layout is swapped + with the existing file layout, which ensures that the user-visible + inode number is kept, and open file handles and locks on the file are + kept. Files to be migrated can be specified as command-line arguments. If a directory is specified on the command-line then all files within the directory are migrated. If no files are specified on the command-line, then a list of files is read from the standard input, making lfs_migrate suitable for use with - lfs find to locate files on specific OSTs and/or - matching other file attributes. - The current file allocation policies on the MDS dictate where the - new files are placed, taking into account whether specific OSTs have been - disabled on the MDS via - lctl(preventing new files from being allocated there), - whether some OSTs are overly full (reducing the number of files placed on - those OSTs), or if there is a specific default file striping for the - target directory (potentially changing the stripe count, stripe size, OST - pool, or OST index of a new file). + lfs find to locate files on specific OSTs and/or + matching other file attributes, and other tools that generate a list + of files on standard output. + Unless otherwise specified through command-line options, the + file allocation policies on the MDS dictate where the new files + are placed, taking into account whether specific OSTs have been + disabled on the MDS via lctl (preventing new + files from being allocated there), whether some OSTs are overly full + (reducing the number of files placed on those OSTs), or if there is + a specific default file striping for the parent directory (potentially + changing the stripe count, stripe size, OST pool, or OST index of a + new file). The lfs_migrate utility can also be used in some cases to -- 1.8.3.1