X-Git-Url: https://git.whamcloud.com/?a=blobdiff_plain;f=LustreMaintenance.xml;h=271d1b38e27affce1a0344050720cd97f57875aa;hb=30dbf01c7386a9514d618281526d78fdea92db6b;hp=dfeaa6fcd734d5cd7a97c0399c184c93927ea2ed;hpb=3eba9c153757b350882c28161e45b4e5815617b2;p=doc%2Fmanual.git diff --git a/LustreMaintenance.xml b/LustreMaintenance.xml index dfeaa6f..271d1b3 100644 --- a/LustreMaintenance.xml +++ b/LustreMaintenance.xml @@ -3,61 +3,67 @@ Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre maintenance tasks: - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + -
+
<indexterm><primary>maintenance</primary></indexterm> <indexterm><primary>maintenance</primary><secondary>inactive OSTs</secondary></indexterm> @@ -74,7 +80,7 @@ <literal>exclude=testfs-OST0000:testfs-OST0001</literal>.</para> </note> </section> - <section xml:id="dbdoclet.50438199_15240"> + <section xml:id="lustremaint.findingNodes"> <title><indexterm><primary>maintenance</primary><secondary>finding nodes</secondary></indexterm> Finding Nodes in the Lustre File System There may be situations in which you need to find all nodes in @@ -105,7 +111,7 @@ Finding Nodes in the Lustre File System 0: testfs-OST0000_UUID ACTIVE 1: testfs-OST0001_UUID ACTIVE
-
+
<indexterm><primary>maintenance</primary><secondary>mounting a server</secondary></indexterm> Mounting a Server Without Lustre Service If you are using a combined MGS/MDT, but you only want to start the MGS and not the MDT, run this command: @@ -114,13 +120,15 @@ Mounting a Server Without Lustre Service In this example, the combined MGS/MDT is testfs-MDT0000 and the mount point is /mnt/test/mdt. $ mount -t lustre -L testfs-MDT0000 -o nosvc /mnt/test/mdt
-
+
<indexterm><primary>maintenance</primary><secondary>regenerating config logs</secondary></indexterm> Regenerating Lustre Configuration Logs - If the Lustre file system configuration logs are in a state where the file system cannot - be started, use the writeconf command to erase them. After the - writeconf command is run and the servers restart, the configuration logs - are re-generated and stored on the MGS (as in a new file system). + If the Lustre file system configuration logs are in a state where + the file system cannot be started, use the + tunefs.lustre --writeconf command to regenerate them. + After the writeconf command is run and the servers + restart, the configuration logs are re-generated and stored on the MGS + (as with a new file system). You should only use the writeconf command if: @@ -130,82 +138,84 @@ Regenerating Lustre Configuration Logs A server NID is being changed - The writeconf command is destructive to some configuration items (i.e., OST pools information and items set via conf_param), and should be used with caution. To avoid problems: - - - Shut down the file system before running the writeconf command - - - Run the writeconf command on all servers (MDT first, then OSTs) - - - Start the file system in this order: - - - MGS (or the combined MGS/MDT) - - - MDT - - - OSTs - - - Lustre clients - - - - + The writeconf command is destructive to some + configuration items (e.g. OST pools information and tunables set via + conf_param), and should be used with caution. - The OST pools feature enables a group of OSTs to be named for file striping purposes. If you use OST pools, be aware that running the writeconf command erases all pools information (as well as any other parameters set via lctl conf_param). We recommend that the pools definitions (and conf_param settings) be executed via a script, so they can be reproduced easily after a writeconf is performed. + The OST pools feature enables a group of OSTs to be named for + file striping purposes. If you use OST pools, be aware that running + the writeconf command erases + all pools information (as well as + any other parameters set via lctl conf_param). + We recommend that the pools definitions (and + conf_param settings) be executed via a script, + so they can be regenerated easily after writeconf + is performed. However, tunables saved with lctl set_param + -P are not erased in this case. + + If the MGS still holds any configuration logs, it may be + possible to dump these logs to save any parameters stored with + lctl conf_param by dumping the config logs on + the MGS and saving the output: + +mgs# lctl --device MGS llog_print fsname-client +mgs# lctl --device MGS llog_print fsname-MDT0000 +mgs# lctl --device MGS llog_print fsname-OST0000 + + To regenerate Lustre file system configuration logs: - Shut down the file system in this order. + Stop the file system services in the following order before + running the tunefs.lustre --writeconf command: + Unmount the clients. - Unmount the MDT. + Unmount the MDT(s). - Unmount all OSTs. + Unmount the OST(s). + + + If the MGS is separate from the MDT it can remain mounted + during this process. - Make sure the the MDT and OST devices are available. + Make sure the MDT and OST devices are available. - Run the writeconf command on all servers. - Run writeconf on the MDT first, and then the OSTs. + Run the tunefs.lustre --writeconf command + on all target devices. + Run writeconf on the MDT(s) first, and then the OST(s). - On the MDT, run: - mdt# tunefs.lustre --writeconf /dev/mdt_device + On each MDS, for each MDT run: + mds# tunefs.lustre --writeconf /dev/mdt_device - - On each OST, run: - - ost# tunefs.lustre --writeconf /dev/ost_device + On each OSS, for each OST run: + oss# tunefs.lustre --writeconf /dev/ost_device - Restart the file system in this order. + Restart the file system in the following order: - Mount the MGS (or the combined MGS/MDT). + Mount the separate MGT, if it is not already mounted. - Mount the MDT. + Mount the MDT(s) in order, starting with MDT0000. - Mount the OSTs. + Mount the OSTs in order, starting with OST0000. Mount the clients. @@ -213,9 +223,11 @@ Regenerating Lustre Configuration Logs - After the writeconf command is run, the configuration logs are re-generated as servers restart. + After the tunefs.lustre --writeconf command is + run, the configuration logs are re-generated as servers connect to the + MGS.
-
+
<indexterm><primary>maintenance</primary><secondary>changing a NID</secondary></indexterm> Changing a Server NID In Lustre software release 2.3 or earlier, the tunefs.lustre @@ -281,7 +293,58 @@ Changing a Server NID The previous configuration log is backed up on the MGS disk with the suffix '.bak'.
-
+
+ <indexterm> + <primary>maintenance</primary> + <secondary>Clearing a config</secondary> + </indexterm> Clearing configuration + + This command runs on MGS node having the MGS device mounted with + -o nosvc. It cleans up configuration files + stored in the CONFIGS/ directory of any records marked SKIP. + If the device name is given, then the specific logs for that + filesystem (e.g. testfs-MDT0000) are processed. Otherwise, if a + filesystem name is given then all configuration files are cleared. + The previous configuration log is backed up on the MGS disk with + the suffix 'config.timestamp.bak'. Eg: Lustre-MDT0000-1476454535.bak. + + To clear a configuration: + + + Shut down the file system in this order: + + + Unmount the clients. + + + Unmount the MDT. + + + Unmount all OSTs. + + + + + + If the MGS and MDS share a partition, start the MGS only + using "nosvc" option. + + mount -t lustre MDT partition -o nosvc mount_point + + + Run the clear_conf command on the MGS: + + lctl clear_conf config + + Example: To clear the configuration for + MDT0000 on a filesystem named + testfs + + mgs# lctl clear_conf testfs-MDT0000 + + +
+
<indexterm> <primary>maintenance</primary> <secondary>adding an MDT</secondary> @@ -334,7 +397,7 @@ client# lfs mkdir -c 4 /mnt/testfs/new_directory_striped_across_4_mdts </listitem> </orderedlist> </section> - <section xml:id="dbdoclet.adding_new_ost"> + <section xml:id="lustremaint.adding_new_ost"> <title><indexterm><primary>maintenance</primary><secondary>adding a OST</secondary></indexterm> Adding a New OST to a Lustre File System A new OST can be added to existing Lustre file system on either @@ -369,7 +432,7 @@ oss# mount -t lustre /dev/sda /mnt/testfs/ost12 This redistributes file data over the entire set of OSTs. For example, to rebalance all files within the directory /mnt/lustre/dir, enter: - client# lfs_migrate /mnt/lustre/file + client# lfs_migrate /mnt/lustre/dir To migrate files within the /test file system on OST0004 that are larger than 4GB in size to other OSTs, enter: @@ -378,7 +441,7 @@ oss# mount -t lustre /dev/sda /mnt/testfs/ost12
-
+
<indexterm><primary>maintenance</primary><secondary>restoring an OST</secondary></indexterm> <indexterm><primary>maintenance</primary><secondary>removing an OST</secondary></indexterm> Removing and Restoring MDTs and OSTs @@ -420,19 +483,19 @@ Removing and Restoring MDTs and OSTs desire to continue using the filesystem before it is repaired. -
+
<indexterm><primary>maintenance</primary><secondary>removing an MDT</secondary></indexterm>Removing an MDT from the File System If the MDT is permanently inaccessible, - lfs rm_entry {directory} can be used to delete the - directory entry for the unavailable MDT. Using rmdir - would otherwise report an IO error due to the remote MDT being inactive. - Please note that if the MDT is available, standard - rm -r should be used to delete the remote directory. - After the remote directory has been removed, the administrator should - mark the MDT as permanently inactive with: -lctl conf_param {MDT name}.mdc.active=0 -A user can identify which MDT holds a remote sub-directory using -the lfs utility. For example: + lfs rm_entry {directory} can be used to delete the + directory entry for the unavailable MDT. Using rmdir + would otherwise report an IO error due to the remote MDT being inactive. + Please note that if the MDT is available, standard + rm -r should be used to delete the remote directory. + After the remote directory has been removed, the administrator should + mark the MDT as permanently inactive with: + lctl conf_param {MDT name}.mdc.active=0 + A user can identify which MDT holds a remote sub-directory using + the lfs utility. For example: client$ lfs getstripe --mdt-index /mnt/lustre/remote_dir1 1 client$ mkdir /mnt/lustre/local_dir0 @@ -441,8 +504,8 @@ client$ lfs getstripe --mdt-index /mnt/lustre/local_dir0 The lfs getstripe --mdt-index command returns the index of the MDT that is serving the given directory. -
-
+
+
<indexterm><primary>maintenance</primary></indexterm> <indexterm><primary>maintenance</primary><secondary>inactive MDTs</secondary></indexterm>Working with Inactive MDTs @@ -450,7 +513,7 @@ client$ lfs getstripe --mdt-index /mnt/lustre/local_dir0 the MDT is activated again. Clients accessing an inactive MDT will receive an EIO error.
-
+
<indexterm> <primary>maintenance</primary> <secondary>removing an OST</secondary> @@ -519,6 +582,11 @@ client$ lfs getstripe --mdt-index /mnt/lustre/local_dir0 files with objects on the deactivated OST, and copy them to other OSTs in the file system to: </para> <screen>client# lfs find --ost <replaceable>ost_name</replaceable> <replaceable>/mount/point</replaceable> | lfs_migrate -y</screen> + <para>Note that if multiple OSTs are being deactivated at one + time, the <literal>lfs find</literal> command can take multiple + <literal>--ost</literal> arguments, and will return files that + are located on <emphasis>any</emphasis> of the specified OSTs. + </para> </listitem> <listitem> <para>If the OST is no longer available, delete the files @@ -554,14 +622,14 @@ client$ lfs getstripe --mdt-index /mnt/lustre/local_dir0 <note><para>A deactivated OST still appears in the file system configuration, though a replacement OST can be created using the <literal>mkfs.lustre --replace</literal> option, see - <xref linkend="section_restore_ost"/>. + <xref linkend="lustremaint.restore_ost"/>. </para></note> </listitem> </orderedlist> </listitem> </orderedlist> </section> - <section remap="h3" xml:id="section_ydg_pgt_tl"> + <section remap="h3" xml:id="lustremaint.ydg_pgt_tl"> <title><indexterm> <primary>maintenance</primary> <secondary>backing up OST config</secondary> @@ -597,7 +665,7 @@ oss# mount -t ldiskfs <replaceable>/dev/ost_device</replaceable> /mnt/ost</scree </listitem> </orderedlist> </section> - <section xml:id="section_restore_ost"> + <section xml:id="lustremaint.restore_ost"> <title><indexterm> <primary>maintenance</primary> <secondary>restoring OST config</secondary> @@ -669,7 +737,7 @@ oss0# dd if=/tmp/mountdata of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 ski </listitem> </orderedlist> </section> - <section xml:id="section_ucf_qgt_tl"> + <section xml:id="lustremaint.ucf_qgt_tl"> <title><indexterm> <primary>maintenance</primary> <secondary>reintroducing an OSTs</secondary> @@ -683,7 +751,7 @@ oss0# dd if=/tmp/mountdata of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 ski client# lctl set_param osc.<replaceable>fsname</replaceable>-OST<replaceable>number</replaceable>-*.active=1</screen></para> </section> </section> - <section xml:id="dbdoclet.50438199_77819"> + <section xml:id="lustremaint.abortRecovery"> <title><indexterm><primary>maintenance</primary><secondary>aborting recovery</secondary></indexterm> <indexterm><primary>backup</primary><secondary>aborting recovery</secondary></indexterm> Aborting Recovery @@ -692,7 +760,7 @@ Aborting Recovery The recovery process is blocked until all OSTs are available.
-
+
<indexterm><primary>maintenance</primary><secondary>identifying OST host</secondary></indexterm> Determining Which Machine is Serving an OST In the course of administering a Lustre file system, you may need to determine which @@ -713,7 +781,7 @@ osc.testfs-OST0002-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp osc.testfs-OST0003-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp osc.testfs-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-
+
<indexterm><primary>maintenance</primary><secondary>changing failover node address</secondary></indexterm> Changing the Address of a Failover Node To change the address of a failover node (e.g, to use node X instead of node Y), run @@ -726,13 +794,13 @@ Changing the Address of a Failover Node --failnode options, see .
-
+
<indexterm><primary>maintenance</primary><secondary>separate a combined MGS/MDT</secondary></indexterm> Separate a combined MGS/MDT These instructions assume the MGS node will be the same as the MDS node. For instructions on how to move MGS to a different node, see - . + . These instructions are for doing the split without shutting down other servers and clients. @@ -752,7 +820,7 @@ Changing the Address of a Failover Node mds# cp -r /mdt_mount_point/CONFIGS/filesystem_name-* /mgs_mount_point/CONFIGS/. mds# umount /mgs_mount_point mds# umount /mdt_mount_point - See for alternative method. + See for alternative method. Start the MGS. @@ -772,4 +840,33 @@ Changing the Address of a Failover Node
+
+ <indexterm><primary>maintenance</primary> + <secondary>set an MDT to readonly</secondary></indexterm> + Set an MDT to read-only + It is sometimes desirable to be able to mark the filesystem + read-only directly on the server, rather than remounting the clients and + setting the option there. This can be useful if there is a rogue client + that is deleting files, or when decommissioning a system to prevent + already-mounted clients from modifying it anymore. + Set the mdt.*.readonly parameter to + 1 to immediately set the MDT to read-only. All future + MDT access will immediately return a "Read-only file system" error + (EROFS) until the parameter is set to + 0 again. + Example of setting the readonly parameter to + 1, verifying the current setting, accessing from a + client, and setting the parameter back to 0: + mds# lctl set_param mdt.fs-MDT0000.readonly=1 +mdt.fs-MDT0000.readonly=1 + +mds# lctl get_param mdt.fs-MDT0000.readonly +mdt.fs-MDT0000.readonly=1 + +client$ touch test_file +touch: cannot touch ‘test_file’: Read-only file system + +mds# lctl set_param mdt.fs-MDT0000.readonly=0 +mdt.fs-MDT0000.readonly=0 +