X-Git-Url: https://git.whamcloud.com/?a=blobdiff_plain;f=LustreMaintenance.xml;h=271d1b38e27affce1a0344050720cd97f57875aa;hb=30dbf01c7386a9514d618281526d78fdea92db6b;hp=dfeaa6fcd734d5cd7a97c0399c184c93927ea2ed;hpb=3eba9c153757b350882c28161e45b4e5815617b2;p=doc%2Fmanual.git
diff --git a/LustreMaintenance.xml b/LustreMaintenance.xml
index dfeaa6f..271d1b3 100644
--- a/LustreMaintenance.xml
+++ b/LustreMaintenance.xml
@@ -3,61 +3,67 @@
Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre maintenance tasks:
-
+
-
+
-
+
-
+
-
+
-
+
-
+
-
+
-
+
-
+
-
+
-
+
-
+
-
+
-
+
-
+
-
+
-
+
+
+
+
+
+
+
-
+
maintenance
maintenanceinactive OSTs
@@ -74,7 +80,7 @@
exclude=testfs-OST0000:testfs-OST0001.
-
+
maintenancefinding nodes
Finding Nodes in the Lustre File System
There may be situations in which you need to find all nodes in
@@ -105,7 +111,7 @@ Finding Nodes in the Lustre File System
0: testfs-OST0000_UUID ACTIVE
1: testfs-OST0001_UUID ACTIVE
-
+
maintenancemounting a server
Mounting a Server Without Lustre Service
If you are using a combined MGS/MDT, but you only want to start the MGS and not the MDT, run this command:
@@ -114,13 +120,15 @@ Mounting a Server Without Lustre Service
In this example, the combined MGS/MDT is testfs-MDT0000 and the mount point is /mnt/test/mdt.
$ mount -t lustre -L testfs-MDT0000 -o nosvc /mnt/test/mdt
-
+
maintenanceregenerating config logs
Regenerating Lustre Configuration Logs
- If the Lustre file system configuration logs are in a state where the file system cannot
- be started, use the writeconf command to erase them. After the
- writeconf command is run and the servers restart, the configuration logs
- are re-generated and stored on the MGS (as in a new file system).
+ If the Lustre file system configuration logs are in a state where
+ the file system cannot be started, use the
+ tunefs.lustre --writeconf command to regenerate them.
+ After the writeconf command is run and the servers
+ restart, the configuration logs are re-generated and stored on the MGS
+ (as with a new file system).
You should only use the writeconf command if:
@@ -130,82 +138,84 @@ Regenerating Lustre Configuration Logs
A server NID is being changed
- The writeconf command is destructive to some configuration items (i.e., OST pools information and items set via conf_param), and should be used with caution. To avoid problems:
-
-
- Shut down the file system before running the writeconf command
-
-
- Run the writeconf command on all servers (MDT first, then OSTs)
-
-
- Start the file system in this order:
-
-
- MGS (or the combined MGS/MDT)
-
-
- MDT
-
-
- OSTs
-
-
- Lustre clients
-
-
-
-
+ The writeconf command is destructive to some
+ configuration items (e.g. OST pools information and tunables set via
+ conf_param), and should be used with caution.
- The OST pools feature enables a group of OSTs to be named for file striping purposes. If you use OST pools, be aware that running the writeconf command erases all pools information (as well as any other parameters set via lctl conf_param). We recommend that the pools definitions (and conf_param settings) be executed via a script, so they can be reproduced easily after a writeconf is performed.
+ The OST pools feature enables a group of OSTs to be named for
+ file striping purposes. If you use OST pools, be aware that running
+ the writeconf command erases
+ all pools information (as well as
+ any other parameters set via lctl conf_param).
+ We recommend that the pools definitions (and
+ conf_param settings) be executed via a script,
+ so they can be regenerated easily after writeconf
+ is performed. However, tunables saved with lctl set_param
+ -P are not erased in this case.
+
+ If the MGS still holds any configuration logs, it may be
+ possible to dump these logs to save any parameters stored with
+ lctl conf_param by dumping the config logs on
+ the MGS and saving the output:
+
+mgs# lctl --device MGS llog_print fsname-client
+mgs# lctl --device MGS llog_print fsname-MDT0000
+mgs# lctl --device MGS llog_print fsname-OST0000
+
+
To regenerate Lustre file system configuration logs:
- Shut down the file system in this order.
+ Stop the file system services in the following order before
+ running the tunefs.lustre --writeconf command:
+
Unmount the clients.
- Unmount the MDT.
+ Unmount the MDT(s).
- Unmount all OSTs.
+ Unmount the OST(s).
+
+
+ If the MGS is separate from the MDT it can remain mounted
+ during this process.
- Make sure the the MDT and OST devices are available.
+ Make sure the MDT and OST devices are available.
- Run the writeconf command on all servers.
- Run writeconf on the MDT first, and then the OSTs.
+ Run the tunefs.lustre --writeconf command
+ on all target devices.
+ Run writeconf on the MDT(s) first, and then the OST(s).
- On the MDT, run:
- mdt# tunefs.lustre --writeconf /dev/mdt_device
+ On each MDS, for each MDT run:
+ mds# tunefs.lustre --writeconf /dev/mdt_device
-
- On each OST, run:
-
- ost# tunefs.lustre --writeconf /dev/ost_device
+ On each OSS, for each OST run:
+ oss# tunefs.lustre --writeconf /dev/ost_device
- Restart the file system in this order.
+ Restart the file system in the following order:
- Mount the MGS (or the combined MGS/MDT).
+ Mount the separate MGT, if it is not already mounted.
- Mount the MDT.
+ Mount the MDT(s) in order, starting with MDT0000.
- Mount the OSTs.
+ Mount the OSTs in order, starting with OST0000.
Mount the clients.
@@ -213,9 +223,11 @@ Regenerating Lustre Configuration Logs
- After the writeconf command is run, the configuration logs are re-generated as servers restart.
+ After the tunefs.lustre --writeconf command is
+ run, the configuration logs are re-generated as servers connect to the
+ MGS.
-
+
maintenancechanging a NID
Changing a Server NID
In Lustre software release 2.3 or earlier, the tunefs.lustre
@@ -281,7 +293,58 @@ Changing a Server NID
The previous configuration log is backed up on the MGS
disk with the suffix '.bak'.
-
+
+
+ maintenance
+ Clearing a config
+ Clearing configuration
+
+ This command runs on MGS node having the MGS device mounted with
+ -o nosvc. It cleans up configuration files
+ stored in the CONFIGS/ directory of any records marked SKIP.
+ If the device name is given, then the specific logs for that
+ filesystem (e.g. testfs-MDT0000) are processed. Otherwise, if a
+ filesystem name is given then all configuration files are cleared.
+ The previous configuration log is backed up on the MGS disk with
+ the suffix 'config.timestamp.bak'. Eg: Lustre-MDT0000-1476454535.bak.
+
+ To clear a configuration:
+
+
+ Shut down the file system in this order:
+
+
+ Unmount the clients.
+
+
+ Unmount the MDT.
+
+
+ Unmount all OSTs.
+
+
+
+
+
+ If the MGS and MDS share a partition, start the MGS only
+ using "nosvc" option.
+
+ mount -t lustre MDT partition -o nosvc mount_point
+
+
+ Run the clear_conf command on the MGS:
+
+ lctl clear_conf config
+
+ Example: To clear the configuration for
+ MDT0000 on a filesystem named
+ testfs
+
+ mgs# lctl clear_conf testfs-MDT0000
+
+
+
+
maintenance
adding an MDT
@@ -334,7 +397,7 @@ client# lfs mkdir -c 4 /mnt/testfs/new_directory_striped_across_4_mdts
-
+
maintenanceadding a OST
Adding a New OST to a Lustre File System
A new OST can be added to existing Lustre file system on either
@@ -369,7 +432,7 @@ oss# mount -t lustre /dev/sda /mnt/testfs/ost12
This redistributes file data over the entire set of OSTs.
For example, to rebalance all files within the directory
/mnt/lustre/dir, enter:
- client# lfs_migrate /mnt/lustre/file
+ client# lfs_migrate /mnt/lustre/dir
To migrate files within the /test file
system on OST0004 that are larger than 4GB in
size to other OSTs, enter:
@@ -378,7 +441,7 @@ oss# mount -t lustre /dev/sda /mnt/testfs/ost12
-
+
maintenancerestoring an OST
maintenanceremoving an OST
Removing and Restoring MDTs and OSTs
@@ -420,19 +483,19 @@ Removing and Restoring MDTs and OSTs
desire to continue using the filesystem before it is repaired.
-
+
maintenanceremoving an MDTRemoving an MDT from the File System
If the MDT is permanently inaccessible,
- lfs rm_entry {directory} can be used to delete the
- directory entry for the unavailable MDT. Using rmdir
- would otherwise report an IO error due to the remote MDT being inactive.
- Please note that if the MDT is available, standard
- rm -r should be used to delete the remote directory.
- After the remote directory has been removed, the administrator should
- mark the MDT as permanently inactive with:
-lctl conf_param {MDT name}.mdc.active=0
-A user can identify which MDT holds a remote sub-directory using
-the lfs utility. For example:
+ lfs rm_entry {directory} can be used to delete the
+ directory entry for the unavailable MDT. Using rmdir
+ would otherwise report an IO error due to the remote MDT being inactive.
+ Please note that if the MDT is available, standard
+ rm -r should be used to delete the remote directory.
+ After the remote directory has been removed, the administrator should
+ mark the MDT as permanently inactive with:
+ lctl conf_param {MDT name}.mdc.active=0
+ A user can identify which MDT holds a remote sub-directory using
+ the lfs utility. For example:
client$ lfs getstripe --mdt-index /mnt/lustre/remote_dir1
1
client$ mkdir /mnt/lustre/local_dir0
@@ -441,8 +504,8 @@ client$ lfs getstripe --mdt-index /mnt/lustre/local_dir0
The lfs getstripe --mdt-index command
returns the index of the MDT that is serving the given directory.
-
-
+
maintenance
maintenanceinactive MDTsWorking with Inactive MDTs
@@ -450,7 +513,7 @@ client$ lfs getstripe --mdt-index /mnt/lustre/local_dir0
the MDT is activated again. Clients accessing an inactive MDT will receive
an EIO error.
-
+
maintenance
removing an OST
@@ -519,6 +582,11 @@ client$ lfs getstripe --mdt-index /mnt/lustre/local_dir0
files with objects on the deactivated OST, and copy them
to other OSTs in the file system to:
client# lfs find --ost ost_name /mount/point | lfs_migrate -y
+ Note that if multiple OSTs are being deactivated at one
+ time, the lfs find command can take multiple
+ --ost arguments, and will return files that
+ are located on any of the specified OSTs.
+
If the OST is no longer available, delete the files
@@ -554,14 +622,14 @@ client$ lfs getstripe --mdt-index /mnt/lustre/local_dir0
A deactivated OST still appears in the file system
configuration, though a replacement OST can be created using the
mkfs.lustre --replace option, see
- .
+ .
-
+
maintenance
backing up OST config
@@ -597,7 +665,7 @@ oss# mount -t ldiskfs /dev/ost_device /mnt/ost
-
+
maintenance
restoring OST config
@@ -669,7 +737,7 @@ oss0# dd if=/tmp/mountdata of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 ski
-
+
maintenance
reintroducing an OSTs
@@ -683,7 +751,7 @@ oss0# dd if=/tmp/mountdata of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 ski
client# lctl set_param osc.fsname-OSTnumber-*.active=1
-
+
maintenanceaborting recovery
backupaborting recovery
Aborting Recovery
@@ -692,7 +760,7 @@ Aborting Recovery
The recovery process is blocked until all OSTs are available.
-
+
maintenanceidentifying OST host
Determining Which Machine is Serving an OST
In the course of administering a Lustre file system, you may need to determine which
@@ -713,7 +781,7 @@ osc.testfs-OST0002-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
osc.testfs-OST0003-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
osc.testfs-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
-
+
maintenancechanging failover node address
Changing the Address of a Failover Node
To change the address of a failover node (e.g, to use node X instead of node Y), run
@@ -726,13 +794,13 @@ Changing the Address of a Failover Node
--failnode options, see .
-
+
maintenanceseparate a
combined MGS/MDT
Separate a combined MGS/MDT
These instructions assume the MGS node will be the same as the MDS
node. For instructions on how to move MGS to a different node, see
- .
+ .
These instructions are for doing the split without shutting down
other servers and clients.
@@ -752,7 +820,7 @@ Changing the Address of a Failover Node
mds# cp -r /mdt_mount_point/CONFIGS/filesystem_name-* /mgs_mount_point/CONFIGS/.
mds# umount /mgs_mount_point
mds# umount /mdt_mount_point
- See for alternative method.
+ See for alternative method.
Start the MGS.
@@ -772,4 +840,33 @@ Changing the Address of a Failover Node
+
+ maintenance
+ set an MDT to readonly
+ Set an MDT to read-only
+ It is sometimes desirable to be able to mark the filesystem
+ read-only directly on the server, rather than remounting the clients and
+ setting the option there. This can be useful if there is a rogue client
+ that is deleting files, or when decommissioning a system to prevent
+ already-mounted clients from modifying it anymore.
+ Set the mdt.*.readonly parameter to
+ 1 to immediately set the MDT to read-only. All future
+ MDT access will immediately return a "Read-only file system" error
+ (EROFS) until the parameter is set to
+ 0 again.
+ Example of setting the readonly parameter to
+ 1, verifying the current setting, accessing from a
+ client, and setting the parameter back to 0:
+ mds# lctl set_param mdt.fs-MDT0000.readonly=1
+mdt.fs-MDT0000.readonly=1
+
+mds# lctl get_param mdt.fs-MDT0000.readonly
+mdt.fs-MDT0000.readonly=1
+
+client$ touch test_file
+touch: cannot touch âtest_fileâ: Read-only file system
+
+mds# lctl set_param mdt.fs-MDT0000.readonly=0
+mdt.fs-MDT0000.readonly=0
+