Clean up some examples seen in the manual while working on LUDOC-14.
Use "ost0" and "ost1" in examples, to match the actual OST indices
used, instead of "ost1" and "ost2". Same for "oss0" and "oss1", etc.
Use "lctl {get,set}_param" instead of direct /proc access in a few
places (there are a lot more that should be fixed).
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibeb4e3318b6fbdb8a81283d5cfad14a69e27443b
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="backupandrestore">
<title xml:id="backupandrestore.title">Backing Up and Restoring a File System</title>
- <para>Lustre provides backups at the file system-level, device-level and file-level. This chapter describes how to backup and restore on Lustre, and includes the following sections:</para>
+ <para>Lustre provides backups at the filesystem-level, device-level and file-level. This chapter describes how to backup and restore on Lustre, and includes the following sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438207_56395"/></para>
<indexterm><primary>rsync</primary><see>backup</see></indexterm>
Backing up a File System</title>
- <para>Backing up a complete file system gives you full control over the files to back up, and allows restoration of individual files as needed. File system-level backups are also the easiest to integrate into existing backup solutions.</para>
+ <para>Backing up a complete file system gives you full control over the files to back up, and allows restoration of individual files as needed. Filesystem-level backups are also the easiest to integrate into existing backup solutions.</para>
<para>File system backups are performed from a Lustre client (or many clients working parallel in different directories) rather than on individual server nodes; this is no different than backing up any other file system.</para>
<para>However, due to the large size of most Lustre file systems, it is not always possible to get a complete backup. We recommend that you back up subsets of a file system. This includes subdirectories of the entire file system, filesets for a single user, files incremented by date, and so on.</para>
<note>
<title><indexterm><primary>backup</primary><secondary>rsync</secondary><tertiary>examples</tertiary></indexterm><literal>lustre_rsync</literal> Examples</title>
<para>Sample <literal>lustre_rsync</literal> commands are listed below.</para>
<para>Register a changelog user for an MDT (e.g. <literal>lustre-MDT0000</literal>).</para>
- <screen># lctl --device lustre-MDT0000 changelog_register lustre-MDT0000 Registered\
- changelog userid 'cl1'</screen>
+ <screen># lctl --device lustre-MDT0000 changelog_register lustre-MDT0000
+Registered changelog userid 'cl1'</screen>
<para>Synchronize a Lustre file system (<literal>/mnt/lustre</literal>) to a target file system (<literal>/mnt/target</literal>).</para>
- <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target --mdt=lustre-MDT0000 \
---user=cl1 --statuslog sync.log --verbose
+ <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target \
+ --mdt=lustre-MDT0000 --user=cl1 --statuslog sync.log --verbose
Lustre filesystem: lustre
MDT device: lustre-MDT0000
Source: /mnt/lustre
lustre_rsync took 2 seconds
Changelog records consumed: 42</screen>
<para>To synchronize a Lustre file system (<literal>/mnt/lustre</literal>) to two target file systems (<literal>/mnt/target1</literal> and <literal>/mnt/target2</literal>).</para>
- <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target1 --target=/mnt/tar\
-get2 \
- --mdt=lustre-MDT0000 --user=cl1
+ <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target1 \
+ --target=/mnt/target2 --mdt=lustre-MDT0000 --user=cl1 \
--statuslog sync.log</screen>
</section>
</section>
<para>If hardware replacement is the reason for the backup or if a spare storage device is available, it is possible to do a raw copy of the MDT or OST from one block device to the other, as long as the new device is at least as large as the original device. To do this, run:</para>
<screen>dd if=/dev/{original} of=/dev/{new} bs=1M</screen>
<para>If hardware errors cause read problems on the original device, use the command below to allow as much data as possible to be read from the original device while skipping sections of the disk with errors:</para>
- <screen>dd if=/dev/{original} of=/dev/{new} bs=4k conv=sync,noerror count={original\
- size in 4kB blocks}</screen>
+ <screen>dd if=/dev/{original} of=/dev/{new} bs=4k conv=sync,noerror /
+ count={original size in 4kB blocks}</screen>
<para>Even in the face of hardware errors, the <literal>ldiskfs</literal> file system is very robust and it may be possible to recover the file system data after running <literal>e2fsck -f</literal> on the new device.</para>
</section>
<section xml:id="dbdoclet.50438207_21638">
<para><emphasis role="bold">Back up the extended attributes.</emphasis></para>
<screen>[oss]# getfattr -R -d -m '.*' -e hex -P . > ea-$(date +%Y%m%d).bak</screen>
<note>
- <para>If the <literal>tar(1)</literal> command supports the <literal>--xattr</literal> option, the <literal>getfattr</literal> step may be unnecessary as long as it does a backup of the "trusted" attributes. However, completing this step is not harmful and can serve as an added safety measure.</para>
+ <para>If the <literal>tar(1)</literal> command supports the <literal>--xattr</literal> option, the <literal>getfattr</literal> step may be unnecessary as long as tar does a backup of the "trusted.*" attributes. However, completing this step is not harmful and can serve as an added safety measure.</para>
</note>
<note>
<para>In most distributions, the <literal>getfattr</literal> command is part of the "<literal>attr</literal>" package. If the <literal>getfattr</literal> command returns errors like <literal>Operation not supported</literal>, then the kernel does not correctly support EAs. Stop and use a different backup method.</para>
<para>Create LVM devices for your MDT and OST targets. Make sure not to use the entire disk for the targets; save some room for the snapshots. The snapshots start out as 0 size, but grow as you make changes to the current file system. If you expect to change 20% of the file system between backups, the most recent snapshot will be 20% of the target size, the next older one will be 40%, etc. Here is an example:</para>
<screen>cfs21:~# pvcreate /dev/sda1
Physical volume "/dev/sda1" successfully created
-cfs21:~# vgcreate volgroup /dev/sda1
- Volume group "volgroup" successfully created
-cfs21:~# lvcreate -L200M -nMDT volgroup
- Logical volume "MDT" created
-cfs21:~# lvcreate -L200M -nOST0 volgroup
+cfs21:~# vgcreate vgmain /dev/sda1
+ Volume group "vgmain" successfully created
+cfs21:~# lvcreate -L200G -nMDT0 vgmain
+ Logical volume "MDT0" created
+cfs21:~# lvcreate -L200G -nOST0 vgmain
Logical volume "OST0" created
cfs21:~# lvscan
- ACTIVE '/dev/volgroup/MDT' [200.00 MB] inherit
- ACTIVE '/dev/volgroup/OST0' [200.00 MB] inherit</screen>
+ ACTIVE '/dev/vgmain/MDT0' [200.00 GB] inherit
+ ACTIVE '/dev/vgmain/OST0' [200.00 GB] inherit</screen>
</listitem>
<listitem>
<para>Format the LVM volumes as Lustre targets.</para>
<para>In this example, the backup file system is called 'main' and designates the current, most up-to-date backup.</para>
- <screen>cfs21:~# mkfs.lustre --fsname=main --mdt --index=0 /dev/volgroup/MDT
+ <screen>cfs21:~# mkfs.lustre --fsname=main --mdt --index=0 /dev/vgmain/MDT0
No management node specified, adding MGS to this MDT.
Permanent disk data:
Target: main-MDT0000
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters:
checking for existing Lustre data
- device size = 200MB
- formatting backing filesystem ldiskfs on /dev/volgroup/MDT
- target name main-MDTffff
+ device size = 200GB
+ formatting backing filesystem ldiskfs on /dev/vgmain/MDT0
+ target name main-MDT0000
4k blocks 0
options -i 4096 -I 512 -q -O dir_index -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L main-MDTffff -i 4096 -I 512 -q
- -O dir_index -F /dev/volgroup/MDT
+ -O dir_index -F /dev/vgmain/MDT0
Writing CONFIGS/mountdata
-cfs21:~# mkfs.lustre --mgsnode=cfs21 --fsname=main --ost --index=0 /dev/volgroup/OST0
+cfs21:~# mkfs.lustre --mgsnode=cfs21 --fsname=main --ost --index=0 /dev/vgmain/OST0
Permanent disk data:
Target: main-OST0000
Index: 0
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: mgsnode=192.168.0.21@tcp
checking for existing Lustre data
- device size = 200MB
- formatting backing filesystem ldiskfs on /dev/volgroup/OST0
+ device size = 200GB
+ formatting backing filesystem ldiskfs on /dev/vgmain/OST0
target name main-OST0000
4k blocks 0
options -I 256 -q -O dir_index -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L lustre-OSTffff -J size=400 -I 256
-i 262144 -O extents,uninit_bg,dir_nlink,huge_file,flex_bg -G 256
- -E resize=4290772992,lazy_journal_init, -F /dev/volgroup/OST0
+ -E resize=4290772992,lazy_journal_init, -F /dev/vgmain/OST0
Writing CONFIGS/mountdata
-cfs21:~# mount -t lustre /dev/volgroup/MDT /mnt/mdt
-cfs21:~# mount -t lustre /dev/volgroup/OST0 /mnt/ost
+cfs21:~# mount -t lustre /dev/vgmain/MDT0 /mnt/mdt
+cfs21:~# mount -t lustre /dev/vgmain/OST0 /mnt/ost
cfs21:~# mount -t lustre cfs21:/main /mnt/main</screen>
</listitem>
</orderedlist>
<para>You can create as many snapshots as you have room for in the volume group. If necessary, you can dynamically add disks to the volume group.</para>
<para>The snapshots of the target MDT and OSTs should be taken at the same point in time. Make sure that the cronjob updating the backup file system is not running, since that is the only thing writing to the disks. Here is an example:</para>
<screen>cfs21:~# modprobe dm-snapshot
-cfs21:~# lvcreate -L50M -s -n MDTb1 /dev/volgroup/MDT
+cfs21:~# lvcreate -L50M -s -n MDT0.b1 /dev/vgmain/MDT0
Rounding up size to full physical extent 52.00 MB
- Logical volume "MDTb1" created
-cfs21:~# lvcreate -L50M -s -n OSTb1 /dev/volgroup/OST0
+ Logical volume "MDT0.b1" created
+cfs21:~# lvcreate -L50M -s -n OST0.b1 /dev/vgmain/OST0
Rounding up size to full physical extent 52.00 MB
- Logical volume "OSTb1" created</screen>
+ Logical volume "OST.b1" created</screen>
<para>After the snapshots are taken, you can continue to back up new/changed files to "main". The snapshots will not contain the new files.</para>
<screen>cfs21:~# cp /etc/termcap /mnt/main
cfs21:~# ls /mnt/main
<listitem>
<para>Rename the LVM snapshot.</para>
<para>Rename the file system snapshot from "main" to "back" so you can mount it without unmounting "main". This is recommended, but not required. Use the <literal>--reformat</literal> flag to <literal>tunefs.lustre</literal> to force the name change. For example:</para>
- <screen>cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/volgroup/MDTb1
+ <screen>cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/vgmain/MDT0.b1
checking for existing Lustre data
found Lustre data
Reading CONFIGS/mountdata
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters:
Writing CONFIGS/mountdata
-cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/volgroup/OSTb1
+cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/vgmain/OST0.b1
checking for existing Lustre data
found Lustre data
Reading CONFIGS/mountdata
Parameters: mgsnode=192.168.0.21@tcp
Writing CONFIGS/mountdata
When renaming an FS, we must also erase the last_rcvd file from the snapshots
-cfs21:~# mount -t ldiskfs /dev/volgroup/MDTb1 /mnt/mdtback
+cfs21:~# mount -t ldiskfs /dev/vgmain/MDT0.b1 /mnt/mdtback
cfs21:~# rm /mnt/mdtback/last_rcvd
cfs21:~# umount /mnt/mdtback
-cfs21:~# mount -t ldiskfs /dev/volgroup/OSTb1 /mnt/ostback
+cfs21:~# mount -t ldiskfs /dev/vgmain/OST0.b1 /mnt/ostback
cfs21:~# rm /mnt/ostback/last_rcvd
cfs21:~# umount /mnt/ostback</screen>
</listitem>
<listitem>
<para>Mount the file system from the LVM snapshot.</para>
<para>For example:</para>
- <screen>cfs21:~# mount -t lustre /dev/volgroup/MDTb1 /mnt/mdtback
+ <screen>cfs21:~# mount -t lustre /dev/vgmain/MDT0.b1 /mnt/mdtback
-cfs21:~# mount -t lustre /dev/volgroup/OSTb1 /mnt/ostback
+cfs21:~# mount -t lustre /dev/vgmain/OST0.b1 /mnt/ostback
cfs21:~# mount -t lustre cfs21:/back /mnt/back</screen>
</listitem>
<listitem>
- <para>Note the old directory contents, as of the snapshot time.</para>
- <para>For example:</para>
+ <para>Note the old directory contents, as of the snapshot time. For example:</para>
<screen>cfs21:~/cfs/b1_5/lustre/utils# ls /mnt/back
fstab passwds</screen>
</listitem>
<section remap="h3">
<title><indexterm><primary>backup</primary><secondary>using LVM</secondary><tertiary>deleting</tertiary></indexterm>Deleting Old Snapshots</title>
<para>To reclaim disk space, you can erase old snapshots as your backup policy dictates. Run:</para>
- <screen>lvremove /dev/volgroup/MDTb1</screen>
+ <screen>lvremove /dev/vgmain/MDT0.b1</screen>
</section>
<section remap="h3">
<title><indexterm><primary>backup</primary><secondary>using LVM</secondary><tertiary>resizing</tertiary></indexterm>Changing Snapshot Volume Size</title>
<para>You can also extend or shrink snapshot volumes if you find your daily deltas are smaller or larger than expected. Run:</para>
- <screen>lvextend -L10G /dev/volgroup/MDTb1</screen>
+ <screen>lvextend -L10G /dev/vgmain/MDT0.b1</screen>
<note>
<para>Extending snapshots seems to be broken in older LVM. It is working in LVM v2.02.01.</para>
</note>
</listitem>
</itemizedlist>
<para>The <literal>stdout</literal> and the <literal>.summary</literal> file will contain lines like this:</para>
- <screen>total_size 8388608K rsz 1024 thr 1 crg 1 180.45 MB/s 1 x 180.50 \=/ 180.50 \
-MB/s
+ <screen>total_size 8388608K rsz 1024 thr 1 crg 1 180.45 MB/s 1 x 180.50 \
+ = 180.50 MB/s
</screen>
<para>Each line corresponds to a run of the test. Each test run will have a different number of threads, record size, or number of regions.</para>
<itemizedlist>
<listitem>
<para>Run the <literal>obdfilter_survey</literal> script with the <literal>target=parameter</literal>.</para>
<para>For example, to run a local test with up to two objects (<literal>nobjhi</literal>), up to two threads (<literal>thrhi</literal>), and 1024 Mb (size) transfer size:</para>
- <screen>$ nobjhi=2 thrhi=2 size=1024 targets='lustre-OST0001 \
-lustre-OST0002' sh obdfilter-survey</screen>
+ <screen>$ nobjhi=2 thrhi=2 size=1024 targets="lustre-OST0001 \
+ lustre-OST0002" sh obdfilter-survey</screen>
</listitem>
</orderedlist>
</section>
</listitem>
<listitem>
<para>Run the <literal>obdfilter_survey</literal> script with the parameters <literal>case=network</literal> and <literal>targets=<replaceable><hostname</replaceable>|<replaceable>ip_of_server></replaceable></literal>. For example:</para>
- <screen>$ nobjhi=2 thrhi=2 size=1024 targets='oss1 oss2' case=network sh obdfilte\
-sh odbfilter-survey</screen>
+ <screen>$ nobjhi=2 thrhi=2 size=1024 targets="oss0 oss1" \
+ case=network sh odbfilter-survey</screen>
</listitem>
<listitem>
<para>On the server side, view the statistics at:</para>
<para>On the OSS nodes to be tested, run the <literal>lctl dl</literal> command. The OSC device names are listed in the fourth column of the output. For example:</para>
<screen>$ lctl dl |grep obdfilter
3 UP osc lustre-OST0000-osc-ffff88007754bc00 \
- 54b91eab-0ea9-1516-b571-5e6df349592e 5
+ 54b91eab-0ea9-1516-b571-5e6df349592e 5
4 UP osc lustre-OST0001-osc-ffff88007754bc00 \
- 54b91eab-0ea9-1516-b571-5e6df349592e 5
+ 54b91eab-0ea9-1516-b571-5e6df349592e 5
...
</screen>
</listitem>
<para>Run the <literal><replaceable role="bold">o</replaceable>bdfilter_survey</literal> script with the <literal>target=parameter</literal> and <literal>case=netdisk</literal>.</para>
<para>An example of a local test run with up to two objects (<literal>nobjhi</literal>), up to two threads (<literal>thrhi</literal>), and 1024 Mb (size) transfer size is shown below:</para>
<screen>$ nobjhi=2 thrhi=2 size=1024 \
- targets="lustre-OST0000-osc-ffff88007754bc00 \
- lustre-OST0001-osc-ffff88007754bc00" \
- sh obdfilter-survey
+ targets="lustre-OST0000-osc-ffff88007754bc00 \
+ lustre-OST0001-osc-ffff88007754bc00" sh obdfilter-survey
</screen>
</listitem>
</orderedlist>
<screen>$ ./ost-survey.sh 10 /mnt/lustre
</screen>
<para>Typical output is:</para>
- <screen>Average read Speed: 6.73
-Average write Speed: 5.41
-read - Worst OST indx 0 5.84 MB/s
-write - Worst OST indx 0 3.77 MB/s
-read - Best OST indx 1 7.38 MB/s
-write - Best OST indx 1 6.31 MB/s
+ <screen>
+Average read Speed: 6.73
+Average write Speed: 5.41
+read - Worst OST indx 0 5.84 MB/s
+write - Worst OST indx 0 3.77 MB/s
+read - Best OST indx 1 7.38 MB/s
+write - Best OST indx 1 6.31 MB/s
3 OST devices found
-Ost index 0 Read speed 5.84 Write speed \
- 3.77
-Ost index 0 Read time 0.17 Write time \
- 0.27
-Ost index 1 Read speed 7.38 Write speed \
- 6.31
-Ost index 1 Read time 0.14 Write time \
- 0.16
-Ost index 2 Read speed 6.98 Write speed \
- 6.16
-Ost index 2 Read time 0.14 Write time \
- 0.16
+Ost index 0 Read speed 5.84 Write speed 3.77
+Ost index 0 Read time 0.17 Write time 0.27
+Ost index 1 Read speed 7.38 Write speed 6.31
+Ost index 1 Read time 0.14 Write time 0.16
+Ost index 2 Read speed 6.98 Write speed 6.16
+Ost index 2 Read time 0.14 Write time 0.16
</screen>
</section>
<section xml:id="dbdoclet.50438212_58201">
<para> <emphasis role="bold">MGS/MDS node</emphasis></para>
</entry>
<entry>
- <para> <literal>mdt1</literal></para>
+ <para> <literal>mdt0</literal></para>
</entry>
<entry>
<para>MDS in Lustre file system temp</para>
<para> <literal>/mnt/mdt</literal></para>
</entry>
<entry>
- <para>Mount point for the <literal>mdt1</literal> block device (<literal>/dev/sdb</literal>) on the MGS/MDS node</para>
+ <para>Mount point for the <literal>mdt0</literal> block device (<literal>/dev/sdb</literal>) on the MGS/MDS node</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">OSS node</emphasis></para>
</entry>
<entry>
- <para><literal> oss1</literal></para>
+ <para><literal> oss0</literal></para>
</entry>
<entry>
<para>First OSS node in Lustre file system temp</para>
<para> <emphasis role="bold">OST</emphasis></para>
</entry>
<entry>
- <para><literal> ost1</literal></para>
+ <para><literal>ost0</literal></para>
</entry>
<entry>
<para>First OST in Lustre file system temp</para>
<para> <literal>/dev/sdc</literal></para>
</entry>
<entry>
- <para>Block device for the first OSS node (<literal>oss1</literal>)</para>
+ <para>Block device for the first OSS node (<literal>oss0</literal>)</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">mount point</emphasis></para>
</entry>
<entry>
- <para> <literal>/mnt/ost1</literal></para>
+ <para> <literal>/mnt/ost0</literal></para>
</entry>
<entry>
- <para> Mount point for the <literal>ost1</literal> block device (<literal>/dev/sdc</literal>) on the <literal>oss1</literal> node</para>
+ <para> Mount point for the <literal>ost0</literal> block device (<literal>/dev/sdc</literal>) on the <literal>oss1</literal> node</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">OSS node</emphasis></para>
</entry>
<entry>
- <para><literal>oss2</literal></para>
+ <para><literal>oss1</literal></para>
</entry>
<entry>
<para>Second OSS node in Lustre file system temp</para>
<para> <emphasis role="bold">OST</emphasis></para>
</entry>
<entry>
- <para> <literal>ost2</literal></para>
+ <para> <literal>ost1</literal></para>
</entry>
<entry>
<para>Second OST in Lustre file system temp</para>
<para><literal>/dev/sdd</literal></para>
</entry>
<entry>
- <para>Block device for the second OSS node (oss2)</para>
+ <para>Block device for the second OSS node (oss1)</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">mount point</emphasis></para>
</entry>
<entry>
- <para><literal>/mnt/ost2</literal></para>
+ <para><literal>/mnt/ost1</literal></para>
</entry>
<entry>
- <para> Mount point for the <literal>ost2</literal> block device (<literal>/dev/sdd</literal>) on the <literal>oss2</literal> node</para>
+ <para> Mount point for the <literal>ost1</literal> block device (<literal>/dev/sdd</literal>) on the <literal>oss1</literal> node</para>
</entry>
</row>
<row>
<screen>[root@mds /]# mount -t lustre /dev/sdb /mnt/mdt</screen>
<para>This command generates this output:</para>
<screen>Lustre: temp-MDT0000: new disk, initializing
-Lustre: 3009:0:(lproc_mds.c:262:lprocfs_wr_group_upcall()) temp-MDT0000: gr\
-oup upcall set to /usr/sbin/l_getgroups
+Lustre: 3009:0:(lproc_mds.c:262:lprocfs_wr_group_upcall()) temp-MDT0000:
+group upcall set to /usr/sbin/l_getgroups
Lustre: temp-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups
Lustre: Server temp-MDT0000 on device /dev/sdb has started </screen>
</listitem>
<listitem xml:id="dbdoclet.50438267_pgfId-1291170">
- <para>Create and mount <literal>ost1</literal>.</para>
- <para>In this example, the OSTs (<literal>ost1</literal> and <literal>ost2</literal>) are being created on different OSSs (<literal>oss1</literal> and <literal>oss2</literal> respectively).</para>
+ <para>Create and mount <literal>ost0</literal>.</para>
+ <para>In this example, the OSTs (<literal>ost0</literal> and <literal>ost1</literal>) are being created on different OSS nodes (<literal>oss0</literal> and <literal>oss1</literal> respectively).</para>
<orderedlist>
<listitem>
- <para>Create <literal>ost1</literal>. On <literal>oss1</literal> node, run:</para>
- <screen>[root@oss1 /]# mkfs.lustre --fsname=temp --mgsnode=10.2.0.1@tcp0 --ost --index=0 /dev/sdc</screen>
+ <para>Create <literal>ost0</literal>. On <literal>oss0</literal> node, run:</para>
+ <screen>[root@oss0 /]# mkfs.lustre --fsname=temp --mgsnode=10.2.0.1@tcp0 --ost --index=0 /dev/sdc</screen>
<para>The command generates this output:</para>
<screen> Permanent disk data:
Target: temp-OST0000
Writing CONFIGS/mountdata </screen>
</listitem>
<listitem>
- <para>Mount ost1 on the OSS on which it was created. On <literal>oss1</literal> node, run:</para>
- <screen>root@oss1 /] mount -t lustre /dev/sdc /mnt/ost1 </screen>
+ <para>Mount ost0 on the OSS on which it was created. On <literal>oss0</literal> node, run:</para>
+ <screen>root@oss0 /] mount -t lustre /dev/sdc /mnt/ost0</screen>
<para>The command generates this output:</para>
<screen>LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
</orderedlist>
</listitem>
<listitem>
- <para>Create and mount <literal>ost2</literal>.</para>
+ <para>Create and mount <literal>ost1</literal>.</para>
<orderedlist>
<listitem>
- <para>Create ost2. On <literal>oss2</literal> node, run:</para>
- <screen>[root@oss2 /]# mkfs.lustre --fsname=temp --mgsnode=10.2.0.1@tcp0 \
---ost --index=1 /dev/sdd</screen>
+ <para>Create ost1. On <literal>oss1</literal> node, run:</para>
+ <screen>[root@oss1 /]# mkfs.lustre --fsname=temp --mgsnode=10.2.0.1@tcp0 \
+ --ost --index=1 /dev/sdd</screen>
<para>The command generates this output:</para>
<screen> Permanent disk data:
Target: temp-OST0001
Writing CONFIGS/mountdata </screen>
</listitem>
<listitem>
- <para>Mount ost2 on the OSS on which it was created. On <literal>oss2</literal> node, run:</para>
- <screen>root@oss2 /] mount -t lustre /dev/sdd /mnt/ost2 </screen>
+ <para>Mount ost1 on the OSS on which it was created. On <literal>oss1</literal> node, run:</para>
+ <screen>root@oss1 /] mount -t lustre /dev/sdd /mnt/ost1 </screen>
<para>The command generates this output:</para>
<screen>LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
<para>Run the <literal>lfs df -h</literal> command:</para>
<screen>[root@client1 /] lfs df -h </screen>
<para>The <literal>lfs df -h</literal> command lists space usage per OST and the MDT in human-readable format. This command generates output similar to this:</para>
- <screen>UUID bytes Used Available \
- Use% Mounted on
-temp-MDT0000_UUID 8.0G 400.0M 7.6G 0% \
-/lustre[MDT:0]
-temp-OST0000_UUID 800.0G 400.0M 799.6G 0% \
-/lustre[OST:0]
-temp-OST0001_UUID 800.0G 400.0M 799.6G 0% \
-/lustre[OST:1]
-filesystem summary: 1.6T 800.0M 1.6T \
-0% /lustre</screen>
+ <screen>
+UUID bytes Used Available Use% Mounted on
+temp-MDT0000_UUID 8.0G 400.0M 7.6G 0% /lustre[MDT:0]
+temp-OST0000_UUID 800.0G 400.0M 799.6G 0% /lustre[OST:0]
+temp-OST0001_UUID 800.0G 400.0M 799.6G 0% /lustre[OST:1]
+filesystem summary: 1.6T 800.0M 1.6T 0% /lustre</screen>
</listitem>
<listitem>
<para>Run the <literal>lfs df -ih</literal> command.</para>
<screen>[root@client1 /] lfs df -ih</screen>
<para>The <literal>lfs df -ih</literal> command lists inode usage per OST and the MDT. This command generates output similar to this:</para>
- <screen>UUID Inodes IUsed IFree IUse% Mounted\
- on
-temp-MDT0000_UUID 2.5M 32 2.5M 0% /lustre[MDT:0]
-temp-OST0000_UUID 5.5M 54 5.5M 0% /lustre[OST:0]
-temp-OST0001_UUID 5.5M 54 5.5M 0% /lustre[OST:1]
-filesystem summary: 2.5M 32 2.5M 0% /lustre</screen>
+ <screen>
+UUID Inodes IUsed IFree IUse% Mounted on
+temp-MDT0000_UUID 2.5M 32 2.5M 0% /lustre[MDT:0]
+temp-OST0000_UUID 5.5M 54 5.5M 0% /lustre[OST:0]
+temp-OST0001_UUID 5.5M 54 5.5M 0% /lustre[OST:1]
+filesystem summary: 2.5M 32 2.5M 0% /lustre</screen>
</listitem>
<listitem>
<para>Run the <literal>dd</literal> command:</para>
</orderedlist>
<para>The Lustre mount command no longer recognizes the <literal>usrquota</literal> and <literal>grpquota</literal> options. If they were previously specified, remove them from <literal>/etc/fstab</literal>.</para>
<para>When quota is enabled, it is enabled for all file system clients (started automatically using <literal>quota_type</literal> or manually with <literal>lfs quotaon</literal>).</para>
- <note>
- <para>Lustre with the Linux kernel 2.4 does <emphasis>not</emphasis> support quotas.</para>
- </note>
<para>To enable quotas automatically when the file system is started, you must set the <literal>mdt.quota_type</literal> and <literal>ost.quota_type</literal> parameters, respectively, on the MDT and OSTs. The parameters can be set to the string <literal>u</literal> (user), <literal>g</literal> (group) or <literal>ug</literal> for both users and groups.</para>
<para>You can enable quotas at <literal>mkfs</literal> time (<literal>mkfs.lustre --param mdt.quota_type=ug</literal>) or with <literal>tunefs.lustre</literal>. As an example:</para>
<screen>tunefs.lustre --param ost.quota_type=ug $ost_dev</screen>
<indexterm><primary>maintance</primary><secondary>inactive OSTs</secondary></indexterm>
Working with Inactive OSTs</title>
<para>To mount a client or an MDT with one or more inactive OSTs, run commands similar to this:</para>
- <screen>client> mount -o exclude=testfs-OST0000 -t lustre uml1:/testfs\ /mnt/testfs
+ <screen>client> mount -o exclude=testfs-OST0000 -t lustre \
+ uml1:/testfs /mnt/testfs
client> cat /proc/fs/lustre/lov/testfs-clilov-*/target_obd</screen>
<para>To activate an inactive OST on a live client or MDT, use the <literal>lctl activate</literal> command on the OSC device. For example:</para>
<screen>lctl --device 7 activate</screen>
<screen>[client]# lfs find --obd <OST UUID> <mount_point> | lfs_migrate -y</screen>
</listitem>
<listitem>
- <para>If the OST is no longer available, delete the files on that OST and restore them from backup: <screen>[client]# lfs find --obd <OST UUID> -print0 <mount_point> | \
- tee /tmp/files_to_restore | xargs -0 -n 1 unlink</screen>The list of files that need to be restored from backup is stored in <literal>/tmp/files_to_restore</literal>. Restoring these files is beyond the scope of this document. </para>
+ <para>If the OST is no longer available, delete the files on that OST and restore them from backup:
+ <screen>[client]# lfs find --obd <OST UUID> -print0 <mount_point> | \
+ tee /tmp/files_to_restore | xargs -0 -n 1 unlink</screen>
+ The list of files that need to be restored from backup is stored in <literal>/tmp/files_to_restore</literal>. Restoring these files is beyond the scope of this document.</para>
</listitem>
</orderedlist>
</listitem>
<listitem>
<para>Back up the OST configuration files.
<screen>[oss]# tar cvf {ostname}.tar -C /mnt/ost last_rcvd \
-CONFIGS/ O/0/LAST_ID</screen>
+ CONFIGS/ O/0/LAST_ID</screen>
</para>
</listitem>
<listitem>
<listitem>
<para>
Format the OST file system.
- <screen>[oss]# mkfs.lustre --ost --index {OST index} {other options} \
-{newdev}</screen>
+ <screen>[oss]# mkfs.lustre --ost --index={old OST index} {other options} \
+ {newdev}</screen>
</para>
</listitem>
<listitem>
The <literal>CONFIGS/mountdata</literal> file is created by <literal>mkfs.lustre</literal> at format time, but has flags set that request it to register itself with the MGS. It is possible to copy these flags from another working OST (which should be the same):
-<screen>[oss2]# debugfs -c -R "dump CONFIGS/mountdata /tmp/ldd" {other_osdev}
-[oss2]# scp /tmp/ldd oss:/tmp/ldd
-[oss]# dd if=/tmp/ldd of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 skip=5</screen></para>
+<screen>[oss1]# debugfs -c -R "dump CONFIGS/mountdata /tmp/ldd" {other_osdev}
+[oss1]# scp /tmp/ldd oss:/tmp/ldd
+[oss0]# dd if=/tmp/ldd of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 skip=5</screen></para>
</listitem>
<listitem>
<para>
<para>Mount-by-label should NOT be used in a multi-path environment.</para>
</caution>
<para>Although the file system name is internally limited to 8 characters, you can mount the clients at any mount point, so file system users are not subjected to short names. Here is an example:</para>
- <screen>mount -t lustre uml1@tcp0:/shortfs /mnt/<long-file_system-name></screen>
+ <screen>mount -t lustre mds0@tcp0:/shortfs /mnt/<long-file_system-name></screen>
</section>
<section xml:id="dbdoclet.50438194_24122">
<title><indexterm><primary>operations</primary><secondary>starting</secondary></indexterm>Starting Lustre</title>
<para> In <emphasis>failover</emphasis> mode, Lustre clients wait for the OST to recover.</para>
</listitem>
</itemizedlist>
- <para>By default, the Lustre file system uses failover mode for OSTs. To specify failout mode instead, run this command:</para>
+ <para>By default, the Lustre file system uses failover mode for OSTs. To specify failout mode instead, use the <literal>--param="failover.mode=failout"</literal> option:</para>
<screen>$ mkfs.lustre --fsname=<fsname> --mgsnode=<MGS node NID> --param="failover.mode=failout" --ost --index="OST index" <block device name></screen>
- <para>In this example, failout mode is specified for the OSTs on MGS <literal>uml1</literal>, file system <literal>testfs</literal>.</para>
- <screen>$ mkfs.lustre --fsname=testfs --mgsnode=uml1 --param="failover.mode=failout" --ost --index=3 /dev/sdb </screen>
+ <para>In this example, failout mode is specified for the OSTs on MGS <literal>mds0</literal>, file system <literal>testfs</literal>.</para>
+ <screen>$ mkfs.lustre --fsname=testfs --mgsnode=mds0 --param="failover.mode=failout" --ost --index=3 /dev/sdb </screen>
<caution>
<para>Before running this command, unmount all OSTs that will be affected by the change in the failover/failout mode.</para>
</caution>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438194_17237">
- <title>Setting Parameters with <literal>mkfs.lustre</literal></title>
- <para>When the file system is created, parameters can simply be added as a <literal>--param</literal> option to the <literal>mkfs.lustre</literal> command. For example:</para>
+ <title>Setting Tunable Parameters with <literal>mkfs.lustre</literal></title>
+ <para>When the file system is first formatted, parameters can simply be added as a <literal>--param</literal> option to the <literal>mkfs.lustre</literal> command. For example:</para>
<screen>$ mkfs.lustre --mdt --param="sys.timeout=50" /dev/sda</screen>
<para>For more details about creating a file system,see <xref linkend="configuringlustre"/>. For more details about <literal>mkfs.lustre</literal>, see <xref linkend="systemconfigurationutilities"/>.</para>
</section>
<section xml:id="dbdoclet.50438194_55253">
<title>Setting Parameters with <literal>tunefs.lustre</literal></title>
- <para>If a server (OSS or MDS) is stopped, parameters can be added using the <literal>--param</literal> option to the <literal>tunefs.lustre</literal> command. For example:</para>
+ <para>If a server (OSS or MDS) is stopped, parameters can be added to an existing filesystem using the <literal>--param</literal> option to the <literal>tunefs.lustre</literal> command. For example:</para>
<screen>$ tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda</screen>
<para>With <literal>tunefs.lustre</literal>, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old <literal>tunefs.lustre</literal> parameters and just use newly-specified parameters, run:</para>
<screen>$ tunefs.lustre --erase-params --param=<new parameters> </screen>
<para>This example reports data on RPC service times.</para>
<screen>$ lctl get_param -n ost.*.ost_io.timeouts
service : cur 1 worst 30 (at 1257150393, 85d23h58m54s ago) 1 1 1 1 </screen>
- <para>This example reports the number of inodes available on each OST.</para>
- <screen># lctl get_param osc.*.filesfree
-osc.myth-OST0000-osc-ffff88006dd20000.filesfree=217623
-osc.myth-OST0001-osc-ffff88006dd20000.filesfree=5075042
-osc.myth-OST0002-osc-ffff88006dd20000.filesfree=3762034
-osc.myth-OST0003-osc-ffff88006dd20000.filesfree=91052
-osc.myth-OST0004-osc-ffff88006dd20000.filesfree=129651</screen>
+ <para>This example reports the amount of space this client has reserved for writeback cache with each OST:</para>
+ <screen># lctl get_param osc.*.cur_grant_bytes
+osc.myth-OST0000-osc-ffff8800376bdc00.cur_grant_bytes=2097152
+osc.myth-OST0001-osc-ffff8800376bdc00.cur_grant_bytes=33890304
+osc.myth-OST0002-osc-ffff8800376bdc00.cur_grant_bytes=35418112
+osc.myth-OST0003-osc-ffff8800376bdc00.cur_grant_bytes=2097152
+osc.myth-OST0004-osc-ffff8800376bdc00.cur_grant_bytes=33808384</screen>
</section>
</section>
</section>
<para>If a node has multiple network interfaces, it may have multiple NIDs. When a node is specified, all of its NIDs must be listed, delimited by commas (<literal>,</literal>) so other nodes can choose the NID that is appropriate for their network interfaces. When failover nodes are specified, they are delimited by a colon (<literal>:</literal>) or by repeating a keyword (<literal>--mgsnode=</literal> or <literal>--failnode=</literal>). To obtain all NIDs from a node (while LNET is running), run:</para>
<screen>lctl list_nids</screen>
<para>This displays the server's NIDs (networks configured to work with Lustre).</para>
- <para>This example has a combined MGS/MDT failover pair on uml1 and uml2, and a OST failover pair on uml3 and uml4. There are corresponding Elan addresses on uml1 and uml2.</para>
- <screen>uml1> mkfs.lustre --fsname=testfs --mgs --mdt --index=0 --failnode=uml2,2@elan /dev/sda1
-uml1> mount -t lustre /dev/sda1 /mnt/test/mdt
-uml3> mkfs.lustre --fsname=testfs --failnode=uml4 --mgsnode=uml1,1@elan \
---mgsnode=uml2,2@elan --ost --index=0 /dev/sdb
-uml3> mount -t lustre /dev/sdb /mnt/test/ost0
-client> mount -t lustre uml1,1@elan:uml2,2@elan:/testfs /mnt/testfs
-uml1> umount /mnt/mdt
-uml2> mount -t lustre /dev/sda1 /mnt/test/mdt
-uml2> cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status</screen>
- <para>Where multiple NIDs are specified, comma-separation (for example, <literal>uml2,2@elan</literal>) means that the two NIDs refer to the same host, and that Lustre needs to choose the "best" one for communication. Colon-separation (for example, <literal>uml1:uml2</literal>) means that the two NIDs refer to two different hosts, and should be treated as failover locations (Lustre tries the first one, and if that fails, it tries the second one.)</para>
+ <para>This example has a combined MGS/MDT failover pair on mds0 and mds1, and a OST failover pair on oss0 and oss1. There are corresponding Elan addresses on mds0 and mds1.</para>
+ <screen>mds0> mkfs.lustre --fsname=testfs --mdt --mgs --failnode=mds1,2@elan /dev/sda1
+mds0> mount -t lustre /dev/sda1 /mnt/test/mdt
+oss0> mkfs.lustre --fsname=testfs --failnode=oss1 --ost --index=0 \
+ --mgsnode=mds0,1@elan --mgsnode=mds1,2@elan /dev/sdb
+oss0> mount -t lustre /dev/sdb /mnt/test/ost0
+client> mount -t lustre mds0,1@elan:mds1,2@elan:/testfs /mnt/testfs
+mds0> umount /mnt/mdt
+mds1> mount -t lustre /dev/sda1 /mnt/test/mdt
+mds1> cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status</screen>
+ <para>Where multiple NIDs are specified, comma-separation (for example, <literal>mds1,2@elan</literal>) means that the two NIDs refer to the same host, and that Lustre needs to choose the "best" one for communication. Colon-separation (for example, <literal>mds0:mds1</literal>) means that the two NIDs refer to two different hosts, and should be treated as failover locations (Lustre tries the first one, and if that fails, it tries the second one.)</para>
<note>
<para>If you have an MGS or MDT configured for failover, perform these steps:</para>
<orderedlist>
<listitem>
- <para>On the OST, list the NIDs of all MGS nodes at <literal>mkfs</literal> time.</para>
- <screen>OST# mkfs.lustre --fsname sunfs --mgsnode=10.0.0.1 \
+ <para>On the oss0 node, list the NIDs of all MGS nodes at <literal>mkfs</literal> time.</para>
+ <screen>oss0# mkfs.lustre --fsname sunfs --mgsnode=10.0.0.1 \
--mgsnode=10.0.0.2 --ost --index=0 /dev/sdb</screen>
</listitem>
<listitem>
<section xml:id="dbdoclet.50438194_70905">
<title><indexterm><primary>operations</primary><secondary>erasing a file system</secondary></indexterm>Erasing a File System</title>
<para>If you want to erase a file system, run this command on your targets:</para>
- <screen>$ "mkfs.lustre -reformat"</screen>
+ <screen>$ "mkfs.lustre --reformat"</screen>
<para>If you are using a separate MGS and want to keep other file systems defined on that MGS, then set the <literal>writeconf</literal> flag on the MDT for that file system. The <literal>writeconf</literal> flag causes the configuration logs to be erased; they are regenerated the next time the servers start.</para>
<para>To set the <literal>writeconf</literal> flag on the MDT:</para>
<orderedlist>
<screen>$ mkfs.lustre --reformat --fsname spfs --mgs --mdt --index=0 /dev/sda</screen>
</listitem>
<listitem>
- <para>If you have a separate MGS (that you do not want to reformat), then add the "writeconf" flag to <literal>mkfs.lustre</literal> on the MDT, run:</para>
- <screen>$ mkfs.lustre --reformat --writeconf -fsname spfs --mgs --mdt --index=0 /dev/sda</screen>
+ <para>If you have a separate MGS (that you do not want to reformat), then add the "--writeconf" flag to <literal>mkfs.lustre</literal> on the MDT, run:</para>
+ <screen>$ mkfs.lustre --reformat --writeconf --fsname spfs --mgs --mdt /dev/sda</screen>
</listitem>
</orderedlist>
<note>
0-106495]</screen>
<para>If this happens, the client will re-read or re-write the affected data up to five times to get a good copy of the data over the network. If it is still not possible, then an I/O error is returned to the application.</para>
<para>To enable both types of checksums (in-memory and wire), run:</para>
- <screen>echo 1 > /proc/fs/lustre/llite/<emphasis><fsname></emphasis>/checksum_pages</screen>
+ <screen>lctl set_param llite.*.checksum_pages=1</screen>
<para>To disable both types of checksums (in-memory and wire), run:</para>
- <screen>echo 0 > /proc/fs/lustre/llite/<emphasis><fsname></emphasis>/checksum_pages</screen>
+ <screen>lctl set_param llite.*.checksum_pages=0</screen>
<para>To check the status of a wire checksum, run:</para>
<screen>lctl get_param osc.*.checksums</screen>
<section remap="h4">
<title>Changing Checksum Algorithms</title>
- <para>By default, Lustre uses the adler32 checksum algorithm, because it is robust and has a lower impact on performance than crc32. The Lustre administrator can change the checksum algorithm via /proc, depending on what is supported in the kernel.</para>
+ <para>By default, Lustre uses the adler32 checksum algorithm, because it is robust and has a lower impact on performance than crc32. The Lustre administrator can change the checksum algorithm via <literal>lctl get_param</literal>, depending on what is supported in the kernel.</para>
<para>To check which checksum algorithm is being used by Lustre, run:</para>
- <screen>$ cat /proc/fs/lustre/osc/<fsname>-OST<index>-osc-*/checksum_type</screen>
+ <screen>$ lctl get_param osc.*.checksum_type</screen>
<para>To change the wire checksum algorithm used by Lustre, run:</para>
- <screen>$ echo <algorithm name> /proc/fs/lustre/osc/<fsname>-OST<index>- \osc-*/checksum_\
-type</screen>
+ <screen>$ lctl set_param osc.*.checksum_type=<algorithm name></screen>
<note>
<para>The in-memory checksum always uses the adler32 algorithm, if available, and only falls back to crc32 if adler32 cannot be used.</para>
</note>
- <para>In the following example, the cat command is used to determine that Lustre is using the adler32 checksum algorithm. Then the echo command is used to change the checksum algorithm to crc32. A second cat command confirms that the crc32 checksum algorithm is now in use.</para>
- <screen>$ cat /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/checksum_ty\
-pe
-crc32 [adler]
-$ echo crc32 > /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/che\
-cksum_type
-$ cat /proc/fs/lustre/osc/lustre-OST0000-osc- \ffff81012b2c48e0/checksum_ty\
-pe
-[crc32] adler</screen>
+ <para>In the following example, the <literal>lctl get_param</literal> command is used to determine that Lustre is using the adler32 checksum algorithm. Then the <literal>lctl set_param</literal> command is used to change the checksum algorithm to crc32. A second <literal>lctl get_param</literal> command confirms that the crc32 checksum algorithm is now in use.</para>
+ <screen>$ lctl get_param osc.*.checksum_type
+osc.lustre-OST0000-osc-ffff81012b2c48e0.checksum_type=crc32 [adler]
+$ lctl set_param osc.*.checksum_type=crc32
+osc.lustre-OST0000-osc-ffff81012b2c48e0.checksum_type=crc32
+$ lctl get_param osc.*.checksum_type
+osc.lustre-OST0000-osc-ffff81012b2c48e0.checksum_type=[crc32] adler</screen>
</section>
</section>
</section>
<para>The <literal>lctl conf_param</literal> value overwrites the parameter's previous value. If the new value uses an incorrect syntax, then the system continues with the old parameters and the previously-correct value is lost on remount. That is, be careful doing root squash tuning.</para>
</listitem>
<listitem>
- <para><literal>mkfs.lustre</literal> and <literal>tunefs.lustre</literal> do not perform syntax checking. If the root squash parameters are incorrect, they are ignored on mount and the default values are used instead.</para>
+ <para><literal>mkfs.lustre</literal> and <literal>tunefs.lustre</literal> do not perform parameter syntax checking. If the root squash parameters are incorrect, they are ignored on mount and the default values are used instead.</para>
</listitem>
<listitem>
<para>Root squash parameters are parsed with rigorous syntax checking. The root_squash parameter should be specified as <literal><decnum>':'<decnum></literal>. The <literal>nosquash_nids</literal> parameter should follow LNET NID range list syntax.</para>
git://git.whamcloud.com / doc / manual.git / commitdiff