</indexterm>Handling Full OSTs</title>
<para>Sometimes a Lustre file system becomes unbalanced, often due to
incorrectly-specified stripe settings, or when very large files are created
- that are not striped over all of the OSTs. If an OST is full and an attempt
- is made to write more information to the file system, an error occurs. The
- procedures below describe how to handle a full OST.</para>
+ that are not striped over all of the OSTs. Lustre will automatically avoid
+ allocating new files on OSTs that are full. If an OST is completely full and
+ more data is written to files already located on that OST, an error occurs.
+ The procedures below describe how to handle a full OST.</para>
<para>The MDS will normally handle space balancing automatically at file
- creation time, and this procedure is normally not needed, but may be
- desirable in certain circumstances (e.g. when creating very large files
+ creation time, and this procedure is normally not needed, but manual data
+ migration may be desirable in some cases (e.g. creating very large files
that would consume more than the total free space of the full OSTs).</para>
<section remap="h3">
<title>
client# lfs df -h
UUID bytes Used Available \
Use% Mounted on
-lustre-MDT0000_UUID 4.4G 214.5M 3.9G \
-4% /mnt/lustre[MDT:0]
-lustre-OST0000_UUID 2.0G 751.3M 1.1G \
-37% /mnt/lustre[OST:0]
-lustre-OST0001_UUID 2.0G 755.3M 1.1G \
-37% /mnt/lustre[OST:1]
-lustre-OST0002_UUID 2.0G 1.7G 155.1M \
-86% /mnt/lustre[OST:2] <-
-lustre-OST0003_UUID 2.0G 751.3M 1.1G \
-37% /mnt/lustre[OST:3]
-lustre-OST0004_UUID 2.0G 747.3M 1.1G \
-37% /mnt/lustre[OST:4]
-lustre-OST0005_UUID 2.0G 743.3M 1.1G \
-36% /mnt/lustre[OST:5]
+testfs-MDT0000_UUID 4.4G 214.5M 3.9G \
+4% /mnt/testfs[MDT:0]
+testfs-OST0000_UUID 2.0G 751.3M 1.1G \
+37% /mnt/testfs[OST:0]
+testfs-OST0001_UUID 2.0G 755.3M 1.1G \
+37% /mnt/testfs[OST:1]
+testfs-OST0002_UUID 2.0G 1.7G 155.1M \
+86% /mnt/testfs[OST:2] ****
+testfs-OST0003_UUID 2.0G 751.3M 1.1G \
+37% /mnt/testfs[OST:3]
+testfs-OST0004_UUID 2.0G 747.3M 1.1G \
+37% /mnt/testfs[OST:4]
+testfs-OST0005_UUID 2.0G 743.3M 1.1G \
+36% /mnt/testfs[OST:5]
filesystem summary: 11.8G 5.4G 5.8G \
-45% /mnt/lustre
+45% /mnt/testfs
</screen>
- <para>In this case, OST:2 is almost full and when an attempt is made to
+ <para>In this case, OST0002 is almost full and when an attempt is made to
write additional information to the file system (even with uniform
striping over all the OSTs), the write command fails as follows:</para>
<screen>
-client# lfs setstripe /mnt/lustre 4M 0 -1
-client# dd if=/dev/zero of=/mnt/lustre/test_3 bs=10M count=100
-dd: writing '/mnt/lustre/test_3': No space left on device
+client# lfs setstripe /mnt/testfs 4M 0 -1
+client# dd if=/dev/zero of=/mnt/testfs/test_3 bs=10M count=100
+dd: writing '/mnt/testfs/test_3': No space left on device
98+0 records in
97+0 records out
1017192448 bytes (1.0 GB) copied, 23.2411 seconds, 43.8 MB/s
<title>
<indexterm>
<primary>I/O</primary>
- <secondary>taking OST offline</secondary>
- </indexterm>Taking a Full OST Offline</title>
+ <secondary>disabling OST creates</secondary>
+ </indexterm>Disabling creates on a Full OST</title>
<para>To avoid running out of space in the file system, if the OST usage
is imbalanced and one or more OSTs are close to being full while there
- are others that have a lot of space, the full OSTs may optionally be
- deactivated at the MDS to prevent the MDS from allocating new objects
- there.</para>
+ are others that have a lot of space, the MDS will typically avoid file
+ creation on the full OST(s) automatically. The full OSTs may optionally
+ be deactivated manually on the MDS to ensure the MDS will not allocate
+ new objects there.</para>
<orderedlist>
<listitem>
- <para>Log into the MDS server:</para>
- <screen>
-client# ssh root@192.168.0.10
-root@192.168.0.10's password:
-Last login: Wed Nov 26 13:35:12 2008 from 192.168.0.6
-</screen>
- </listitem>
- <listitem>
- <para>Use the
- <literal>lctl dl</literal> command to show the status of all file
- system components:</para>
- <screen>
-mds# lctl dl
-0 UP mgs MGS MGS 9
-1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
-2 UP mdt MDS MDS_uuid 3
-3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
-4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
-5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
-6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
-7 UP osc lustre-OST0002-osc lustre-mdtlov_UUID 5
-8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
-9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
-10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5
-</screen>
- </listitem>
- <listitem>
- <para>Use
- <literal>lctl</literal> deactivate to take the full OST
- offline:</para>
- <screen>
-mds# lctl --device 7 deactivate
-</screen>
- </listitem>
- <listitem>
- <para>Display the status of the file system components:</para>
+ <para>Log into the MDS server and use the <literal>lctl</literal>
+ command to stop new object creation on the full OST(s):
+ </para>
<screen>
-mds# lctl dl
-0 UP mgs MGS MGS 9
-1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
-2 UP mdt MDS MDS_uuid 3
-3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
-4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
-5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
-6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
-7 IN osc lustre-OST0002-osc lustre-mdtlov_UUID 5
-8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
-9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
-10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5
+mds# lctl set_param osp.<replaceable>fsname</replaceable>-OST<replaceable>nnnn</replaceable>*.max_create_count=0
</screen>
</listitem>
</orderedlist>
- <para>The device list shows that OST0002 is now inactive. When new files
- are created in the file system, they will only use the remaining active
- OSTs. Either manual space rebalancing can be done by migrating data to
- other OSTs, as shown in the next section, or normal file deletion and
- creation can be allowed to passively rebalance the space usage.</para>
+ <para>When new files are created in the file system, they will only use
+ the remaining OSTs. Either manual space rebalancing can be done by
+ migrating data to other OSTs, as shown in the next section, or normal
+ file deletion and creation can passively rebalance the space usage.</para>
</section>
<section remap="h3">
<title>
<secondary>migrating data</secondary>
</indexterm>
<indexterm>
- <primary>migrating metadata</primary>
- </indexterm>
- <indexterm>
<primary>maintenance</primary>
<secondary>full OSTs</secondary>
</indexterm>Migrating Data within a File System</title>
- <para condition='l28'>Lustre software version 2.8 includes a
- feature to migrate metadata between MDTs. This migration can only be
- performed on whole directories. To migrate the contents of
- <literal>/lustre/testremote</literal> from the current MDT to
- MDT index 0, the sequence of commands is as follows:</para>
- <screen>$ cd /lustre
-lfs getdirstripe -M ./testremote <lineannotation>which MDT is dir on?</lineannotation>
-1
-$ for i in $(seq 3); do touch ./testremote/${i}.txt; done <lineannotation>create test files</lineannotation>
-$ for i in $(seq 3); do lfs getstripe -M ./testremote/${i}.txt; done <lineannotation>check files are on MDT 1</lineannotation>
-1
-1
-1
-$ lfs migrate -m 0 ./testremote <lineannotation>migrate testremote to MDT 0</lineannotation>
-$ lfs getdirstripe -M ./testremote <lineannotation>which MDT is dir on now?</lineannotation>
-0
-$ for i in $(seq 3); do lfs getstripe -M ./testremote/${i}.txt; done <lineannotation>check files are on MDT 0 too</lineannotation>
-0
-0
-0</screen>
- <para>For more information, see <literal>man lfs</literal></para>
- <warning><para>Currently, only whole directories can be migrated
- between MDTs. During migration each file receives a new identifier
- (FID). As a consequence, the file receives a new inode number. File
- system tools (for example, backup and archiving tools) may behave
- incorrectly with files that are unchanged except for a new inode number.
- </para></warning>
- <para>As stripes cannot be moved within the file system, data must be
- migrated manually by copying and renaming the file, removing the original
- file, and renaming the new file with the original file name. The simplest
- way to do this is to use the
- <literal>lfs_migrate</literal> command (see
- <xref linkend="dbdoclet.50438206_42260" />). However, the steps for
- migrating a file by hand are also shown here for reference.</para>
- <orderedlist>
- <listitem>
- <para>Identify the file(s) to be moved.</para>
- <para>In the example below, output from the
- <literal>getstripe</literal> command indicates that the file
- <literal>test_2</literal> is located entirely on OST2:</para>
- <screen>
-client# lfs getstripe /mnt/lustre/test_2
-/mnt/lustre/test_2
-obdidx objid objid group
- 2 8 0x8 0
-</screen>
- </listitem>
- <listitem>
- <para>To move single object(s), create a new copy and remove the
- original. Enter:</para>
- <screen>
-client# cp -a /mnt/lustre/test_2 /mnt/lustre/test_2.tmp
-client# mv /mnt/lustre/test_2.tmp /mnt/lustre/test_2
-</screen>
- </listitem>
- <listitem>
- <para>To migrate large files from one or more OSTs, enter:</para>
- <screen>
-client# lfs find --ost
-<replaceable>ost_name</replaceable> -size +1G | lfs_migrate -y
-</screen>
- </listitem>
- <listitem>
- <para>Check the file system balance.</para>
- <para>The
- <literal>df</literal> output in the example below shows a more
- balanced system compared to the
- <literal>df</literal> output in the example in
- <xref linkend="dbdoclet.50438211_17536" />.</para>
- <screen>
-client# lfs df -h
-UUID bytes Used Available Use% \
- Mounted on
-lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% \
- /mnt/lustre[MDT:0]
-lustre-OST0000_UUID 2.0G 1.3G 598.1M 65% \
- /mnt/lustre[OST:0]
-lustre-OST0001_UUID 2.0G 1.3G 594.1M 65% \
- /mnt/lustre[OST:1]
-lustre-OST0002_UUID 2.0G 913.4M 1000.0M 45% \
- /mnt/lustre[OST:2]
-lustre-OST0003_UUID 2.0G 1.3G 602.1M 65% \
- /mnt/lustre[OST:3]
-lustre-OST0004_UUID 2.0G 1.3G 606.1M 64% \
- /mnt/lustre[OST:4]
-lustre-OST0005_UUID 2.0G 1.3G 610.1M 64% \
- /mnt/lustre[OST:5]
-
-filesystem summary: 11.8G 7.3G 3.9G 61% \
-/mnt/lustre
-</screen>
- </listitem>
- </orderedlist>
+ <para>If there is a need to move the file data from the current
+ OST(s) to new OST(s), the data must be migrated (copied)
+ to the new location. The simplest way to do this is to use the
+ <literal>lfs_migrate</literal> command, as described in
+ <xref linkend="lustremaint.adding_new_ost" />.</para>
</section>
<section remap="h3">
<title>
<primary>maintenance</primary>
<secondary>bringing OST online</secondary>
</indexterm>Returning an Inactive OST Back Online</title>
- <para>Once the deactivated OST(s) no longer are severely imbalanced, due
+ <para>Once the full OST(s) no longer are severely imbalanced, due
to either active or passive data redistribution, they should be
reactivated so they will again have new files allocated on them.</para>
<screen>
-[mds]# lctl --device 7 activate
-[mds]# lctl dl
- 0 UP mgs MGS MGS 9
- 1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-816dd1e813 5
- 2 UP mdt MDS MDS_uuid 3
- 3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
- 4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
- 5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
- 6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
- 7 UP osc lustre-OST0002-osc lustre-mdtlov_UUID 5
- 8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
- 9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
- 10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID
+[mds]# lctl set_param osp.testfs-OST0002.max_create_count=20000
</screen>
</section>
+ <section remap="h3">
+ <title><indexterm>
+ <primary>migrating metadata</primary>
+ </indexterm>Migrating Metadata within a Filesystem</title>
+ <section remap="h3" condition='l28'>
+ <title><indexterm>
+ <primary>migrating metadata</primary>
+ </indexterm>Whole Directory Migration</title>
+ <para>Lustre software version 2.8 includes a feature
+ to migrate metadata (directories and inodes therein) between MDTs.
+ This migration can only be performed on whole directories. Striped
+ directories are not supported until Lustre 2.12. For example, to
+ migrate the contents of the <literal>/testfs/remotedir</literal>
+ directory from the MDT where it currently is located to MDT0000 to
+ allow that MDT to be removed, the sequence of commands is as follows:
+ </para>
+ <screen>$ cd /testfs
+$ lfs getdirstripe -m ./remotedir <lineannotation>which MDT is dir on?</lineannotation>
+1
+$ touch ./remotedir/file.{1,2,3}.txt<lineannotation>create test files</lineannotation>
+$ lfs getstripe -m ./remotedir/file.*.txt<lineannotation>check files are on MDT0001</lineannotation>
+1
+1
+1
+$ lfs migrate -m 0 ./remotedir <lineannotation>migrate testremote to MDT0000</lineannotation>
+$ lfs getdirstripe -m ./remotedir <lineannotation>which MDT is dir on now?</lineannotation>
+0
+$ lfs getstripe -m ./remotedir/file.*.txt<lineannotation>check files are on MDT0000</lineannotation>
+0
+0
+0</screen>
+ <para>For more information, see <literal>man lfs-migrate</literal>.
+ </para>
+ <warning><para>During migration each file receives a new identifier
+ (FID). As a consequence, the file will report a new inode number to
+ userspace applications. Some system tools (for example, backup and
+ archiving tools, NFS, Samba) that identify files by inode number may
+ consider the migrated files to be new, even though the contents are
+ unchanged. If a Lustre system is re-exporting to NFS, the migrated
+ files may become inaccessible during and after migration if the
+ client or server are caching a stale file handle with the old FID.
+ Restarting the NFS service will flush the local file handle cache,
+ but clients may also need to be restarted as they may cache stale
+ file handles as well.
+ </para></warning>
+ </section>
+ <section remap="h3" condition='l2C'>
+ <title><indexterm>
+ <primary>migrating metadata</primary>
+ </indexterm>Striped Directory Migration</title>
+ <para>Lustre 2.8 included a feature to migrate metadata (directories
+ and inodes therein) between MDTs, however it did not support migration
+ of striped directories, or changing the stripe count of an existing
+ directory. Lustre 2.12 adds support for migrating and restriping
+ directories. The <literal>lfs migrate -m</literal> command can only
+ only be performed on whole directories, though it will migrate both
+ the specified directory and its sub-entries recursively.
+ For example, to migrate the contents of a large directory
+ <literal>/testfs/largedir</literal> from its current location on
+ MDT0000 to MDT0001 and MDT0003, run the following command:</para>
+ <screen>$ lfs migrate -m 1,3 /testfs/largedir</screen>
+ <para>Metadata migration will migrate file dirent and inode to other
+ MDTs, but it won't touch file data. During migration, directory and
+ its sub-files can be accessed like normal ones, though the same
+ warning above applies to tools that depend on the file inode number.
+ Migration may fail for various reasons such as MDS restart, or disk
+ full. In those cases, some of the sub-files may have been migrated to
+ the new MDTs, while others are still on the original MDT. The files
+ can be accessed normally. The same <literal>lfs migrate -m</literal>
+ command should be executed again when these issues are fixed to finish
+ this migration. However, you cannot abort a failed migration, or
+ migrate to different MDTs from previous migration command.</para>
+ </section>
+ </section>
</section>
<section xml:id="dbdoclet.50438211_75549">
<title>
<replaceable>poolname</replaceable>
</screen>
<note>
- <para>The pool name is an ASCII string up to 16 characters.</para>
+ <para>The pool name is an ASCII string up to 15 characters.</para>
</note>
<para>To add the named OST to a pool, run:</para>
<screen>
<literal>_UUID</literal> are missing, they are automatically added.</para>
<para>For example, to add even-numbered OSTs to
<literal>pool1</literal> on file system
- <literal>lustre</literal>, run a single command (
+ <literal>testfs</literal>, run a single command (
<literal>pool_add</literal>) to add many OSTs to the pool at one
time:</para>
<para>
<screen>
-lctl pool_add lustre.pool1 OST[0-10/2]
+lctl pool_add testfs.pool1 OST[0-10/2]
</screen>
</para>
<note>
<para>To set striping patterns, run:</para>
<screen>
client# lfs setstripe [--size|-s stripe_size] [--offset|-o start_ost]
- [--count|-c stripe_count] [--pool|-p pool_name]
+ [--stripe-count|-c stripe_count] [--overstripe-count|-C stripe_count]
+ [--pool|-p pool_name]
<replaceable>dir|filename</replaceable>
</screen>
<listitem>
<para>Add a new OST by passing on the following commands, run:</para>
<screen>
-oss# mkfs.lustre --fsname=spfs --mgsnode=mds16@tcp0 --ost --index=12 /dev/sda
-oss# mkdir -p /mnt/test/ost12
-oss# mount -t lustre /dev/sda /mnt/test/ost12
+oss# mkfs.lustre --fsname=testfs --mgsnode=mds16@tcp0 --ost --index=12 /dev/sda
+oss# mkdir -p /mnt/testfs/ost12
+oss# mount -t lustre /dev/sda /mnt/testfs/ost12
</screen>
</listitem>
<listitem>
checksum algorithm is now in use.</para>
<screen>
$ lctl get_param osc.*.checksum_type
-osc.lustre-OST0000-osc-ffff81012b2c48e0.checksum_type=crc32 [adler]
+osc.testfs-OST0000-osc-ffff81012b2c48e0.checksum_type=crc32 [adler]
$ lctl set_param osc.*.checksum_type=crc32
-osc.lustre-OST0000-osc-ffff81012b2c48e0.checksum_type=crc32
+osc.testfs-OST0000-osc-ffff81012b2c48e0.checksum_type=crc32
$ lctl get_param osc.*.checksum_type
-osc.lustre-OST0000-osc-ffff81012b2c48e0.checksum_type=[crc32] adler
+osc.testfs-OST0000-osc-ffff81012b2c48e0.checksum_type=[crc32] adler
</screen>
</section>
</section>
git://git.whamcloud.com / doc / manual.git / blobdiff