From ff25c07facab19c2e373d662a343ee2a25955552 Mon Sep 17 00:00:00 2001 From: Richard Henwood Date: Mon, 30 Nov 2015 12:55:53 -0600 Subject: [PATCH] LUDOC-306 dne2: update with new command for DNE2 Distributed metadata phase 2 (DNE2) introduces the concept of a the metadata contents of a given directory striped across multiple MDTs. Update the manual to ensure the operation of this new feature is documented. Include description of migration tool. See also: LU-3531 Change-Id: I96f7012638358b9985480ec67539cf72adf3ced5 Signed-off-by: Richard Henwood Reviewed-on: http://review.whamcloud.com/17394 Tested-by: Jenkins --- ConfiguringLustre.xml | 4 +- Glossary.xml | 51 +++++++++++++---- LustreOperations.xml | 136 +++++++++++++++++++--------------------------- ManagingFileSystemIO.xml | 68 ++++++++++++----------- UnderstandingFailover.xml | 2 +- UnderstandingLustre.xml | 7 +++ 6 files changed, 139 insertions(+), 129 deletions(-) diff --git a/ConfiguringLustre.xml b/ConfiguringLustre.xml index 13e3106..3326556 100644 --- a/ConfiguringLustre.xml +++ b/ConfiguringLustre.xml @@ -127,8 +127,8 @@ mkfs.lustre --fsname= - Optional for Lustre software release 2.4 and later. Add in - additional MDTs. + Optional for Lustre software release 2.4 and later. + Add in additional MDTs. mkfs.lustre --fsname= fsname --mgsnode= diff --git a/Glossary.xml b/Glossary.xml index 980511e..a7d104d 100644 --- a/Glossary.xml +++ b/Glossary.xml @@ -79,11 +79,21 @@ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"> new subdirectories at the time they are created. - + Distributed namespace (DNE) - A collection of metadata targets implementing a single file - system namespace. + A collection of metadata targets serving a single file + system namespace. Prior to DNE, Lustre file systems were limited to a + single metadata target for the entire name space. Without the ability + to distribute metadata load over multiple targets, Lustre file system + performance is limited. Lustre was enhanced with DNE functionality in + two development phases. After completing the first phase of development + in Lustre software version 2.4, Remote Directories + allowed the metadata for sub-directories to be serviced by an + independent MDT(s). After completing the second phase of development in + Lustre software version 2.8, Striped Directories + allowed files in a single directory to be serviced by multiple MDTs. + @@ -653,6 +663,19 @@ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"> server restarts. + + Remote directory + + A remote directory describes a feature of + Lustre where metadata for files in a given directory may be + stored on a different MDT than the metadata for the parent + directory. Remote directories only became possible with the + advent of DNE Phase 1, which arrived in Lustre version + 2.4. Remote directories are available to system + administrators who wish to provide individual metadata + targets for individual workloads. + + Replay request @@ -712,7 +735,7 @@ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"> S - + Stripe A contiguous, logical extent of a Lustre file written to a single @@ -720,6 +743,18 @@ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"> of a file visible to user applications. + + Striped Directory + + A striped directory is a feature of Lustre + software where metadata for files in a given directory are + distributed evenly over multiple MDTs. Striped directories + are only available in Lustre software version 2.8 or later. + An administrator can use a striped directory to increase + metadata performance by distributing the metadata requests + in a single directory over two or more MDTs. + + Stripe size @@ -737,14 +772,6 @@ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"> file. - - Striping metadata - - The extended attribute associated with a file that describes how - its data is distributed over storage objects. See also - Default stripe pattern. - - T diff --git a/LustreOperations.xml b/LustreOperations.xml index 5b429f4..aa46def 100644 --- a/LustreOperations.xml +++ b/LustreOperations.xml @@ -5,79 +5,7 @@ xml:id="lustreoperations"> Lustre Operations Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre administration - tasks: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + tasks.
<indexterm> @@ -442,16 +370,15 @@ client# mount -t lustre mgsnode@tcp0:/bar /mnt/bar unique MDTs. An administrator can allocate a sub-directory to a given MDT using the command:</para> <screen> -client# lfs mkdir –i -<replaceable>mdt_index</replaceable> +client# lfs mkdir –i +<replaceable>mdt_index</replaceable> <replaceable>/mount_point/remote_dir</replaceable> - </screen> - <para>This command will allocate the sub-directory - <literal>remote_dir</literal> onto the MDT of index + <para>This command will allocate the sub-directory + <literal>remote_dir</literal> onto the MDT of index <literal>mdtindex</literal>. For more information on adding additional MDTs and - <literal>mdtindex</literal> see + <literal>mdtindex</literal> see <xref linkend='dbdoclet.addmdtindex' />.</para> <warning> <para>An administrator can allocate remote sub-directories to separate @@ -460,9 +387,56 @@ client# lfs mkdir –i will leave the namespace below it inaccessible. For this reason, by default it is only possible to create remote sub-directories off MDT0. To relax this restriction and enable remote sub-directories off any MDT, an - administrator must issue the command - <literal>lctl set_param mdd.*.enable_remote_dir=1</literal>.</para> + administrator must issue the command + <literal>lctl set_param mdt.*.enable_remote_dir=1</literal>.</para> </warning> + <para condition='l28'>With Lustre software version 2.8, a new + tunable is available to allow users with a specific group ID to create + and delete remote and striped directories. This tunable is + <literal>enable_remote_dir_gid</literal>. For example, setting this + parameter to the 'wheel' or 'admin' group ID allows users with that GID + to create and delete remote and striped directories. Setting this + parameter to <literal>-1</literal> on MDT0 to permanently allow any + non-root users create and delete remote and striped directories. For + example: + <screen>lctl set_param -P mdt.*.enable_remote_dir_gid=-1</screen> + </para> + </section> + <section xml:id="dbdoclet.lfsmkdirdne2" condition='l28'> + <title> + <indexterm> + <primary>operations</primary> + <secondary>striped directory</secondary> + </indexterm> + <indexterm> + <primary>operations</primary> + <secondary>mkdir</secondary> + </indexterm> + <indexterm> + <primary>operations</primary> + <secondary>setdirstripe</secondary> + </indexterm> + <indexterm> + <primary>striping</primary> + <secondary>metadata</secondary> + </indexterm>Creating a directory striped across multiple MDTs + Lustre 2.8 enables individual files in a given directory to + record their metadata on separate MDTs (a striped + directory). The result of this is that metadata requests for + files in a striped directory are serviced by multiple MDTs and metadata + service load is distributed over all the MDTs that service a given + directory. By distributing metadata service load over multiple MDTs, + performance can be improved beyond the limit of single MDT + performance. Prior to the development of this feature all files in a + directory must record their metadata on a single MDT. + This command to stripe a directory over + mdt_count MDTs is: + + +client# lfs setdirstripe -c +mdt_count +/mount_point/new_directory +
diff --git a/ManagingFileSystemIO.xml b/ManagingFileSystemIO.xml index 94ebf96..3f3142b 100644 --- a/ManagingFileSystemIO.xml +++ b/ManagingFileSystemIO.xml @@ -4,35 +4,6 @@ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingfilesystemio"> <title xml:id="managingfilesystemio.title">Managing the File System and I/O - This chapter describes file striping and I/O options, and includes the - following sections: - - - - - - - - - - - - - - - - - - - - - - - - - - -
<indexterm> @@ -170,21 +141,52 @@ mds# lctl dl <secondary>migrating data</secondary> </indexterm> <indexterm> + <primary>migrating metadata</primary> + </indexterm> + <indexterm> <primary>maintenance</primary> <secondary>full OSTs</secondary> </indexterm>Migrating Data within a File System + + Lustre software version 2.8 includes a + feature to migrate metadata between MDTs. This migration can only be + performed on whole directories. To migrate the contents of + /lustre/testremote from the current MDT to + MDT index 0, the sequence of commands is as follows: + $ cd /lustre +lfs getdirstripe -M ./testremote which MDT is dir on? +1 +$ for i in $(seq 3); do touch ./testremote/${i}.txt; done create test files +$ for i in $(seq 3); do lfs getstripe -M ./testremote/${i}.txt; done check files are on MDT 1 +1 +1 +1 +$ lfs migrate -m 0 ./testremote migrate testremote to MDT 0 +$ lfs getdirstripe -M ./testremote which MDT is dir on now? +0 +$ for i in $(seq 3); do lfs getstripe -M ./testremote/${i}.txt; done check files are on MDT 0 too +0 +0 +0 + For more information, see man lfs + Currently, only whole directories can be migrated + between MDTs. During migration each file receives a new identifier + (FID). As a consequence, the file receives a new inode number. File + system tools (for example, backup and archiving tools) may behave + incorrectly with files that are unchanged except for a new inode number. + As stripes cannot be moved within the file system, data must be migrated manually by copying and renaming the file, removing the original file, and renaming the new file with the original file name. The simplest - way to do this is to use the - lfs_migrate command (see + way to do this is to use the + lfs_migrate command (see ). However, the steps for migrating a file by hand are also shown here for reference. Identify the file(s) to be moved. - In the example below, output from the - getstripe command indicates that the file + In the example below, output from the + getstripe command indicates that the file test_2 is located entirely on OST2: client# lfs getstripe /mnt/lustre/test_2 diff --git a/UnderstandingFailover.xml b/UnderstandingFailover.xml index ef215cc..5317617 100644 --- a/UnderstandingFailover.xml +++ b/UnderstandingFailover.xml @@ -150,7 +150,7 @@ xml:id="understandingfailover"> For MDT failover, two MDSs can be configured to serve the same MDT. Only one MDS node can serve an MDT at a time. - Lustresoftware release 2.4 allows multiple MDTs. + Lustre software release 2.4 allows multiple MDTs. By placing two or more MDT partitions on storage shared by two MDSs, one MDS can fail and the remaining MDS can begin serving the unserved MDT. This is described as an active/active failover pair. diff --git a/UnderstandingLustre.xml b/UnderstandingLustre.xml index 4ee65f7..5332990 100644 --- a/UnderstandingLustre.xml +++ b/UnderstandingLustre.xml @@ -512,6 +512,13 @@ xml:id="understandinglustre"> machines share storage for two or more MDTs. After the failure of one MDS, the remaining MDS begins serving the MDT(s) of the failed MDS. + Since Lustre software release 2.8, + multiple MDTs can be employed to share the inode records for files + contained in a single directory. A directory for which inode records + are distributed across multiple MDTs is known as a striped + directory. In the case of a Lustre filesystem the inode + records maybe also be referred to as the 'metadata' portion of the + file record. -- 1.8.3.1