From: Andreas Dilger Date: Thu, 23 Jul 2020 00:28:22 +0000 (-0600) Subject: LUDOC-11 misc: remove references for older Lustre X-Git-Url: https://git.whamcloud.com/?a=commitdiff_plain;h=83d99d7c931163028a92ff92234d4b17289a3417;p=doc%2Fmanual.git LUDOC-11 misc: remove references for older Lustre Remove references to Lustre 2.4 and earlier from the manual. There were even places with referencs to Lustre 1.8 and 2.0. The text still exists in earlier builds and in Git for reference. This avoids confusing users with commands that no longer exist, or highlighting features that are included with all releases. Add conditions for Lustre 2.15 and 2.16 as they will be used soon. Signed-off-by: Andreas Dilger Change-Id: I0786d88fcc2543f542b7a880bf042928fd3ebbe5 Reviewed-on: https://review.whamcloud.com/39511 Tested-by: jenkins --- diff --git a/BackupAndRestore.xml b/BackupAndRestore.xml index 978c542..c26152c 100644 --- a/BackupAndRestore.xml +++ b/BackupAndRestore.xml @@ -518,8 +518,9 @@ Changelog records consumed: 42 backup OST and MDT Backing Up an OST or MDT - If backing up an MDT, substitute mdt for - ost in the instructions below. + The below examples show backing up an OST filesystem. When backing + up an MDT, substitute mdt for ost + in the instructions below. Umount the target diff --git a/BenchmarkingTests.xml b/BenchmarkingTests.xml index a971491..ae5a217 100644 --- a/BenchmarkingTests.xml +++ b/BenchmarkingTests.xml @@ -409,16 +409,16 @@ $ nobjhi=2 thrhi=2 size=1024 case=disk sh obdfilter-survey - Performance measurements for write, rewrite, read etc are + Performance measurements for write, rewrite, read etc are provided below: - # example output + # example output Fri Sep 25 11:14:03 EDT 2015 Obdfilter-survey for case=disk from hds1fnb6123 ost 10 sz 167772160K rsz 1024K obj 10 thr 10 write 10982.73 [ 601.97,2912.91] rewrite 15696.54 [1160.92,3450.85] read 12358.60 [ 938.96,2634.87] ... - The file + The file ./lustre-iokit/obdfilter-survey/README.obdfilter-survey provides an explaination for the output as follows: - ost 10 is the total number of OSTs under test. + ost 10 is the total number of OSTs under test. sz 167772160K is the total amount of data read or written (in bytes). rsz 1024K is the record size (size of each echo_client I/O, in bytes). obj 10 is the total number of objects over all OSTs @@ -473,7 +473,7 @@ over all OSTs. (nobjhi), up to two threads ( thrhi), and 1024 Mb (size) transfer size: $ nobjhi=2 thrhi=2 size=1024 targets="lustre-OST0001 \ - lustre-OST0002" sh obdfilter-survey + lustre-OST0002" sh obdfilter-survey @@ -510,7 +510,7 @@ over all OSTs. targets=hostname|ip_of_server . For example: $ nobjhi=2 thrhi=2 size=1024 targets="oss0 oss1" \ - case=network sh obdfilter-survey + case=network sh obdfilter-survey On the server side, view the statistics at: @@ -826,9 +826,9 @@ Ost# Read(MB/s) Write(MB/s) Read-time Write-time MDS performance Testing MDS Performance (mds-survey) The mds-survey script tests the local - metadata performance using the echo_client to drive different layers of - the MDS stack: mdd, mdt, osd (the Lustre software only supports mdd - stack). It can be used with the following classes of operations: + metadata performance using the echo_client to + drive the MDD layer of the MDS stack. + It can be used with the following classes of operations: Open-create/mkdir/create diff --git a/ConfiguringFailover.xml b/ConfiguringFailover.xml index fd918d7..a478d5c 100644 --- a/ConfiguringFailover.xml +++ b/ConfiguringFailover.xml @@ -152,25 +152,6 @@ associated with each target and the service node on which the target is mounted. Later, when the client attempts to access data on a target, it will try the NID for each specified service node until it connects to the target. - Previous to Lustre software release 2.0, the --failnode option to - mkfs.lustre was used to designate a failover service node for a primary - server for a target. When the --failnode option is used, certain - restrictions apply: - - The target must be initially mounted on the primary service node, not the failover - node designated by the --failnode option. - - - If the tunefs.lustre –-writeconf option is used to erase and - regenerate the configuration log for the file system, a target cannot be initially - mounted on a designated failnode. - - - If a --failnode option is added to a target to designate a - failover server for the target, the target must be re-mounted on the primary node before - the --failnode option takes effect - -
Administering Failover in a Lustre File System diff --git a/ConfiguringQuotas.xml b/ConfiguringQuotas.xml index 852eebe..0a7b88f 100644 --- a/ConfiguringQuotas.xml +++ b/ConfiguringQuotas.xml @@ -59,6 +59,11 @@ grpquota options to mount. Space accounting is enabled by default and quota enforcement can be enabled/disabled on a per-filesystem basis with lctl conf_param. + It is worth noting that the + lfs quotaon, lfs quotaoff, + lfs quotacheck and quota_type + sub-commands are deprecated as of Lustre 2.4.0, and removed completely + in Lustre 2.8.0. @@ -156,7 +161,9 @@ to fix the quota files when the QUOTA feature flag is present. The project quota feature is disabled by default, and tune2fs needs to be run to enable every target - manually. + manually. If user, group, and project quota usage is inconsistent, + run e2fsck -f on all unmounted MDTs and OSTs. + For ZFS backend, the project quota feature is not @@ -186,6 +193,11 @@ + To (re-)enable space usage quota on ldiskfs filesystems, run + tune2fs -O quota against all targets. This command + sets the QUOTA feature flag in the superblock and runs e2fsck internally. + As a result, the target must be offline to build the per-UID/GID disk + usage database. Lustre filesystems formatted with a Lustre release prior to 2.10 can be still safely upgraded to release 2.10, but will not have project quota usage reporting functional until @@ -200,7 +212,7 @@ to be installed on the server nodes when using the ldiskfs backend (e2fsprogs is not needed with ZFS backend). In general, we recommend to use the latest e2fsprogs version available on - + http://downloads.whamcloud.com/public/e2fsprogs/. The ldiskfs OSD relies on the standard Linux quota to maintain accounting information on disk. As a consequence, the Linux kernel diff --git a/Glossary.xml b/Glossary.xml index b2828b0..570a7eb 100644 --- a/Glossary.xml +++ b/Glossary.xml @@ -80,7 +80,7 @@ - Distributed namespace (DNE) + Distributed Namespace Environment (DNE) A collection of metadata targets serving a single file system namespace. Without DNE, Lustre file systems are limited to a @@ -104,7 +104,7 @@ EA Extended attribute. A small amount of data that can be retrieved - through a name (EA or attr) associated with a particular inode. A + through a name (EA or xattr) associated with a particular inode. A Lustre file system uses EAs to store striping information (indicating the location of file data on OSTs). Examples of extended attributes are ACLs, striping information, and the FID of the file. diff --git a/LustreOperations.xml b/LustreOperations.xml index 7939c0f..8fea523 100644 --- a/LustreOperations.xml +++ b/LustreOperations.xml @@ -161,12 +161,12 @@ XXX.XXX.0.11@tcp:/testfs on /mnt/testfs type lustre (rw,lazystatfs) [154523.177714] Lustre: Unmounted testfs-client Unmount the MDT and MGT - On the MGS and MDS node(s), use the umount - command: + On the MGS and MDS node(s), run the + umount command: umount -a -t lustre The example below shows the unmount of the MDT and MGT for - the testfs filesystem on a combined MGS/MDS: - + the testfs filesystem on a combined MGS/MDS: + [root@mds1 ~]# mount |grep lustre /dev/sda on /mnt/mgt type lustre (ro) /dev/sdb on /mnt/mdt type lustre (ro) @@ -175,7 +175,7 @@ XXX.XXX.0.11@tcp:/testfs on /mnt/testfs type lustre (rw,lazystatfs) [155263.566230] Lustre: Failing over testfs-MDT0000 [155263.775355] Lustre: server umount testfs-MDT0000 complete [155269.843862] Lustre: server umount MGS complete - For a seperate MGS and MDS, the same command is used, first on + For a seperate MGS and MDS, the same command is used, first on the MDS and then followed by the MGS. Unmount all the OSTs @@ -183,8 +183,8 @@ XXX.XXX.0.11@tcp:/testfs on /mnt/testfs type lustre (rw,lazystatfs) umount -a -t lustre The example below shows the unmount of all OSTs for the - testfs filesystem on server - OSS1: + testfs filesystem on server + OSS1: [root@oss1 ~]# mount |grep lustre /dev/sda on /mnt/ost0 type lustre (ro) @@ -966,7 +966,7 @@ tune2fs [-m reserved_blocks_percent] /dev/ The command output is: -debugfs 1.42.3.wc3 (15-Aug-2012) +debugfs 1.45.6.wc1 (20-Mar-2020) /dev/lustre/ost_test2: catastrophic mode - not reading inode or group bitmaps Inode: 352365 Type: regular Mode: 0666 Flags: 0x80000 Generation: 2393149953 Version: 0x0000002a:00005f81 @@ -982,25 +982,23 @@ Size of extra inode fields: 24 Extended attributes stored in inode body: fid = "b9 da 24 00 00 00 00 00 6a fa 0d 3f 01 00 00 00 eb 5b 0b 00 00 00 0000 00 00 00 00 00 00 00 00 " (32) - fid: objid=34976 seq=0 parent=[0x24dab9:0x3f0dfa6a:0x0] stripe=1 + fid: objid=34976 seq=0 parent=[0x200000400:0x122:0x0] stripe=1 EXTENTS: (0-64):4620544-4620607 - For Lustre software release 2.x file systems, the parent FID will - be of the form [0x200000400:0x122:0x0] and can be resolved directly - using the - lfs fid2path [0x200000404:0x122:0x0] - /mnt/lustre command on any Lustre client, and the process is + The parent FID will be of the form + [0x200000400:0x122:0x0] and can be resolved directly + using the command lfs fid2path [0x200000404:0x122:0x0] + /mnt/lustre on any Lustre client, and the process is complete. - In this example the parent inode FID is an upgraded 1.x inode - (due to the first part of the FID being below 0x200000400), the MDT - inode number is + In cases of an upgraded 1.x inode (if the first part of the + FID is below 0x200000400), the MDT inode number is 0x24dab9 and generation - 0x3f0dfa6a and the pathname needs to be resolved + 0x3f0dfa6a and the pathname can also be resolved using debugfs. @@ -1013,7 +1011,7 @@ EXTENTS: Here is the command output: -debugfs 1.42.3.wc2 (15-Aug-2012) +debugfs 1.42.3.wc3 (15-Aug-2012) /dev/lustre/mdt_test: catastrophic mode - not reading inode or group bitmap\ s Inode Pathname diff --git a/LustreRecovery.xml b/LustreRecovery.xml index 8dab07f..07f081b 100644 --- a/LustreRecovery.xml +++ b/LustreRecovery.xml @@ -406,13 +406,17 @@
<indexterm><primary>imperative recovery</primary></indexterm>Imperative Recovery - Large-scale Lustre file system implementations have historically experienced problems - recovering in a timely manner after a server failure. This is due to the way that clients - detect the server failure and how the servers perform their recovery. Many of the processes - are driven by the RPC timeout, which must be scaled with system size to prevent false - diagnosis of server death. The purpose of imperative recovery is to reduce the recovery window - by actively informing clients of server failure. The resulting reduction in the recovery - window will minimize target downtime and therefore increase overall system availability. + Large-scale Lustre filesystems will experience server hardware + failures over their lifetime, and it is important that servers can + recover in a timely manner after such failures. High Availability + software can move storage targets over to a backup server automatically. + Clients can detect the server failure by RPC timeouts, which must be + scaled with system size to prevent false diagnosis of server death in + cases of heavy load. The purpose of imperative recovery is to reduce + the recovery window by actively informing clients of server failure. + The resulting reduction in the recovery window will minimize target + downtime and therefore increase overall system availability. + Imperative Recovery does not remove previous recovery mechanisms, and client timeout-based recovery actions can occur in a cluster when IR is enabled as each client can still independently disconnect and reconnect from a target. In case of a mix of IR and non-IR diff --git a/LustreTuning.xml b/LustreTuning.xml index 4b955f7..91e5ae8 100644 --- a/LustreTuning.xml +++ b/LustreTuning.xml @@ -158,12 +158,6 @@ lctl {get,set}_param {service}.thread_{min,max,started} in providing the read page service. The read page service handles file close and readdir operations. - - - mds_attr_num_threads controls the number of threads - in providing the setattr service to clients running Lustre software - release 1.8. -
@@ -201,14 +195,8 @@ lctl {get,set}_param {service}.thread_{min,max,started} to CPT4. - - - mds_attr_num_cpts=[EXPRESSION] binds the setattr - service threads to CPTs defined by - EXPRESSION. - - Parameters must be set before module load in the file + Parameters must be set before module load in the file /etc/modprobe.d/lustre.conf. For example: lustre.conf options lnet networks=tcp0(eth0) @@ -615,6 +603,13 @@ cpu_partition_table= ksocklnd using the nscheds parameter. This adjusts the number of threads for each partition, not the overall number of threads on the LND. + + The default number of threads for + ko2iblnd and + ksocklnd are automatically set and are chosen to + work well across a number of typical scenarios, for systems with both + high and low core counts. +
ko2iblnd Tuning The following table outlines the ko2iblnd module parameters to be used diff --git a/ManagingFileSystemIO.xml b/ManagingFileSystemIO.xml index c619e9d..b4db9ce 100644 --- a/ManagingFileSystemIO.xml +++ b/ManagingFileSystemIO.xml @@ -591,21 +591,24 @@ osc.testfs-OST0000-osc-ffff81012b2c48e0.checksum_type=[crc32] adler
- Ptlrpc Thread Pool - The increasing use of large SMP nodes for Lustre servers requires - good scaling of many application threads generating large amounts of IO. - Lustre implements a ptlrpc thread pool, so - that multiple threads can be created to serve asynchronous RPC requests. - The number of threads spawned is controlled at module load time using - module options. By default one thread is spawned per CPU, with a minimum - of 2 threads spawned irrespective of module options. + PtlRPC Client Thread Pool + The use of large SMP nodes for Lustre clients + requires significant parallelism within the kernel to avoid + cases where a single CPU would be 100% utilized and other CPUs would be + relativity idle. This is especially noticeable when a single thread + traverses a large directory. + The Lustre client implements a PtlRPC daemon thread pool, so that + multiple threads can be created to serve asynchronous RPC requests, even + if only a single userspace thread is running. The number of ptlrpcd + threads spawned is controlled at module load time using module options. + By default two service threads are spawned per CPU socket. One of the issues with thread operations is the cost of moving a thread context from one CPU to another with the resulting loss of CPU - cache warmth. To reduce this cost, ptlrpc threads can be bound to a CPU. + cache warmth. To reduce this cost, PtlRPC threads can be bound to a CPU. However, if the CPUs are busy, a bound thread may not be able to respond quickly, as the bound CPU may be busy with other tasks and the thread must wait to schedule. - Because of these considerations, the pool of ptlrpc threads can be + Because of these considerations, the pool of ptlrpcd threads can be a mixture of bound and unbound threads. The system operator can balance the thread mixture based on system size and workload.
@@ -615,11 +618,11 @@ osc.testfs-OST0000-osc-ffff81012b2c48e0.checksum_type=[crc32] adler etc/modprobe.d directory, as options for the ptlrpc module. -options ptlrpcd max_ptlrpcds=XXX +options ptlrpcd ptlrpcd_per_cpt_max=XXX - Sets the number of ptlrpcd threads created at module load time. - The default if not specified is one thread per CPU, including - hyper-threaded CPUs. The lower bound is 2 (old prlrpcd behaviour) + Sets the number of ptlrpcd threads created per socket. + The default if not specified is two threads per CPU socket, including + hyper-threaded CPUs. The lower bound is 2 threads per socket. options ptlrpcd ptlrpcd_bind_policy=[1-4] diff --git a/ManagingStripingFreeSpace.xml b/ManagingStripingFreeSpace.xml index 6e54acb..8f795e6 100644 --- a/ManagingStripingFreeSpace.xml +++ b/ManagingStripingFreeSpace.xml @@ -2210,6 +2210,9 @@ File 4: OST6, OST7, OST0 ea_inode feature on the MDT: tune2fs -O ea_inode /dev/mdtdev + Since Lustre 2.13 the + ea_inode feature is enabled by default on all newly + formatted ldiskfs MDT filesystems. The maximum stripe count for a single file does not limit the maximum number of OSTs that are in the filesystem as a whole, only the maximum possible size and maximum aggregate bandwidth for the file. diff --git a/Revision.xml b/Revision.xml index d2d95bb..4dcade3 100644 --- a/Revision.xml +++ b/Revision.xml @@ -12,18 +12,21 @@ http://wiki.lustre.org/Lustre_Manual_Changes. - This manual covers a range of Lustre 2.x software - releases. Features that are specific to individual releases are - identified within the table of contents using a short hand notation - (i.e. this paragraph is tagged for a Lustre 2.8 specific feature), - and within the text using a distinct box. For example: + This manual covers a range of Lustre 2.x software + releases, currently starting with the 2.5 release. Features specific + to individual releases are identified within the table of contents + using a shorthand notation (e.g. this paragraph is tagged as a Lustre + 2.5 specific feature so that it will be updated when the 2.5-specific + tagging is removed), and within the text using a distinct box. + - which version? + Which version amd I running? version which version of Lustre am I running? The current version of Lustre that is in use on the node can be found - using the command lctl get_param version, for example: + using the command lctl get_param version on any Lustre + client or server, for example: $ lctl get_param version version=2.10.5 diff --git a/SettingUpLustreSystem.xml b/SettingUpLustreSystem.xml index 03af700..658ce2b 100644 --- a/SettingUpLustreSystem.xml +++ b/SettingUpLustreSystem.xml @@ -55,12 +55,14 @@ - Only servers running on 64-bit CPUs are tested and supported. 64-bit CPU clients are - typically used for testing to match expected customer usage and avoid limitations due to the 4 - GB limit for RAM size, 1 GB low-memory limitation, and 16 TB file size limit of 32-bit CPUs. - Also, due to kernel API limitations, performing backups of Lustre software release 2.x. file - systems on 32-bit clients may cause backup tools to confuse files that have the same 32-bit - inode number. + Only servers running on 64-bit CPUs are tested and supported. + 64-bit CPU clients are typically used for testing to match expected + customer usage and avoid limitations due to the 4 GB limit for RAM + size, 1 GB low-memory limitation, and 16 TB file size limit of 32-bit + CPUs. Also, due to kernel API limitations, performing backups of Lustre + filesystems on 32-bit clients may cause backup tools to confuse files + that report the same 32-bit inode number, if the backup tools depend + on the inode number for correct operation. The storage attached to the servers typically uses RAID to provide fault tolerance and can optionally be organized with logical volume management (LVM), which is then formatted as a Lustre file system. Lustre OSS and MDS servers read, write and modify data in the format @@ -523,8 +525,8 @@ OSTs formatted with ldiskfs can use a maximum of approximately 320 million objects per MDT, up to a maximum of 4 billion inodes. - Specifying a very small bytes-per-inode ratio for a large OST that - exceeds this limit can cause either premature out-of-space errors and prevent + Specifying a very small bytes-per-inode ratio for a large OST that + exceeds this limit can cause either premature out-of-space errors and prevent the full OST space from being used, or will waste space and slow down e2fsck more than necessary. The default inode ratios are chosen to ensure that the total number of inodes remain below this limit. @@ -598,10 +600,13 @@ 256 - A single MDS can host - multiple MDTs, either for separate file systems, or up to 255 - additional MDTs can be added to the filesystem and attached into - the namespace with DNE remote or striped directories. + A single MDS can host one or more MDTs, either for separate + filesystems, or aggregated into a single namespace. Each + filesystem requires a separate MDT for the filesystem root + directory. + Up to 255 more MDTs can be added to the filesystem and are + attached into the filesystem namespace with creation of DNE + remote or striped directories. diff --git a/UnderstandingLustre.xml b/UnderstandingLustre.xml index 67efcd4..377b1d1 100644 --- a/UnderstandingLustre.xml +++ b/UnderstandingLustre.xml @@ -342,12 +342,6 @@ systems that would otherwise cause file system corruption. - It is possible to configure active/active failover of multiple - MDTs. This allows scaling the metadata performance of Lustre - filesystems with the addition of MDT storage devices and MDS nodes. - - - Security:By default TCP connections are only allowed from privileged ports. UNIX group membership is @@ -505,22 +499,22 @@ Metadata Targets (MDT) - Each - filesystem has at least one MDT. The + filesystem has at least one MDT, which holds the root directory. The MDT stores metadata (such as filenames, directories, permissions and file layout) on storage attached to an MDS. Each file system has one MDT. An MDT on a shared storage target can be available to multiple MDSs, although only one can access it at a time. If an active MDS - fails, a standby MDS can serve the MDT and make it available to + fails, a second MDS node can serve the MDT and make it available to clients. This is referred to as MDS failover. - Multiple MDTs are supported in the Distributed Namespace - Environment (DNE). + Multiple MDTs are supported with the Distributed Namespace + Environment (). In addition to the primary MDT that holds the filesystem root, it is possible to add additional MDS nodes, each with their own MDTs, to hold sub-directory trees of the filesystem. Since Lustre software release 2.8, DNE also allows the filesystem to distribute files of a single directory over multiple MDT nodes. A directory which is distributed across multiple - MDTs is known as a striped directory. + MDTs is known as a . @@ -701,28 +695,29 @@ Lustre I/O Lustre File System Storage and I/O - Lustre uses file identifiers (FIDs) to replace UNIX inode numbers - for identifying files or objects. A FID is a 128-bit identifier that - contains a unique 64-bit sequence number, a 32-bit object ID (OID), and - a 32-bit version number. The sequence number is unique across all Lustre - targets in a file system (OSTs and MDTs). This allows Lustre to identify - files on multiple MDTs independent of the underlying filesystem type. - - The LFSCK file system consistency checking tool verifies the - consistency of file objects between MDTs and OSTs. It includes the - following functionality: + Lustre File IDentifiers (FIDs) are used internally for identifying + files or objects, similar to inode numbers in local filesystems. A FID + is a 128-bit identifier, which contains a unique 64-bit sequence number + (SEQ), a 32-bit object ID (OID), and a 32-bit version number. The sequence + number is unique across all Lustre targets in a file system (OSTs and + MDTs). This allows multiple MDTs and OSTs to uniquely identify objects + without depending on identifiers in the underlying filesystem (e.g. inode + numbers) that are likely to be duplicated between targets. The FID SEQ + number also allows mapping a FID to a particular MDT or OST. + The LFSCK file system consistency checking tool provides + functionality that enables FID-in-dirent for existing files. It + includes the following functionality: - Verifies the FID-in-dirent for each file and regenerates the - FID-in-dirent if it is invalid or missing. + Verifies the FID stored with each directory entry and regenerates + it from the inode if it is invalid or missing. - Verifies the linkEA entry for each and regenerates the linkEA - if it is invalid or missing. The - linkEA consists of the file name and - parent FID. It is stored as an extended attribute in the file - itself. Thus, the linkEA can be used to reconstruct the full path name - of a file. + Verifies the linkEA entry for each inode and regenerates it if + invalid or missing. The linkEA + stores of the file name and parent FID. It is stored as an extended + attribute in each inode. Thus, the linkEA can be used to + reconstruct the full path name of a file from only the FID. Information about where file data is located on the OST(s) is stored @@ -867,6 +862,11 @@ 31.25 PiB for ldiskfs or 8EiB with ZFS. Note that a Lustre file system can support files up to 2^63 bytes (8EiB), limited only by the space available on the OSTs. + + ldiskfs filesystems without the ea_inode + feature limit the maximum stripe count for a single file to 160 OSTs. + + Although a single file can only be striped over 2000 objects, Lustre file systems can have thousands of OSTs. The I/O bandwidth to access a single file is the aggregated I/O bandwidth to the objects in a diff --git a/UpgradingLustre.xml b/UpgradingLustre.xml index 895d414..2c22271 100644 --- a/UpgradingLustre.xml +++ b/UpgradingLustre.xml @@ -4,12 +4,11 @@ xml:id="upgradinglustre"> Upgrading a Lustre File System This chapter describes interoperability between Lustre software - releases. It also provides procedures for upgrading from Lustre software - release 1.8 to Lustre software release 2.x , from a Lustre software release - 2.x to a more recent Lustre software release 2.x (major release upgrade), and - from a a Lustre software release 2.x.y to a more recent Lustre software - release 2.x.y (minor release upgrade). It includes the following - sections: + releases. It also provides procedures for upgrading from older Lustre 2.x + software releases to a more recent 2.y Lustre release a (major release + upgrade), and from a Lustre software release 2.x.y to a more recent + Lustre software release 2.x.z (minor release upgrade). It includes the + following sections: @@ -52,7 +51,7 @@ All servers must be be upgraded to a Linux kernel supported by the Lustre software. See the Lustre Release Notes for your Lustre - version for a list of tested Linux distributions. + version for a list of tested Linux distributions. Clients to be upgraded must be running a compatible Linux @@ -94,11 +93,6 @@ ea_inode large_xattr - - - wide striping - ea_inode - large_xattr Upgrading to Lustre Software Release 2.x (Major Release) The procedure for upgrading from a Lustre software release 2.x to a @@ -127,9 +121,8 @@ (tested) Linux distribution and reboot. - Upgrade the Linux operating system on all clients to Red Hat - Enterprise Linux 6 or other compatible (tested) distribution and - reboot. + Upgrade the Linux operating system on all clients to a + compatible (tested) distribution and reboot. Download the Lustre server RPMs for your platform from the @@ -206,18 +199,14 @@ - Lustre allows striping a single file across up to - 2000 OSTs. Before Lustre 2.13, the "wide striping" feature that - allowed creating files with more than 160 stripes was not enabled by - default. From the 2.13 release onward, the ea_inode - feature is enabled for newly-formatted MDTs. The feature can also - be enabled by the tune2fs command on existing MDTs: - mds# tune2fs -O ea_inode /dev/mdtdev - - For more information about wide striping, see - . - - + The DNE feature allows using multiple MDTs within a single + filesystem namespace, and each MDT can each serve one or more remote + sub-directories in the file system. The root + directory is always located on MDT0. + Note that clients running a release prior to the Lustre software + release 2.4 can only see the namespace hosted by MDT0 and will return an + IO error if an attempt is made to access a directory on another + MDT. (Optional) To format an additional MDT, complete these steps: @@ -229,30 +218,44 @@ In this example, the next available index is 1. - Add the new block device as a new MDT at the next available - index by entering (on one line): + Format the new block device as a new MDT at the next + available MDT index by entering (on one line): mds# mkfs.lustre --reformat --fsname=filesystem_name --mdt \ - --mgsnode=mgsnode --index 1 + --mgsnode=mgsnode --index new_mdt_index /dev/mdt1_device - (Optional) If you are upgrading before Lustre software release + (Optional) If you are upgrading from a release before Lustre 2.10, to enable the project quota feature enter the following on every - ldiskfs backend target: + ldiskfs backend target while unmounted: tune2fs –O project /dev/dev Enabling the project feature will prevent the filesystem from being used by older versions of ldiskfs, so it - should only be enabled if the project quota feature is required and/or - after it is known that the upgraded release does not need to be - downgraded. + should only be enabled if the project quota feature is required + and/or after it is known that the upgraded release does not need + to be downgraded. + + When setting up the file system, enter: conf_param $FSNAME.quota.mdt=$QUOTA_TYPE conf_param $FSNAME.quota.ost=$QUOTA_TYPE + (Optional) If upgrading an ldiskfs MDT formatted + prior to Lustre 2.13, the "wide striping" feature that allows files + to have more than 160 stripes and store other large xattrs was not + enabled by default. This feature can be enabled on existing MDTs + by running the following command on all MDT devices: + mds# tune2fs -O ea_inode /dev/mdtdev + + For more information about wide striping, see + . + + Start the Lustre file system by starting the components in the order shown in the following steps: @@ -283,6 +286,42 @@ conf_param $FSNAME.quota.ost=$QUOTA_TYPE + + (Optional) If you are upgrading from a release before Lustre + 2.7, to enable OST FIDs to also store the OST index (to improve + reliability of LFSCK and debug messages), after + the OSTs are mounted run once on each OSS: + oss# lctl set_param osd-ldiskfs.*.osd_index_in_idif=1 + + Enabling the index_in_idif feature will + prevent the OST from being used by older versions of Lustre, so it + should only be enabled once it is known there is no need for the + OST to be downgraded to an earlier release. + + + If a new MDT was added to the filesystem, the new MDT must be + attached into the namespace by creating one or more + new DNE subdirectories with the + lfs mkdir command that use the new MDT: + +client# lfs mkdir -i new_mdt_index /testfs/new_dir + + + In Lustre 2.8 and later, it is possible to + split a new directory across multiple MDTs by creating it with + multiple stripes: + +client# lfs mkdir -c 2 /testfs/new_striped_dir + + + In Lustre 2.13 and later, it is possible to set + the default striping on existing directories + so that new remote subdirectories are created on less-full MDTs: + +client# lfs setdirstripe -c 1 -i -1 /testfs/some_dir + + + The mounting order described in the steps above must be followed diff --git a/UserUtilities.xml b/UserUtilities.xml index 26c9c84..449d115 100644 --- a/UserUtilities.xml +++ b/UserUtilities.xml @@ -84,10 +84,11 @@ lfs help -v option with lfs quota. - - The quotacheck, quotaon and quotaoff - sub-commands were deprecated in the Lustre 2.4 release, and removed completely in - the Lustre 2.8 release. See for details on + + The quotacheck, quotaon and + quotaoff sub-commands were deprecated in the + Lustre 2.4 release, and removed completely in the Lustre 2.8 release. + See for details on configuring and checking quotas.
diff --git a/style/customstyle_common.xsl b/style/customstyle_common.xsl index 624f1ee..7e0e807 100644 --- a/style/customstyle_common.xsl +++ b/style/customstyle_common.xsl @@ -86,9 +86,27 @@ + + + + + + + + + + + + + + + + + + - + @@ -182,6 +200,15 @@ L 2.13 + + L 2.14 + + + L 2.15 + + + L 2.16 + L ?.? diff --git a/style/customstyle_fo.xsl b/style/customstyle_fo.xsl index a689a92..fe87867 100644 --- a/style/customstyle_fo.xsl +++ b/style/customstyle_fo.xsl @@ -146,7 +146,6 @@ - Introduced in Lustre 2.4 Introduced in Lustre 2.5 Introduced in Lustre 2.6 Introduced in Lustre 2.7 @@ -159,7 +158,7 @@ Introduced in Lustre 2.14 Introduced in Lustre 2.15 Introduced in Lustre 2.16 - Documentation Error: unrecognised condition attribute + Introduced in before Lustre 2.5 @@ -189,8 +188,6 @@ - L 2.3 - L 2.4 L 2.5 L 2.6 L 2.7 @@ -200,6 +197,9 @@ L 2.11 L 2.12 L 2.13 + L 2.14 + L 2.15 + L 2.16