From e65b9b34b15fac9c275e41f9d346f465ea6c7bf3 Mon Sep 17 00:00:00 2001 From: Andreas Dilger Date: Wed, 10 May 2017 17:35:24 -0600 Subject: [PATCH] LUDOC-376 build: add condition 'l2A' for 2.10 Add support for annotating content with "condition='l2A'" to indicate new content for Lustre 2.10. Also add "condition='l2B'" for 2.11 in advance. Remove support for "condition='l21'" which was never used in the manual. Remove markers for "condition='l22'" which is now very old, and fix the content to indicate this is "normal" behaviour. Signed-off-by: Andreas Dilger Change-Id: I1daa97cbfe58247cda912b300f1cea31c1dc2e7c Reviewed-on: https://review.whamcloud.com/27055 Reviewed-by: Joseph Gmitter Tested-by: Jenkins --- Glossary.xml | 2 +- LustreProc.xml | 120 +++++++++++++++++++++--------------------- ManagingStripingFreeSpace.xml | 48 ++++++++++------- SettingUpLustreSystem.xml | 21 +++----- UpgradingLustre.xml | 38 ++++++------- style/customstyle_common.xsl | 38 ++++++------- style/customstyle_fo.xsl | 8 +-- 7 files changed, 136 insertions(+), 139 deletions(-) diff --git a/Glossary.xml b/Glossary.xml index ad1e829..f0ddd65 100644 --- a/Glossary.xml +++ b/Glossary.xml @@ -793,7 +793,7 @@ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"> utilization of many OSTs. For more information about wide striping, see . + linkend="wide_striping" />. diff --git a/LustreProc.xml b/LustreProc.xml index 02dd0c8..a5916a0 100644 --- a/LustreProc.xml +++ b/LustreProc.xml @@ -1246,86 +1246,88 @@ write RPCs in flight: 0 proc readahead Tuning File Readahead and Directory Statahead - File readahead and directory statahead enable reading of data into memory before a - process requests the data. File readahead reads file content data into memory and directory - statahead reads metadata into memory. When readahead and statahead work well, a process that - accesses data finds that the information it needs is available immediately when requested in - memory without the delay of network I/O. - In Lustre software release 2.2.0, the directory statahead feature was - improved to enhance directory traversal performance. The improvements primarily addressed - two issues: - - A race condition existed between the statahead thread and other VFS operations - while processing asynchronous getattr RPC replies, causing - duplicate entries in dcache. This issue was resolved by using statahead local dcache. - - - - File size/block attributes pre-fetching was not supported, so the traversing - thread had to send synchronous glimpse size RPCs to OST(s). This issue was resolved by - using asynchronous glimpse lock (AGL) RPCs to pre-fetch file size/block attributes - from OST(s). - - + File readahead and directory statahead enable reading of data + into memory before a process requests the data. File readahead prefetches + file content data into memory for read() related + calls, while directory statahead fetches file metadata into memory for + readdir() and stat() related + calls. When readahead and statahead work well, a process that accesses + data finds that the information it needs is available immediately in + memory on the client when requested without the delay of network I/O.
Tuning File Readahead - File readahead is triggered when two or more sequential reads by an application fail - to be satisfied by data in the Linux buffer cache. The size of the initial readahead is 1 - MB. Additional readaheads grow linearly and increment until the readahead cache on the - client is full at 40 MB. + File readahead is triggered when two or more sequential reads + by an application fail to be satisfied by data in the Linux buffer + cache. The size of the initial readahead is 1 MB. Additional + readaheads grow linearly and increment until the readahead cache on + the client is full at 40 MB. Readahead tunables include: - llite.fsname-instance.max_read_ahead_mb - - Controls the maximum amount of data readahead on a file. Files are read ahead in - RPC-sized chunks (1 MB or the size of the read() call, if larger) - after the second sequential read on a file descriptor. Random reads are done at the - size of the read() call only (no readahead). Reads to - non-contiguous regions of the file reset the readahead algorithm, and readahead is not - triggered again until sequential reads take place again. - To disable readahead, set this tunable to 0. The default value is 40 MB. + llite.fsname-instance.max_read_ahead_mb - + Controls the maximum amount of data readahead on a file. + Files are read ahead in RPC-sized chunks (1 MB or the size of + the read() call, if larger) after the second + sequential read on a file descriptor. Random reads are done at + the size of the read() call only (no + readahead). Reads to non-contiguous regions of the file reset + the readahead algorithm, and readahead is not triggered again + until sequential reads take place again. + + To disable readahead, set + max_read_ahead_mb=0. The default value is 40 MB. + - llite.fsname-instance.max_read_ahead_whole_mb - - Controls the maximum size of a file that is read in its entirety, regardless of the - size of the read(). + llite.fsname-instance.max_read_ahead_whole_mb - + Controls the maximum size of a file that is read in its entirety, + regardless of the size of the read(). This + avoids multiple small read RPCs on relatively small files, when + it is not possible to efficiently detect a sequential read + pattern before the whole file has been read. +
Tuning Directory Statahead and AGL - Many system commands, such as ls –l, du, and - find, traverse a directory sequentially. To make these commands run - efficiently, the directory statahead and asynchronous glimpse lock (AGL) can be enabled to - improve the performance of traversing. + Many system commands, such as ls –l, + du, and find, traverse a + directory sequentially. To make these commands run efficiently, the + directory statahead can be enabled to improve the performance of + directory traversal. The statahead tunables are: - statahead_max - Controls whether directory statahead is enabled - and the maximum statahead window size (i.e., how many files can be pre-fetched by the - statahead thread). By default, statahead is enabled and the value of - statahead_max is 32. - To disable statahead, run: + statahead_max - + Controls the maximum number of file attributes that will be + prefetched by the statahead thread. By default, statahead is + enabled and statahead_max is 32 files. + To disable statahead, set statahead_max + to zero via the following command on the client: lctl set_param llite.*.statahead_max=0 - To set the maximum statahead window size (n), - run: + To change the maximum statahead window size on a client: lctl set_param llite.*.statahead_max=n - The maximum value of n is 8192. - The AGL can be controlled by entering: - lctl set_param llite.*.statahead_agl=n - The default value for n is 1, which enables the AGL. If - n is 0, the AGL is disabled. + The maximum statahead_max is 8192 files. + + The directory statahead thread will also prefetch the file + size/block attributes from the OSTs, so that all file attributes + are available on the client when requested by an application. + This is controlled by the asynchronous glimpse lock (AGL) setting. + The AGL behaviour can be disabled by setting: + lctl set_param llite.*.statahead_agl=0 - statahead_stats - A read-only interface that indicates the - current statahead and AGL statistics, such as how many times statahead/AGL has been - triggered since the last mount, how many statahead/AGL failures have occurred due to - an incorrect prediction or other causes. + statahead_stats - + A read-only interface that provides current statahead and AGL + statistics, such as how many times statahead/AGL has been triggered + since the last mount, how many statahead/AGL failures have occurred + due to an incorrect prediction or other causes. - The AGL is affected by statahead because the inodes processed by AGL are built - by the statahead thread, which means the statahead thread is the input of the AGL - pipeline. So if statahead is disabled, then the AGL is disabled by force. + AGL behaviour is affected by statahead since the inodes + processed by AGL are built by the statahead thread. If + statahead is disabled, then AGL is also disabled. diff --git a/ManagingStripingFreeSpace.xml b/ManagingStripingFreeSpace.xml index 26945dd..5a1c68f 100644 --- a/ManagingStripingFreeSpace.xml +++ b/ManagingStripingFreeSpace.xml @@ -20,7 +20,7 @@ - +
@@ -54,11 +54,14 @@ objects on OSTs with more free space. (This can reduce I/O performance until space usage is rebalanced again.) For a more detailed description of how striping is allocated, see . - Files can only be striped over a finite number of OSTs. Prior to Lustre - software release 2.2, the maximum number of OSTs that a file could be striped across was - limited to 160. As of Lustre software release 2.2, the maximum number of OSTs is 2000. For - more information, see . + Files can only be striped over a finite number of OSTs, based on the + maximum size of the attributes that can be stored on the MDT. If the MDT + is ldiskfs-based without the ea_inode feature, a file + can be striped across at most 160 OSTs. With a ZFS-based MDT, or if the + ea_inode feature is enabled for an ldiskfs-based MDT, + a file can be striped across up to 2000 OSTs. For more information, see + . +
<indexterm> @@ -585,24 +588,31 @@ File 4: OST6, OST7, OST0</screen> </note> </section> </section> - <section xml:id="section_syy_gcl_qk"> + <section xml:id="wide_striping"> <title><indexterm> <primary>striping</primary> <secondary>wide striping</secondary> </indexterm><indexterm> <primary>wide striping</primary> </indexterm>Lustre Striping Internals - For Lustre releases prior to Lustre software release 2.2, files can be striped across a - maximum of 160 OSTs. Lustre inodes use an extended attribute to record the location of each - object (the object ID and the number of the OST on which it is stored). The size of the - extended attribute limits the maximum stripe count to 160 objects. - In Lustre software release 2.2 and subsequent releases, the maximum number - of OSTs over which files can be striped has been raised to 2000 by allocating a new block on - which to store the extended attribute that holds the object information. This feature, known - as "wide striping," only allocates the additional extended attribute data block if the file is - striped with a stripe count greater than 160. The file layout (object ID, OST number) is - stored on the new data block with a pointer to this block stored in the original Lustre inode - for the file. For files smaller than 160 objects, the Lustre inode is used to store the file - layout. + Individual files can only be striped over a finite number of OSTs, + based on the maximum size of the attributes that can be stored on the MDT. + If the MDT is ldiskfs-based without the ea_inode + feature, a file can be striped across at most 160 OSTs. With ZFS-based + MDTs, or if the ea_inode feature is enabled for an + ldiskfs-based MDT, a file can be striped across up to 2000 OSTs. + + Lustre inodes use an extended attribute to record on which OST each + object is located, and the identifier each object on that OST. The size of + the extended attribute is a function of the number of stripes. + If using an ldiskfs-based MDT, the maximum number of OSTs over which + files can be striped can been raised to 2000 by enabling the + ea_inode feature on the MDT: + tune2fs -O ea_inode /dev/mdtdev + + The maximum stripe count for a single file does not limit the + maximum number of OSTs that are in the filesystem as a whole, only the + maximum possible size and maximum aggregate bandwidth for the file. +
diff --git a/SettingUpLustreSystem.xml b/SettingUpLustreSystem.xml index a90d133..898f7e5 100644 --- a/SettingUpLustreSystem.xml +++ b/SettingUpLustreSystem.xml @@ -787,21 +787,12 @@   - - In Lustre software releases prior to version 2.2, - the maximum stripe count for a single file was limited to 160 OSTs. - In version 2.2, the wide striping feature was added to support files - striped over up to 2000 OSTs. In order to store the large layout for - such files in ldiskfs, the ea_inode feature must - be enabled on the MDT, but no similar tunable is needed for ZFS MDTs. - This feature is disabled by default at - mkfs.lustre time. In order to enable this feature, - specify --mkfsoptions="-O ea_inode" at MDT format - time, or use tune2fs -O ea_inode to enable it after - the MDT has been formatted. Using either the deprecated - large_xattr or preferred ea_inode - feature name results in ea_inode being shown in - the file system feature list. + By default for ldiskfs MDTs the maximum stripe count for a + single file is limited to 160 OSTs. In order to + increase the maximum file stripe count, use + --mkfsoptions="-O ea_inode" when formatting the MDT, + or use tune2fs -O ea_inode to enable it after the + MDT has been formatted.
diff --git a/UpgradingLustre.xml b/UpgradingLustre.xml index 646df00..4c7b524 100644 --- a/UpgradingLustre.xml +++ b/UpgradingLustre.xml @@ -111,16 +111,14 @@ xml:id="upgradinglustre"> provider. - In Lustre software release 2.2, a feature has been - added that allows striping across up to 2000 OSTs. By default, this "wide - striping" feature is disabled. It is activated by setting the - large_xattr or - ea_inode option on the MDT using either - mkfs.lustre or - tune2fs. For example after upgrading an existing file - system to Lustre software release 2.2 or later, wide striping can be - enabled by running the following command on the MDT device before - mounting it: + In Lustre software release 2.2, a feature has been added for + ldiskfs-based MDTs that allows striping a single file across up to 2000 + OSTs. By default, this "wide striping" feature is disabled. It is + activated by setting the ea_inode option on the MDT + using either mkfs.lustre or tune2fs. + For example after upgrading an existing file system to Lustre software + release 2.2 or later, wide striping can be enabled by running the + following command on the MDT device before mounting it: tune2fs -O large_xattr Once the wide striping feature is enabled and in use on the MDT, it is not possible to directly downgrade the MDT file system to an earlier @@ -128,14 +126,10 @@ xml:id="upgradinglustre"> disable wide striping: - Delete all wide-striped files. - OR - Use - lfs_migrate with the option - -c - stripe_count(set - stripe_countto 160) to move the files to - another location. + Delete all wide-striped files, OR + use lfs_migrate -c 160 (or fewer stripes) + to migrate the files to use fewer OSTs. This does not affect the + total number of OSTs that the whole filesystem can access. Unmount the MDT. @@ -288,11 +282,11 @@ xml:id="upgradinglustre"> (Optional) For upgrades to Lustre software release 2.2 or higher, to enable wide striping on an existing MDT, run the following command - on the MDT : - mdt# tune2fs -O large_xattr device + on the MDT: + tune2fs -O ea_inode /dev/mdtdev + For more information about wide striping, see - . + . (Optional) For upgrades to Lustre software release 2.4 or higher, diff --git a/style/customstyle_common.xsl b/style/customstyle_common.xsl index 6a09b16..6c54395 100644 --- a/style/customstyle_common.xsl +++ b/style/customstyle_common.xsl @@ -32,18 +32,6 @@ - - - - - - - - - - - - @@ -86,9 +74,21 @@ + + + + + + + + + + + + - + @@ -155,12 +155,6 @@ - - L 2.1 - - - L 2.2 - L 2.3 @@ -182,6 +176,12 @@ L 2.9 + + L 2.10 + + + L 2.11 + L ?.? diff --git a/style/customstyle_fo.xsl b/style/customstyle_fo.xsl index 87e6d0c..6848ac2 100644 --- a/style/customstyle_fo.xsl +++ b/style/customstyle_fo.xsl @@ -148,8 +148,6 @@ - Introduced in Lustre 2.1 - Introduced in Lustre 2.2 Introduced in Lustre 2.3 Introduced in Lustre 2.4 Introduced in Lustre 2.5 @@ -157,6 +155,8 @@ Introduced in Lustre 2.7 Introduced in Lustre 2.8 Introduced in Lustre 2.9 + Introduced in Lustre 2.10 + Introduced in Lustre 2.11 Documentation Error: unrecognised condition attribute @@ -186,8 +186,6 @@ - L 2.1 - L 2.2 L 2.3 L 2.4 L 2.5 @@ -195,6 +193,8 @@ L 2.7 L 2.8 L 2.9 + L 2.10 + L 2.11 -- 1.8.3.1