X-Git-Url: https://git.whamcloud.com/?a=blobdiff_plain;f=ManagingStripingFreeSpace.xml;h=d213d1a12b619eab4ccf5f0c2e7287d7abd1f782;hb=e096f06b81ed50da7c6f21867162f272299465ef;hp=137303ca4e277314d52efd4ff3b0b3dacafddf93;hpb=25106a04c49c4acef6322b68ce22217fac9e523c;p=doc%2Fmanual.git diff --git a/ManagingStripingFreeSpace.xml b/ManagingStripingFreeSpace.xml index 137303c..d213d1a 100644 --- a/ManagingStripingFreeSpace.xml +++ b/ManagingStripingFreeSpace.xml @@ -1,29 +1,32 @@ - + + Managing File Layout (Striping) and Free Space This chapter describes file layout (striping) and I/O options, and includes the following sections: - + - + - + - + - + -
+
<indexterm> <primary>space</primary> @@ -53,17 +56,18 @@ default), the MDS then uses weighted random allocations with a preference for allocating objects on OSTs with more free space. (This can reduce I/O performance until space usage is rebalanced again.) For a more detailed description of how striping is allocated, see <xref - linkend="dbdoclet.50438209_10424"/>.</para> + linkend="file_striping.managing_free_space"/>.</para> <para>Files can only be striped over a finite number of OSTs, based on the maximum size of the attributes that can be stored on the MDT. If the MDT is ldiskfs-based without the <literal>ea_inode</literal> feature, a file can be striped across at most 160 OSTs. With a ZFS-based MDT, or if the - <literal>ea_inode</literal> feature is enabled for an ldiskfs-based MDT, + <literal>ea_inode</literal> feature is enabled for an ldiskfs-based MDT + (the default since Lustre 2.13.0), a file can be striped across up to 2000 OSTs. For more information, see <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/>. </para> </section> - <section xml:id="dbdoclet.50438209_48033"> + <section xml:id="file_striping.considerations"> <title><indexterm> <primary>file layout</primary> <secondary>See striping</secondary> @@ -178,7 +182,7 @@ </itemizedlist> </section> </section> - <section xml:id="dbdoclet.50438209_78664"> + <section xml:id="file_striping.lfs_setstripe"> <title><indexterm> <primary>striping</primary> <secondary>configuration</secondary> @@ -230,14 +234,16 @@ <literal>pool_name</literal> </emphasis> </para> - <para>The <literal>pool_name</literal> specifies the OST pool to which the file will be written. - This allows limiting the OSTs used to a subset of all OSTs in the file system. For more - details about using OST pools, see <link xl:href="ManagingFileSystemIO.html#50438211_75549" - >Creating and Managing OST Pools</link>.</para> + <para>The <literal>pool_name</literal> specifies the OST pool to which the + file will be written. This allows limiting the OSTs used to a subset of + all OSTs in the file system. For more details about using OST pools, see + <link xl:href="managingfilesystemio.managing_ost_pools"> + Creating and Managing OST Pools + </link>.</para> <section remap="h3"> <title>Specifying a File Layout (Striping Pattern) for a Single File It is possible to specify the file layout when a new file is created using the command lfs setstripe. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an lfs setstripe command fails if the file already exists. -
+
Setting the Stripe Size The command to create a new file with a specified stripe size is similar to: [client]# lfs setstripe -s 4M /mnt/lustre/new_file @@ -260,7 +266,8 @@ obdidx objid objid group The command below creates a new file with a stripe count of -1 to specify striping over all available OSTs: [client]# lfs setstripe -c -1 /mnt/lustre/full_stripe - The example below indicates that the file full_stripe is striped + The example below indicates that the file + full_stripe is striped over all six active OSTs in the configuration: [client]# lfs getstripe /mnt/lustre/full_stripe /mnt/lustre/full_stripe @@ -271,8 +278,9 @@ obdidx objid objid group 3 5 0x5 0 4 4 0x4 0 5 2 0x2 0 - This is in contrast to the output in , which - shows only a single object for the file. + This is in contrast to the output in + , + which shows only a single object for the file.
@@ -329,7 +337,7 @@ lmm_stripe_offset: 0 0 37364 0x91f4 0
-
+
<indexterm><primary>striping</primary><secondary>getting information</secondary></indexterm>Retrieving File Layout/Striping Information (<literal>getstripe</literal>) The lfs getstripe command is used to display information that shows over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along @@ -1959,7 +1967,7 @@ STRIPE OPTIONS:
-
+
<indexterm> <primary>space</primary> <secondary>free space</secondary> @@ -1988,16 +1996,17 @@ STRIPE OPTIONS: <literal>lctl set_param</literal> command, for example the next command reserve 1GB space for all OSTs. <screen>lctl set_param -P osp.*.reserved_mb_low=1024</screen></para> - <para>This section describes how to check available free space on disks and how free space is - allocated. It then describes how to set the threshold and weighting factors for the allocation - algorithms.</para> - <section xml:id="dbdoclet.checking_free_space"> + <para>This section describes how to check available free space on disks + and how free space is allocated. It then describes how to set the + threshold and weighting factors for the allocation algorithms.</para> + <section xml:id="file_striping.checking_free_space"> <title>Checking File System Free Space - Free space is an important consideration in assigning file stripes. The lfs - df command can be used to show available disk space on the mounted Lustre file - system and space consumption per OST. If multiple Lustre file systems are mounted, a path - may be specified, but is not required. Options to the lfs df command are - shown below. + Free space is an important consideration in assigning file stripes. + The lfs df command can be used to show available + disk space on the mounted Lustre file system and space consumption per + OST. If multiple Lustre file systems are mounted, a path may be + specified, but is not required. Options to the lfs df + command are shown below. @@ -2015,10 +2024,25 @@ STRIPE OPTIONS: - -h + + -h, --human-readable + - Displays sizes in human readable format (for example: 1K, 234M, 5G). + Displays sizes in human readable format (for example: 1K, + 234M, 5G) using base-2 (binary) values (i.e. 1G = 1024M). + + + + + + -H, --si + + + + Like -h, this displays counts in human + readable format, but using base-10 (decimal) values + (i.e. 1G = 1000M). @@ -2029,46 +2053,151 @@ STRIPE OPTIONS: Lists inodes instead of block usage. + + + -l, --lazy + + + Do not attempt to contact any OST or MDT not currently + connected to the client. This avoids blocking the + lfs df output if a target is offline or + unreachable, and only returns the space on OSTs that can + currently be accessed. + + + + + -p, --pool + + + Limit the usage to report only OSTs that are in the + specified pool. If multiple + Lustre filesystems are mounted, list the OSTs in + pool for each filesystem, or + limit the display to only a pool for a specific filesystem + if fsname.pool is given. + Specifying both fsname and + pool is equivalent to providing + a specific mountpoint. + + + + + + + -v, --verbose + + + + Display verbose status of MDTs and OSTs. This may + include one or more optional flags at the end of each line. + + + + + lfs df may also report additional target status + as the last column in the display, if there are issues with that target. + Target states include: + + + + D: OST/MDT is Degraded. + The target has a failed drive in the RAID device, or is + undergoing RAID reconstruction. This state is marked on + the server automatically for ZFS targets via + zed, or a (user-supplied) script that + monitors the target device and sets + "lctl set_param obdfilter.target.degraded=1" + on the OST. This target will be avoided for new + allocations, but will still be used to read existing files + located there or if there are not enough non-degraded OSTs + to make up a widely-striped file. + + + R: OST/MDT is Read-only. + The target filesystem is marked read-only due to filesystem + corruption detected by ldiskfs or ZFS. No modifications + are allowed on this OST, and it needs to be unmounted and + e2fsck or zpool scrub + run to repair the underlying filesystem. + + + N: OST/MDT is No-precreate. + The target is configured to deny object precreation set by + "lctl set_param obdfilter.target.no_precreate=1" + parameter or the "-o no_precreate" mount option. + This may be done to add an OST to the filesystem without allowing + objects to be allocated on it yet, or for other reasons. + + + S: OST/MDT is out of Space. + The target filesystem has less than the minimum required + free space and will not be used for new object allocations + until it has more free space. + + + I: OST/MDT is out of Inodes. + The target filesystem has less than the minimum required + free inodes and will not be used for new object allocations + until it has more free inodes. + + + f: OST/MDT is on flash. + The target filesystem is using a flash (non-rotational) + storage device. This is normally detected from the + underlying Linux block device, but can be set manually + with "lctl set_param osd-*.*.nonrotational=1 + on the respective OSTs. This lower-case status is only + shown in conjunction with the -v option, + since it is not an error condition. + + - The df -i and lfs df -i commands show the - minimum number of inodes that can be created in the - file system at the current time. If the total number of objects available across all of - the OSTs is smaller than those available on the MDT(s), taking into account the default - file striping, then df -i will also report a smaller number of inodes - than could be created. Running lfs df -i will report the actual number - of inodes that are free on each target. - For ZFS file systems, the number of inodes that can be created is dynamic and depends - on the free space in the file system. The Free and Total inode counts reported for a ZFS - file system are only an estimate based on the current usage for each target. The Used - inode count is the actual number of inodes used by the file system. + The df -i and lfs df -i + commands show the minimum number + of inodes that can be created in the file system at the current time. + If the total number of objects available across all of the OSTs is + smaller than those available on the MDT(s), taking into account the + default file striping, then df -i will also + report a smaller number of inodes than could be created. Running + lfs df -i will report the actual number of inodes + that are free on each target. + + For ZFS file systems, the number of inodes that can be created + is dynamic and depends on the free space in the file system. The + Free and Total inode counts reported for a ZFS file system are only + an estimate based on the current usage for each target. The Used + inode count is the actual number of inodes used by the file system. + Examples - [client1] $ lfs df -UUID 1K-blockS Used Available Use% Mounted on -mds-lustre-0_UUID 9174328 1020024 8154304 11% /mnt/lustre[MDT:0] -ost-lustre-0_UUID 94181368 56330708 37850660 59% /mnt/lustre[OST:0] -ost-lustre-1_UUID 94181368 56385748 37795620 59% /mnt/lustre[OST:1] -ost-lustre-2_UUID 94181368 54352012 39829356 57% /mnt/lustre[OST:2] -filesystem summary: 282544104 167068468 39829356 57% /mnt/lustre + client$ lfs df +UUID 1K-blocks Used Available Use% Mounted on +testfs-OST0000_UUID 9174328 1020024 8154304 11% /mnt/lustre[MDT:0] +testfs-OST0000_UUID 94181368 56330708 37850660 59% /mnt/lustre[OST:0] +testfs-OST0001_UUID 94181368 56385748 37795620 59% /mnt/lustre[OST:1] +testfs-OST0002_UUID 94181368 54352012 39829356 57% /mnt/lustre[OST:2] +filesystem summary: 282544104 167068468 39829356 57% /mnt/lustre -[client1] $ lfs df -h -UUID bytes Used Available Use% Mounted on -mds-lustre-0_UUID 8.7G 996.1M 7.8G 11% /mnt/lustre[MDT:0] -ost-lustre-0_UUID 89.8G 53.7G 36.1G 59% /mnt/lustre[OST:0] -ost-lustre-1_UUID 89.8G 53.8G 36.0G 59% /mnt/lustre[OST:1] -ost-lustre-2_UUID 89.8G 51.8G 38.0G 57% /mnt/lustre[OST:2] -filesystem summary: 269.5G 159.3G 110.1G 59% /mnt/lustre +[client1] $ lfs df -hv +UUID bytes Used Available Use% Mounted on +testfs-MDT0000_UUID 8.7G 996.1M 7.8G 11% /mnt/lustre[MDT:0] +testfs-OST0000_UUID 89.8G 53.7G 36.1G 59% /mnt/lustre[OST:0] f +testfs-OST0001_UUID 89.8G 53.8G 36.0G 59% /mnt/lustre[OST:1] f +testfs-OST0002_UUID 89.8G 51.8G 38.0G 57% /mnt/lustre[OST:2] f +filesystem summary: 269.5G 159.3G 110.1G 59% /mnt/lustre -[client1] $ lfs df -i -UUID Inodes IUsed IFree IUse% Mounted on -mds-lustre-0_UUID 2211572 41924 2169648 1% /mnt/lustre[MDT:0] -ost-lustre-0_UUID 737280 12183 725097 1% /mnt/lustre[OST:0] -ost-lustre-1_UUID 737280 12232 725048 1% /mnt/lustre[OST:1] -ost-lustre-2_UUID 737280 12214 725066 1% /mnt/lustre[OST:2] -filesystem summary: 2211572 41924 2169648 1% /mnt/lustre[OST:2] +[client1] $ lfs df -iH +UUID Inodes IUsed IFree IUse% Mounted on +testfs-MDT0000_UUID 2.21M 41.9k 2.17M 1% /mnt/lustre[MDT:0] +testfs-OST0000_UUID 737.3k 12.1k 725.1k 1% /mnt/lustre[OST:0] +testfs-OST0001_UUID 737.3k 12.2k 725.0k 1% /mnt/lustre[OST:1] +testfs-OST0002_UUID 737.3k 12.2k 725.0k 1% /mnt/lustre[OST:2] +filesystem summary: 2.21M 41.9k 2.17M 1% /mnt/lustre[OST:2] +
<indexterm> @@ -2207,9 +2336,15 @@ File 4: OST6, OST7, OST0</screen> <literal>ea_inode</literal> feature on the MDT: <screen>tune2fs -O ea_inode /dev/<replaceable>mdtdev</replaceable></screen> </para> + <note condition='l2D'><para>Since Lustre 2.13 the + <literal>ea_inode</literal> feature is enabled by default on all newly + formatted ldiskfs MDT filesystems.</para></note> <note><para>The maximum stripe count for a single file does not limit the maximum number of OSTs that are in the filesystem as a whole, only the maximum possible size and maximum aggregate bandwidth for the file. </para></note> </section> </chapter> +<!-- + vim:expandtab:shiftwidth=2:tabstop=8: + -->