From: Andreas Dilger Date: Wed, 20 Jan 2021 02:18:41 +0000 (-0700) Subject: LUDOC-419 utils: document 'lfs df' flags X-Git-Url: https://git.whamcloud.com/?a=commitdiff_plain;h=refs%2Fchanges%2F73%2F41273%2F2;p=doc%2Fmanual.git LUDOC-419 utils: document 'lfs df' flags Document the options and flags for 'lfs df'. Signed-off-by: Andreas Dilger Change-Id: I15f8d018d31005a87eb129b4510d386fd33ebbe5 Reviewed-on: https://review.whamcloud.com/41273 Tested-by: jenkins Reviewed-by: johann peyrard --- diff --git a/ConfiguringQuotas.xml b/ConfiguringQuotas.xml index c681fb1..ea6db93 100644 --- a/ConfiguringQuotas.xml +++ b/ConfiguringQuotas.xml @@ -595,7 +595,7 @@ $ cp: writing `/mnt/testfs/foo`: Disk quota exceeded. lctl get_param *.*.filestotal). For more information on using the lfs df -i command and the command output, see - . + . Unfortunately, the statfs interface does not report the free inode count directly, but instead reports the total inode and used inode counts. The diff --git a/LustreProc.xml b/LustreProc.xml index 327a428..03c059b 100644 --- a/LustreProc.xml +++ b/LustreProc.xml @@ -2416,8 +2416,9 @@ nid refs peer max tx min space is more than this. The default is 0.2% of total OST size. - For more information about monitoring and managing free space, see . + For more information about monitoring and managing free space, see + .
<indexterm> diff --git a/LustreTroubleshooting.xml b/LustreTroubleshooting.xml index 33bbbb0..53d736a 100644 --- a/LustreTroubleshooting.xml +++ b/LustreTroubleshooting.xml @@ -607,7 +607,7 @@ filesystem_summary: 1910263 1910263 0 100% /myth such as <literal>No space left on device</literal>. The numeric error message may also appear in the system log.</para> <para>For more information about the <literal>lfs df</literal> command, - see <xref linkend="dbdoclet.checking_free_space"/>.</para> + see <xref linkend="file_striping.checking_free_space"/>.</para> <para>You can also use the <literal>lctl get_param</literal> command to monitor the space and object usage on the OSTs and MDTs from any client:</para> diff --git a/ManagingFileSystemIO.xml b/ManagingFileSystemIO.xml index 49350a0..15d7bc9 100644 --- a/ManagingFileSystemIO.xml +++ b/ManagingFileSystemIO.xml @@ -240,7 +240,7 @@ $ lfs getstripe -m ./remotedir/file.*.txt<lineannotation>check files are on MDT0 </section> </section> </section> - <section xml:id="dbdoclet.50438211_75549"> + <section xml:id="managingfilesystemio.managing_ost_pools"> <title> <indexterm> <primary>I/O</primary> diff --git a/ManagingStripingFreeSpace.xml b/ManagingStripingFreeSpace.xml index 8f795e6..d213d1a 100644 --- a/ManagingStripingFreeSpace.xml +++ b/ManagingStripingFreeSpace.xml @@ -8,25 +8,25 @@ sections:</para> <itemizedlist> <listitem> - <para><xref linkend="dbdoclet.50438209_79324"/></para> + <para><xref linkend="file_striping.how_it_works"/></para> </listitem> <listitem> - <para><xref linkend="dbdoclet.50438209_48033"/></para> + <para><xref linkend="file_striping.considerations"/></para> </listitem> <listitem> - <para><xref linkend="dbdoclet.50438209_78664"/></para> + <para><xref linkend="file_striping.lfs_setstripe"/></para> </listitem> <listitem> - <para><xref linkend="dbdoclet.50438209_44776"/></para> + <para><xref linkend="file_striping.lfs_getstripe"/></para> </listitem> <listitem> - <para><xref linkend="dbdoclet.50438209_10424"/></para> + <para><xref linkend="file_striping.managing_free_space"/></para> </listitem> <listitem> <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/></para> </listitem> </itemizedlist> - <section xml:id="dbdoclet.50438209_79324"> + <section xml:id="file_striping.how_it_works"> <title> <indexterm> <primary>space</primary> @@ -56,17 +56,18 @@ default), the MDS then uses weighted random allocations with a preference for allocating objects on OSTs with more free space. (This can reduce I/O performance until space usage is rebalanced again.) For a more detailed description of how striping is allocated, see <xref - linkend="dbdoclet.50438209_10424"/>.</para> + linkend="file_striping.managing_free_space"/>.</para> <para>Files can only be striped over a finite number of OSTs, based on the maximum size of the attributes that can be stored on the MDT. If the MDT is ldiskfs-based without the <literal>ea_inode</literal> feature, a file can be striped across at most 160 OSTs. With a ZFS-based MDT, or if the - <literal>ea_inode</literal> feature is enabled for an ldiskfs-based MDT, + <literal>ea_inode</literal> feature is enabled for an ldiskfs-based MDT + (the default since Lustre 2.13.0), a file can be striped across up to 2000 OSTs. For more information, see <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/>. </para> </section> - <section xml:id="dbdoclet.50438209_48033"> + <section xml:id="file_striping.considerations"> <title><indexterm> <primary>file layout</primary> <secondary>See striping</secondary> @@ -181,7 +182,7 @@ </itemizedlist> </section> </section> - <section xml:id="dbdoclet.50438209_78664"> + <section xml:id="file_striping.lfs_setstripe"> <title><indexterm> <primary>striping</primary> <secondary>configuration</secondary> @@ -233,14 +234,16 @@ <literal>pool_name</literal> </emphasis> </para> - <para>The <literal>pool_name</literal> specifies the OST pool to which the file will be written. - This allows limiting the OSTs used to a subset of all OSTs in the file system. For more - details about using OST pools, see <link xl:href="ManagingFileSystemIO.html#50438211_75549" - >Creating and Managing OST Pools</link>.</para> + <para>The <literal>pool_name</literal> specifies the OST pool to which the + file will be written. This allows limiting the OSTs used to a subset of + all OSTs in the file system. For more details about using OST pools, see + <link xl:href="managingfilesystemio.managing_ost_pools"> + Creating and Managing OST Pools + </link>.</para> <section remap="h3"> <title>Specifying a File Layout (Striping Pattern) for a Single File It is possible to specify the file layout when a new file is created using the command lfs setstripe. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an lfs setstripe command fails if the file already exists. -
+
Setting the Stripe Size The command to create a new file with a specified stripe size is similar to: [client]# lfs setstripe -s 4M /mnt/lustre/new_file @@ -263,7 +266,8 @@ obdidx objid objid group The command below creates a new file with a stripe count of -1 to specify striping over all available OSTs: [client]# lfs setstripe -c -1 /mnt/lustre/full_stripe - The example below indicates that the file full_stripe is striped + The example below indicates that the file + full_stripe is striped over all six active OSTs in the configuration: [client]# lfs getstripe /mnt/lustre/full_stripe /mnt/lustre/full_stripe @@ -274,8 +278,9 @@ obdidx objid objid group 3 5 0x5 0 4 4 0x4 0 5 2 0x2 0 - This is in contrast to the output in , which - shows only a single object for the file. + This is in contrast to the output in + , + which shows only a single object for the file.
@@ -332,7 +337,7 @@ lmm_stripe_offset: 0 0 37364 0x91f4 0
-
+
<indexterm><primary>striping</primary><secondary>getting information</secondary></indexterm>Retrieving File Layout/Striping Information (<literal>getstripe</literal>) The lfs getstripe command is used to display information that shows over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along @@ -1962,7 +1967,7 @@ STRIPE OPTIONS:
-
+
<indexterm> <primary>space</primary> <secondary>free space</secondary> @@ -1991,16 +1996,17 @@ STRIPE OPTIONS: <literal>lctl set_param</literal> command, for example the next command reserve 1GB space for all OSTs. <screen>lctl set_param -P osp.*.reserved_mb_low=1024</screen></para> - <para>This section describes how to check available free space on disks and how free space is - allocated. It then describes how to set the threshold and weighting factors for the allocation - algorithms.</para> - <section xml:id="dbdoclet.checking_free_space"> + <para>This section describes how to check available free space on disks + and how free space is allocated. It then describes how to set the + threshold and weighting factors for the allocation algorithms.</para> + <section xml:id="file_striping.checking_free_space"> <title>Checking File System Free Space - Free space is an important consideration in assigning file stripes. The lfs - df command can be used to show available disk space on the mounted Lustre file - system and space consumption per OST. If multiple Lustre file systems are mounted, a path - may be specified, but is not required. Options to the lfs df command are - shown below. + Free space is an important consideration in assigning file stripes. + The lfs df command can be used to show available + disk space on the mounted Lustre file system and space consumption per + OST. If multiple Lustre file systems are mounted, a path may be + specified, but is not required. Options to the lfs df + command are shown below. @@ -2018,10 +2024,25 @@ STRIPE OPTIONS: - -h + + -h, --human-readable + - Displays sizes in human readable format (for example: 1K, 234M, 5G). + Displays sizes in human readable format (for example: 1K, + 234M, 5G) using base-2 (binary) values (i.e. 1G = 1024M). + + + + + + -H, --si + + + + Like -h, this displays counts in human + readable format, but using base-10 (decimal) values + (i.e. 1G = 1000M). @@ -2032,46 +2053,151 @@ STRIPE OPTIONS: Lists inodes instead of block usage. + + + -l, --lazy + + + Do not attempt to contact any OST or MDT not currently + connected to the client. This avoids blocking the + lfs df output if a target is offline or + unreachable, and only returns the space on OSTs that can + currently be accessed. + + + + + -p, --pool + + + Limit the usage to report only OSTs that are in the + specified pool. If multiple + Lustre filesystems are mounted, list the OSTs in + pool for each filesystem, or + limit the display to only a pool for a specific filesystem + if fsname.pool is given. + Specifying both fsname and + pool is equivalent to providing + a specific mountpoint. + + + + + + + -v, --verbose + + + + Display verbose status of MDTs and OSTs. This may + include one or more optional flags at the end of each line. + + + + + lfs df may also report additional target status + as the last column in the display, if there are issues with that target. + Target states include: + + + + D: OST/MDT is Degraded. + The target has a failed drive in the RAID device, or is + undergoing RAID reconstruction. This state is marked on + the server automatically for ZFS targets via + zed, or a (user-supplied) script that + monitors the target device and sets + "lctl set_param obdfilter.target.degraded=1" + on the OST. This target will be avoided for new + allocations, but will still be used to read existing files + located there or if there are not enough non-degraded OSTs + to make up a widely-striped file. + + + R: OST/MDT is Read-only. + The target filesystem is marked read-only due to filesystem + corruption detected by ldiskfs or ZFS. No modifications + are allowed on this OST, and it needs to be unmounted and + e2fsck or zpool scrub + run to repair the underlying filesystem. + + + N: OST/MDT is No-precreate. + The target is configured to deny object precreation set by + "lctl set_param obdfilter.target.no_precreate=1" + parameter or the "-o no_precreate" mount option. + This may be done to add an OST to the filesystem without allowing + objects to be allocated on it yet, or for other reasons. + + + S: OST/MDT is out of Space. + The target filesystem has less than the minimum required + free space and will not be used for new object allocations + until it has more free space. + + + I: OST/MDT is out of Inodes. + The target filesystem has less than the minimum required + free inodes and will not be used for new object allocations + until it has more free inodes. + + + f: OST/MDT is on flash. + The target filesystem is using a flash (non-rotational) + storage device. This is normally detected from the + underlying Linux block device, but can be set manually + with "lctl set_param osd-*.*.nonrotational=1 + on the respective OSTs. This lower-case status is only + shown in conjunction with the -v option, + since it is not an error condition. + + - The df -i and lfs df -i commands show the - minimum number of inodes that can be created in the - file system at the current time. If the total number of objects available across all of - the OSTs is smaller than those available on the MDT(s), taking into account the default - file striping, then df -i will also report a smaller number of inodes - than could be created. Running lfs df -i will report the actual number - of inodes that are free on each target. - For ZFS file systems, the number of inodes that can be created is dynamic and depends - on the free space in the file system. The Free and Total inode counts reported for a ZFS - file system are only an estimate based on the current usage for each target. The Used - inode count is the actual number of inodes used by the file system. + The df -i and lfs df -i + commands show the minimum number + of inodes that can be created in the file system at the current time. + If the total number of objects available across all of the OSTs is + smaller than those available on the MDT(s), taking into account the + default file striping, then df -i will also + report a smaller number of inodes than could be created. Running + lfs df -i will report the actual number of inodes + that are free on each target. + + For ZFS file systems, the number of inodes that can be created + is dynamic and depends on the free space in the file system. The + Free and Total inode counts reported for a ZFS file system are only + an estimate based on the current usage for each target. The Used + inode count is the actual number of inodes used by the file system. + Examples - [client1] $ lfs df -UUID 1K-blockS Used Available Use% Mounted on -mds-lustre-0_UUID 9174328 1020024 8154304 11% /mnt/lustre[MDT:0] -ost-lustre-0_UUID 94181368 56330708 37850660 59% /mnt/lustre[OST:0] -ost-lustre-1_UUID 94181368 56385748 37795620 59% /mnt/lustre[OST:1] -ost-lustre-2_UUID 94181368 54352012 39829356 57% /mnt/lustre[OST:2] -filesystem summary: 282544104 167068468 39829356 57% /mnt/lustre + client$ lfs df +UUID 1K-blocks Used Available Use% Mounted on +testfs-OST0000_UUID 9174328 1020024 8154304 11% /mnt/lustre[MDT:0] +testfs-OST0000_UUID 94181368 56330708 37850660 59% /mnt/lustre[OST:0] +testfs-OST0001_UUID 94181368 56385748 37795620 59% /mnt/lustre[OST:1] +testfs-OST0002_UUID 94181368 54352012 39829356 57% /mnt/lustre[OST:2] +filesystem summary: 282544104 167068468 39829356 57% /mnt/lustre -[client1] $ lfs df -h -UUID bytes Used Available Use% Mounted on -mds-lustre-0_UUID 8.7G 996.1M 7.8G 11% /mnt/lustre[MDT:0] -ost-lustre-0_UUID 89.8G 53.7G 36.1G 59% /mnt/lustre[OST:0] -ost-lustre-1_UUID 89.8G 53.8G 36.0G 59% /mnt/lustre[OST:1] -ost-lustre-2_UUID 89.8G 51.8G 38.0G 57% /mnt/lustre[OST:2] -filesystem summary: 269.5G 159.3G 110.1G 59% /mnt/lustre +[client1] $ lfs df -hv +UUID bytes Used Available Use% Mounted on +testfs-MDT0000_UUID 8.7G 996.1M 7.8G 11% /mnt/lustre[MDT:0] +testfs-OST0000_UUID 89.8G 53.7G 36.1G 59% /mnt/lustre[OST:0] f +testfs-OST0001_UUID 89.8G 53.8G 36.0G 59% /mnt/lustre[OST:1] f +testfs-OST0002_UUID 89.8G 51.8G 38.0G 57% /mnt/lustre[OST:2] f +filesystem summary: 269.5G 159.3G 110.1G 59% /mnt/lustre -[client1] $ lfs df -i -UUID Inodes IUsed IFree IUse% Mounted on -mds-lustre-0_UUID 2211572 41924 2169648 1% /mnt/lustre[MDT:0] -ost-lustre-0_UUID 737280 12183 725097 1% /mnt/lustre[OST:0] -ost-lustre-1_UUID 737280 12232 725048 1% /mnt/lustre[OST:1] -ost-lustre-2_UUID 737280 12214 725066 1% /mnt/lustre[OST:2] -filesystem summary: 2211572 41924 2169648 1% /mnt/lustre[OST:2] +[client1] $ lfs df -iH +UUID Inodes IUsed IFree IUse% Mounted on +testfs-MDT0000_UUID 2.21M 41.9k 2.17M 1% /mnt/lustre[MDT:0] +testfs-OST0000_UUID 737.3k 12.1k 725.1k 1% /mnt/lustre[OST:0] +testfs-OST0001_UUID 737.3k 12.2k 725.0k 1% /mnt/lustre[OST:1] +testfs-OST0002_UUID 737.3k 12.2k 725.0k 1% /mnt/lustre[OST:2] +filesystem summary: 2.21M 41.9k 2.17M 1% /mnt/lustre[OST:2] +
<indexterm> diff --git a/UnderstandingLustre.xml b/UnderstandingLustre.xml index def502a..7769017 100644 --- a/UnderstandingLustre.xml +++ b/UnderstandingLustre.xml @@ -810,7 +810,7 @@ ability to stripe is also useful when a single OST does not have enough free space to hold an entire file. For more information about benefits and drawbacks of file striping, see - <xref linkend="dbdoclet.50438209_48033" />.</para> + <xref linkend="file_striping.considerations" />.</para> <para>Striping allows segments or 'chunks' of data in a file to be stored on different OSTs, as shown in <xref linkend="understandinglustre.fig.filestripe" />. In the Lustre file @@ -830,7 +830,7 @@ for <literal>stripe_size</literal> is 1MB. The user may change these values on a per directory or per file basis. For more details, see - <xref linkend="dbdoclet.50438209_78664" />.</para> + <xref linkend="file_striping.lfs_setstripe" />.</para> <para> <xref linkend="understandinglustre.fig.filestripe" />, the <literal>stripe_size</literal> for File C is larger than the