1 <?xml version="1.0" encoding="utf-8"?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
5 xmlns:xi="http://www.w3.org/2001/XInclude">
6 <title xml:id="userutilities.title">User Utilities</title>
7 <para>This chapter describes user utilities.</para>
8 <section xml:id="dbdoclet.50438206_94597">
11 <primary>lfs</primary>
13 <literal>lfs</literal>
16 <literal>lfs</literal> utility can be used for user configuration routines
17 and monitoring.</para>
19 <title>Synopsis</title>
22 lfs changelog [--follow] <replaceable>mdt_name</replaceable> [startrec [endrec]]
23 lfs changelog_clear <replaceable>mdt_name id endrec</replaceable>
24 lfs check <replaceable>mds|osts|servers</replaceable>
25 lfs data_version [-nrw] <replaceable>filename</replaceable>
26 lfs df [-i] [-h] [--pool]-p <replaceable>fsname</replaceable>[.<replaceable>pool</replaceable>] [<replaceable>path</replaceable>] [--lazy]
27 lfs find [[!] --atime|-A [-+]N] [[!] --mtime|-M [-+]N]
28 [[!] --ctime|-C [-+]N] [--maxdepth|-D N] [--name|-n <replaceable>pattern</replaceable>]
29 [--print|-p] [--print0|-P] [[!] --obd|-O <replaceable>ost_name</replaceable>[,<replaceable>ost_name...</replaceable>]]
30 [[!] --size|-S [+-]N[kMGTPE]] --type |-t {bcdflpsD}]
31 [[!] --gid|-g|--group|-G <replaceable>gname|gid</replaceable>]
32 [[!] --uid|-u|--user|-U <replaceable>uname|uid</replaceable>]
33 <replaceable>dirname|filename</replaceable>
34 lfs getname [-h]|[path...]
35 lfs getstripe [--obd|-O <replaceable>ost_name</replaceable>] [--quiet|-q] [--verbose|-v]
36 [--stripe-count|-c] [--stripe-index|-i]
37 [--stripe-size|-s] [--pool|-p] [--directory|-d]
38 [--mdt-index|-M] [--recursive|-r] [--raw|-R]
40 <replaceable>dirname|filename</replaceable> ...
41 lfs setstripe [--size|-s stripe_size] [--stripe-count|-c <replaceable>stripe_count</replaceable>]
42 [--overstripe-count|-C <replaceable>stripe_count</replaceable>]
43 [--stripe-index|-i <replaceable>start_ost_index</replaceable>]
44 [--ost-list|-o <replaceable>ost_indicies</replaceable>]
45 [--pool|-p <replaceable>pool</replaceable>]
46 <replaceable>dirname|filename</replaceable>
47 lfs setstripe -d <replaceable>dir</replaceable>
49 lfs pool_list <replaceable>filesystem</replaceable>[.<replaceable>pool</replaceable>]| <replaceable>pathname</replaceable>
50 lfs quota [-q] [-v] [-h] [-o <replaceable>obd_uuid</replaceable>|-I <replaceable>ost_idx</replaceable>|-i <replaceable>mdt_idx</replaceable>]
51 [-u <replaceable>username|uid|-g</replaceable> <replaceable>group|gid</replaceable>|-p <replaceable>projid</replaceable>] <replaceable>/mount_point</replaceable>
52 lfs quota -t -u|-g|-p <replaceable>/mount_point</replaceable>
53 lfs quotacheck [-ug] <replaceable>/mount_point</replaceable>
54 lfs quotachown [-i] <replaceable>/mount_point</replaceable>
55 lfs quotainv [-ug] [-f] <replaceable>/mount_point</replaceable>
56 lfs quotaon [-ugf] <replaceable>/mount_point</replaceable>
57 lfs quotaoff [-ug] <replaceable>/mount_point</replaceable>
58 lfs setquota {-u|--user|-g|--group|-p|--project} <replaceable>uname|uid|gname|gid|projid</replaceable>
59 [--block-softlimit <replaceable>block_softlimit</replaceable>]
60 [--block-hardlimit <replaceable>block_hardlimit</replaceable>]
61 [--inode-softlimit <replaceable>inode_softlimit</replaceable>]
62 [--inode-hardlimit <replaceable>inode_hardlimit</replaceable>]
63 <replaceable>/mount_point</replaceable>
64 lfs setquota -u|--user|-g|--group|-p|--project <replaceable>uname|uid|gname|gid|projid</replaceable>
65 [-b <replaceable>block_softlimit</replaceable>] [-B <replaceable>block_hardlimit</replaceable>]
66 [-i <replaceable>inode-softlimit</replaceable>] [-I <replaceable>inode_hardlimit</replaceable>]
67 <replaceable>/mount_point</replaceable>
68 lfs setquota -t -u|-g|-p [--block-grace <replaceable>block_grace</replaceable>]
69 [--inode-grace <replaceable>inode_grace</replaceable>]
70 <replaceable>/mount_point</replaceable>
71 lfs setquota -t -u|-g|-p [-b <replaceable>block_grace</replaceable>] [-i <replaceable>inode_grace</replaceable>]
72 <replaceable>/mount_point</replaceable>
76 <para>In the above example, the
78 <replaceable>/mount_point</replaceable>
79 </literal> parameter refers to the mount point of the Lustre file
83 <para>The old lfs quota output was very detailed and contained
84 cluster-wide quota statistics (including cluster-wide limits for a
85 user/group and cluster-wide usage for a user/group), as well as
86 statistics for each MDS/OST. Now,
87 <literal>lfs quota</literal> has been updated to provide only
88 cluster-wide statistics, by default. To obtain the full report of
89 cluster-wide limits, usage and statistics, use the
90 <literal>-v</literal> option with
91 <literal>lfs quota</literal>.</para>
95 <title>Description</title>
97 <literal>lfs</literal> utility is used to create a new file with a
98 specific striping pattern, determine the default striping pattern, gather
99 the extended attributes (object numbers and location) for a specific
100 file, find files with specific attributes, list OST information or set
101 quota limits. It can be invoked interactively without any arguments or in
102 a non-interactive mode with one of the supported arguments.</para>
105 <title>Options</title>
107 <literal>lfs</literal> options are listed and described below. For a
108 complete list of available options, type help at the
109 <literal>lfs</literal> prompt.</para>
110 <informaltable frame="all">
112 <colspec colname="c1" colwidth="20*" />
113 <colspec colname="c2" colwidth="30*" />
114 <colspec colname="c3" colwidth="50*" />
117 <entry nameend="c2" namest="c1">
119 <emphasis role="bold">Option</emphasis>
124 <emphasis role="bold">Description</emphasis>
131 <entry nameend="c2" namest="c1">
133 <literal>changelog</literal>
137 <para>Shows the metadata changes on an MDT. Start and end
138 points are optional. The
139 <literal>--follow</literal> option blocks on new changes; this
140 option is only valid when run directly on the MDT node.</para>
144 <entry nameend="c2" namest="c1">
146 <literal>changelog_clear</literal>
150 <para>Indicates that changelog records previous to
152 <replaceable>endrec</replaceable>
153 </literal> are no longer of interest to a particular consumer
155 <replaceable>id</replaceable>
156 </literal>, potentially allowing the MDT to free up disk space.
159 <replaceable>endrec</replaceable>
160 </literal> of 0 indicates the current last record. Changelog
161 consumers must be registered on the MDT node using
162 <literal>lctl</literal>.</para>
166 <entry nameend="c2" namest="c1">
167 <literal>check</literal>
170 <para>Displays the status of MDS or OSTs (as specified in the
171 command) or all servers (MDS and OSTs).</para>
175 <entry nameend="c2" namest="c1">
177 <literal>data_version [-nrw]
178 <replaceable>filename</replaceable></literal>
182 <para>Displays the current version of file data. If
183 <literal>-n</literal> is specified, the data version is read
184 without taking a lock. As a consequence, the data version could
185 be outdated if there are dirty caches on filesystem clients, but
186 this option will not force data flushes and has less of an
187 impact on the filesystem. If <literal>-r</literal> is specified,
188 the data version is read after dirty pages on clients are
189 flushed. If <literal>-w</literal> is specified, the data version
190 is read after all caching pages on clients are flushed.
193 Even with <literal>-r</literal> or <literal>-w</literal>, race
194 conditions are possible and the data version should be checked
195 before and after an operation to be confident the data did not
199 The data version is the sum of the last committed transaction
200 numbers of all data objects of a file. It is used by HSM policy
201 engines for verifying that file data has not been changed during
202 an archive operation or before a release operation, and by OST
203 migration, primarily for verifying that file data has not been
204 changed during a data copy, when done in non-blocking mode.
209 <entry nameend="c2" namest="c1">
210 <literal>df [-i] [-h] [--pool|-p
211 <replaceable>fsname</replaceable>[.
212 <replaceable>pool</replaceable>] [
213 <replaceable>path</replaceable>] [--lazy]</literal>
217 <literal>-i</literal> to report file system disk space usage or
218 inode usage of each MDT or OST or, if a pool is specified with
220 <literal>-p</literal> option, a subset of OSTs.</para>
221 <para>By default, the usage of all mounted Lustre file systems
223 <literal>path</literal> option is included, only the usage for
224 the specified file system is reported. If the
225 <literal>-h</literal> option is included, the output is printed
226 in human-readable format, using SI base-2 suffixes for
227 <emphasis role="bold">M</emphasis>ega-,
228 <emphasis role="bold">G</emphasis>iga-,
229 <emphasis role="bold">T</emphasis>era-,
230 <emphasis role="bold">P</emphasis>eta-, or
231 <emphasis role="bold">E</emphasis>xabytes.</para>
233 <literal>--lazy</literal> option is specified, any OST that is
234 currently disconnected from the client will be skipped. Using
236 <literal>--lazy</literal> option prevents the
237 <literal>df</literal> output from being blocked when an OST is
238 offline. Only the space on the OSTs that can currently be
239 accessed are returned. The
240 <literal>llite.*.lazystatfs</literal> tunable can be enabled to
241 make this the default behaviour for all
242 <literal>statfs()</literal> operations.</para>
246 <entry nameend="c2" namest="c1">
248 <literal>find</literal>
252 <para>Searches the directory tree rooted at the given
253 directory/filename for files that match the given
256 <literal>!</literal> before an option negates its meaning (files
257 NOT matching the parameter). Using
258 <literal>+</literal> before a numeric value means files with the
259 parameter OR MORE. Using
260 <literal>-</literal> before a numeric value means files with the
261 parameter OR LESS.</para>
267 <literal>--atime</literal>
270 <para>File was last accessed N*24 hours ago. (There is no
272 <literal>atime</literal> is kept coherent across the
274 <para>OSTs store a transient
275 <literal>atime</literal> that is updated when clients do read
277 <literal>atime</literal> is written to the MDS when the file is
278 closed. However, on-disk atime is only updated if it is more
279 than 60 seconds old (
280 <literal>/proc/fs/lustre/mds/*/max_atime_diff</literal>). The
281 Lustre software considers the latest
282 <literal>atime</literal> from all OSTs. If a
283 <literal>setattr</literal> is set by user, then it is updated on
284 both the MDS and OST, allowing the
285 <literal>atime</literal> to go backward.</para>
294 <literal>--ctime</literal>
298 <para>File status was last changed N*24 hours ago.</para>
307 <literal>--mtime</literal>
311 <para>File data was last modified N*24 hours ago.</para>
320 <literal>--obd</literal>
324 <para>File has an object on a specific OST(s).</para>
333 <literal>--size</literal>
337 <para>File has a size in bytes, or kilo-, Mega-, Giga-, Tera-,
338 Peta- or Exabytes if a suffix is given.</para>
347 <literal>--type</literal>
351 <para>File has the type - block, character, directory, pipe,
352 file, symlink, socket or door (used in Solaris operating
362 <literal>--uid</literal>
366 <para>File has a specific numeric user ID.</para>
375 <literal>--user</literal>
379 <para>File owned by a specific user (numeric user ID
389 <literal>--gid</literal>
393 <para>File has a specific group ID.</para>
402 <literal>--group</literal>
406 <para>File belongs to a specific group (numeric group ID
416 <literal>-maxdepth</literal></para>
419 <para>Limits find to descend at most N levels of the directory
429 <literal>--print</literal>/
430 <literal>--print0</literal></para>
433 <para>Prints the full filename, followed by a new line or NULL
434 character correspondingly.</para>
438 <entry nameend="c2" namest="c1">
440 <literal>osts [path]</literal>
444 <para>Lists all OSTs for the file system. If a path located on
445 a mounted Lustre file system is specified, then only OSTs
446 belonging to this file system are displayed.</para>
450 <entry nameend="c2" namest="c1">
452 <literal>getname [path...]</literal>
456 <para>List each Lustre file system instance associated with
457 each Lustre mount point. If no path is specified, all Lustre
458 mount points are interrogated. If a list of paths is provided,
459 the instance of each path is provided. If the path is not a
460 Lustre instance 'No such device' is returned.</para>
464 <entry nameend="c2" namest="c1">
466 <literal>getstripe</literal>
470 <para>Lists striping information for a given filename or
471 directory. By default, the stripe count, stripe size and offset
473 <para>If you only want specific striping information, then the
475 <literal>--stripe-count</literal>,
476 <literal>--stripe-size</literal>,
477 <literal>--stripe-index</literal>,
478 <literal>--layout</literal>, or
479 <literal>--pool</literal> plus various combinations of these
480 options can be used to retrieve specific information.</para>
482 <literal>--raw</literal> option is specified, the stripe
483 information is printed without substituting the file system
484 default values for unspecified fields. If the striping EA is
485 not set, 0, 0, and -1 will be printed for the stripe count,
486 size, and offset respectively.</para>
487 <para condition="l24">The
488 <literal>--mdt-index</literal> prints the index of the MDT for a given
490 <xref linkend="lustremaint.rmremotedir" />.</para>
500 <replaceable>ost_name</replaceable></literal>
504 <para>Lists files that have an object on a specific OST.</para>
513 <literal>--quiet</literal>
517 <para>Lists details about the file's object ID
527 <literal>--verbose</literal>
531 <para>Prints additional striping information.</para>
540 <literal>--stripe-count</literal>
544 <para>Lists the stripe count (how many OSTs to use).</para>
553 <literal>--index</literal>
557 <para>Lists the index for each OST in the file system.</para>
566 <literal>--offset</literal>
570 <para>Lists the OST index on which file striping starts.</para>
579 <literal>--pool</literal>
583 <para>Lists the pools to which a file belongs.</para>
592 <literal>--size</literal>
596 <para>Lists the stripe size (how much data to write to one OST
597 before moving to the next OST).</para>
606 <literal>--directory</literal>
610 <para>Lists entries about a specified directory instead of its
611 contents (in the same manner as
612 <literal>ls -d</literal>).</para>
621 <literal>--recursive</literal>
625 <para>Recurses into all sub-directories.</para>
629 <entry nameend="c2" namest="c1">
631 <literal>setstripe</literal>
635 <para>Create new files with a specific file layout (stripe
636 pattern) configuration.
638 <para>The file cannot exist prior to using
639 <literal>setstripe</literal>. A directory must exist prior to
641 <literal>setstripe</literal>.</para>
651 <literal>--stripe-count stripe_cnt</literal>
655 <para>Number of OSTs over which to stripe a file. A
656 <literal>stripe_cnt</literal> of 0 uses the file system-wide
657 default stripe count (default is 1). A
658 <literal>stripe_cnt</literal> of -1 stripes over all available
668 <literal>--overstripe-count stripe_cnt</literal>
672 <para>The same as --stripe-count, but allows overstriping,
673 which will place more than one stripe per OST if
674 <literal>stripe_cnt</literal> is greater than the number of
675 OSTs. Overstriping is useful for matching the number of stripes
676 to the number of processes, or with very fast OSTs, where one
677 stripe per OST is not enough to get full performance.</para>
686 <literal>--size stripe_size</literal>
688 <para>The default stripe-size is 0. The default start-ost is
689 -1. Do NOT confuse them! If you set start-ost to 0, all new
690 file creations occur on OST 0 (seldom a good idea).</para>
691 </footnote> </para>
694 <para>Number of bytes to store on an OST before moving to the
695 next OST. A stripe_size of 0 uses the file system's default
696 stripe size, (default is 1 MB). Can be specified with
697 <emphasis role="bold">k</emphasis>(KB),
698 <emphasis role="bold">m</emphasis>(MB), or
699 <emphasis role="bold">g</emphasis>(GB), respectively.</para>
708 <literal>--stripe-index start_ost_index</literal>
712 <para>The OST index (base 10, starting at 0) on which to start
713 striping for this file. A start_ost_index value of -1 allows
714 the MDS to choose the starting index. This is the default
715 value, and it means that the MDS selects the starting OST as it
716 wants. We strongly recommend selecting this default, as it
717 allows space and load balancing to be done by the MDS as
719 <literal>start_ost_index</literal> value has no relevance on
720 whether the MDS will use round-robin or QoS weighted allocation
721 for the remaining stripes in the file.</para>
730 <literal>--ost-index ost_indices</literal>
734 <para>This option is used to specify the exact stripe
735 layout on the the file system. <literal>ost_indices</literal>
736 is a list of OSTs referenced by their indices and index ranges
737 separated by commas, e.g. <literal>1,2-4,7</literal>.</para>
747 <replaceable>pool</replaceable></literal>
751 <para>Name of the pre-defined pool of OSTs (see
752 <xref linkend="dbdoclet.50438219_38274" />) that will be used
754 <literal>stripe_cnt</literal>,
755 <literal>stripe_size</literal> and
756 <literal>start_ost</literal> values are used as well. The
757 start-ost value must be part of the pool or an error is
762 <entry nameend="c2" namest="c1">
764 <literal>setstripe -d</literal>
768 <para>Deletes default striping on the specified
773 <entry nameend="c2" namest="c1">
775 <literal>pool_list {filesystem}[.poolname]|{pathname}</literal>
779 <para>Lists pools in the file system or pathname, or OSTs in
780 the file system's pool.</para>
784 <entry nameend="c2" namest="c1">
786 <literal>quota [-q] [-v] [-o
787 <replaceable>obd_uuid</replaceable>|-i
788 <replaceable>mdt_idx</replaceable>|-I
789 <replaceable>ost_idx</replaceable>] [-u|-g|-p
790 <replaceable>uname|uid|gname|gid|projid]</replaceable>
791 <replaceable>/mount_point</replaceable></literal>
796 <para>Displays disk usage and limits, either for the full file
797 system or for objects on a specific OBD. A user or group name
798 or an usr, group and project ID can be specified. If all user,
799 group project ID are omitted, quotas for the current UID/GID
800 are shown. The <literal>-q</literal> option disables printing
801 of additional descriptions (including column titles). It fills
802 in blank spaces in the
803 <literal>grace</literal> column with zeros (when there is no
804 grace period set), to ensure that the number of columns is
806 <literal>-v</literal> option provides more verbose (per-OBD
807 statistics) output.</para>
811 <entry nameend="c2" namest="c1">
814 <replaceable>-u|-g|-p</replaceable>
815 <replaceable>/mount_point</replaceable></literal>
819 <para>Displays block and inode grace times for user (
820 <literal>-u</literal>) or group (
821 <literal>-g</literal>) or project (
822 <literal>-p</literal>) quotas.</para>
826 <entry nameend="c2" namest="c1">
828 <literal>quotachown</literal>
832 <para>Changes the file's owner and group on OSTs of the
833 specified file system.</para>
837 <entry nameend="c2" namest="c1">
839 <literal>quotacheck [-ugf]
840 <replaceable>/mount_point</replaceable></literal>
844 <para>Scans the specified file system for disk usage, and
845 creates or updates quota files. Options specify quota for users
847 <literal>-u</literal>), groups (
848 <literal>-g</literal>), and force (
849 <literal>-f</literal>).</para>
853 <entry nameend="c2" namest="c1">
855 <literal>quotaon [-ugf]
856 <replaceable>/mount_point</replaceable></literal>
860 <para>Turns on file system quotas. Options specify quota for
862 <literal>-u</literal>), groups (
863 <literal>-g</literal>), and force (
864 <literal>-f</literal>).</para>
868 <entry nameend="c2" namest="c1">
870 <literal>quotaoff [-ugf]
871 <replaceable>/mount_point</replaceable></literal>
875 <para>Turns off file system quotas. Options specify quota for
877 <literal>-u</literal>), groups (
878 <literal>-g</literal>), and force (
879 <literal>-f</literal>).</para>
883 <entry nameend="c2" namest="c1">
885 <literal>quotainv [-ug] [-f]
886 <replaceable>/mount_point</replaceable></literal>
890 <para>Clears quota files (administrative quota files if used
892 <literal>-f</literal>, operational quota files otherwise), all
893 of their quota entries for users (
894 <literal>-u</literal>) or groups (
895 <literal>-g</literal>). After running
896 <literal>quotainv</literal>, you must run
897 <literal>quotacheck</literal> before using quotas.</para>
899 <para>Use extreme caution when using this command; its
900 results cannot be undone.</para>
905 <entry nameend="c2" namest="c1">
907 <literal>setquota {-u|-g|-p
908 <replaceable>uname|uid|gname|gid|projid}</replaceable>
910 <replaceable>block_softlimit</replaceable>]
912 <replaceable>block_hardlimit</replaceable>]
914 <replaceable>inode_softlimit</replaceable>]
916 <replaceable>inode_hardlimit</replaceable>]
917 <replaceable>/mount_point</replaceable></literal>
921 <para>Sets file system quotas for users, groups or one project.
922 Limits can be specified with
923 <literal>--{block|inode}-{softlimit|hardlimit}</literal> or
924 their short equivalents
925 <literal>-b</literal>,
926 <literal>-B</literal>,
927 <literal>-i</literal>,
928 <literal>-I</literal>. Users can set 1, 2, 3 or 4 limits.
931 <literal>setquota</literal> interface is supported, but it may
932 be removed in a future Lustre software release.</para>
933 </footnote>Also, limits can be specified with special suffixes,
934 -b, -k, -m, -g, -t, and -p to indicate units of 1, 2^10, 2^20,
935 2^30, 2^40 and 2^50, respectively. By default, the block limits
936 unit is 1 kilobyte (1,024), and block limits are always
937 kilobyte-grained (even if specified in bytes). See
938 <xref linkend="dbdoclet.50438206_11903" />.</para>
942 <entry nameend="c2" namest="c1">
944 <literal>setquota -t -u|-g|-p [--block-grace
945 <replaceable>block_grace</replaceable>] [--inode-grace
946 <replaceable>inode_grace</replaceable>]
947 <replaceable>/mount_point</replaceable></literal>
951 <para>Sets the file system quota grace times for users or
952 groups. Grace time is specified in '
953 <literal>XXwXXdXXhXXmXXs</literal>' format or as an integer
955 <xref linkend="dbdoclet.50438206_11903" />.</para>
959 <entry nameend="c2" namest="c1">
961 <literal>help</literal>
965 <para>Provides brief help on various
966 <literal>lfs</literal> arguments.</para>
970 <entry nameend="c2" namest="c1">
972 <literal>exit/quit</literal>
976 <para>Quits the interactive
977 <literal>lfs</literal> session.</para>
984 <section remap="h5" xml:id="dbdoclet.50438206_11903">
985 <title>Examples</title>
986 <para>Creates a file striped on two OSTs with 128 KB on each
989 $ lfs setstripe -s 128k -c 2 /mnt/lustre/file1
991 <para>Deletes a default stripe pattern on a given directory. New files
992 use the default striping pattern.</para>
994 $ lfs setstripe -d /mnt/lustre/dir
996 <para>Lists the detailed object allocation of a given file.</para>
998 $ lfs getstripe -v /mnt/lustre/file1
1000 <para>List all the mounted Lustre file systems and corresponding Lustre
1005 <para>Efficiently lists all files in a given directory and its
1006 subdirectories.</para>
1008 $ lfs find /mnt/lustre
1010 <para>Recursively lists all regular files in a given directory more than
1013 $ lfs find /mnt/lustre -mtime +30 -type f -print
1015 <para>Recursively lists all files in a given directory that have objects
1016 on OST2-UUID. The lfs check servers command checks the status of all
1017 servers (MDT and OSTs).</para>
1019 $ lfs find --obd OST2-UUID /mnt/lustre/
1021 <para>Lists all OSTs in the file system.</para>
1025 <para>Lists space usage per OST and MDT in human-readable format.</para>
1029 <para>Lists inode usage per OST and MDT.</para>
1033 <para>List space or inode usage for a specific OST pool.</para>
1036 <replaceable>filesystem</replaceable>[.
1037 <replaceable>pool</replaceable>] |
1038 <replaceable>pathname</replaceable>
1040 <para>List quotas of user 'bob'.</para>
1042 $ lfs quota -u bob /mnt/lustre
1044 <para>List quotas of project ID '1'.</para>
1046 $ lfs quota -p 1 /mnt/lustre
1048 <para>Show grace times for user quotas on
1049 <literal>/mnt/lustre</literal>.</para>
1051 $ lfs quota -t -u /mnt/lustre
1053 <para>Changes file owner and group.</para>
1055 $ lfs quotachown -i /mnt/lustre
1057 <para>Checks quotas for user and group. Turns on quotas after making the
1060 $ lfs quotacheck -ug /mnt/lustre
1062 <para>Turns on quotas of user and group.</para>
1064 $ lfs quotaon -ug /mnt/lustre
1066 <para>Turns off quotas of user and group.</para>
1068 $ lfs quotaoff -ug /mnt/lustre
1070 <para>Sets quotas of user 'bob', with a 1 GB block quota hardlimit and a
1071 2 GB block quota softlimit.</para>
1073 $ lfs setquota -u bob --block-softlimit 2000000 --block-hardlimit 1000000
1076 <para>Sets grace times for user quotas: 1000 seconds for block quotas, 1
1077 week and 4 days for inode quotas.</para>
1079 $ lfs setquota -t -u --block-grace 1000 --inode-grace 1w4d /mnt/lustre
1081 <para>Checks the status of all servers (MDT, OST)</para>
1085 <para>Creates a file striped on two OSTs from the pool
1086 <literal>my_pool</literal></para>
1088 $ lfs setstripe --pool my_pool -c 2 /mnt/lustre/file
1090 <para>Lists the pools defined for the mounted Lustre file system
1091 <literal>/mnt/lustre</literal></para>
1093 $ lfs pool_list /mnt/lustre/
1095 <para>Lists the OSTs which are members of the pool
1096 <literal>my_pool</literal> in file system
1097 <literal>my_fs</literal></para>
1099 $ lfs pool_list my_fs.my_pool
1101 <para>Finds all directories/files associated with
1102 <literal>poolA</literal>.</para>
1104 $ lfs find /mnt/lustre --pool poolA
1106 <para>Finds all directories/files not associated with a pool.</para>
1108 $ lfs find /mnt//lustre --pool ""
1110 <para>Finds all directories/files associated with pool.</para>
1112 $ lfs find /mnt/lustre ! --pool ""
1114 <para>Associates a directory with the pool
1115 <literal>my_pool</literal>, so all new files and directories are created
1118 $ lfs setstripe --pool my_pool /mnt/lustre/dir
1121 <section remap="h5">
1122 <title>See Also</title>
1124 <xref linkend="dbdoclet.50438219_38274" />
1128 <section xml:id="dbdoclet.lfs_migrate">
1131 <primary>lfs_migrate</primary>
1133 <literal>lfs_migrate</literal>
1136 <literal>lfs_migrate</literal> utility is a simple to migrate file
1137 <emphasis>data</emphasis> between OSTs.</para>
1138 <section remap="h5">
1139 <title>Synopsis</title>
1141 lfs_migrate [<replaceable>lfs_setstripe_options</replaceable>]
1142 [-h] [-n] [-q] [-R] [-s] [-y] [-0] [file|directory ...]
1145 <section remap="h5">
1146 <title>Description</title>
1148 <literal>lfs_migrate</literal> utility is a tool to assist migration
1149 of file data between Lustre OSTs. The utility copies each specified
1150 file to a temporary file using supplied <literal>lfs setstripe</literal>
1151 options, if any, optionally verifies the file contents have not changed,
1152 and then swaps the layout (OST objects) from the temporary file and the
1153 original file (for Lustre 2.5 and later), or renames the temporary file
1154 to the original filename. This allows the user/administrator to balance
1155 space usage between OSTs, or move files off OSTs that are starting to show
1156 hardware problems (though are still functional) or will be removed.</para>
1158 <para>For versions of Lustre before 2.5,
1159 <literal>lfs_migrate</literal> was not integrated with the MDS at all.
1160 That made it UNSAFE for use on files that were being modified by other
1161 applications, since the file was migrated through a copy and rename of
1162 the file. With Lustre 2.5 and later, the new file layout is swapped
1163 with the existing file layout, which ensures that the user-visible
1164 inode number is kept, and open file handles and locks on the file are
1167 <para>Files to be migrated can be specified as command-line arguments. If
1168 a directory is specified on the command-line then all files within the
1169 directory are migrated. If no files are specified on the command-line,
1170 then a list of files is read from the standard input, making
1171 <literal>lfs_migrate</literal> suitable for use with
1172 <literal>lfs find</literal> to locate files on specific OSTs and/or
1173 matching other file attributes, and other tools that generate a list
1174 of files on standard output.</para>
1175 <para>Unless otherwise specified through command-line options, the
1176 file allocation policies on the MDS dictate where the new files
1177 are placed, taking into account whether specific OSTs have been
1178 disabled on the MDS via <literal>lctl</literal> (preventing new
1179 files from being allocated there), whether some OSTs are overly full
1180 (reducing the number of files placed on those OSTs), or if there is
1181 a specific default file striping for the parent directory (potentially
1182 changing the stripe count, stripe size, OST pool, or OST index of a
1186 <literal>lfs_migrate</literal> utility can also be used in some cases to
1189 <primary>fragmentation</primary>
1190 </indexterm>fragmentation. File fragmentation will typically reduce
1191 Lustre file system performance. File fragmentation may be observed on
1192 an aged file system and will commonly occur if the file was written by
1193 many threads. Provided there is sufficient free space (or if it was
1194 written when the file system was nearly full) that is less fragmented
1195 than the file being copied, re-writing a file with
1196 <literal>lfs_migrate</literal> will result in a migrated file with
1197 reduced fragmentation. The tool
1198 <literal>filefrag</literal> can be used to report file fragmentation.
1200 <xref linkend="dbdoclet.50438206_75125" /></para>
1203 <para>As long as a file has extent lengths of tens of megabytes (
1204 <replaceable>read_bandwidth * seek_time</replaceable>) or more, the
1205 read performance for the file will not be significantly impacted by
1206 fragmentation, since the read pipeline can be filled by large reads
1207 from disk even with an occasional disk seek.</para>
1210 <section remap="h5">
1211 <title>Options</title>
1212 <para>Options supporting
1213 <literal>lfs_migrate</literal> are described below.</para>
1214 <informaltable frame="all">
1216 <colspec colname="c1" colwidth="50*" />
1217 <colspec colname="c2" colwidth="50*" />
1222 <emphasis role="bold">Option</emphasis>
1227 <emphasis role="bold">Description</emphasis>
1237 <replaceable>stripecount</replaceable></literal>
1241 <para>Restripe file using the specified stripe count. This
1242 option may not be specified at the same time as the
1243 <literal>-R</literal> option.</para>
1249 <literal>-h</literal>
1253 <para>Display help information.</para>
1258 <literal>-l</literal>
1261 <para>Migrate files with hard links (skips, by default). Files
1262 with multiple hard links are split into multiple separate files
1264 <literal>lfs_migrate</literal>, so they are skipped, by
1265 default, to avoid breaking the hard links.</para>
1270 <literal>-n</literal>
1273 <para>Only print the names of files to be migrated.</para>
1279 <literal>-q</literal>
1283 <para>Run quietly (does not print filenames or status).</para>
1288 <literal>-R</literal>
1290 <entry>Restripe file using default directory striping instead of
1291 keeping striping. This option may not be specified at the same
1293 <literal>-c</literal> option.</entry>
1297 <literal>-s</literal>
1299 <entry>Skip file data comparison after migrate. Default is to
1300 compare migrated file against original to verify
1301 correctness.</entry>
1306 <literal>-y</literal>
1311 <literal>y</literal>' to usage warning without prompting (for
1312 scripts, use with caution).</para>
1317 <literal>-0</literal>
1319 <entry>Expect NUL-terminated filenames on standard input, as
1320 generated by <literal>lfs find -print0</literal> or
1321 <literal>find -print0</literal>. This allows filenames with
1322 embedded newlines to be handled correctly.
1329 <section remap="h5">
1330 <title>Examples</title>
1331 <para>Rebalance all files in
1332 <literal>/mnt/lustre/dir</literal>:</para>
1334 $ lfs_migrate /mnt/lustre/dir
1336 <para>Migrate files in /test filesystem on OST0004 larger than 4 GB in
1337 size and older than a day old:</para>
1339 $ lfs find /test -obd test-OST0004 -size +4G -mtime +1 | lfs_migrate -y
1342 <section remap="h5">
1343 <title>See Also</title>
1345 <xref linkend="dbdoclet.50438206_94597" />
1349 <section xml:id="dbdoclet.50438206_75125">
1352 <primary>filefrag</primary>
1354 <literal>filefrag</literal>
1357 <literal>e2fsprogs</literal> package contains the
1358 <literal>filefrag</literal> tool which reports the extent of file
1359 fragmentation.</para>
1360 <section remap="h5">
1361 <title>Synopsis</title>
1363 filefrag [ -belsv ] [ files... ]
1366 <section remap="h5">
1367 <title>Description</title>
1369 <literal>filefrag</literal> utility reports the extent of fragmentation in
1371 <literal>filefrag</literal> utility obtains the extent information from
1372 Lustre files using the
1373 <literal>FIEMAP ioctl</literal>, which is efficient and fast, even for
1374 very large files.</para>
1375 <para>In default mode
1377 <para>The default mode is faster than the verbose/extent mode since it
1378 only counts the number of extents.</para>
1380 <literal>filefrag</literal> prints the number of physically discontiguous
1381 extents in the file. In extent or verbose mode, each extent is printed
1382 with details such as the blocks allocated on each OST. For a Lustre file
1383 system, the extents are printed in device offset order (i.e. all of the
1384 extents for one OST first, then the next OST, etc.), not file logical
1385 offset order. If the file logical offset order was used, the Lustre
1386 striping would make the output very verbose and difficult to see if there
1387 was file fragmentation or not.</para>
1389 <para>Note that as long as a file has extent lengths of tens of
1390 megabytes or more (i.e.
1391 <replaceable>read_bandwidth * seek_time >
1392 extent_length</replaceable>), the read performance for the file will
1393 not be significantly impacted by fragmentation, since file readahead
1394 can fully utilize the disk disk bandwidth even with occasional
1397 <para>In default mode
1399 <para>The default mode is faster than the verbose/extent mode.</para>
1401 <literal>filefrag</literal> returns the number of physically discontiguous
1402 extents in the file. In extent or verbose mode, each extent is printed
1403 with details. For a Lustre file system, the extents are printed in device
1404 offset order, not logical offset order.</para>
1406 <section remap="h5">
1407 <title>Options</title>
1408 <para>The options and descriptions for the
1409 <literal>filefrag</literal> utility are listed below.</para>
1410 <informaltable frame="all">
1412 <colspec colname="c1" colwidth="50*" />
1413 <colspec colname="c2" colwidth="50*" />
1418 <emphasis role="bold">Option</emphasis>
1423 <emphasis role="bold">Description</emphasis>
1432 <literal>-b</literal>
1436 <para>Uses the 1024-byte blocksize for the output. By default,
1437 this blocksize is used by the Lustre file system, since OSTs
1438 may use different block sizes.</para>
1444 <literal>-e</literal>
1448 <para>Uses the extent mode when printing the output. This is
1449 the default for Lustre files in verbose mode.</para>
1455 <literal>-l</literal>
1459 <para>Displays extents in LUN offset order. This is the only
1460 available mode for Lustre.</para>
1466 <literal>-s</literal>
1470 <para>Synchronizes any unwritten file data to disk before
1471 requesting the mapping.</para>
1477 <literal>-v</literal>
1481 <para>Prints the file's layout in verbose mode when checking
1482 file fragmentation, including the logical to physical mapping
1483 for each extent in the file and the OST index.</para>
1490 <section remap="h5">
1491 <title>Examples</title>
1492 <para>Lists default output.</para>
1494 $ filefrag /mnt/lustre/foo
1495 /mnt/lustre/foo: 13 extents found
1497 <para>Lists verbose output in extent format.</para>
1499 $ filefrag -v /mnt/lustre/foo
1500 Filesystem type is: bd00bd0
1501 File size of /mnt/lustre/foo is 1468297786 (1433888 blocks of 1024 bytes)
1502 ext: device_logical: physical_offset: length: dev: flags:
1503 0: 0.. 122879: 2804679680..2804802559: 122880: 0002: network
1504 1: 122880.. 245759: 2804817920..2804940799: 122880: 0002: network
1505 2: 245760.. 278527: 2804948992..2804981759: 32768: 0002: network
1506 3: 278528.. 360447: 2804982784..2805064703: 81920: 0002: network
1507 4: 360448.. 483327: 2805080064..2805202943: 122880: 0002: network
1508 5: 483328.. 606207: 2805211136..2805334015: 122880: 0002: network
1509 6: 606208.. 729087: 2805342208..2805465087: 122880: 0002: network
1510 7: 729088.. 851967: 2805473280..2805596159: 122880: 0002: network
1511 8: 851968.. 974847: 2805604352..2805727231: 122880: 0002: network
1512 9: 974848.. 1097727: 2805735424..2805858303: 122880: 0002: network
1513 10: 1097728.. 1220607: 2805866496..2805989375: 122880: 0002: network
1514 11: 1220608.. 1343487: 2805997568..2806120447: 122880: 0002: network
1515 12: 1343488.. 1433599: 2806128640..2806218751: 90112: 0002: network
1516 /mnt/lustre/foo: 13 extents found
1520 <section xml:id="dbdoclet.50438206_86244">
1523 <primary>mount</primary>
1525 <literal>mount</literal>
1528 <literal>mount(8)</literal> Linux command is used to mount a Lustre file
1529 system. When mounting a Lustre file system, mount(8) executes the
1530 <literal>/sbin/mount.lustre</literal> command to complete the mount. The
1531 mount command supports these options specific to a Lustre file
1533 <informaltable frame="all">
1535 <colspec colname="c1" colwidth="50*" />
1536 <colspec colname="c2" colwidth="50*" />
1541 <emphasis role="bold">Server options</emphasis>
1546 <emphasis role="bold">Description</emphasis>
1555 <literal>abort_recov</literal>
1559 <para>Aborts recovery when starting a target</para>
1565 <literal>nosvc</literal>
1569 <para>Starts only MGS/MGC servers</para>
1575 <literal>nomgs</literal>
1579 <para>Start a MDT with a co-located MGS without starting the
1586 <literal>exclude</literal>
1590 <para>Starts with a dead OST</para>
1596 <literal>md_stripe_cache_size</literal>
1600 <para>Sets the stripe cache size for server side disk with a
1601 striped raid configuration</para>
1607 <informaltable frame="all">
1609 <colspec colname="c1" colwidth="50*" />
1610 <colspec colname="c2" colwidth="50*" />
1615 <emphasis role="bold">Client options</emphasis>
1620 <emphasis role="bold">Description</emphasis>
1629 <literal>flock/noflock/localflock</literal>
1633 <para>Enables/disables global flock or local flock support</para>
1639 <literal>user_xattr/nouser_xattr</literal>
1643 <para>Enables/disables user-extended attributes</para>
1649 <literal>user_fid2path/nouser_fid2path</literal>
1653 <para>Enables/disables FID to path translation by
1654 regular users</para>
1660 <literal>retry=</literal>
1664 <para>Number of times a client will retry to mount the file
1672 <section xml:id="dbdoclet.50438206_56217">
1673 <title>Handling Timeouts</title>
1674 <para>Timeouts are the most common cause of hung applications. After a
1675 timeout involving an MDS or failover OST, applications attempting to access
1676 the disconnected resource wait until the connection gets
1678 <para>When a client performs any remote operation, it gives the server a
1679 reasonable amount of time to respond. If a server does not reply either due
1680 to a down network, hung server, or any other reason, a timeout occurs which
1681 requires a recovery.</para>
1682 <para>If a timeout occurs, a message (similar to this one), appears on the
1683 console of the client, and in
1684 <literal>/var/log/messages</literal>:</para>
1686 LustreError: 26597:(client.c:810:ptlrpc_expire_one_request()) @@@ timeout
1688 req@a2d45200 x5886/t0 o38->mds_svc_UUID@NID_mds_UUID:12 lens 168/64 ref 1 fl