1 <?xml version='1.0' encoding='UTF-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="managingstripingfreespace">
5 <title xml:id="managingstripingfreespace.title">Managing File Layout (Striping) and Free
7 <para>This chapter describes file layout (striping) and I/O options, and includes the following
11 <para><xref linkend="file_striping.how_it_works"/></para>
14 <para><xref linkend="file_striping.considerations"/></para>
17 <para><xref linkend="file_striping.lfs_setstripe"/></para>
20 <para><xref linkend="file_striping.lfs_getstripe"/></para>
23 <para><xref linkend="file_striping.managing_free_space"/></para>
26 <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/></para>
29 <section xml:id="file_striping.how_it_works">
32 <primary>space</primary>
35 <primary>striping</primary>
36 <secondary>how it works</secondary>
39 <primary>striping</primary>
43 <primary>space</primary>
44 <secondary>striping</secondary>
45 </indexterm>How Lustre File System Striping Works</title>
46 <para>In a Lustre file system, the MDS allocates objects to OSTs using either a round-robin
47 algorithm or a weighted algorithm. When the amount of free space is well balanced (i.e., by
48 default, when the free space across OSTs differs by less than 17%), the round-robin algorithm
49 is used to select the next OST to which a stripe is to be written. Periodically, the MDS
50 adjusts the striping layout to eliminate some degenerated cases in which applications that
51 create very regular file layouts (striping patterns) preferentially use a particular OST in
53 <para> Normally the usage of OSTs is well balanced. However, if users create a small number of
54 exceptionally large files or incorrectly specify striping parameters, imbalanced OST usage may
55 result. When the free space across OSTs differs by more than a specific amount (17% by
56 default), the MDS then uses weighted random allocations with a preference for allocating
57 objects on OSTs with more free space. (This can reduce I/O performance until space usage is
58 rebalanced again.) For a more detailed description of how striping is allocated, see <xref
59 linkend="file_striping.managing_free_space"/>.</para>
60 <para>Files can only be striped over a finite number of OSTs, based on the
61 maximum size of the attributes that can be stored on the MDT. If the MDT
62 is ldiskfs-based without the <literal>ea_inode</literal> feature, a file
63 can be striped across at most 160 OSTs. With a ZFS-based MDT, or if the
64 <literal>ea_inode</literal> feature is enabled for an ldiskfs-based MDT
65 (the default since Lustre 2.13.0),
66 a file can be striped across up to 2000 OSTs. For more information, see
67 <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/>.
70 <section xml:id="file_striping.considerations">
72 <primary>file layout</primary>
73 <secondary>See striping</secondary>
74 </indexterm><indexterm>
75 <primary>striping</primary>
76 <secondary>considerations</secondary>
79 <primary>space</primary>
80 <secondary>considerations</secondary>
81 </indexterm> Lustre File Layout (Striping) Considerations</title>
82 <para>Whether you should set up file striping and what parameter values you select depends on
83 your needs. A good rule of thumb is to stripe over as few objects as will meet those needs and
85 <para>Some reasons for using striping include:</para>
88 <para><emphasis role="bold">Providing high-bandwidth access.</emphasis> Many applications
89 require high-bandwidth access to a single file, which may be more bandwidth than can be
90 provided by a single OSS. Examples are a scientific application that writes to a single
91 file from hundreds of nodes, or a binary executable that is loaded by many nodes when an
92 application starts.</para>
93 <para>In cases like these, a file can be striped over as many OSSs as it takes to achieve
94 the required peak aggregate bandwidth for that file. Striping across a larger number of
95 OSSs should only be used when the file size is very large and/or is accessed by many nodes
96 at a time. Currently, Lustre files can be striped across up to 2000 OSTs</para>
99 <para><emphasis role="bold">Improving performance when OSS bandwidth is exceeded.</emphasis>
100 Striping across many OSSs can improve performance if the aggregate client bandwidth
101 exceeds the server bandwidth and the application reads and writes data fast enough to take
102 advantage of the additional OSS bandwidth. The largest useful stripe count is bounded by
103 the I/O rate of the clients/jobs divided by the performance per OSS.</para>
106 <para condition="l2D"><emphasis role="bold">Matching stripes to I/O
107 pattern.</emphasis>When writing to a single file from multiple nodes,
108 having more than one client writing to a stripe can lead to issues
109 with lock exchange, where clients contend over writing to that stripe,
110 even if their I/Os do not overlap. This can be avoided if I/O can be
111 stripe aligned so that each stripe is accessed by only one client.
112 Since Lustre 2.13, the 'overstriping' feature is available, allowing more
113 than one stripe per OST. This is particularly helpful for the case where
114 thread count exceeds OST count, making it possible to match stripe count
115 to thread count even in this case.</para>
118 <para><emphasis role="bold">Providing space for very large files.</emphasis> Striping is
119 useful when a single OST does not have enough free space to hold the entire file.</para>
122 <para>Some reasons to minimize or avoid striping:</para>
125 <para><emphasis role="bold">Increased overhead.</emphasis> Striping results in more locks
126 and extra network operations during common operations such as <literal>stat</literal> and
127 <literal>unlink</literal>. Even when these operations are performed in parallel, one
128 network operation takes less time than 100 operations.</para>
129 <para>Increased overhead also results from server contention. Consider a cluster with 100
130 clients and 100 OSSs, each with one OST. If each file has exactly one object and the load
131 is distributed evenly, there is no contention and the disks on each server can manage
132 sequential I/O. If each file has 100 objects, then the clients all compete with one
133 another for the attention of the servers, and the disks on each node seek in 100 different
134 directions resulting in needless contention.</para>
137 <para><emphasis role="bold">Increased risk.</emphasis> When files are striped across all
138 servers and one of the servers breaks down, a small part of each striped file is lost. By
139 comparison, if each file has exactly one stripe, fewer files are lost, but they are lost
140 in their entirety. Many users would prefer to lose some of their files entirely than all
141 of their files partially.</para>
144 <para><emphasis role="bold">Small files.</emphasis> Small files do not benefit from striping
145 because they can be efficiently stored and accessed as a single OST object or even with
149 <para><emphasis role="bold">O_APPEND mode.</emphasis> When files are opened for append, they
150 instantiate all uninitialized components expressed in the layout. Typically, log files are
151 opened for append, and complex layouts can be inefficient.</para>
153 <para>The <literal>mdd.*.append_stripe_count</literal> and <literal>mdd.*.append_pool
154 </literal> options can be used to specify special default striping for files created
155 with <literal> O_APPEND</literal>.</para>
160 <title><indexterm><primary>striping</primary><secondary>size</secondary></indexterm>
161 Choosing a Stripe Size</title>
162 <para>Choosing a stripe size is a balancing act, but reasonable defaults are described below.
163 The stripe size has no effect on a single-stripe file.</para>
166 <para><emphasis role="bold">The stripe size must be a multiple of the page
167 size.</emphasis> Lustre software tools enforce a multiple of 64 KB (the maximum page
168 size on ia64 and PPC64 nodes) so that users on platforms with smaller pages do not
169 accidentally create files that might cause problems for ia64 clients.</para>
172 <para><emphasis role="bold">The smallest recommended stripe size is 512 KB.</emphasis>
173 Although you can create files with a stripe size of 64 KB, the smallest practical stripe
174 size is 512 KB because the Lustre file system sends 1MB chunks over the network.
175 Choosing a smaller stripe size may result in inefficient I/O to the disks and reduced
179 <para><emphasis role="bold">A good stripe size for sequential I/O using high-speed
180 networks is between 1 MB and 4 MB.</emphasis> In most situations, stripe sizes larger
181 than 4 MB may result in longer lock hold times and contention during shared file
185 <para><emphasis role="bold">The maximum stripe size is 4 GB.</emphasis> Using a large
186 stripe size can improve performance when accessing very large files. It allows each
187 client to have exclusive access to its own part of a file. However, a large stripe size
188 can be counterproductive in cases where it does not match your I/O pattern.</para>
191 <para><emphasis role="bold">Choose a stripe pattern that takes into account the write
192 patterns of your application.</emphasis> Writes that cross an object boundary are
193 slightly less efficient than writes that go entirely to one server. If the file is
194 written in a consistent and aligned way, make the stripe size a multiple of the
195 <literal>write()</literal> size.</para>
200 <section xml:id="file_striping.lfs_setstripe">
202 <primary>striping</primary>
203 <secondary>configuration</secondary>
204 </indexterm>Setting the File Layout/Striping Configuration (<literal>lfs
205 setstripe</literal>)</title>
206 <para>Use the <literal>lfs setstripe</literal> command to create new files with a specific file layout (stripe pattern) configuration.</para>
207 <screen>lfs setstripe [--size|-s stripe_size] [--stripe-count|-c stripe_count] [--overstripe-count|-C stripe_count] \
208 [--index|-i start_ost] [--pool|-p pool_name] <replaceable>filename|dirname</replaceable> </screen>
209 <para><emphasis role="bold">
210 <literal>stripe_size</literal>
213 <para>The <literal>stripe_size</literal> indicates how much data to write to one OST before
214 moving to the next OST. The default <literal>stripe_size</literal> is 1 MB. Passing a
215 <literal>stripe_size</literal> of 0 causes the default stripe size to be used. Otherwise,
216 the <literal>stripe_size</literal> value must be a multiple of 64 KB.</para>
217 <para><emphasis role="bold">
218 <literal>stripe_count (--stripe-count, --overstripe-count)</literal>
221 <para>The <literal>stripe_count</literal> indicates how many stripes to use.
222 The default <literal>stripe_count</literal> value is 1. Setting
223 <literal>stripe_count</literal> to 0 causes the default stripe count to be
224 used. Setting <literal>stripe_count</literal> to -1 means stripe over all
225 available OSTs (full OSTs are skipped). When --overstripe-count is used,
226 per OST if necessary.</para>
227 <para><emphasis role="bold">
228 <literal>start_ost</literal>
231 <para>The start OST is the first OST to which files are written. The default value for
232 <literal>start_ost</literal> is -1, which allows the MDS to choose the starting index. This
233 setting is strongly recommended, as it allows space and load balancing to be done by the MDS
234 as needed. If the value of <literal>start_ost</literal> is set to a value other than -1, the
235 file starts on the specified OST index. OST index numbering starts at 0.</para>
237 <para>If the specified OST is inactive or in a degraded mode, the MDS will silently choose
238 another target.</para>
241 <para>If you pass a <literal>start_ost</literal> value of 0 and a
242 <literal>stripe_count</literal> value of <emphasis>1</emphasis>, all files are written to
243 OST 0, until space is exhausted. <emphasis role="italic">This is probably not what you meant
244 to do.</emphasis> If you only want to adjust the stripe count and keep the other
245 parameters at their default settings, do not specify any of the other parameters:</para>
246 <para><screen>client# lfs setstripe -c <replaceable>stripe_count</replaceable> <replaceable>filename</replaceable></screen></para>
248 <para><emphasis role="bold">
249 <literal>pool_name</literal>
252 <para>The <literal>pool_name</literal> specifies the OST pool to which the
253 file will be written. This allows limiting the OSTs used to a subset of
254 all OSTs in the file system. For more details about using OST pools, see
255 <xref linkend="managingfilesystemio.managing_ost_pools" />.</para>
257 <title>Specifying a File Layout (Striping Pattern) for a Single File</title>
258 <para>It is possible to specify the file layout when a new file is created using the command <literal>lfs setstripe</literal>. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an <literal>lfs setstripe</literal> command fails if the file already exists.</para>
259 <section xml:id="file_striping.stripe_size">
260 <title>Setting the Stripe Size</title>
261 <para>The command to create a new file with a specified stripe size is similar to:</para>
262 <screen>[client]# lfs setstripe -s 4M /mnt/lustre/new_file</screen>
263 <para>This example command creates the new file <literal>/mnt/lustre/new_file</literal> with a stripe size of 4 MB.</para>
264 <para>Now, when the file is created, the new stripe setting creates the file on a single OST with a stripe size of 4M:</para>
265 <screen> [client]# lfs getstripe /mnt/lustre/new_file
268 lmm_stripe_size: 4194304
272 obdidx objid objid group
273 1 690550 0xa8976 0 </screen>
274 <para>In this example, the stripe size is 4 MB.</para>
277 <title><indexterm><primary>striping</primary><secondary>count</secondary></indexterm>
278 Setting the Stripe Count</title>
279 <para>The command below creates a new file with a stripe count of <literal>-1</literal> to
280 specify striping over all available OSTs:</para>
281 <screen>[client]# lfs setstripe -c -1 /mnt/lustre/full_stripe</screen>
282 <para>The example below indicates that the file
283 <literal>full_stripe</literal> is striped
284 over all six active OSTs in the configuration:</para>
285 <screen>[client]# lfs getstripe /mnt/lustre/full_stripe
286 /mnt/lustre/full_stripe
287 obdidx objid objid group
294 <para> This is in contrast to the output in
295 <xref linkend="file_striping.stripe_size"/>,
296 which shows only a single object for the file.</para>
301 <primary>striping</primary>
302 <secondary>per directory</secondary>
303 </indexterm>Setting the Striping Layout for a Directory</title>
304 <para>In a directory, the <literal>lfs setstripe</literal> command sets a default striping
305 configuration for files created in the directory. The usage is the same as <literal>lfs
306 setstripe</literal> for a regular file, except that the directory must exist prior to
307 setting the default striping configuration. If a file is created in a directory with a
308 default stripe configuration (without otherwise specifying striping), the Lustre file system
309 uses those striping parameters instead of the file system default for the new file.</para>
310 <para>To change the striping pattern for a sub-directory, create a directory with desired file
311 layout as described above. Sub-directories inherit the file layout of the root/parent
314 <para>Special default striping can be used for files created with <literal>O_APPEND</literal>.
315 Files with uninitialized layouts opened with <literal>O_APPEND</literal> will
316 override a directory's default striping configuration and abide by the <literal>
317 mdd.*.append_pool</literal> and <literal>mdd.*.append_stripe_count</literal> options (if
318 they are specified).</para>
323 <primary>striping</primary>
324 <secondary>per file system</secondary>
325 </indexterm>Setting the Striping Layout for a File System</title>
326 <para>Setting the striping specification on the <literal>root</literal> directory determines
327 the striping for all new files created in the file system unless an overriding striping
328 specification takes precedence (such as a striping layout specified by the application, or
329 set using <literal>lfs setstripe</literal>, or specified for the parent directory).</para>
331 <para>The striping settings for a <literal>root</literal> directory are, by default, applied
332 to any new child directories created in the root directory, unless striping settings have
333 been specified for the child directory.</para>
336 <para>Special default striping can be used for files created with <literal>O_APPEND</literal>.
337 Files with uninitialized layouts opened with <literal>O_APPEND</literal> will
338 override a file system's default striping configuration and abide by the <literal>
339 mdd.*.append_pool</literal> and <literal>mdd.*.append_stripe_count</literal> options (if
340 they are specified).</para>
345 <primary>striping</primary>
346 <secondary>stripe count limit</secondary>
347 </indexterm>Per File System Stripe Count Limit</title>
348 <para>Sometime there are many OSTs in a filesystem, but it is not always
349 desirable to stripe file to across all OSTs, even if the given
350 <literal>stripe_count=-1</literal> (unlimited).
351 In this case, the per-filesystem tunable parameter
352 <literal>lod.*.max_stripecount</literal> can be used to limit the real
353 stripe count of file to a lower number than the OST count.
354 If <literal>lod.*.max_stripecount</literal> is not 0, and the file
355 <literal>stripe_count=-1</literal>, the real stripe count will be
356 the minimum of the OST count and <literal>max_stripecount</literal>. If
357 <literal>lod.*.max_stripecount=0</literal>, or an explicit stripe count
358 is given for the file, it is ignored.</para>
359 <para>To set <literal>max_stripecount</literal>, on all MDSes of
362 mgs# lctl set_param -P lod.$fsname-MDTxxxx-mdtlov.max_stripecount=<N>
365 <para>To check <literal>max_stripecount</literal>, run:
367 mds# lctl get_param lod.$fsname-MDTxxxx-mdtlov.max_stripecount
370 <para>To reset <literal>max_stripecount</literal>, run:
372 mgs# lctl set_param -P -d lod.$fsname-MDTxxxx-mdtlov.max_stripecount
379 <primary>striping</primary>
380 <secondary>on specific OST</secondary>
381 </indexterm>Creating a File on a Specific OST</title>
382 <para>You can use <literal>lfs setstripe</literal> to create a file on a specific OST. In the
383 following example, the file <literal>file1</literal> is created on the first OST (OST index
385 <screen>$ lfs setstripe --stripe-count 1 --index 0 file1
386 $ dd if=/dev/zero of=file1 count=1 bs=100M
390 $ lfs getstripe file1
393 lmm_stripe_size: 1048576
397 obdidx objid objid group
398 0 37364 0x91f4 0</screen>
401 <section xml:id="file_striping.lfs_getstripe">
402 <title><indexterm><primary>striping</primary><secondary>getting information</secondary></indexterm>Retrieving File Layout/Striping Information (<literal>getstripe</literal>)</title>
403 <para>The <literal>lfs getstripe</literal> command is used to display information that shows
404 over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along
405 with the OST index and object ID for each stripe in the file. For directories, the default
406 settings for files created in that directory are displayed.</para>
408 <title>Displaying the Current Stripe Size</title>
409 <para>To see the current stripe size for a Lustre file or directory, use the <literal>lfs
410 getstripe</literal> command. For example, to view information for a directory, enter a
411 command similar to:</para>
412 <screen>[client]# lfs getstripe /mnt/lustre </screen>
413 <para>This command produces output similar to:</para>
415 (Default) stripe_count: 1 stripe_size: 1M stripe_offset: -1</screen>
416 <para>In this example, the default stripe count is <literal>1</literal> (data blocks are
417 striped over a single OST), the default stripe size is 1 MB, and the objects are created
418 over all available OSTs.</para>
419 <para>To view information for a file, enter a command similar to:</para>
420 <screen>$ lfs getstripe /mnt/lustre/foo
423 lmm_stripe_size: 1048576
427 obdidx objid objid group
428 2 835487 m0xcbf9f 0 </screen>
429 <para>In this example, the file is located on <literal>obdidx 2</literal>, which corresponds
430 to the OST <literal>lustre-OST0002</literal>. To see which node is serving that OST, run:
431 <screen>$ lctl get_param osc.lustre-OST0002-osc.ost_conn_uuid
432 osc.lustre-OST0002-osc.ost_conn_uuid=192.168.20.1@tcp</screen></para>
435 <title>Inspecting the File Tree</title>
436 <para>To inspect an entire tree of files, use the <literal>lfs find</literal> command:</para>
437 <screen>lfs find [--recursive | -r] <replaceable>file|directory</replaceable> ...</screen>
441 <primary>striping</primary>
442 <secondary>remote directories</secondary>
443 </indexterm>Locating the MDT for a remote directory</title>
444 <para>Lustre can be configured with multiple MDTs in the same file
445 system. Each directory and file could be located on a different MDT.
446 To identify which MDT a given subdirectory is located, pass the
447 <literal>getstripe [--mdt-index|-M]</literal> parameter to
448 <literal>lfs</literal>. An example of this command is provided in
449 the section <xref linkend="lustremaint.rmremotedir"/>.</para>
452 <section xml:id="pfl" condition='l2A'>
454 <primary>striping</primary>
455 <secondary>PFL</secondary>
456 </indexterm>Progressive File Layout(PFL)</title>
457 <para>The Lustre Progressive File Layout (PFL) feature simplifies the use
458 of Lustre so that users can expect reasonable performance for a variety of
459 normal file IO patterns without the need to explicitly understand their IO
460 model or Lustre usage details in advance. In particular, users do not
461 necessarily need to know the size or concurrency of output files in
462 advance of their creation and explicitly specify an optimal layout for
463 each file in order to achieve good performance for both highly concurrent
464 shared-single-large-file IO or parallel IO to many smaller per-process
466 <para>The layout of a PFL file is stored on disk as <literal>composite
467 layout</literal>. A PFL file is essentially an array of
468 <literal>sub-layout components</literal>, with each sub-layout component
469 being a plain layout covering different and non-overlapped extents of
470 the file. For PFL files, the file layout is composed of a series of
471 components, therefore it's possible that there are some file extents are
472 not described by any components.</para>
473 <para>An example of how data blocks of PFL files are mapped to OST objects
474 of components is shown in the following PFL object mapping diagram:</para>
475 <figure xml:id="managinglayout.fig.pfl">
476 <title>PFL object mapping diagram</title>
479 <imagedata scalefit="1" width="100%"
480 fileref="figures/PFL_object_mapping_diagram.png" />
483 <phrase>PFL object mapping diagram</phrase>
487 <para>The PFL file in <xref linkend="managinglayout.fig.pfl"/> has 3
488 components and shows the mapping for the blocks of a 2055MB file.
489 The stripe size for the first two components is 1MB, while the stripe size
490 for the third component is 4MB. The stripe count is increasing for each
491 successive component. The first component only has two 1MB blocks and the
492 single object has a size of 2MB. The second component holds the next 254MB
493 of the file spread over 4 separate OST objects in RAID-0, each one will
494 have a size of 256MB / 4 objects = 64MB per object. Note the first two
495 objects <literal>obj 2,0</literal> and <literal>obj 2,1</literal>
496 have a 1MB hole at the start where the data is stored in the first
497 component. The final component holds the next 1800MB spread over 32 OST
498 objects. There is a 256MB / 32 = 8MB hole at the start each one for the
499 data stored in the first two components. Each object will be
500 2048MB / 32 objects = 64MB per object, except the
501 <literal>obj 3,0</literal> that holds an extra 4MB chunk and
502 <literal>obj 3,1</literal> that holds an extra 3MB chunk. If more data
503 was written to the file, only the objects in component 3 would increase
505 <para>When a file range with defined but not instantiated component is
506 accessed, clients will send a Layout Intent RPC to the MDT, and the MDT
507 would instantiate the objects of the components covering that range.
509 <para>Next, some commands for user to operate PFL files are introduced and
510 some examples of possible composite layout are illustrated as well.
511 Lustre provides commands
512 <literal>lfs setstripe</literal> and <literal>lfs migrate</literal> for
513 users to operate PFL files. <literal>lfs setstripe</literal> commands
514 are used to create PFL files, add or delete components to or from an
515 existing composite file; <literal>lfs migrate</literal> commands are used
516 to re-layout the data in existing files using the new layout parameter by
517 copying the data from the existing OST(s) to the new OST(s). Also,
518 as introduced in the previous sections, <literal>lfs getstripe</literal>
519 commands can be used to list the striping/component information for a
520 given PFL file, and <literal>lfs find</literal> commands can be used to
521 search the directory tree rooted at the given directory or file name for
522 the files that match the given PFL component parameters.</para>
523 <note><para>Using PFL files requires both the client and server to
524 understand the PFL file layout, which isn't available for Lustre 2.9 and
525 earlier. And it will not prevent older clients from accessing non-PFL
526 files in the filesystem.</para></note>
528 <title><literal>lfs setstripe</literal></title>
529 <para><literal>lfs setstripe</literal> commands are used to create PFL
530 files, add or delete components to or from an existing composite file.
531 (Suppose we have 8 OSTs in the following examples and stripe size is 1MB
534 <title>Create a PFL file</title>
535 <para><emphasis role="bold">Command</emphasis></para>
536 <screen>lfs setstripe
537 [--component-end|-E end1] [STRIPE_OPTIONS]
538 [--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable></screen>
539 <para>The <literal>-E</literal> option is used to specify the end offset
540 (in bytes or using a suffix “kMGTP”, e.g. 256M) of each component, and
541 it also indicates the following <literal>STRIPE_OPTIONS</literal> are
542 for this component. Each component defines the stripe pattern of the
543 file in the range of [start, end). The first component must start from
544 offset 0 and all components must be adjacent with each other, no holes
545 are allowed, so each extent will start at the end of previous extent.
546 A <literal>-1</literal> end offset or <literal>eof</literal> indicates
547 this is the last component extending to the end of file.</para>
548 <para><emphasis role="bold">Example</emphasis></para>
549 <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 -E -1 -c -1 -i 4 \
550 /mnt/testfs/create_comp</screen>
551 <para>This command creates a file with composite layout illustrated in
552 the following figure. The first component has 1 stripe and covers
553 [0, 4M), the second component has 4 stripes and covers [4M, 64M), and
554 the last component stripes start at OST4, cross over all available
555 OSTs and covers [64M, EOF).</para>
556 <figure xml:id="managinglayout.fig.pfl_create">
557 <title>Example: create a composite file</title>
560 <imagedata scalefit="1" depth="2.75in" align="center"
561 fileref="figures/PFL_createfile.png" />
564 <phrase>Example: create a composite file</phrase>
568 <para>The composite layout can be output by the following command:</para>
569 <screen>$ lfs getstripe /mnt/testfs/create_comp
570 /mnt/testfs/create_comp
575 lcme_extent.e_start: 0
576 lcme_extent.e_end: 4194304
578 lmm_stripe_size: 1048576
583 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
587 lcme_extent.e_start: 4194304
588 lcme_extent.e_end: 67108864
590 lmm_stripe_size: 1048576
593 lmm_stripe_offset: -1
596 lcme_extent.e_start: 67108864
597 lcme_extent.e_end: EOF
599 lmm_stripe_size: 1048576
602 lmm_stripe_offset: 4</screen>
603 <note><para>Only the first component’s OST objects of the PFL file are
604 instantiated when the layout is being set. Other instantiation is
605 delayed to later write/truncate operations.</para></note>
606 <para>If we write 128M data to this PFL file, the second and third
607 components will be instantiated:</para>
608 <screen>$ dd if=/dev/zero of=/mnt/testfs/create_comp bs=1M count=128
609 $ lfs getstripe /mnt/testfs/create_comp
610 /mnt/testfs/create_comp
615 lcme_extent.e_start: 0
616 lcme_extent.e_end: 4194304
618 lmm_stripe_size: 1048576
623 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
627 lcme_extent.e_start: 4194304
628 lcme_extent.e_end: 67108864
630 lmm_stripe_size: 1048576
635 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
636 - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
637 - 2: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }
638 - 3: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
642 lcme_extent.e_start: 67108864
643 lcme_extent.e_end: EOF
645 lmm_stripe_size: 1048576
650 - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x3:0x0] }
651 - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
652 - 2: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
653 - 3: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
654 - 4: { l_ost_idx: 0, l_fid: [0x100000000:0x3:0x0] }
655 - 5: { l_ost_idx: 1, l_fid: [0x100010000:0x3:0x0] }
656 - 6: { l_ost_idx: 2, l_fid: [0x100020000:0x3:0x0] }
657 - 7: { l_ost_idx: 3, l_fid: [0x100030000:0x3:0x0] }</screen>
660 <title>Add component(s) to an existing composite file</title>
661 <para><emphasis role="bold">Command</emphasis></para>
662 <screen>lfs setstripe --component-add
663 [--component-end|-E end1] [STRIPE_OPTIONS]
664 [--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable></screen>
665 <para>The option <literal>--component-add</literal> is used to add
666 components to an existing composite file. The extent start of
667 the first component to be added is equal to the extent end of last
668 component in the existing file, and all components to be added must
669 be adjacent with each other.</para>
670 <note><para>If the last existing component is specified by
671 <literal>-E -1</literal> or <literal>-E eof</literal>, which covers
672 to the end of the file, it must be deleted before a new one is added.
674 <para><emphasis role="bold">Example</emphasis></para>
675 <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 /mnt/testfs/add_comp
676 $ lfs setstripe --component-add -E -1 -c 4 -o 6-7,0,5 \
677 /mnt/testfs/add_comp</screen>
678 <para>This command adds a new component which starts from the end of
679 the last existing component to the end of file. The layout of this
680 example is illustrated in
681 <xref linkend="managinglayout.fig.pfl_addcomp"/>. The last component
682 stripes across 4 OSTs in sequence OST6, OST7, OST0 and OST5, covers
684 <figure xml:id="managinglayout.fig.pfl_addcomp">
685 <title>Example: add a component to an existing composite file</title>
688 <imagedata scalefit="1" depth="2.75in" align="center"
689 fileref="figures/PFL_addcomp.png" />
692 <phrase>Example: add a component to an existing composite file
697 <para>The layout can be printed out by the following command:</para>
698 <screen>$ lfs getstripe /mnt/testfs/add_comp
704 lcme_extent.e_start: 0
705 lcme_extent.e_end: 4194304
707 lmm_stripe_size: 1048576
712 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
716 lcme_extent.e_start: 4194304
717 lcme_extent.e_end: 67108864
719 lmm_stripe_size: 1048576
724 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
725 - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
726 - 2: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }
727 - 3: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
731 lcme_extent.e_start: 67108864
732 lcme_extent.e_end: EOF
734 lmm_stripe_size: 1048576
737 lmm_stripe_offset: -1</screen>
738 <para>The component ID "lcme_id" changes as layout generation
739 changes. It is not necessarily sequential and does not imply ordering
740 of individual components.</para>
741 <note><para>Similar to specifying a full-file composite layout at file
742 creation time, <literal>--component-add</literal> won't instantiate
743 OST objects, the instantiation is delayed to later write/truncate
744 operations. For example, after writing beyond the 64MB start of the
745 file's last component, the new component has had objects allocated:
747 <screen>$ lfs getstripe -I5 /mnt/testfs/add_comp
753 lcme_extent.e_start: 67108864
754 lcme_extent.e_end: EOF
756 lmm_stripe_size: 1048576
761 - 0: { l_ost_idx: 6, l_fid: [0x100060000:0x4:0x0] }
762 - 1: { l_ost_idx: 7, l_fid: [0x100070000:0x4:0x0] }
763 - 2: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
764 - 3: { l_ost_idx: 5, l_fid: [0x100050000:0x4:0x0] }</screen>
767 <title>Delete component(s) from an existing file</title>
768 <para><emphasis role="bold">Command</emphasis></para>
769 <screen>lfs setstripe --component-del
770 [--component-id|-I comp_id | --component-flags comp_flags]
771 <replaceable>filename</replaceable></screen>
772 <para>The option <literal>--component-del</literal> is used to remove
773 the component(s) specified by component ID or flags from an existing
774 file. Any data stored in the deleted component will be lost after
775 this operation.</para>
776 <para>The ID specified by <literal>-I</literal> option is the numerical
777 unique ID of the component, which can be obtained by command
778 <literal>lfs getstripe -I</literal> command, and the flag specified by
779 <literal>--component-flags</literal> option is a certain type of
780 components, which can be obtained by command
781 <literal>lfs getstripe --component-flags</literal>. For now, we only
782 have two flags <literal>init</literal> and <literal>^init</literal>
783 for instantiated and un-instantiated components respectively.</para>
784 <note><para>Deletion must start with the last component because creation
785 of a hole in the middle of a file layout is not allowed.</para></note>
786 <para><emphasis role="bold">Example</emphasis></para>
787 <screen>$ lfs getstripe -I /mnt/testfs/del_comp
791 $ lfs setstripe --component-del -I 5 /mnt/testfs/del_comp</screen>
792 <para>This example deletes the component with ID 5 from file
793 <literal>/mnt/testfs/del_comp</literal>. If we still use the last
794 example, the final result is illustrated in
795 <xref linkend="managinglayout.fig.pfl_delcomp"/>.</para>
796 <figure xml:id="managinglayout.fig.pfl_delcomp">
797 <title>Example: delete a component from an existing file</title>
800 <imagedata scalefit="1" depth="2.75in" align="center"
801 fileref="figures/PFL_delcomp.png" />
804 <phrase>Example: delete a component from an existing file</phrase>
808 <para>If you try to delete a non-last component, you will see the
809 following error:</para>
810 <screen>$ lfs setstripe -component-del -I 2 /mnt/testfs/del_comp
811 Delete component 0x2 from /mnt/testfs/del_comp failed. Invalid argument
812 error: setstripe: delete component of file '/mnt/testfs/del_comp' failed: Invalid argument</screen>
815 <title>Set default PFL layout to an existing directory</title>
816 <para>Similar to create a PFL file, you can set default PFL layout to
817 an existing directory. After that, all the files created will inherit
818 this layout by default.</para>
819 <para><emphasis role="bold">Command</emphasis></para>
820 <screen>lfs setstripe
821 [--component-end|-E end1] [STRIPE_OPTIONS]
822 [--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>dirname</replaceable></screen>
823 <para><emphasis role="bold">Example</emphasis></para>
825 $ mkdir /mnt/testfs/pfldir
826 $ lfs setstripe -E 256M -c 1 -E 16G -c 4 -E -1 -S 4M -c -1 /mnt/testfs/pfldir
828 <para>When you run <literal>lfs getstripe</literal>, you will see:
831 $ lfs getstripe /mnt/testfs/pfldir
837 lcme_extent.e_start: 0
838 lcme_extent.e_end: 268435456
839 stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
842 lcme_extent.e_start: 268435456
843 lcme_extent.e_end: 17179869184
844 stripe_count: 4 stripe_size: 1048576 stripe_offset: -1
847 lcme_extent.e_start: 17179869184
848 lcme_extent.e_end: EOF
849 stripe_count: -1 stripe_size: 4194304 stripe_offset: -1
851 <para>If you create a file under <literal>/mnt/testfs/pfldir</literal>,
852 the layout of that file will inherit the layout from its parent
855 $ touch /mnt/testfs/pfldir/pflfile
856 $ lfs getstripe /mnt/testfs/pfldir/pflfile
857 /mnt/testfs/pfldir/pflfile
862 lcme_extent.e_start: 0
863 lcme_extent.e_end: 268435456
865 lmm_stripe_size: 1048576
870 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0xa:0x0] }
874 lcme_extent.e_start: 268435456
875 lcme_extent.e_end: 17179869184
877 lmm_stripe_size: 1048576
880 lmm_stripe_offset: -1
884 lcme_extent.e_start: 17179869184
885 lcme_extent.e_end: EOF
887 lmm_stripe_size: 4194304
890 lmm_stripe_offset: -1
893 <literal>lfs setstripe --component-add/del</literal> can't be run
894 on a directory, because the default layout in directory is like a config,
895 which can be arbitrarily changed by <literal>lfs setstripe</literal>,
896 while the layout of a file may have data (OST objects) attached.
897 If you want to delete the default layout in a directory, run
898 <literal>lfs setstripe -d <replaceable>dirname</replaceable></literal>
899 to return the directory to the filesystem-wide defaults, like:
901 $ lfs setstripe -d /mnt/testfs/pfldir
902 $ lfs getstripe -d /mnt/testfs/pfldir
904 stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
905 /mnt/testfs/pfldir/commonfile
907 lmm_stripe_size: 1048576
911 obdidx objid objid group
918 <title><literal>lfs migrate</literal></title>
919 <para><literal>lfs migrate</literal> commands are used to re-layout the
920 data in the existing files with the new layout parameter by copying the
921 data from the existing OST(s) to the new OST(s).</para>
922 <para><emphasis role="bold">Command</emphasis></para>
923 <screen>lfs migrate [--component-end|-E comp_end] [STRIPE_OPTIONS] ...
924 <replaceable>filename</replaceable></screen>
925 <para>The difference between <literal>migrate</literal> and
926 <literal>setstripe</literal> is that <literal>migrate</literal> is to
927 re-layout the data in the existing files, while
928 <literal>setstripe</literal> is to create new files with the specified
930 <para><emphasis role="bold">Example</emphasis></para>
931 <para><emphasis role="bold">Case1. Migrate a normal one to a composite
932 layout</emphasis></para>
933 <screen>$ lfs setstripe -c 1 -S 128K /mnt/testfs/norm_to_2comp
934 $ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
935 $ lfs getstripe /mnt/testfs/norm_to_2comp --yaml
936 /mnt/testfs/norm_to_comp
938 lmm_stripe_size: 131072
944 l_fid: 0x100070000:0x2:0x0
945 $ lfs migrate -E 1M -S 512K -c 1 -E -1 -S 1M -c 2 \
946 /mnt/testfs/norm_to_2comp</screen>
947 <para>In this example, a 5MB size file with 1 stripe and 128K stripe size
948 is migrated to a composite layout file with 2 components, illustrated in
949 <xref linkend="managinglayout.fig.pfl_norm_to_comp"/>.</para>
950 <figure xml:id="managinglayout.fig.pfl_norm_to_comp">
951 <title>Example: migrate normal to composite</title>
954 <imagedata scalefit="1" depth="2.75in" align="center"
955 fileref="figures/PFL_norm_to_comp.png" />
958 <phrase>Example: migrate normal to composite</phrase>
962 <para>The stripe information after migration is like:</para>
963 <screen>$ lfs getstripe /mnt/testfs/norm_to_2comp
964 /mnt/testfs/norm_to_2comp
969 lcme_extent.e_start: 0
970 lcme_extent.e_end: 1048576
972 lmm_stripe_size: 524288
977 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
981 lcme_extent.e_start: 1048576
982 lcme_extent.e_end: EOF
984 lmm_stripe_size: 1048576
989 - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
990 - 1: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }</screen>
991 <para><emphasis role="bold">Case2. Migrate a composite layout to another
992 composite layout</emphasis></para>
993 <screen>$ lfs setstripe -E 1M -S 512K -c 1 -E -1 -S 1M -c 2 \
994 /mnt/testfs/2comp_to_3comp
995 $ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
996 $ lfs migrate -E 1M -S 1M -c 2 -E 4M -S 1M -c 2 -E -1 -S 3M -c 3 \
997 /mnt/testfs/2comp_to_3comp</screen>
998 <para>In this example, a composite layout file with 2 components is
999 migrated a composite layout file with 3 components. If we still use
1000 the example in case1, the migration process is illustrated in
1001 <xref linkend="managinglayout.fig.pfl_comp_to_comp"/>.</para>
1002 <figure xml:id="managinglayout.fig.pfl_comp_to_comp">
1003 <title>Example: migrate composite to composite</title>
1006 <imagedata scalefit="1" depth="2.75in" align="center"
1007 fileref="figures/PFL_comp_to_comp.png" />
1010 <phrase>Example: migrate composite to composite</phrase>
1014 <para>The stripe information is like:</para>
1015 <screen>$ lfs getstripe /mnt/testfs/2comp_to_3comp
1016 /mnt/testfs/2comp_to_3comp
1021 lcme_extent.e_start: 0
1022 lcme_extent.e_end: 1048576
1024 lmm_stripe_size: 1048576
1027 lmm_stripe_offset: 4
1029 - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
1030 - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
1034 lcme_extent.e_start: 1048576
1035 lcme_extent.e_end: 4194304
1037 lmm_stripe_size: 1048576
1040 lmm_stripe_offset: 6
1042 - 0: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
1043 - 1: { l_ost_idx: 7, l_fid: [0x100070000:0x3:0x0] }
1047 lcme_extent.e_start: 4194304
1048 lcme_extent.e_end: EOF
1050 lmm_stripe_size: 3145728
1053 lmm_stripe_offset: 0
1055 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x3:0x0] }
1056 - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
1057 - 2: { l_ost_idx: 2, l_fid: [0x100020000:0x3:0x0] }</screen>
1058 <para><emphasis role="bold">Case3. Migrate a composite layout to a
1059 normal one</emphasis></para>
1060 <screen>$ lfs migrate -E 1M -S 1M -c 2 -E 4M -S 1M -c 2 -E -1 -S 3M -c 3 \
1061 /mnt/testfs/3comp_to_norm
1062 $ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
1063 $ lfs migrate -c 2 -S 2M /mnt/testfs/3comp_to_normal</screen>
1064 <para>In this example, a composite file with 3 components is migrated to
1065 a normal file with 2 stripes and 2M stripe size. If we still use the
1066 example in Case2, the migration process is illustrated in
1067 <xref linkend="managinglayout.fig.pfl_comp_to_norm"/>.</para>
1068 <figure xml:id="managinglayout.fig.pfl_comp_to_norm">
1069 <title>Example: migrate composite to normal</title>
1072 <imagedata scalefit="1" depth="2.75in" align="center"
1073 fileref="figures/PFL_comp_to_norm.png" />
1076 <phrase>Example: migrate composite to normal</phrase>
1080 <para>The stripe information is like:</para>
1081 <screen>$ lfs getstripe /mnt/testfs/3comp_to_norm --yaml
1082 /mnt/testfs/3comp_to_norm
1084 lmm_stripe_size: 2097152
1087 lmm_stripe_offset: 4
1090 l_fid: 0x100040000:0x3:0x0
1092 l_fid: 0x100050000:0x3:0x0</screen>
1094 <section remap="h3">
1095 <title><literal>lfs getstripe</literal></title>
1096 <para><literal>lfs getstripe</literal> commands can be used to list the
1097 striping/component information for a given PFL file. Here, only those
1098 parameters new for PFL files are shown.</para>
1099 <para><emphasis role="bold">Command</emphasis></para>
1100 <screen>lfs getstripe
1101 [--component-id|-I [comp_id]]
1102 [--component-flags [comp_flags]]
1104 [--component-start [+-][N][kMGTPE]]
1105 [--component-end|-E [+-][N][kMGTPE]]
1106 <replaceable>dirname|filename</replaceable></screen>
1107 <para><emphasis role="bold">Example</emphasis></para>
1108 <para>Suppose we already have a composite file
1109 <literal>/mnt/testfs/3comp</literal>, created by the following
1111 <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 -E -1 -c -1 -i 4 \
1112 /mnt/testfs/3comp</screen>
1113 <para>And write some data</para>
1114 <screen>$ dd if=/dev/zero of=/mnt/testfs/3comp bs=1M count=5</screen>
1115 <para><emphasis role="bold">Case1. List component ID and its related
1116 information</emphasis></para>
1119 <para>List all the components ID</para>
1120 <screen>$ lfs getstripe -I /mnt/testfs/3comp
1126 <para>List the detailed striping information of component ID=2</para>
1127 <screen>$ lfs getstripe -I2 /mnt/testfs/3comp
1133 lcme_extent.e_start: 4194304
1134 lcme_extent.e_end: 67108864
1136 lmm_stripe_size: 1048576
1139 lmm_stripe_offset: 5
1141 - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
1142 - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
1143 - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
1144 - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
1147 <para>List the stripe offset and stripe count of component ID=2</para>
1148 <screen>$ lfs getstripe -I2 -i -c /mnt/testfs/3comp
1150 lmm_stripe_offset: 5</screen>
1153 <para><emphasis role="bold">Case2. List the component which contains the
1154 specified flag</emphasis></para>
1157 <para>List the flag of each component</para>
1158 <screen>$ lfs getstripe -component-flag -I /mnt/testfs/3comp
1164 lcme_flags: 0</screen>
1167 <para>List component(s) who is not instantiated</para>
1168 <screen>$ lfs getstripe --component-flags=^init /mnt/testfs/3comp
1174 lcme_extent.e_start: 67108864
1175 lcme_extent.e_end: EOF
1176 lmm_stripe_count: -1
1177 lmm_stripe_size: 1048576
1180 lmm_stripe_offset: 4</screen>
1183 <para><emphasis role="bold">Case3. List the total number of all the
1184 component(s)</emphasis></para>
1187 <para>List the total number of all the components</para>
1188 <screen>$ lfs getstripe --component-count /mnt/testfs/3comp
1192 <para><emphasis role="bold">Case4. List the component with the specified
1193 extent start or end positions</emphasis></para>
1196 <para>List the start position in bytes of each component</para>
1197 <screen>$ lfs getstripe --component-start /mnt/testfs/3comp
1203 <para>List the start position in bytes of component ID=3</para>
1204 <screen>$ lfs getstripe --component-start -I3 /mnt/testfs/3comp
1208 <para>List the component with start = 64M</para>
1209 <screen>$ lfs getstripe --component-start=64M /mnt/testfs/3comp
1215 lcme_extent.e_start: 67108864
1216 lcme_extent.e_end: EOF
1217 lmm_stripe_count: -1
1218 lmm_stripe_size: 1048576
1221 lmm_stripe_offset: 4</screen>
1224 <para>List the component(s) with start > 5M</para>
1225 <screen>$ lfs getstripe --component-start=+5M /mnt/testfs/3comp
1231 lcme_extent.e_start: 67108864
1232 lcme_extent.e_end: EOF
1233 lmm_stripe_count: -1
1234 lmm_stripe_size: 1048576
1237 lmm_stripe_offset: 4</screen>
1240 <para>List the component(s) with start < 5M</para>
1241 <screen>$ lfs getstripe --component-start=-5M /mnt/testfs/3comp
1247 lcme_extent.e_start: 0
1248 lcme_extent.e_end: 4194304
1250 lmm_stripe_size: 1048576
1253 lmm_stripe_offset: 4
1255 - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
1259 lcme_extent.e_start: 4194304
1260 lcme_extent.e_end: 67108864
1262 lmm_stripe_size: 1048576
1265 lmm_stripe_offset: 5
1267 - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
1268 - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
1269 - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
1270 - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
1273 <para>List the component(s) with start > 3M and end < 70M</para>
1274 <screen>$ lfs getstripe --component-start=+3M --component-end=-70M \
1281 lcme_extent.e_start: 4194304
1282 lcme_extent.e_end: 67108864
1284 lmm_stripe_size: 1048576
1287 lmm_stripe_offset: 5
1289 - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
1290 - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
1291 - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
1292 - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
1296 <section remap="h3">
1297 <title><literal>lfs find</literal></title>
1298 <para><literal>lfs find</literal> commands can be used to search the
1299 directory tree rooted at the given directory or file name for the files
1300 that match the given PFL component parameters. Here, only those
1301 parameters new for PFL files are shown. Their usages are similar to
1302 <literal>lfs getstripe</literal> commands.</para>
1303 <para><emphasis role="bold">Command</emphasis></para>
1304 <screen>lfs find <replaceable>directory|filename</replaceable>
1305 [[!] --component-count [+-=]<replaceable>comp_cnt</replaceable>]
1306 [[!] --component-start [+-=]<replaceable>N</replaceable>[kMGTPE]]
1307 [[!] --component-end|-E [+-=]<replaceable>N</replaceable>[kMGTPE]]
1308 [[!] --component-flags=<replaceable>comp_flags</replaceable>]</screen>
1309 <note><para>If you use <literal>--component-xxx</literal> options, only
1310 the composite files will be searched; but if you use
1311 <literal>! --component-xxx</literal> options, all the files will be
1312 searched.</para></note>
1313 <para><emphasis role="bold">Example</emphasis></para>
1314 <para>We use the following directory and composite files to show how
1315 <literal>lfs find</literal> works.</para>
1316 <screen>$ mkdir /mnt/testfs/testdir
1317 $ lfs setstripe -E 1M -E 10M -E eof /mnt/testfs/testdir/3comp
1318 $ lfs setstripe -E 4M -E 20M -E 30M -E eof /mnt/testfs/testdir/4comp
1319 $ mkdir -p /mnt/testfs/testdir/dir_3comp
1320 $ lfs setstripe -E 6M -E 30M -E eof /mnt/testfs/testdir/dir_3comp
1321 $ lfs setstripe -E 8M -E eof /mnt/testfs/testdir/dir_3comp/2comp
1322 $ lfs setstripe -c 1 /mnt/testfs/testdir/dir_3comp/commnfile</screen>
1323 <para><emphasis role="bold">Case1. Find the files that match the specified
1324 component count condition</emphasis></para>
1325 <para>Find the files under directory /mnt/testfs/testdir whose number of
1326 components is not equal to 3.</para>
1327 <screen>$ lfs find /mnt/testfs/testdir ! --component-count=3
1329 /mnt/testfs/testdir/4comp
1330 /mnt/testfs/testdir/dir_3comp/2comp
1331 /mnt/testfs/testdir/dir_3comp/commonfile</screen>
1332 <para><emphasis role="bold">Case2. Find the files/dirs that match the
1333 specified component start/end condition</emphasis></para>
1334 <para>Find the file(s) under directory /mnt/testfs/testdir with component
1335 start = 4M and end < 70M</para>
1336 <screen>$ lfs find /mnt/testfs/testdir --component-start=4M -E -30M
1337 /mnt/testfs/testdir/4comp</screen>
1338 <para><emphasis role="bold">Case3. Find the files/dirs that match the
1339 specified component flag condition</emphasis></para>
1340 <para>Find the file(s) under directory /mnt/testfs/testdir whose component
1341 flags contain <literal>init</literal></para>
1342 <screen>$ lfs find /mnt/testfs/testdir --component-flag=init
1343 /mnt/testfs/testdir/3comp
1344 /mnt/testfs/testdir/4comp
1345 /mnt/testfs/testdir/dir_3comp/2comp</screen>
1346 <note><para>Since <literal>lfs find</literal> uses
1347 "<literal>!</literal>" to do negative search, we don’t support
1348 flag <literal>^init</literal> here.</para></note>
1352 <section xml:id="striping.sel" condition='l2D'>
1354 <indexterm><primary>striping</primary><secondary>SEL</secondary>
1355 </indexterm>Self-Extending Layout (SEL)</title>
1356 <para>The Lustre Self-Extending Layout (SEL) feature is an extension of the
1357 <xref linkend="pfl"/> feature, which allows the MDS to change the defined
1358 PFL layout dynamically. With this feature, the MDS monitors the used space
1359 on OSTs and swaps the OSTs for the current file when they are low on space.
1360 This avoids <literal>ENOSPC</literal> problems for SEL files when
1361 applications are writing to them.</para>
1362 <para>Whereas PFL delays the instantiation of some components until an IO
1363 operation occurs on this region, SEL allows splitting such non-instantiated
1364 components in two parts: an “extendable” component and an “extension”
1365 component. The extendable component is a regular PFL component, covering
1366 just a part of the region, which is small originally. The extension (or SEL)
1367 component is a new component type which is always non-instantiated and
1368 unassigned, covering the other part of the region. When a write reaches this
1369 unassigned space, and the client calls the MDS to have it instantiated, the
1370 MDS makes a decision as to whether to grant additional space to the extendable
1371 component. The granted region moves from the head of the extension
1372 component to the tail of the extendable component, thus the extendable
1373 component grows and the SEL one is shortened. Therefore, it allows the file
1374 to continue on the same OSTs, or in the case where space is low on one of
1375 the current OSTs, to modify the layout to switch to a new component on new
1376 OSTs. In particular, it lets IO automatically spill over to a large HDD OST
1377 pool once a small SSD OST pool is getting low on space.</para>
1378 <para>The default extension policy modifies the layout in the following
1380 <orderedlist numeration="arabic">
1382 <para>Extension: continue on the same OSTs – used when not low on space
1383 on any of the OSTs of the current component; a particular extent is
1384 granted to the extendable component.</para>
1387 <para>Spill over: switch to next component OSTs – it is used only for
1388 not the last component when <emphasis>at least one</emphasis>
1389 of the current OSTs is low on space; the whole region of the SEL
1390 component moves to the next component and the SEL component is removed
1394 <para>Repeating: create a new component with the same layout but on
1395 free OSTs – it is used only for the last component when <emphasis>
1396 at least one</emphasis> of the current OSTs is low on space; a new
1397 component has the same layout but instantiated on different OSTs (from
1398 the same pool) which have enough space.</para>
1401 <para>Forced extension: continue with the current component OSTs despite
1402 the low on space condition – it is used only for the last component when
1403 a repeating attempt detected low on space condition as well - spillover
1404 is impossible and there is no sense in the repeating.</para>
1407 <para>Each spill event increments the <literal>spill_hit</literal>
1408 counter, which can be accessed with:
1409 <literal>lctl lod.*.<replaceable>POOLNAME</replaceable>.spill_hit</literal></para>
1412 <note><para>The SEL feature does not require clients to understand the SEL
1413 format of already created files, only the MDS support is needed which is
1414 introduced in Lustre 2.13. However, old clients will have some limitations
1415 as the Lustre tools will not support it.</para></note>
1417 <title><literal>lfs setstripe</literal></title>
1418 <para>The <literal>lfs setstripe</literal> command is used to create files
1419 with composite layouts, as well as add or delete components to or from an
1420 existing file. It is extended to support SEL components.</para>
1422 <title>Create a SEL file</title>
1423 <para><emphasis role="bold">Command</emphasis></para>
1424 <screen>lfs setstripe
1425 [--component-end|-E end1] [STRIPE_OPTIONS] ... <replaceable>FILENAME</replaceable>
1428 --extension-size, --ext-size, -z <ext_size></screen>
1429 <para>The <literal>-z</literal> option is added to specify the size of
1430 the region which is granted to the extendable component on each
1431 iteration. While declaring any component, this option turns the declared
1432 component to a pair of components: extendable and extension ones.</para>
1433 <para><emphasis role="bold">Example</emphasis></para>
1434 <para>The following command creates 2 pairs of extendable and
1435 extension components:
1436 <screen># lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/file</screen>
1437 <figure xml:id="managinglayout.fig.sel_createfile">
1438 <title>Example: create a SEL file</title>
1441 <imagedata scalefit="1" depth="0.8in" align="center"
1442 fileref="figures/SEL_Createfile.png" />
1445 <phrase>Example: create a SEL file</phrase>
1450 <note><para>As usual, only the first PFL component is instantiated at
1451 the creation time, thus it is immediately extended to the extension
1452 size (64M for the first component), whereas the third component is left
1453 zero-length.</para></note>
1454 <screen># lfs getstripe /mnt/lustre/file
1462 lcme_extent.e_start: 0
1463 lcme_extent.e_end: 67108864
1465 lmm_stripe_size: 1048576
1468 lmm_stripe_offset: 0
1470 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1474 lcme_flags: extension
1475 lcme_extent.e_start: 67108864
1476 lcme_extent.e_end: 1073741824
1478 lmm_extension_size: 67108864
1481 lmm_stripe_offset: -1
1486 lcme_extent.e_start: 1073741824
1487 lcme_extent.e_end: 1073741824
1489 lmm_stripe_size: 1048576
1492 lmm_stripe_offset: -1
1496 lcme_flags: extension
1497 lcme_extent.e_start: 1073741824
1498 lcme_extent.e_end: EOF
1500 lmm_extension_size: 268435456
1503 lmm_stripe_offset: -1</screen>
1506 <title>Create a SEL layout template</title>
1507 <para>Similar to PFL, it is possible to set a SEL layout template to
1508 a directory. After that, all the files created under it will inherit this
1509 layout by default.</para>
1510 <screen># lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/dir
1511 # ./lustre/utils/lfs getstripe /mnt/lustre/dir
1519 lcme_extent.e_start: 0
1520 lcme_extent.e_end: 67108864
1521 stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1
1525 lcme_flags: extension
1526 lcme_extent.e_start: 67108864
1527 lcme_extent.e_end: 1073741824
1528 stripe_count: 1 extension_size: 67108864 pattern: raid0 stripe_offset: -1
1533 lcme_extent.e_start: 1073741824
1534 lcme_extent.e_end: 1073741824
1535 stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1
1539 lcme_flags: extension
1540 lcme_extent.e_start: 1073741824
1541 lcme_extent.e_end: EOF
1542 stripe_count: 1 extension_size: 268435456 pattern: raid0 stripe_offset: -1
1547 <title><literal>lfs getstripe</literal></title>
1548 <para><literal>lfs getstripe</literal> commands can be used to list the
1549 striping/component information for a given SEL file. Here, only those parameters
1550 new for SEL files are shown.</para>
1551 <para><emphasis role="bold">Command</emphasis></para>
1552 <screen>lfs getstripe
1553 [--extension-size|--ext-size|-z] <replaceable>filename</replaceable></screen>
1554 <para>The <literal>-z</literal> option is added to print the extension
1555 size in bytes. For composite files this is the extension size of the
1556 first extension component. If a particular component is identified by
1557 other options (<literal>--component-id, --component-start</literal>,
1558 etc...), this component extension size is printed.</para>
1559 <para><emphasis role="bold">Example 1: List a SEL component information
1561 <para>Suppose we already have a composite file
1562 <literal>/mnt/lustre/file</literal>, created by the following command:</para>
1563 <screen># lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/file</screen>
1564 <para>The 2nd component could be listed with the following command:</para>
1565 <screen># lfs getstripe -I2 /mnt/lustre/file
1572 lcme_flags: extension
1573 lcme_extent.e_start: 67108864
1574 lcme_extent.e_end: 1073741824
1576 lmm_extension_size: 67108864
1579 lmm_stripe_offset: -1
1581 <note><para>As you can see the SEL components are marked by the <literal>
1582 extension</literal> flag and <literal>lmm_extension_size</literal> field
1583 keeps the specified extension size.</para></note>
1584 <para><emphasis role="bold">Example 2: List the extension size</emphasis></para>
1585 <para>Having the same file as in the above example, the extension size of
1586 the second component could be listed with:</para>
1587 <screen># lfs getstripe -z -I2 /mnt/lustre/file
1589 <para><emphasis role="bold">Example 3: Extension</emphasis></para>
1590 <para>Having the same file as in the above example, suppose there is a
1591 write which crosses the end of the first component (64M), and then another
1592 write another write which crosses the end of the first component (128M) again,
1593 the layout changes as following:</para>
1594 <figure xml:id="managinglayout.fig.sel_extension">
1595 <title>Example: an extension of a SEL file</title>
1598 <imagedata scalefit="1" depth="3.5in" align="center"
1599 fileref="figures/SEL_extension.png" />
1602 <phrase>Example: an extension of a SEL file</phrase>
1606 <para>The layout can be printed out by the following command:</para>
1607 <screen># lfs getstripe /mnt/lustre/file
1615 lcme_extent.e_start: 0
1616 lcme_extent.e_end: 201326592
1618 lmm_stripe_size: 1048576
1621 lmm_stripe_offset: 0
1623 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1627 lcme_flags: extension
1628 lcme_extent.e_start: 201326592
1629 lcme_extent.e_end: 1073741824
1631 lmm_extension_size: 67108864
1634 lmm_stripe_offset: -1
1639 lcme_extent.e_start: 1073741824
1640 lcme_extent.e_end: 1073741824
1642 lmm_stripe_size: 1048576
1645 lmm_stripe_offset: -1
1649 lcme_flags: extension
1650 lcme_extent.e_start: 1073741824
1651 lcme_extent.e_end: EOF
1653 lmm_extension_size: 268435456
1656 lmm_stripe_offset: -1</screen>
1657 <para><emphasis role="bold">Example 4: Spillover</emphasis></para>
1658 <para>In case where <literal>OST0</literal> is low on space and an IO
1659 happens to a SEL component, a spillover happens: the full region of the
1660 SEL component is added to the next component, e.g. in the example above
1661 the next layout modification will look like:</para>
1662 <figure xml:id="managinglayout.fig.sel_spillover">
1663 <title>Example: a spillover in a SEL file</title>
1666 <imagedata scalefit="1" depth="2.25in" align="center"
1667 fileref="figures/SEL_spillover.png" />
1670 <phrase>Example: a spillover in a SEL file</phrase>
1674 <note><para>Despite the fact the third component was [1G, 1G] originally,
1675 while it is not instantiated, instead of getting extended backward, it is
1676 moved backward to the start of the previous SEL component (192M) and
1677 extended on its extension size (256M) from that position, thus it becomes
1678 <literal>[192M, 448M]</literal>.</para></note>
1679 <screen># lfs getstripe /mnt/lustre/file
1687 lcme_extent.e_start: 0
1688 lcme_extent.e_end: 201326592
1690 lmm_stripe_size: 1048576
1693 lmm_stripe_offset: 0
1695 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1700 lcme_extent.e_start: 201326592
1701 lcme_extent.e_end: 469762048
1703 lmm_stripe_size: 1048576
1706 lmm_stripe_offset: 1
1708 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x8:0x0] }
1712 lcme_flags: extension
1713 lcme_extent.e_start: 469762048
1714 lcme_extent.e_end: EOF
1716 lmm_extension_size: 268435456
1719 lmm_stripe_offset: -1</screen>
1720 <para><emphasis role="bold">Example 5: Repeating</emphasis></para>
1721 <para>Suppose in the example above, <literal>OST0</literal> got
1722 enough free space back but <literal>OST1</literal> is low on space,
1723 the following write to the last SEL component leads to a new component
1724 allocation before the SEL component, which repeats the previous
1725 component layout but instantiated on free OSTs:</para>
1726 <figure xml:id="managinglayout.fig.sel_repeat">
1727 <title>Example: repeat a SEL component</title>
1730 <imagedata scalefit="1" depth="2.25in" align="center"
1731 fileref="figures/SEL_repeating.png" />
1734 <phrase>Example: repeat a SEL component
1739 <screen># lfs getstripe /mnt/lustre/file
1747 lcme_extent.e_start: 0
1748 lcme_extent.e_end: 201326592
1750 lmm_stripe_size: 1048576
1753 lmm_stripe_offset: 0
1755 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1760 lcme_extent.e_start: 201326592
1761 lcme_extent.e_end: 469762048
1763 lmm_stripe_size: 1048576
1766 lmm_stripe_offset: 1
1768 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x8:0x0] }
1773 lcme_extent.e_start: 469762048
1774 lcme_extent.e_end: 738197504
1776 lmm_stripe_size: 1048576
1778 lmm_layout_gen: 65535
1779 lmm_stripe_offset: 0
1781 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6:0x0] }
1785 lcme_flags: extension
1786 lcme_extent.e_start: 738197504
1787 lcme_extent.e_end: EOF
1789 lmm_extension_size: 268435456
1792 lmm_stripe_offset: -1</screen>
1793 <para><emphasis role="bold">Example 6: Forced extension</emphasis></para>
1794 <para>Suppose in the example above, both <literal>OST0</literal> and
1795 <literal>OST1</literal> are low on space, the following write to the
1796 last SEL component will behave as an extension as there is no sense to
1798 <figure xml:id="managinglayout.fig.pfl_forced">
1799 <title>Example: forced extension in a SEL file</title>
1802 <imagedata scalefit="1" depth="2.25in" align="center"
1803 fileref="figures/SEL_forced.png" />
1806 <phrase>Example: forced extension in a SEL file.
1811 <screen># lfs getstripe /mnt/lustre/file
1819 lcme_extent.e_start: 0
1820 lcme_extent.e_end: 201326592
1822 lmm_stripe_size: 1048576
1825 lmm_stripe_offset: 0
1827 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1832 lcme_extent.e_start: 201326592
1833 lcme_extent.e_end: 469762048
1835 lmm_stripe_size: 1048576
1838 lmm_stripe_offset: 1
1840 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x8:0x0] }
1845 lcme_extent.e_start: 469762048
1846 lcme_extent.e_end: 1006632960
1848 lmm_stripe_size: 1048576
1850 lmm_layout_gen: 65535
1851 lmm_stripe_offset: 0
1853 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6:0x0] }
1857 lcme_flags: extension
1858 lcme_extent.e_start: 1006632960
1859 lcme_extent.e_end: EOF
1861 lmm_extension_size: 268435456
1864 lmm_stripe_offset: -1</screen>
1867 <title><literal>lfs find</literal></title>
1868 <para><literal>lfs find</literal> commands can be used to search for
1869 the files that match the given SEL component paremeters. Here, only
1870 those parameters new for the SEL files are shown.</para>
1872 [[!] --extension-size|--ext-size|-z [+-]ext-size[KMG]
1873 [[!] --component-flags=extension]</screen>
1874 <para>The <literal>-z</literal> option is added to specify the extension
1875 size to search for. The files which have any component with the
1876 extension size matched the given criteria are printed out. As always
1877 “+” and “-“ signs are allowed to specify the least and the most size.
1879 <para>A new <literal>extension</literal> component flag is added. Only
1880 files which have at least one SEL component are printed.</para>
1881 <note><para>The negative search for flags searches the files which
1882 <emphasis role="strong">have</emphasis> a non-SEL component (not files
1883 which <emphasis role="strong">do not have</emphasis> any SEL component).
1885 <para><emphasis role="bold">Example</emphasis></para>
1886 <screen># lfs setstripe --extension-size 64M -c 1 -E -1 /mnt/lustre/file
1888 # lfs find --comp-flags extension /mnt/lustre/*
1891 # lfs find ! --comp-flags extension /mnt/lustre/*
1894 # lfs find -z 64M /mnt/lustre/*
1897 # lfs find -z +64M /mnt/lustre/*
1899 # lfs find -z -64M /mnt/lustre/*
1901 # lfs find -z +63M /mnt/lustre/*
1904 # lfs find -z -65M /mnt/lustre/*
1907 # lfs find -z 65M /mnt/lustre/*
1909 # lfs find ! -z 64M /mnt/lustre/*
1911 # lfs find ! -z +64M /mnt/lustre/*
1914 # lfs find ! -z -64M /mnt/lustre/*
1917 # lfs find ! -z +63M /mnt/lustre/*
1919 # lfs find ! -z -65M /mnt/lustre/*
1921 # lfs find ! -z 65M /mnt/lustre/*
1922 /mnt/lustre/file</screen>
1926 <section xml:id="foreign_layout" condition='l2D'>
1928 <indexterm><primary>striping</primary><secondary>Foreign</secondary>
1929 </indexterm>Foreign Layout</title>
1930 <para>The Lustre Foreign Layout feature is an extension of both the
1931 LOV and LMV formats which allows the creation of empty files and directories
1932 with the necessary specifications to point to corresponding objects outside
1933 from Lustre namespace.</para>
1934 <para>The new LOV/LMV foreign internal format can be represented as:</para>
1935 <figure xml:id="managinglayout.fig.foreign_format">
1936 <title>LOV/LMV foreign format</title>
1939 <imagedata scalefit="1" width="100%"
1940 fileref="figures/Foreign_Format.png" />
1943 <phrase>LOV/LMV foreign format</phrase>
1948 <title><literal>lfs set[dir]stripe</literal></title>
1949 <para>The <literal>lfs set[dir]stripe</literal> commands are used to
1950 create files or directories with foreign layouts, by calling the
1951 corresponding API, itself invoking the appropriate ioctl().</para>
1953 <title>Create a Foreign file/dir</title>
1954 <para><emphasis role="bold">Command</emphasis></para>
1955 <screen>lfs set[dir]stripe \
1956 --foreign[=<foreign_type>] --xattr|-x <layout_string> \
1957 [--flags <hex_bitmask>] [--mode <mode_bits>] \
1958 <replaceable>{file,dir}name</replaceable></screen>
1959 <para>Both the <literal>--foreign</literal> and
1960 <literal>--xattr|-x</literal> options are mandatory.
1961 The <literal><foreign_type></literal> (default is "none", meaning
1962 no special behavior), and both <literal>--flags</literal> and
1963 <literal>--mode</literal> (default is 0666) options are optional.</para>
1964 <para><emphasis role="bold">Example</emphasis></para>
1965 <para>The following command creates a foreign file of "none" type and
1966 with "foo@bar" LOV content and specific mode and flags:
1967 <screen># lfs setstripe --foreign=none --flags=0xda08 --mode=0640 \
1968 --xattr=foo@bar /mnt/lustre/file</screen>
1969 <figure xml:id="managinglayout.fig.foreign_createfile">
1970 <title>Example: create a foreign file</title>
1973 <imagedata scalefit="1" width="100%" align="center"
1974 fileref="figures/Foreign_Createfile.png" />
1977 <phrase>Example: create a foreign file</phrase>
1985 <title><literal>lfs get[dir]stripe</literal></title>
1986 <para><literal>lfs get[dir]stripe</literal> commands can be used to
1987 retrieve foreign LOV/LMV informations and content.</para>
1988 <para><emphasis role="bold">Command</emphasis></para>
1989 <screen>lfs get[dir]stripe [-v] <replaceable>filename</replaceable></screen>
1990 <para><emphasis role="bold">List foreign layout information
1992 <para>Suppose we already have a foreign file
1993 <literal>/mnt/lustre/file</literal>, created by the following command:</para>
1994 <screen># lfs setstripe --foreign=none --flags=0xda08 --mode=0640 \
1995 --xattr=foo@bar /mnt/lustre/file</screen>
1996 <para>The full foreign layout informations can be listed using the
1997 following command:</para>
1998 <screen># lfs getstripe -v /mnt/lustre/file
2000 lfm_magic: 0x0BD70BD0
2003 lfm_flags: 0x0000DA08
2006 <note><para>As you can see the <literal>lfm_length</literal> field
2007 value is the characters number in the variable length
2008 <literal>lfm_value</literal> field.</para></note>
2011 <title><literal>lfs find</literal></title>
2012 <para><literal>lfs find</literal> commands can be used to search for
2013 all the foreign files/directories or those that match the given
2014 selection paremeters.</para>
2016 [[!] --foreign[=<foreign_type>]</screen>
2017 <para>The <literal>--foreign[=<foreign_type>]</literal> option
2018 has been added to specify that all [!,but not] files and/or directories
2019 with a foreign layout [and [!,but not] of
2020 <literal><foreign_type></literal>] will be retrieved.</para>
2021 <para><emphasis role="bold">Example</emphasis></para>
2022 <screen># lfs setstripe --foreign=none --xattr=foo@bar /mnt/lustre/file
2023 # touch /mnt/lustre/file2
2025 # lfs find --foreign /mnt/lustre/*
2028 # lfs find ! --foreign /mnt/lustre/*
2031 # lfs find --foreign=none /mnt/lustre/*
2032 /mnt/lustre/file</screen>
2036 <section xml:id="file_striping.managing_free_space">
2038 <primary>space</primary>
2039 <secondary>free space</secondary>
2040 </indexterm><indexterm>
2041 <primary>striping</primary>
2042 <secondary>round-robin algorithm</secondary>
2043 </indexterm><indexterm>
2044 <primary>striping</primary>
2045 <secondary>weighted algorithm</secondary>
2046 </indexterm><indexterm>
2047 <primary>round-robin algorithm</primary>
2048 </indexterm><indexterm>
2049 <primary>weighted algorithm</primary>
2050 </indexterm>Managing Free Space</title>
2051 <para>To optimize file system performance, the MDT assigns file stripes to OSTs based on two
2052 allocation algorithms. The <emphasis role="italic">round-robin</emphasis> allocator gives
2053 preference to location (spreading out stripes across OSSs to increase network bandwidth
2054 utilization) and the weighted allocator gives preference to available space (balancing loads
2055 across OSTs). Threshold and weighting factors for these two algorithms can be adjusted by the
2056 user. The MDT reserves 0.1 percent of total OST space and 32 inodes for each OST. The MDT
2057 stops object allocation for the OST if available space is less than reserved or the OST has
2058 fewer than 32 free inodes. The MDT starts object allocation when available space is twice
2059 as big as the reserved space and the OST has more than 64 free inodes. Note, clients
2060 could append existing files no matter what object allocation state is.</para>
2061 <para condition="l29"> The reserved space for each OST can be adjusted by the user. Use the
2062 <literal>lctl set_param</literal> command, for example the next command reserve 1GB space
2064 <screen>lctl set_param -P osp.*.reserved_mb_low=1024</screen></para>
2065 <para>This section describes how to check available free space on disks
2066 and how free space is allocated. It then describes how to set the
2067 threshold and weighting factors for the allocation algorithms.</para>
2068 <section xml:id="file_striping.checking_free_space">
2069 <title>Checking File System Free Space</title>
2070 <para>Free space is an important consideration in assigning file stripes.
2071 The <literal>lfs df</literal> command can be used to show available
2072 disk space on the mounted Lustre file system and space consumption per
2073 OST. If multiple Lustre file systems are mounted, a path may be
2074 specified, but is not required. Options to the <literal>lfs df</literal>
2075 command are shown below.</para>
2076 <informaltable frame="all">
2078 <colspec colname="c1" colwidth="50*"/>
2079 <colspec colname="c2" colwidth="50*"/>
2083 <para><emphasis role="bold">Option</emphasis></para>
2086 <para><emphasis role="bold">Description</emphasis></para>
2094 <literal>-h</literal>, <literal>--human-readable</literal>
2098 <para> Displays sizes in human readable format (for example: 1K,
2099 234M, 5G) using base-2 (binary) values (i.e. 1G = 1024M).</para>
2105 <literal>-H</literal>, <literal>--si</literal>
2109 <para>Like <literal>-h</literal>, this displays counts in human
2110 readable format, but using base-10 (decimal) values
2111 (i.e. 1G = 1000M).</para>
2116 <para> <literal role="bold">-i, --inodes</literal></para>
2119 <para> Lists inodes instead of block usage.</para>
2124 <para> <literal role="bold">-l, --lazy</literal></para>
2127 <para>Do not attempt to contact any OST or MDT not currently
2128 connected to the client. This avoids blocking the
2129 <literal>lfs df</literal> output if a target is offline or
2130 unreachable, and only returns the space on OSTs that can
2131 currently be accessed.</para>
2136 <para> <literal role="bold">-p, --pool</literal></para>
2139 <para>Limit the usage to report only OSTs that are in the
2140 specified <replaceable>pool</replaceable>. If multiple
2141 Lustre filesystems are mounted, list the OSTs in
2142 <replaceable>pool</replaceable> for each filesystem, or
2143 limit the display to only a pool for a specific filesystem
2144 if <replaceable>fsname.pool</replaceable> is given.
2145 Specifying both <replaceable>fsname</replaceable> and
2146 <replaceable>pool</replaceable> is equivalent to providing
2147 a specific mountpoint.
2154 <literal>-v</literal>, <literal>--verbose</literal>
2158 <para>Display verbose status of MDTs and OSTs. This may
2159 include one or more optional flags at the end of each line.
2167 <literal>lfs df</literal> may also report additional target status
2168 as the last column in the display, if there are issues with that target.
2169 Target states include:
2173 <literal>D</literal>: OST/MDT is <literal>Degraded</literal>.
2174 The target has a failed drive in the RAID device, or is
2175 undergoing RAID reconstruction. This state is marked on
2176 the server automatically for ZFS targets via
2177 <literal>zed</literal>, or a (user-supplied) script that
2178 monitors the target device and sets
2179 "<literal>lctl set_param obdfilter.<replaceable>target</replaceable>.degraded=1</literal>"
2180 on the OST. This target will be avoided for new
2181 allocations, but will still be used to read existing files
2182 located there or if there are not enough non-degraded OSTs
2183 to make up a widely-striped file.
2186 <literal>R</literal>: OST/MDT is <literal>Read-only</literal>.
2187 The target filesystem is marked read-only due to filesystem
2188 corruption detected by ldiskfs or ZFS. No modifications
2189 are allowed on this OST, and it needs to be unmounted and
2190 <literal>e2fsck</literal> or <literal>zpool scrub</literal>
2191 run to repair the underlying filesystem.
2194 <literal>N</literal>: OST/MDT is <literal>No-precreate</literal>.
2195 The target is configured to deny object precreation set by
2196 "<literal>lctl set_param obdfilter.<replaceable>target</replaceable>.no_precreate=1</literal>"
2197 parameter or the "<literal>-o no_precreate</literal>" mount option.
2198 This may be done to add an OST to the filesystem without allowing
2199 objects to be allocated on it yet, or for other reasons.
2202 <literal>S</literal>: OST/MDT is out of <literal>Space</literal>.
2203 The target filesystem has less than the minimum required
2204 free space and will not be used for new object allocations
2205 until it has more free space.
2208 <literal>I</literal>: OST/MDT is out of <literal>Inodes</literal>.
2209 The target filesystem has less than the minimum required
2210 free inodes and will not be used for new object allocations
2211 until it has more free inodes.
2214 <literal>f</literal>: OST/MDT is on <literal>flash</literal>.
2215 The target filesystem is using a flash (non-rotational)
2216 storage device. This is normally detected from the
2217 underlying Linux block device, but can be set manually
2218 with "<literal>lctl set_param osd-*.*.nonrotational=1</literal>
2219 on the respective OSTs. This lower-case status is only
2220 shown in conjunction with the <literal>-v</literal> option,
2221 since it is not an error condition.
2225 <para>The <literal>df -i</literal> and <literal>lfs df -i</literal>
2226 commands show the <emphasis role="italic">minimum</emphasis> number
2227 of inodes that can be created in the file system at the current time.
2228 If the total number of objects available across all of the OSTs is
2229 smaller than those available on the MDT(s), taking into account the
2230 default file striping, then <literal>df -i</literal> will also
2231 report a smaller number of inodes than could be created. Running
2232 <literal>lfs df -i</literal> will report the actual number of inodes
2233 that are free on each target.
2235 <para>For ZFS file systems, the number of inodes that can be created
2236 is dynamic and depends on the free space in the file system. The
2237 Free and Total inode counts reported for a ZFS file system are only
2238 an estimate based on the current usage for each target. The Used
2239 inode count is the actual number of inodes used by the file system.
2242 <para><emphasis role="bold">Examples</emphasis></para>
2243 <screen>client$ lfs df
2244 UUID 1K-blocks Used Available Use% Mounted on
2245 testfs-OST0000_UUID 9174328 1020024 8154304 11% /mnt/lustre[MDT:0]
2246 testfs-OST0000_UUID 94181368 56330708 37850660 59% /mnt/lustre[OST:0]
2247 testfs-OST0001_UUID 94181368 56385748 37795620 59% /mnt/lustre[OST:1]
2248 testfs-OST0002_UUID 94181368 54352012 39829356 57% /mnt/lustre[OST:2]
2249 filesystem summary: 282544104 167068468 39829356 57% /mnt/lustre
2251 [client1] $ lfs df -hv
2252 UUID bytes Used Available Use% Mounted on
2253 testfs-MDT0000_UUID 8.7G 996.1M 7.8G 11% /mnt/lustre[MDT:0]
2254 testfs-OST0000_UUID 89.8G 53.7G 36.1G 59% /mnt/lustre[OST:0] f
2255 testfs-OST0001_UUID 89.8G 53.8G 36.0G 59% /mnt/lustre[OST:1] f
2256 testfs-OST0002_UUID 89.8G 51.8G 38.0G 57% /mnt/lustre[OST:2] f
2257 filesystem summary: 269.5G 159.3G 110.1G 59% /mnt/lustre
2259 [client1] $ lfs df -iH
2260 UUID Inodes IUsed IFree IUse% Mounted on
2261 testfs-MDT0000_UUID 2.21M 41.9k 2.17M 1% /mnt/lustre[MDT:0]
2262 testfs-OST0000_UUID 737.3k 12.1k 725.1k 1% /mnt/lustre[OST:0]
2263 testfs-OST0001_UUID 737.3k 12.2k 725.0k 1% /mnt/lustre[OST:1]
2264 testfs-OST0002_UUID 737.3k 12.2k 725.0k 1% /mnt/lustre[OST:2]
2265 filesystem summary: 2.21M 41.9k 2.17M 1% /mnt/lustre[OST:2]
2268 <section remap="h3">
2270 <primary>striping</primary>
2271 <secondary>allocations</secondary>
2272 </indexterm> Stripe Allocation Methods</title>
2273 <para>Two stripe allocation methods are provided:</para>
2276 <para><emphasis role="bold">Round-robin allocator</emphasis> - When the OSTs have
2277 approximately the same amount of free space, the round-robin allocator alternates
2278 stripes between OSTs on different OSSs, so the OST used for stripe 0 of each file is
2279 evenly distributed among OSTs, regardless of the stripe count. In a simple example with
2280 eight OSTs numbered 0-7, objects would be allocated like this:</para>
2282 <screen>File 1: OST1, OST2, OST3, OST4
2283 File 2: OST5, OST6, OST7
2284 File 3: OST0, OST1, OST2, OST3, OST4, OST5
2285 File 4: OST6, OST7, OST0</screen>
2287 <para>Here are several more sample round-robin stripe orders (each letter represents a
2288 different OST on a single OSS):</para>
2289 <informaltable frame="none">
2291 <colspec colname="c1" colwidth="50*"/>
2292 <colspec colname="c2" colwidth="50*"/>
2296 <para> 3: AAA</para>
2299 <para> One 3-OST OSS</para>
2304 <para> 3x3: ABABAB</para>
2307 <para> Two 3-OST OSSs</para>
2312 <para> 3x4: BBABABA</para>
2315 <para> One 3-OST OSS (A) and one 4-OST OSS (B)</para>
2320 <para> 3x5: BBABBABA</para>
2323 <para> One 3-OST OSS (A) and one 5-OST OSS (B)</para>
2328 <para> 3x3x3: ABCABCABC</para>
2331 <para> Three 3-OST OSSs</para>
2339 <para><emphasis role="bold">Weighted allocator</emphasis> - When the free space difference
2340 between the OSTs becomes significant, the weighting algorithm is used to influence OST
2341 ordering based on size (amount of free space available on each OST) and location
2342 (stripes evenly distributed across OSTs). The weighted allocator fills the emptier OSTs
2343 faster, but uses a weighted random algorithm, so the OST with the most free space is not
2344 necessarily chosen each time.</para>
2347 <para>The allocation method is determined by the amount of free-space
2348 imbalance on the OSTs. When free space is relatively balanced across
2349 OSTs, the faster round-robin allocator is used, which maximizes network
2350 balancing. The weighted allocator is used when any two OSTs are out of
2351 balance by more than the specified threshold (17% by default). The
2352 threshold between the two allocation methods is defined by the
2353 <literal>qos_threshold_rr</literal> parameter. </para>
2354 <para>To temporarily set the <literal>qos_threshold_rr</literal> to
2355 <literal>25</literal>, enter the folowing on each MDS:
2356 <screen>mds# lctl set_param lod.<replaceable>fsname</replaceable>*.qos_threshold_rr=25</screen></para>
2358 <section remap="h3">
2360 <primary>space</primary>
2361 <secondary>location weighting</secondary>
2362 </indexterm>Adjusting the Weighting Between Free Space and Location</title>
2363 <para>The weighting priority used by the weighted allocator is set by the
2364 the <literal>qos_prio_free</literal> parameter.
2365 Increasing the value of <literal>qos_prio_free</literal> puts more
2366 weighting on the amount of free space available on each OST and less
2367 on how stripes are distributed across OSTs. The default value is
2368 <literal>91</literal> (percent). When the free space priority is set to
2369 <literal>100</literal> (percent), weighting is based entirely on free space and location
2370 is no longer used by the striping algorithm. </para>
2371 <para>To permanently change the allocator weighting to <literal>100</literal>, enter this command on the
2373 <screen>lctl conf_param <replaceable>fsname</replaceable>-MDT0000-*.lod.qos_prio_free=100</screen>
2376 <para>When <literal>qos_prio_free</literal> is set to <literal>100</literal>, a weighted
2377 random algorithm is still used to assign stripes, so, for example, if OST2 has twice as
2378 much free space as OST1, OST2 is twice as likely to be used, but it is not guaranteed to
2383 <section xml:id="wide_striping">
2385 <primary>striping</primary>
2386 <secondary>wide striping</secondary>
2387 </indexterm><indexterm>
2388 <primary>wide striping</primary>
2389 </indexterm>Lustre Striping Internals</title>
2390 <para>Individual files can only be striped over a finite number of OSTs,
2391 based on the maximum size of the attributes that can be stored on the MDT.
2392 If the MDT is ldiskfs-based without the <literal>ea_inode</literal>
2393 feature, a file can be striped across at most 160 OSTs. With ZFS-based
2394 MDTs, or if the <literal>ea_inode</literal> feature is enabled for an
2395 ldiskfs-based MDT, a file can be striped across up to 2000 OSTs.
2397 <para>Lustre inodes use an extended attribute to record on which OST each
2398 object is located, and the identifier each object on that OST. The size of
2399 the extended attribute is a function of the number of stripes.</para>
2400 <para>If using an ldiskfs-based MDT, the maximum number of OSTs over which
2401 files can be striped can been raised to 2000 by enabling the
2402 <literal>ea_inode</literal> feature on the MDT:
2403 <screen>tune2fs -O ea_inode /dev/<replaceable>mdtdev</replaceable></screen>
2405 <note condition='l2D'><para>Since Lustre 2.13 the
2406 <literal>ea_inode</literal> feature is enabled by default on all newly
2407 formatted ldiskfs MDT filesystems.</para></note>
2408 <note><para>The maximum stripe count for a single file does not limit the
2409 maximum number of OSTs that are in the filesystem as a whole, only the
2410 maximum possible size and maximum aggregate bandwidth for the file.
2415 vim:expandtab:shiftwidth=2:tabstop=8:
git://git.whamcloud.com / doc / manual.git / blob