1 <?xml version='1.0' encoding='UTF-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="managingstripingfreespace">
5 <title xml:id="managingstripingfreespace.title">Managing File Layout (Striping) and Free
7 <para>This chapter describes file layout (striping) and I/O options, and includes the following
11 <para><xref linkend="file_striping.how_it_works"/></para>
14 <para><xref linkend="file_striping.considerations"/></para>
17 <para><xref linkend="file_striping.lfs_setstripe"/></para>
20 <para><xref linkend="file_striping.lfs_getstripe"/></para>
23 <para><xref linkend="file_striping.managing_free_space"/></para>
26 <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/></para>
29 <section xml:id="file_striping.how_it_works">
32 <primary>space</primary>
35 <primary>striping</primary>
36 <secondary>how it works</secondary>
39 <primary>striping</primary>
43 <primary>space</primary>
44 <secondary>striping</secondary>
45 </indexterm>How Lustre File System Striping Works</title>
46 <para>In a Lustre file system, the MDS allocates objects to OSTs using either a round-robin
47 algorithm or a weighted algorithm. When the amount of free space is well balanced (i.e., by
48 default, when the free space across OSTs differs by less than 17%), the round-robin algorithm
49 is used to select the next OST to which a stripe is to be written. Periodically, the MDS
50 adjusts the striping layout to eliminate some degenerated cases in which applications that
51 create very regular file layouts (striping patterns) preferentially use a particular OST in
53 <para> Normally the usage of OSTs is well balanced. However, if users create a small number of
54 exceptionally large files or incorrectly specify striping parameters, imbalanced OST usage may
55 result. When the free space across OSTs differs by more than a specific amount (17% by
56 default), the MDS then uses weighted random allocations with a preference for allocating
57 objects on OSTs with more free space. (This can reduce I/O performance until space usage is
58 rebalanced again.) For a more detailed description of how striping is allocated, see <xref
59 linkend="file_striping.managing_free_space"/>.</para>
60 <para>Files can only be striped over a finite number of OSTs, based on the
61 maximum size of the attributes that can be stored on the MDT. If the MDT
62 is ldiskfs-based without the <literal>ea_inode</literal> feature, a file
63 can be striped across at most 160 OSTs. With a ZFS-based MDT, or if the
64 <literal>ea_inode</literal> feature is enabled for an ldiskfs-based MDT
65 (the default since Lustre 2.13.0),
66 a file can be striped across up to 2000 OSTs. For more information, see
67 <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/>.
70 <section xml:id="file_striping.considerations">
72 <primary>file layout</primary>
73 <secondary>See striping</secondary>
74 </indexterm><indexterm>
75 <primary>striping</primary>
76 <secondary>considerations</secondary>
79 <primary>space</primary>
80 <secondary>considerations</secondary>
81 </indexterm> Lustre File Layout (Striping) Considerations</title>
82 <para>Whether you should set up file striping and what parameter values you select depends on
83 your needs. A good rule of thumb is to stripe over as few objects as will meet those needs and
85 <para>Some reasons for using striping include:</para>
88 <para><emphasis role="bold">Providing high-bandwidth access.</emphasis> Many applications
89 require high-bandwidth access to a single file, which may be more bandwidth than can be
90 provided by a single OSS. Examples are a scientific application that writes to a single
91 file from hundreds of nodes, or a binary executable that is loaded by many nodes when an
92 application starts.</para>
93 <para>In cases like these, a file can be striped over as many OSSs as it takes to achieve
94 the required peak aggregate bandwidth for that file. Striping across a larger number of
95 OSSs should only be used when the file size is very large and/or is accessed by many nodes
96 at a time. Currently, Lustre files can be striped across up to 2000 OSTs</para>
99 <para><emphasis role="bold">Improving performance when OSS bandwidth is exceeded.</emphasis>
100 Striping across many OSSs can improve performance if the aggregate client bandwidth
101 exceeds the server bandwidth and the application reads and writes data fast enough to take
102 advantage of the additional OSS bandwidth. The largest useful stripe count is bounded by
103 the I/O rate of the clients/jobs divided by the performance per OSS.</para>
106 <para condition="l2D"><emphasis role="bold">Matching stripes to I/O
107 pattern.</emphasis>When writing to a single file from multiple nodes,
108 having more than one client writing to a stripe can lead to issues
109 with lock exchange, where clients contend over writing to that stripe,
110 even if their I/Os do not overlap. This can be avoided if I/O can be
111 stripe aligned so that each stripe is accessed by only one client.
112 Since Lustre 2.13, the 'overstriping' feature is available, allowing more
113 than stripe per OST. This is particularly helpful for the case where
114 thread count exceeds OST count, making it possible to match stripe count
115 to thread count even in this case.</para>
118 <para><emphasis role="bold">Providing space for very large files.</emphasis> Striping is
119 useful when a single OST does not have enough free space to hold the entire file.</para>
122 <para>Some reasons to minimize or avoid striping:</para>
125 <para><emphasis role="bold">Increased overhead.</emphasis> Striping results in more locks
126 and extra network operations during common operations such as <literal>stat</literal> and
127 <literal>unlink</literal>. Even when these operations are performed in parallel, one
128 network operation takes less time than 100 operations.</para>
129 <para>Increased overhead also results from server contention. Consider a cluster with 100
130 clients and 100 OSSs, each with one OST. If each file has exactly one object and the load
131 is distributed evenly, there is no contention and the disks on each server can manage
132 sequential I/O. If each file has 100 objects, then the clients all compete with one
133 another for the attention of the servers, and the disks on each node seek in 100 different
134 directions resulting in needless contention.</para>
137 <para><emphasis role="bold">Increased risk.</emphasis> When files are striped across all
138 servers and one of the servers breaks down, a small part of each striped file is lost. By
139 comparison, if each file has exactly one stripe, fewer files are lost, but they are lost
140 in their entirety. Many users would prefer to lose some of their files entirely than all
141 of their files partially.</para>
145 <title><indexterm><primary>striping</primary><secondary>size</secondary></indexterm>
146 Choosing a Stripe Size</title>
147 <para>Choosing a stripe size is a balancing act, but reasonable defaults are described below.
148 The stripe size has no effect on a single-stripe file.</para>
151 <para><emphasis role="bold">The stripe size must be a multiple of the page
152 size.</emphasis> Lustre software tools enforce a multiple of 64 KB (the maximum page
153 size on ia64 and PPC64 nodes) so that users on platforms with smaller pages do not
154 accidentally create files that might cause problems for ia64 clients.</para>
157 <para><emphasis role="bold">The smallest recommended stripe size is 512 KB.</emphasis>
158 Although you can create files with a stripe size of 64 KB, the smallest practical stripe
159 size is 512 KB because the Lustre file system sends 1MB chunks over the network.
160 Choosing a smaller stripe size may result in inefficient I/O to the disks and reduced
164 <para><emphasis role="bold">A good stripe size for sequential I/O using high-speed
165 networks is between 1 MB and 4 MB.</emphasis> In most situations, stripe sizes larger
166 than 4 MB may result in longer lock hold times and contention during shared file
170 <para><emphasis role="bold">The maximum stripe size is 4 GB.</emphasis> Using a large
171 stripe size can improve performance when accessing very large files. It allows each
172 client to have exclusive access to its own part of a file. However, a large stripe size
173 can be counterproductive in cases where it does not match your I/O pattern.</para>
176 <para><emphasis role="bold">Choose a stripe pattern that takes into account the write
177 patterns of your application.</emphasis> Writes that cross an object boundary are
178 slightly less efficient than writes that go entirely to one server. If the file is
179 written in a consistent and aligned way, make the stripe size a multiple of the
180 <literal>write()</literal> size.</para>
185 <section xml:id="file_striping.lfs_setstripe">
187 <primary>striping</primary>
188 <secondary>configuration</secondary>
189 </indexterm>Setting the File Layout/Striping Configuration (<literal>lfs
190 setstripe</literal>)</title>
191 <para>Use the <literal>lfs setstripe</literal> command to create new files with a specific file layout (stripe pattern) configuration.</para>
192 <screen>lfs setstripe [--size|-s stripe_size] [--stripe-count|-c stripe_count] [--overstripe-count|-C stripe_count] \
193 [--index|-i start_ost] [--pool|-p pool_name] <replaceable>filename|dirname</replaceable> </screen>
194 <para><emphasis role="bold">
195 <literal>stripe_size</literal>
198 <para>The <literal>stripe_size</literal> indicates how much data to write to one OST before
199 moving to the next OST. The default <literal>stripe_size</literal> is 1 MB. Passing a
200 <literal>stripe_size</literal> of 0 causes the default stripe size to be used. Otherwise,
201 the <literal>stripe_size</literal> value must be a multiple of 64 KB.</para>
202 <para><emphasis role="bold">
203 <literal>stripe_count (--stripe-count, --overstripe-count)</literal>
206 <para>The <literal>stripe_count</literal> indicates how many stripes to use.
207 The default <literal>stripe_count</literal> value is 1. Setting
208 <literal>stripe_count</literal> to 0 causes the default stripe count to be
209 used. Setting <literal>stripe_count</literal> to -1 means stripe over all
210 available OSTs (full OSTs are skipped). When --overstripe-count is used,
211 per OST if necessary.</para>
212 <para><emphasis role="bold">
213 <literal>start_ost</literal>
216 <para>The start OST is the first OST to which files are written. The default value for
217 <literal>start_ost</literal> is -1, which allows the MDS to choose the starting index. This
218 setting is strongly recommended, as it allows space and load balancing to be done by the MDS
219 as needed. If the value of <literal>start_ost</literal> is set to a value other than -1, the
220 file starts on the specified OST index. OST index numbering starts at 0.</para>
222 <para>If the specified OST is inactive or in a degraded mode, the MDS will silently choose
223 another target.</para>
226 <para>If you pass a <literal>start_ost</literal> value of 0 and a
227 <literal>stripe_count</literal> value of <emphasis>1</emphasis>, all files are written to
228 OST 0, until space is exhausted. <emphasis role="italic">This is probably not what you meant
229 to do.</emphasis> If you only want to adjust the stripe count and keep the other
230 parameters at their default settings, do not specify any of the other parameters:</para>
231 <para><screen>client# lfs setstripe -c <replaceable>stripe_count</replaceable> <replaceable>filename</replaceable></screen></para>
233 <para><emphasis role="bold">
234 <literal>pool_name</literal>
237 <para>The <literal>pool_name</literal> specifies the OST pool to which the
238 file will be written. This allows limiting the OSTs used to a subset of
239 all OSTs in the file system. For more details about using OST pools, see
240 <link xl:href="managingfilesystemio.managing_ost_pools">
241 Creating and Managing OST Pools
244 <title>Specifying a File Layout (Striping Pattern) for a Single File</title>
245 <para>It is possible to specify the file layout when a new file is created using the command <literal>lfs setstripe</literal>. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an <literal>lfs setstripe</literal> command fails if the file already exists.</para>
246 <section xml:id="file_striping.stripe_size">
247 <title>Setting the Stripe Size</title>
248 <para>The command to create a new file with a specified stripe size is similar to:</para>
249 <screen>[client]# lfs setstripe -s 4M /mnt/lustre/new_file</screen>
250 <para>This example command creates the new file <literal>/mnt/lustre/new_file</literal> with a stripe size of 4 MB.</para>
251 <para>Now, when the file is created, the new stripe setting creates the file on a single OST with a stripe size of 4M:</para>
252 <screen> [client]# lfs getstripe /mnt/lustre/new_file
255 lmm_stripe_size: 4194304
259 obdidx objid objid group
260 1 690550 0xa8976 0 </screen>
261 <para>In this example, the stripe size is 4 MB.</para>
264 <title><indexterm><primary>striping</primary><secondary>count</secondary></indexterm>
265 Setting the Stripe Count</title>
266 <para>The command below creates a new file with a stripe count of <literal>-1</literal> to
267 specify striping over all available OSTs:</para>
268 <screen>[client]# lfs setstripe -c -1 /mnt/lustre/full_stripe</screen>
269 <para>The example below indicates that the file
270 <literal>full_stripe</literal> is striped
271 over all six active OSTs in the configuration:</para>
272 <screen>[client]# lfs getstripe /mnt/lustre/full_stripe
273 /mnt/lustre/full_stripe
274 obdidx objid objid group
281 <para> This is in contrast to the output in
282 <xref linkend="file_striping.stripe_size"/>,
283 which shows only a single object for the file.</para>
288 <primary>striping</primary>
289 <secondary>per directory</secondary>
290 </indexterm>Setting the Striping Layout for a Directory</title>
291 <para>In a directory, the <literal>lfs setstripe</literal> command sets a default striping
292 configuration for files created in the directory. The usage is the same as <literal>lfs
293 setstripe</literal> for a regular file, except that the directory must exist prior to
294 setting the default striping configuration. If a file is created in a directory with a
295 default stripe configuration (without otherwise specifying striping), the Lustre file system
296 uses those striping parameters instead of the file system default for the new file.</para>
297 <para>To change the striping pattern for a sub-directory, create a directory with desired file
298 layout as described above. Sub-directories inherit the file layout of the root/parent
303 <primary>striping</primary>
304 <secondary>per file system</secondary>
305 </indexterm>Setting the Striping Layout for a File System</title>
306 <para>Setting the striping specification on the <literal>root</literal> directory determines
307 the striping for all new files created in the file system unless an overriding striping
308 specification takes precedence (such as a striping layout specified by the application, or
309 set using <literal>lfs setstripe</literal>, or specified for the parent directory).</para>
311 <para>The striping settings for a <literal>root</literal> directory are, by default, applied
312 to any new child directories created in the root directory, unless striping settings have
313 been specified for the child directory.</para>
318 <primary>striping</primary>
319 <secondary>on specific OST</secondary>
320 </indexterm>Creating a File on a Specific OST</title>
321 <para>You can use <literal>lfs setstripe</literal> to create a file on a specific OST. In the
322 following example, the file <literal>file1</literal> is created on the first OST (OST index
324 <screen>$ lfs setstripe --stripe-count 1 --index 0 file1
325 $ dd if=/dev/zero of=file1 count=1 bs=100M
329 $ lfs getstripe file1
332 lmm_stripe_size: 1048576
336 obdidx objid objid group
337 0 37364 0x91f4 0</screen>
340 <section xml:id="file_striping.lfs_getstripe">
341 <title><indexterm><primary>striping</primary><secondary>getting information</secondary></indexterm>Retrieving File Layout/Striping Information (<literal>getstripe</literal>)</title>
342 <para>The <literal>lfs getstripe</literal> command is used to display information that shows
343 over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along
344 with the OST index and object ID for each stripe in the file. For directories, the default
345 settings for files created in that directory are displayed.</para>
347 <title>Displaying the Current Stripe Size</title>
348 <para>To see the current stripe size for a Lustre file or directory, use the <literal>lfs
349 getstripe</literal> command. For example, to view information for a directory, enter a
350 command similar to:</para>
351 <screen>[client]# lfs getstripe /mnt/lustre </screen>
352 <para>This command produces output similar to:</para>
354 (Default) stripe_count: 1 stripe_size: 1M stripe_offset: -1</screen>
355 <para>In this example, the default stripe count is <literal>1</literal> (data blocks are
356 striped over a single OST), the default stripe size is 1 MB, and the objects are created
357 over all available OSTs.</para>
358 <para>To view information for a file, enter a command similar to:</para>
359 <screen>$ lfs getstripe /mnt/lustre/foo
362 lmm_stripe_size: 1048576
366 obdidx objid objid group
367 2 835487 m0xcbf9f 0 </screen>
368 <para>In this example, the file is located on <literal>obdidx 2</literal>, which corresponds
369 to the OST <literal>lustre-OST0002</literal>. To see which node is serving that OST, run:
370 <screen>$ lctl get_param osc.lustre-OST0002-osc.ost_conn_uuid
371 osc.lustre-OST0002-osc.ost_conn_uuid=192.168.20.1@tcp</screen></para>
374 <title>Inspecting the File Tree</title>
375 <para>To inspect an entire tree of files, use the <literal>lfs find</literal> command:</para>
376 <screen>lfs find [--recursive | -r] <replaceable>file|directory</replaceable> ...</screen>
380 <primary>striping</primary>
381 <secondary>remote directories</secondary>
382 </indexterm>Locating the MDT for a remote directory</title>
383 <para>Lustre can be configured with multiple MDTs in the same file
384 system. Each directory and file could be located on a different MDT.
385 To identify which MDT a given subdirectory is located, pass the
386 <literal>getstripe [--mdt-index|-M]</literal> parameter to
387 <literal>lfs</literal>. An example of this command is provided in
388 the section <xref linkend="lustremaint.rmremotedir"/>.</para>
391 <section xml:id="pfl" condition='l2A'>
393 <primary>striping</primary>
394 <secondary>PFL</secondary>
395 </indexterm>Progressive File Layout(PFL)</title>
396 <para>The Lustre Progressive File Layout (PFL) feature simplifies the use
397 of Lustre so that users can expect reasonable performance for a variety of
398 normal file IO patterns without the need to explicitly understand their IO
399 model or Lustre usage details in advance. In particular, users do not
400 necessarily need to know the size or concurrency of output files in
401 advance of their creation and explicitly specify an optimal layout for
402 each file in order to achieve good performance for both highly concurrent
403 shared-single-large-file IO or parallel IO to many smaller per-process
405 <para>The layout of a PFL file is stored on disk as <literal>composite
406 layout</literal>. A PFL file is essentially an array of
407 <literal>sub-layout components</literal>, with each sub-layout component
408 being a plain layout covering different and non-overlapped extents of
409 the file. For PFL files, the file layout is composed of a series of
410 components, therefore it's possible that there are some file extents are
411 not described by any components.</para>
412 <para>An example of how data blocks of PFL files are mapped to OST objects
413 of components is shown in the following PFL object mapping diagram:</para>
414 <figure xml:id="managinglayout.fig.pfl">
415 <title>PFL object mapping diagram</title>
418 <imagedata scalefit="1" width="100%"
419 fileref="figures/PFL_object_mapping_diagram.png" />
422 <phrase>PFL object mapping diagram</phrase>
426 <para>The PFL file in <xref linkend="managinglayout.fig.pfl"/> has 3
427 components and shows the mapping for the blocks of a 2055MB file.
428 The stripe size for the first two components is 1MB, while the stripe size
429 for the third component is 4MB. The stripe count is increasing for each
430 successive component. The first component only has two 1MB blocks and the
431 single object has a size of 2MB. The second component holds the next 254MB
432 of the file spread over 4 separate OST objects in RAID-0, each one will
433 have a size of 256MB / 4 objects = 64MB per object. Note the first two
434 objects <literal>obj 2,0</literal> and <literal>obj 2,1</literal>
435 have a 1MB hole at the start where the data is stored in the first
436 component. The final component holds the next 1800MB spread over 32 OST
437 objects. There is a 256MB / 32 = 8MB hole at the start each one for the
438 data stored in the first two components. Each object will be
439 2048MB / 32 objects = 64MB per object, except the
440 <literal>obj 3,0</literal> that holds an extra 4MB chunk and
441 <literal>obj 3,1</literal> that holds an extra 3MB chunk. If more data
442 was written to the file, only the objects in component 3 would increase
444 <para>When a file range with defined but not instantiated component is
445 accessed, clients will send a Layout Intent RPC to the MDT, and the MDT
446 would instantiate the objects of the components covering that range.
448 <para>Next, some commands for user to operate PFL files are introduced and
449 some examples of possible composite layout are illustrated as well.
450 Lustre provides commands
451 <literal>lfs setstripe</literal> and <literal>lfs migrate</literal> for
452 users to operate PFL files. <literal>lfs setstripe</literal> commands
453 are used to create PFL files, add or delete components to or from an
454 existing composite file; <literal>lfs migrate</literal> commands are used
455 to re-layout the data in existing files using the new layout parameter by
456 copying the data from the existing OST(s) to the new OST(s). Also,
457 as introduced in the previous sections, <literal>lfs getstripe</literal>
458 commands can be used to list the striping/component information for a
459 given PFL file, and <literal>lfs find</literal> commands can be used to
460 search the directory tree rooted at the given directory or file name for
461 the files that match the given PFL component parameters.</para>
462 <note><para>Using PFL files requires both the client and server to
463 understand the PFL file layout, which isn't available for Lustre 2.9 and
464 earlier. And it will not prevent older clients from accessing non-PFL
465 files in the filesystem.</para></note>
467 <title><literal>lfs setstripe</literal></title>
468 <para><literal>lfs setstripe</literal> commands are used to create PFL
469 files, add or delete components to or from an existing composite file.
470 (Suppose we have 8 OSTs in the following examples and stripe size is 1MB
473 <title>Create a PFL file</title>
474 <para><emphasis role="bold">Command</emphasis></para>
475 <screen>lfs setstripe
476 [--component-end|-E end1] [STRIPE_OPTIONS]
477 [--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable></screen>
478 <para>The <literal>-E</literal> option is used to specify the end offset
479 (in bytes or using a suffix “kMGTP”, e.g. 256M) of each component, and
480 it also indicates the following <literal>STRIPE_OPTIONS</literal> are
481 for this component. Each component defines the stripe pattern of the
482 file in the range of [start, end). The first component must start from
483 offset 0 and all components must be adjacent with each other, no holes
484 are allowed, so each extent will start at the end of previous extent.
485 A <literal>-1</literal> end offset or <literal>eof</literal> indicates
486 this is the last component extending to the end of file.</para>
487 <para><emphasis role="bold">Example</emphasis></para>
488 <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 -E -1 -c -1 -i 4 \
489 /mnt/testfs/create_comp</screen>
490 <para>This command creates a file with composite layout illustrated in
491 the following figure. The first component has 1 stripe and covers
492 [0, 4M), the second component has 4 stripes and covers [4M, 64M), and
493 the last component stripes start at OST4, cross over all available
494 OSTs and covers [64M, EOF).</para>
495 <figure xml:id="managinglayout.fig.pfl_create">
496 <title>Example: create a composite file</title>
499 <imagedata scalefit="1" depth="2.75in" align="center"
500 fileref="figures/PFL_createfile.png" />
503 <phrase>Example: create a composite file</phrase>
507 <para>The composite layout can be output by the following command:</para>
508 <screen>$ lfs getstripe /mnt/testfs/create_comp
509 /mnt/testfs/create_comp
514 lcme_extent.e_start: 0
515 lcme_extent.e_end: 4194304
517 lmm_stripe_size: 1048576
522 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
526 lcme_extent.e_start: 4194304
527 lcme_extent.e_end: 67108864
529 lmm_stripe_size: 1048576
532 lmm_stripe_offset: -1
535 lcme_extent.e_start: 67108864
536 lcme_extent.e_end: EOF
538 lmm_stripe_size: 1048576
541 lmm_stripe_offset: 4</screen>
542 <note><para>Only the first component’s OST objects of the PFL file are
543 instantiated when the layout is being set. Other instantiation is
544 delayed to later write/truncate operations.</para></note>
545 <para>If we write 128M data to this PFL file, the second and third
546 components will be instantiated:</para>
547 <screen>$ dd if=/dev/zero of=/mnt/testfs/create_comp bs=1M count=128
548 $ lfs getstripe /mnt/testfs/create_comp
549 /mnt/testfs/create_comp
554 lcme_extent.e_start: 0
555 lcme_extent.e_end: 4194304
557 lmm_stripe_size: 1048576
562 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
566 lcme_extent.e_start: 4194304
567 lcme_extent.e_end: 67108864
569 lmm_stripe_size: 1048576
574 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
575 - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
576 - 2: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }
577 - 3: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
581 lcme_extent.e_start: 67108864
582 lcme_extent.e_end: EOF
584 lmm_stripe_size: 1048576
589 - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x3:0x0] }
590 - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
591 - 2: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
592 - 3: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
593 - 4: { l_ost_idx: 0, l_fid: [0x100000000:0x3:0x0] }
594 - 5: { l_ost_idx: 1, l_fid: [0x100010000:0x3:0x0] }
595 - 6: { l_ost_idx: 2, l_fid: [0x100020000:0x3:0x0] }
596 - 7: { l_ost_idx: 3, l_fid: [0x100030000:0x3:0x0] }</screen>
599 <title>Add component(s) to an existing composite file</title>
600 <para><emphasis role="bold">Command</emphasis></para>
601 <screen>lfs setstripe --component-add
602 [--component-end|-E end1] [STRIPE_OPTIONS]
603 [--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable></screen>
604 <para>The option <literal>--component-add</literal> is used to add
605 components to an existing composite file. The extent start of
606 the first component to be added is equal to the extent end of last
607 component in the existing file, and all components to be added must
608 be adjacent with each other.</para>
609 <note><para>If the last existing component is specified by
610 <literal>-E -1</literal> or <literal>-E eof</literal>, which covers
611 to the end of the file, it must be deleted before a new one is added.
613 <para><emphasis role="bold">Example</emphasis></para>
614 <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 /mnt/testfs/add_comp
615 $ lfs setstripe --component-add -E -1 -c 4 -o 6-7,0,5 \
616 /mnt/testfs/add_comp</screen>
617 <para>This command adds a new component which starts from the end of
618 the last existing component to the end of file. The layout of this
619 example is illustrated in
620 <xref linkend="managinglayout.fig.pfl_addcomp"/>. The last component
621 stripes across 4 OSTs in sequence OST6, OST7, OST0 and OST5, covers
623 <figure xml:id="managinglayout.fig.pfl_addcomp">
624 <title>Example: add a component to an existing composite file</title>
627 <imagedata scalefit="1" depth="2.75in" align="center"
628 fileref="figures/PFL_addcomp.png" />
631 <phrase>Example: add a component to an existing composite file
636 <para>The layout can be printed out by the following command:</para>
637 <screen>$ lfs getstripe /mnt/testfs/add_comp
643 lcme_extent.e_start: 0
644 lcme_extent.e_end: 4194304
646 lmm_stripe_size: 1048576
651 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
655 lcme_extent.e_start: 4194304
656 lcme_extent.e_end: 67108864
658 lmm_stripe_size: 1048576
663 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
664 - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
665 - 2: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }
666 - 3: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
670 lcme_extent.e_start: 67108864
671 lcme_extent.e_end: EOF
673 lmm_stripe_size: 1048576
676 lmm_stripe_offset: -1</screen>
677 <para>The component ID "lcme_id" changes as layout generation
678 changes. It is not necessarily sequential and does not imply ordering
679 of individual components.</para>
680 <note><para>Similar to specifying a full-file composite layout at file
681 creation time, <literal>--component-add</literal> won't instantiate
682 OST objects, the instantiation is delayed to later write/truncate
683 operations. For example, after writing beyond the 64MB start of the
684 file's last component, the new component has had objects allocated:
686 <screen>$ lfs getstripe -I5 /mnt/testfs/add_comp
692 lcme_extent.e_start: 67108864
693 lcme_extent.e_end: EOF
695 lmm_stripe_size: 1048576
700 - 0: { l_ost_idx: 6, l_fid: [0x100060000:0x4:0x0] }
701 - 1: { l_ost_idx: 7, l_fid: [0x100070000:0x4:0x0] }
702 - 2: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
703 - 3: { l_ost_idx: 5, l_fid: [0x100050000:0x4:0x0] }</screen>
706 <title>Delete component(s) from an existing file</title>
707 <para><emphasis role="bold">Command</emphasis></para>
708 <screen>lfs setstripe --component-del
709 [--component-id|-I comp_id | --component-flags comp_flags]
710 <replaceable>filename</replaceable></screen>
711 <para>The option <literal>--component-del</literal> is used to remove
712 the component(s) specified by component ID or flags from an existing
713 file. This operation will result in any data stored in the deleted
714 component will be lost.</para>
715 <para>The ID specified by <literal>-I</literal> option is the numerical
716 unique ID of the component, which can be obtained by command
717 <literal>lfs getstripe -I</literal> command, and the flag specified by
718 <literal>--component-flags</literal> option is a certain type of
719 components, which can be obtained by command
720 <literal>lfs getstripe --component-flags</literal>. For now, we only
721 have two flags <literal>init</literal> and <literal>^init</literal>
722 for instantiated and un-instantiated components respectively.</para>
723 <note><para>Deletion must start with the last component because hole is
724 not allowed.</para></note>
725 <para><emphasis role="bold">Example</emphasis></para>
726 <screen>$ lfs getstripe -I /mnt/testfs/del_comp
730 $ lfs setstripe --component-del -I 5 /mnt/testfs/del_comp</screen>
731 <para>This example deletes the component with ID 5 from file
732 <literal>/mnt/testfs/del_comp</literal>. If we still use the last
733 example, the final result is illustrated in
734 <xref linkend="managinglayout.fig.pfl_delcomp"/>.</para>
735 <figure xml:id="managinglayout.fig.pfl_delcomp">
736 <title>Example: delete a component from an existing file</title>
739 <imagedata scalefit="1" depth="2.75in" align="center"
740 fileref="figures/PFL_delcomp.png" />
743 <phrase>Example: delete a component from an existing file</phrase>
747 <para>If you try to delete a non-last component, you will see the
748 following error:</para>
749 <screen>$ lfs setstripe -component-del -I 2 /mnt/testfs/del_comp
750 Delete component 0x2 from /mnt/testfs/del_comp failed. Invalid argument
751 error: setstripe: delete component of file '/mnt/testfs/del_comp' failed: Invalid argument</screen>
754 <title>Set default PFL layout to an existing directory</title>
755 <para>Similar to create a PFL file, you can set default PFL layout to
756 an existing directory. After that, all the files created will inherit
757 this layout by default.</para>
758 <para><emphasis role="bold">Command</emphasis></para>
759 <screen>lfs setstripe
760 [--component-end|-E end1] [STRIPE_OPTIONS]
761 [--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>dirname</replaceable></screen>
762 <para><emphasis role="bold">Example</emphasis></para>
764 $ mkdir /mnt/testfs/pfldir
765 $ lfs setstripe -E 256M -c 1 -E 16G -c 4 -E -1 -S 4M -c -1 /mnt/testfs/pfldir
767 <para>When you run <literal>lfs getstripe</literal>, you will see:
770 $ lfs getstripe /mnt/testfs/pfldir
776 lcme_extent.e_start: 0
777 lcme_extent.e_end: 268435456
778 stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
781 lcme_extent.e_start: 268435456
782 lcme_extent.e_end: 17179869184
783 stripe_count: 4 stripe_size: 1048576 stripe_offset: -1
786 lcme_extent.e_start: 17179869184
787 lcme_extent.e_end: EOF
788 stripe_count: -1 stripe_size: 4194304 stripe_offset: -1
790 <para>If you create a file under <literal>/mnt/testfs/pfldir</literal>,
791 the layout of that file will inherit the layout from its parent
794 $ touch /mnt/testfs/pfldir/pflfile
795 $ lfs getstripe /mnt/testfs/pfldir/pflfile
796 /mnt/testfs/pfldir/pflfile
801 lcme_extent.e_start: 0
802 lcme_extent.e_end: 268435456
804 lmm_stripe_size: 1048576
809 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0xa:0x0] }
813 lcme_extent.e_start: 268435456
814 lcme_extent.e_end: 17179869184
816 lmm_stripe_size: 1048576
819 lmm_stripe_offset: -1
823 lcme_extent.e_start: 17179869184
824 lcme_extent.e_end: EOF
826 lmm_stripe_size: 4194304
829 lmm_stripe_offset: -1
832 <literal>lfs setstripe --component-add/del</literal> can't be run
833 on a directory, because default layout in directory is likea config,
834 which can be arbitrarily changed by <literal>lfs setstripe</literal>,
835 while layout in file may have data (OST objects) attached. If you want
836 to delete default layout in a directory, run
837 <literal>lfs setstripe -d <replaceable>dirname</replaceable></literal>
838 to return the directory to the filesystem-wide defaults, like:
840 $ lfs setstripe -d /mnt/testfs/pfldir
841 $ lfs getstripe -d /mnt/testfs/pfldir
843 stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
844 /mnt/testfs/pfldir/commonfile
846 lmm_stripe_size: 1048576
850 obdidx objid objid group
857 <title><literal>lfs migrate</literal></title>
858 <para><literal>lfs migrate</literal> commands are used to re-layout the
859 data in the existing files with the new layout parameter by copying the
860 data from the existing OST(s) to the new OST(s).</para>
861 <para><emphasis role="bold">Command</emphasis></para>
862 <screen>lfs migrate [--component-end|-E comp_end] [STRIPE_OPTIONS] ...
863 <replaceable>filename</replaceable></screen>
864 <para>The difference between <literal>migrate</literal> and
865 <literal>setstripe</literal> is that <literal>migrate</literal> is to
866 re-layout the data in the existing files, while
867 <literal>setstripe</literal> is to create new files with the specified
869 <para><emphasis role="bold">Example</emphasis></para>
870 <para><emphasis role="bold">Case1. Migrate a normal one to a composite
871 layout</emphasis></para>
872 <screen>$ lfs setstripe -c 1 -S 128K /mnt/testfs/norm_to_2comp
873 $ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
874 $ lfs getstripe /mnt/testfs/norm_to_2comp --yaml
875 /mnt/testfs/norm_to_comp
877 lmm_stripe_size: 131072
883 l_fid: 0x100070000:0x2:0x0
884 $ lfs migrate -E 1M -S 512K -c 1 -E -1 -S 1M -c 2 \
885 /mnt/testfs/norm_to_2comp</screen>
886 <para>In this example, a 5MB size file with 1 stripe and 128K stripe size
887 is migrated to a composite layout file with 2 components, illustrated in
888 <xref linkend="managinglayout.fig.pfl_norm_to_comp"/>.</para>
889 <figure xml:id="managinglayout.fig.pfl_norm_to_comp">
890 <title>Example: migrate normal to composite</title>
893 <imagedata scalefit="1" depth="2.75in" align="center"
894 fileref="figures/PFL_norm_to_comp.png" />
897 <phrase>Example: migrate normal to composite</phrase>
901 <para>The stripe information after migration is like:</para>
902 <screen>$ lfs getstripe /mnt/testfs/norm_to_2comp
903 /mnt/testfs/norm_to_2comp
908 lcme_extent.e_start: 0
909 lcme_extent.e_end: 1048576
911 lmm_stripe_size: 524288
916 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
920 lcme_extent.e_start: 1048576
921 lcme_extent.e_end: EOF
923 lmm_stripe_size: 1048576
928 - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
929 - 1: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }</screen>
930 <para><emphasis role="bold">Case2. Migrate a composite layout to another
931 composite layout</emphasis></para>
932 <screen>$ lfs setstripe -E 1M -S 512K -c 1 -E -1 -S 1M -c 2 \
933 /mnt/testfs/2comp_to_3comp
934 $ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
935 $ lfs migrate -E 1M -S 1M -c 2 -E 4M -S 1M -c 2 -E -1 -S 3M -c 3 \
936 /mnt/testfs/2comp_to_3comp</screen>
937 <para>In this example, a composite layout file with 2 components is
938 migrated a composite layout file with 3 components. If we still use
939 the example in case1, the migration process is illustrated in
940 <xref linkend="managinglayout.fig.pfl_comp_to_comp"/>.</para>
941 <figure xml:id="managinglayout.fig.pfl_comp_to_comp">
942 <title>Example: migrate composite to composite</title>
945 <imagedata scalefit="1" depth="2.75in" align="center"
946 fileref="figures/PFL_comp_to_comp.png" />
949 <phrase>Example: migrate composite to composite</phrase>
953 <para>The stripe information is like:</para>
954 <screen>$ lfs getstripe /mnt/testfs/2comp_to_3comp
955 /mnt/testfs/2comp_to_3comp
960 lcme_extent.e_start: 0
961 lcme_extent.e_end: 1048576
963 lmm_stripe_size: 1048576
968 - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
969 - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
973 lcme_extent.e_start: 1048576
974 lcme_extent.e_end: 4194304
976 lmm_stripe_size: 1048576
981 - 0: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
982 - 1: { l_ost_idx: 7, l_fid: [0x100070000:0x3:0x0] }
986 lcme_extent.e_start: 4194304
987 lcme_extent.e_end: EOF
989 lmm_stripe_size: 3145728
994 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x3:0x0] }
995 - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
996 - 2: { l_ost_idx: 2, l_fid: [0x100020000:0x3:0x0] }</screen>
997 <para><emphasis role="bold">Case3. Migrate a composite layout to a
998 normal one</emphasis></para>
999 <screen>$ lfs migrate -E 1M -S 1M -c 2 -E 4M -S 1M -c 2 -E -1 -S 3M -c 3 \
1000 /mnt/testfs/3comp_to_norm
1001 $ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
1002 $ lfs migrate -c 2 -S 2M /mnt/testfs/3comp_to_normal</screen>
1003 <para>In this example, a composite file with 3 components is migrated to
1004 a normal file with 2 stripes and 2M stripe size. If we still use the
1005 example in Case2, the migration process is illustrated in
1006 <xref linkend="managinglayout.fig.pfl_comp_to_norm"/>.</para>
1007 <figure xml:id="managinglayout.fig.pfl_comp_to_norm">
1008 <title>Example: migrate composite to normal</title>
1011 <imagedata scalefit="1" depth="2.75in" align="center"
1012 fileref="figures/PFL_comp_to_norm.png" />
1015 <phrase>Example: migrate composite to normal</phrase>
1019 <para>The stripe information is like:</para>
1020 <screen>$ lfs getstripe /mnt/testfs/3comp_to_norm --yaml
1021 /mnt/testfs/3comp_to_norm
1023 lmm_stripe_size: 2097152
1026 lmm_stripe_offset: 4
1029 l_fid: 0x100040000:0x3:0x0
1031 l_fid: 0x100050000:0x3:0x0</screen>
1033 <section remap="h3">
1034 <title><literal>lfs getstripe</literal></title>
1035 <para><literal>lfs getstripe</literal> commands can be used to list the
1036 striping/component information for a given PFL file. Here, only those
1037 parameters new for PFL files are shown.</para>
1038 <para><emphasis role="bold">Command</emphasis></para>
1039 <screen>lfs getstripe
1040 [--component-id|-I [comp_id]]
1041 [--component-flags [comp_flags]]
1043 [--component-start [+-][N][kMGTPE]]
1044 [--component-end|-E [+-][N][kMGTPE]]
1045 <replaceable>dirname|filename</replaceable></screen>
1046 <para><emphasis role="bold">Example</emphasis></para>
1047 <para>Suppose we already have a composite file
1048 <literal>/mnt/testfs/3comp</literal>, created by the following
1050 <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 -E -1 -c -1 -i 4 \
1051 /mnt/testfs/3comp</screen>
1052 <para>And write some data</para>
1053 <screen>$ dd if=/dev/zero of=/mnt/testfs/3comp bs=1M count=5</screen>
1054 <para><emphasis role="bold">Case1. List component ID and its related
1055 information</emphasis></para>
1058 <para>List all the components ID</para>
1059 <screen>$ lfs getstripe -I /mnt/testfs/3comp
1065 <para>List the detailed striping information of component ID=2</para>
1066 <screen>$ lfs getstripe -I2 /mnt/testfs/3comp
1072 lcme_extent.e_start: 4194304
1073 lcme_extent.e_end: 67108864
1075 lmm_stripe_size: 1048576
1078 lmm_stripe_offset: 5
1080 - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
1081 - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
1082 - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
1083 - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
1086 <para>List the stripe offset and stripe count of component ID=2</para>
1087 <screen>$ lfs getstripe -I2 -i -c /mnt/testfs/3comp
1089 lmm_stripe_offset: 5</screen>
1092 <para><emphasis role="bold">Case2. List the component which contains the
1093 specified flag</emphasis></para>
1096 <para>List the flag of each component</para>
1097 <screen>$ lfs getstripe -component-flag -I /mnt/testfs/3comp
1103 lcme_flags: 0</screen>
1106 <para>List component(s) who is not instantiated</para>
1107 <screen>$ lfs getstripe --component-flags=^init /mnt/testfs/3comp
1113 lcme_extent.e_start: 67108864
1114 lcme_extent.e_end: EOF
1115 lmm_stripe_count: -1
1116 lmm_stripe_size: 1048576
1119 lmm_stripe_offset: 4</screen>
1122 <para><emphasis role="bold">Case3. List the total number of all the
1123 component(s)</emphasis></para>
1126 <para>List the total number of all the components</para>
1127 <screen>$ lfs getstripe --component-count /mnt/testfs/3comp
1131 <para><emphasis role="bold">Case4. List the component with the specified
1132 extent start or end positions</emphasis></para>
1135 <para>List the start position in bytes of each component</para>
1136 <screen>$ lfs getstripe --component-start /mnt/testfs/3comp
1142 <para>List the start position in bytes of component ID=3</para>
1143 <screen>$ lfs getstripe --component-start -I3 /mnt/testfs/3comp
1147 <para>List the component with start = 64M</para>
1148 <screen>$ lfs getstripe --component-start=64M /mnt/testfs/3comp
1154 lcme_extent.e_start: 67108864
1155 lcme_extent.e_end: EOF
1156 lmm_stripe_count: -1
1157 lmm_stripe_size: 1048576
1160 lmm_stripe_offset: 4</screen>
1163 <para>List the component(s) with start > 5M</para>
1164 <screen>$ lfs getstripe --component-start=+5M /mnt/testfs/3comp
1170 lcme_extent.e_start: 67108864
1171 lcme_extent.e_end: EOF
1172 lmm_stripe_count: -1
1173 lmm_stripe_size: 1048576
1176 lmm_stripe_offset: 4</screen>
1179 <para>List the component(s) with start < 5M</para>
1180 <screen>$ lfs getstripe --component-start=-5M /mnt/testfs/3comp
1186 lcme_extent.e_start: 0
1187 lcme_extent.e_end: 4194304
1189 lmm_stripe_size: 1048576
1192 lmm_stripe_offset: 4
1194 - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
1198 lcme_extent.e_start: 4194304
1199 lcme_extent.e_end: 67108864
1201 lmm_stripe_size: 1048576
1204 lmm_stripe_offset: 5
1206 - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
1207 - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
1208 - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
1209 - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
1212 <para>List the component(s) with start > 3M and end < 70M</para>
1213 <screen>$ lfs getstripe --component-start=+3M --component-end=-70M \
1220 lcme_extent.e_start: 4194304
1221 lcme_extent.e_end: 67108864
1223 lmm_stripe_size: 1048576
1226 lmm_stripe_offset: 5
1228 - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
1229 - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
1230 - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
1231 - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
1235 <section remap="h3">
1236 <title><literal>lfs find</literal></title>
1237 <para><literal>lfs find</literal> commands can be used to search the
1238 directory tree rooted at the given directory or file name for the files
1239 that match the given PFL component parameters. Here, only those
1240 parameters new for PFL files are shown. Their usages are similar to
1241 <literal>lfs getstripe</literal> commands.</para>
1242 <para><emphasis role="bold">Command</emphasis></para>
1243 <screen>lfs find <replaceable>directory|filename</replaceable>
1244 [[!] --component-count [+-=]<replaceable>comp_cnt</replaceable>]
1245 [[!] --component-start [+-=]<replaceable>N</replaceable>[kMGTPE]]
1246 [[!] --component-end|-E [+-=]<replaceable>N</replaceable>[kMGTPE]]
1247 [[!] --component-flags=<replaceable>comp_flags</replaceable>]</screen>
1248 <note><para>If you use <literal>--component-xxx</literal> options, only
1249 the composite files will be searched; but if you use
1250 <literal>! --component-xxx</literal> options, all the files will be
1251 searched.</para></note>
1252 <para><emphasis role="bold">Example</emphasis></para>
1253 <para>We use the following directory and composite files to show how
1254 <literal>lfs find</literal> works.</para>
1255 <screen>$ mkdir /mnt/testfs/testdir
1256 $ lfs setstripe -E 1M -E 10M -E eof /mnt/testfs/testdir/3comp
1257 $ lfs setstripe -E 4M -E 20M -E 30M -E eof /mnt/testfs/testdir/4comp
1258 $ mkdir -p /mnt/testfs/testdir/dir_3comp
1259 $ lfs setstripe -E 6M -E 30M -E eof /mnt/testfs/testdir/dir_3comp
1260 $ lfs setstripe -E 8M -E eof /mnt/testfs/testdir/dir_3comp/2comp
1261 $ lfs setstripe -c 1 /mnt/testfs/testdir/dir_3comp/commnfile</screen>
1262 <para><emphasis role="bold">Case1. Find the files that match the specified
1263 component count condition</emphasis></para>
1264 <para>Find the files under directory /mnt/testfs/testdir whose number of
1265 components is not equal to 3.</para>
1266 <screen>$ lfs find /mnt/testfs/testdir ! --component-count=3
1268 /mnt/testfs/testdir/4comp
1269 /mnt/testfs/testdir/dir_3comp/2comp
1270 /mnt/testfs/testdir/dir_3comp/commonfile</screen>
1271 <para><emphasis role="bold">Case2. Find the files/dirs that match the
1272 specified component start/end condition</emphasis></para>
1273 <para>Find the file(s) under directory /mnt/testfs/testdir with component
1274 start = 4M and end < 70M</para>
1275 <screen>$ lfs find /mnt/testfs/testdir --component-start=4M -E -30M
1276 /mnt/testfs/testdir/4comp</screen>
1277 <para><emphasis role="bold">Case3. Find the files/dirs that match the
1278 specified component flag condition</emphasis></para>
1279 <para>Find the file(s) under directory /mnt/testfs/testdir whose component
1280 flags contain <literal>init</literal></para>
1281 <screen>$ lfs find /mnt/testfs/testdir --component-flag=init
1282 /mnt/testfs/testdir/3comp
1283 /mnt/testfs/testdir/4comp
1284 /mnt/testfs/testdir/dir_3comp/2comp</screen>
1285 <note><para>Since <literal>lfs find</literal> uses
1286 "<literal>!</literal>" to do negative search, we don’t support
1287 flag <literal>^init</literal> here.</para></note>
1291 <section xml:id="striping.sel" condition='l2D'>
1293 <indexterm><primary>striping</primary><secondary>SEL</secondary>
1294 </indexterm>Self-Extending Layout (SEL)</title>
1295 <para>The Lustre Self-Extending Layout (SEL) feature is an extension of the
1296 <xref linkend="pfl"/> feature, which allows the MDS to change the defined
1297 PFL layout dynamically. With this feature, the MDS monitors the used space
1298 on OSTs and swaps the OSTs for the current file when they are low on space.
1299 This avoids <literal>ENOSPC</literal> problems for SEL files when
1300 applications are writing to them.</para>
1301 <para>Whereas PFL delays the instantiation of some components until an IO
1302 operation occurs on this region, SEL allows splitting such non-instantiated
1303 components in two parts: an “extendable” component and an “extension”
1304 component. The extendable component is a regular PFL component, covering
1305 just a part of the region, which is small originally. The extension (or SEL)
1306 component is a new component type which is always non-instantiated and
1307 unassigned, covering the other part of the region. When a write reaches this
1308 unassigned space, and the client calls the MDS to have it instantiated, the
1309 MDS makes a decision as to whether to grant additional space to the extendable
1310 component. The granted region moves from the head of the extension
1311 component to the tail of the extendable component, thus the extendable
1312 component grows and the SEL one is shortened. Therefore, it allows the file
1313 to continue on the same OSTs, or in the case where space is low on one of
1314 the current OSTs, to modify the layout to switch to a new component on new
1315 OSTs. In particular, it lets IO automatically spill over to a large HDD OST
1316 pool once a small SSD OST pool is getting low on space.</para>
1317 <para>The default extension policy modifies the layout in the following
1319 <orderedlist numeration="arabic">
1321 <para>Extension: continue on the same OSTs – used when not low on space
1322 on any of the OSTs of the current component; a particular extent is
1323 granted to the extendable component.</para>
1326 <para>Spill over: switch to next component OSTs – it is used only for
1327 not the last component when <emphasis>at least one</emphasis>
1328 of the current OSTs is low on space; the whole region of the SEL
1329 component moves to the next component and the SEL component is removed
1333 <para>Repeating: create a new component with the same layout but on
1334 free OSTs – it is used only for the last component when <emphasis>
1335 at least one</emphasis> of the current OSTs is low on space; a new
1336 component has the same layout but instantiated on different OSTs (from
1337 the same pool) which have enough space.</para>
1340 <para>Forced extension: continue with the current component OSTs despite
1341 the low on space condition – it is used only for the last component when
1342 a repeating attempt detected low on space condition as well - spillover
1343 is impossible and there is no sense in the repeating.</para>
1346 <note><para>The SEL feature does not require clients to understand the SEL
1347 format of already created files, only the MDS support is needed which is
1348 introduced in Lustre 2.13. However, old clients will have some limitations
1349 as the Lustre tools will not support it.</para></note>
1351 <title><literal>lfs setstripe</literal></title>
1352 <para>The <literal>lfs setstripe</literal> command is used to create files
1353 with composite layouts, as well as add or delete components to or from an
1354 existing file. It is extended to support SEL components.</para>
1356 <title>Create a SEL file</title>
1357 <para><emphasis role="bold">Command</emphasis></para>
1358 <screen>lfs setstripe
1359 [--component-end|-E end1] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable>
1362 --extension-size, --ext-size, -z <ext_size></screen>
1363 <para>The <literal>-z</literal> option is added to specify the size of
1364 the region which is granted to the extendable component on each
1365 iteration. While declaring any component, this option turns the declared
1366 component to a pair of components: extendable and extension ones.</para>
1367 <para><emphasis role="bold">Example</emphasis></para>
1368 <para>The following command creates 2 pairs of extendable and
1369 extension components:
1370 <screen># lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/file</screen>
1371 <figure xml:id="managinglayout.fig.sel_createfile">
1372 <title>Example: create a SEL file</title>
1375 <imagedata scalefit="1" depth="0.8in" align="center"
1376 fileref="figures/SEL_Createfile.png" />
1379 <phrase>Example: create a SEL file</phrase>
1384 <note><para>As usual, only the first PFL component is instantiated at
1385 the creation time, thus it is immediately extended to the extension
1386 size (64M for the first component), whereas the third component is left
1387 zero-length.</para></note>
1388 <screen># lfs getstripe /mnt/lustre/file
1396 lcme_extent.e_start: 0
1397 lcme_extent.e_end: 67108864
1399 lmm_stripe_size: 1048576
1402 lmm_stripe_offset: 0
1404 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1408 lcme_flags: extension
1409 lcme_extent.e_start: 67108864
1410 lcme_extent.e_end: 1073741824
1412 lmm_extension_size: 67108864
1415 lmm_stripe_offset: -1
1420 lcme_extent.e_start: 1073741824
1421 lcme_extent.e_end: 1073741824
1423 lmm_stripe_size: 1048576
1426 lmm_stripe_offset: -1
1430 lcme_flags: extension
1431 lcme_extent.e_start: 1073741824
1432 lcme_extent.e_end: EOF
1434 lmm_extension_size: 268435456
1437 lmm_stripe_offset: -1</screen>
1440 <title>Create a SEL layout template</title>
1441 <para>Similar to PFL, it is possible to set a SEL layout template to
1442 a directory. After that, all the files created under it will inherit this
1443 layout by default.</para>
1444 <screen># lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/dir
1445 # ./lustre/utils/lfs getstripe /mnt/lustre/dir
1453 lcme_extent.e_start: 0
1454 lcme_extent.e_end: 67108864
1455 stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1
1459 lcme_flags: extension
1460 lcme_extent.e_start: 67108864
1461 lcme_extent.e_end: 1073741824
1462 stripe_count: 1 extension_size: 67108864 pattern: raid0 stripe_offset: -1
1467 lcme_extent.e_start: 1073741824
1468 lcme_extent.e_end: 1073741824
1469 stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1
1473 lcme_flags: extension
1474 lcme_extent.e_start: 1073741824
1475 lcme_extent.e_end: EOF
1476 stripe_count: 1 extension_size: 268435456 pattern: raid0 stripe_offset: -1
1481 <title><literal>lfs getstripe</literal></title>
1482 <para><literal>lfs getstripe</literal> commands can be used to list the
1483 striping/component information for a given SEL file. Here, only those parameters
1484 new for SEL files are shown.</para>
1485 <para><emphasis role="bold">Command</emphasis></para>
1486 <screen>lfs getstripe
1487 [--extension-size|--ext-size|-z] <replaceable>filename</replaceable></screen>
1488 <para>The <literal>-z</literal> option is added to print the extension
1489 size in bytes. For composite files this is the extension size of the
1490 first extension component. If a particular component is identified by
1491 other options (<literal>--component-id, --component-start</literal>,
1492 etc...), this component extension size is printed.</para>
1493 <para><emphasis role="bold">Example 1: List a SEL component information
1495 <para>Suppose we already have a composite file
1496 <literal>/mnt/lustre/file</literal>, created by the following command:</para>
1497 <screen># lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/file</screen>
1498 <para>The 2nd component could be listed with the following command:</para>
1499 <screen># lfs getstripe -I2 /mnt/lustre/file
1506 lcme_flags: extension
1507 lcme_extent.e_start: 67108864
1508 lcme_extent.e_end: 1073741824
1510 lmm_extension_size: 67108864
1513 lmm_stripe_offset: -1
1515 <note><para>As you can see the SEL components are marked by the <literal>
1516 extension</literal> flag and <literal>lmm_extension_size</literal> field
1517 keeps the specified extension size.</para></note>
1518 <para><emphasis role="bold">Example 2: List the extension size</emphasis></para>
1519 <para>Having the same file as in the above example, the extension size of
1520 the second component could be listed with:</para>
1521 <screen># lfs getstripe -z -I2 /mnt/lustre/file
1523 <para><emphasis role="bold">Example 3: Extension</emphasis></para>
1524 <para>Having the same file as in the above example, suppose there is a
1525 write which crosses the end of the first component (64M), and then another
1526 write another write which crosses the end of the first component (128M) again,
1527 the layout changes as following:</para>
1528 <figure xml:id="managinglayout.fig.sel_extension">
1529 <title>Example: an extension of a SEL file</title>
1532 <imagedata scalefit="1" depth="3.5in" align="center"
1533 fileref="figures/SEL_extension.png" />
1536 <phrase>Example: an extension of a SEL file</phrase>
1540 <para>The layout can be printed out by the following command:</para>
1541 <screen># lfs getstripe /mnt/lustre/file
1549 lcme_extent.e_start: 0
1550 lcme_extent.e_end: 201326592
1552 lmm_stripe_size: 1048576
1555 lmm_stripe_offset: 0
1557 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1561 lcme_flags: extension
1562 lcme_extent.e_start: 201326592
1563 lcme_extent.e_end: 1073741824
1565 lmm_extension_size: 67108864
1568 lmm_stripe_offset: -1
1573 lcme_extent.e_start: 1073741824
1574 lcme_extent.e_end: 1073741824
1576 lmm_stripe_size: 1048576
1579 lmm_stripe_offset: -1
1583 lcme_flags: extension
1584 lcme_extent.e_start: 1073741824
1585 lcme_extent.e_end: EOF
1587 lmm_extension_size: 268435456
1590 lmm_stripe_offset: -1</screen>
1591 <para><emphasis role="bold">Example 4: Spillover</emphasis></para>
1592 <para>In case where <literal>OST0</literal> is low on space and an IO
1593 happens to a SEL component, a spillover happens: the full region of the
1594 SEL component is added to the next component, e.g. in the example above
1595 the next layout modification will look like:</para>
1596 <figure xml:id="managinglayout.fig.sel_spillover">
1597 <title>Example: a spillover in a SEL file</title>
1600 <imagedata scalefit="1" depth="2.25in" align="center"
1601 fileref="figures/SEL_spillover.png" />
1604 <phrase>Example: a spillover in a SEL file</phrase>
1608 <note><para>Despite the fact the third component was [1G, 1G] originally,
1609 while it is not instantiated, instead of getting extended backward, it is
1610 moved backward to the start of the previous SEL component (192M) and
1611 extended on its extension size (256M) from that position, thus it becomes
1612 <literal>[192M, 448M]</literal>.</para></note>
1613 <screen># lfs getstripe /mnt/lustre/file
1621 lcme_extent.e_start: 0
1622 lcme_extent.e_end: 201326592
1624 lmm_stripe_size: 1048576
1627 lmm_stripe_offset: 0
1629 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1634 lcme_extent.e_start: 201326592
1635 lcme_extent.e_end: 469762048
1637 lmm_stripe_size: 1048576
1640 lmm_stripe_offset: 1
1642 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x8:0x0] }
1646 lcme_flags: extension
1647 lcme_extent.e_start: 469762048
1648 lcme_extent.e_end: EOF
1650 lmm_extension_size: 268435456
1653 lmm_stripe_offset: -1</screen>
1654 <para><emphasis role="bold">Example 5: Repeating</emphasis></para>
1655 <para>Suppose in the example above, <literal>OST0</literal> got
1656 enough free space back but <literal>OST1</literal> is low on space,
1657 the following write to the last SEL component leads to a new component
1658 allocation before the SEL component, which repeats the previous
1659 component layout but instantiated on free OSTs:</para>
1660 <figure xml:id="managinglayout.fig.sel_repeat">
1661 <title>Example: repeat a SEL component</title>
1664 <imagedata scalefit="1" depth="2.25in" align="center"
1665 fileref="figures/SEL_repeating.png" />
1668 <phrase>Example: repeat a SEL component
1673 <screen># lfs getstripe /mnt/lustre/file
1681 lcme_extent.e_start: 0
1682 lcme_extent.e_end: 201326592
1684 lmm_stripe_size: 1048576
1687 lmm_stripe_offset: 0
1689 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1694 lcme_extent.e_start: 201326592
1695 lcme_extent.e_end: 469762048
1697 lmm_stripe_size: 1048576
1700 lmm_stripe_offset: 1
1702 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x8:0x0] }
1707 lcme_extent.e_start: 469762048
1708 lcme_extent.e_end: 738197504
1710 lmm_stripe_size: 1048576
1712 lmm_layout_gen: 65535
1713 lmm_stripe_offset: 0
1715 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6:0x0] }
1719 lcme_flags: extension
1720 lcme_extent.e_start: 738197504
1721 lcme_extent.e_end: EOF
1723 lmm_extension_size: 268435456
1726 lmm_stripe_offset: -1</screen>
1727 <para><emphasis role="bold">Example 6: Forced extension</emphasis></para>
1728 <para>Suppose in the example above, both <literal>OST0</literal> and
1729 <literal>OST1</literal> are low on space, the following write to the
1730 last SEL component will behave as an extension as there is no sense to
1732 <figure xml:id="managinglayout.fig.pfl_forced">
1733 <title>Example: forced extension in a SEL file</title>
1736 <imagedata scalefit="1" depth="2.25in" align="center"
1737 fileref="figures/SEL_forced.png" />
1740 <phrase>Example: forced extension in a SEL file.
1745 <screen># lfs getstripe /mnt/lustre/file
1753 lcme_extent.e_start: 0
1754 lcme_extent.e_end: 201326592
1756 lmm_stripe_size: 1048576
1759 lmm_stripe_offset: 0
1761 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1766 lcme_extent.e_start: 201326592
1767 lcme_extent.e_end: 469762048
1769 lmm_stripe_size: 1048576
1772 lmm_stripe_offset: 1
1774 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x8:0x0] }
1779 lcme_extent.e_start: 469762048
1780 lcme_extent.e_end: 1006632960
1782 lmm_stripe_size: 1048576
1784 lmm_layout_gen: 65535
1785 lmm_stripe_offset: 0
1787 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6:0x0] }
1791 lcme_flags: extension
1792 lcme_extent.e_start: 1006632960
1793 lcme_extent.e_end: EOF
1795 lmm_extension_size: 268435456
1798 lmm_stripe_offset: -1</screen>
1801 <title><literal>lfs find</literal></title>
1802 <para><literal>lfs find</literal> commands can be used to search for
1803 the files that match the given SEL component paremeters. Here, only
1804 those parameters new for the SEL files are shown.</para>
1806 [[!] --extension-size|--ext-size|-z [+-]ext-size[KMG]
1807 [[!] --component-flags=extension]</screen>
1808 <para>The <literal>-z</literal> option is added to specify the extension
1809 size to search for. The files which have any component with the
1810 extension size matched the given criteria are printed out. As always
1811 “+” and “-“ signs are allowed to specify the least and the most size.
1813 <para>A new <literal>extension</literal> component flag is added. Only
1814 files which have at least one SEL component are printed.</para>
1815 <note><para>The negative search for flags searches the files which
1816 <emphasis role="strong">have</emphasis> a non-SEL component (not files
1817 which <emphasis role="strong">do not have</emphasis> any SEL component).
1819 <para><emphasis role="bold">Example</emphasis></para>
1820 <screen># lfs setstripe --extension-size 64M -c 1 -E -1 /mnt/lustre/file
1822 # lfs find --comp-flags extension /mnt/lustre/*
1825 # lfs find ! --comp-flags extension /mnt/lustre/*
1828 # lfs find -z 64M /mnt/lustre/*
1831 # lfs find -z +64M /mnt/lustre/*
1833 # lfs find -z -64M /mnt/lustre/*
1835 # lfs find -z +63M /mnt/lustre/*
1838 # lfs find -z -65M /mnt/lustre/*
1841 # lfs find -z 65M /mnt/lustre/*
1843 # lfs find ! -z 64M /mnt/lustre/*
1845 # lfs find ! -z +64M /mnt/lustre/*
1848 # lfs find ! -z -64M /mnt/lustre/*
1851 # lfs find ! -z +63M /mnt/lustre/*
1853 # lfs find ! -z -65M /mnt/lustre/*
1855 # lfs find ! -z 65M /mnt/lustre/*
1856 /mnt/lustre/file</screen>
1860 <section xml:id="foreign_layout" condition='l2D'>
1862 <indexterm><primary>striping</primary><secondary>Foreign</secondary>
1863 </indexterm>Foreign Layout</title>
1864 <para>The Lustre Foreign Layout feature is an extension of both the
1865 LOV and LMV formats which allows the creation of empty files and directories
1866 with the necessary specifications to point to corresponding objects outside
1867 from Lustre namespace.</para>
1868 <para>The new LOV/LMV foreign internal format can be represented as:</para>
1869 <figure xml:id="managinglayout.fig.foreign_format">
1870 <title>LOV/LMV foreign format</title>
1873 <imagedata scalefit="1" width="100%"
1874 fileref="figures/Foreign_Format.png" />
1877 <phrase>LOV/LMV foreign format</phrase>
1882 <title><literal>lfs set[dir]stripe</literal></title>
1883 <para>The <literal>lfs set[dir]stripe</literal> commands are used to
1884 create files or directories with foreign layouts, by calling the
1885 corresponding API, itself invoking the appropriate ioctl().</para>
1887 <title>Create a Foreign file/dir</title>
1888 <para><emphasis role="bold">Command</emphasis></para>
1889 <screen>lfs set[dir]stripe \
1890 --foreign[=<foreign_type>] --xattr|-x <layout_string> \
1891 [--flags <hex_bitmask>] [--mode <mode_bits>] \
1892 <replaceable>{file,dir}name</replaceable></screen>
1893 <para>Both the <literal>--foreign</literal> and
1894 <literal>--xattr|-x</literal> options are mandatory.
1895 The <literal><foreign_type></literal> (default is "none", meaning
1896 no special behavior), and both <literal>--flags</literal> and
1897 <literal>--mode</literal> (default is 0666) options are optional.</para>
1898 <para><emphasis role="bold">Example</emphasis></para>
1899 <para>The following command creates a foreign file of "none" type and
1900 with "foo@bar" LOV content and specific mode and flags:
1901 <screen># lfs setstripe --foreign=none --flags=0xda08 --mode=0640 \
1902 --xattr=foo@bar /mnt/lustre/file</screen>
1903 <figure xml:id="managinglayout.fig.foreign_createfile">
1904 <title>Example: create a foreign file</title>
1907 <imagedata scalefit="1" width="100%" align="center"
1908 fileref="figures/Foreign_Createfile.png" />
1911 <phrase>Example: create a foreign file</phrase>
1919 <title><literal>lfs get[dir]stripe</literal></title>
1920 <para><literal>lfs get[dir]stripe</literal> commands can be used to
1921 retrieve foreign LOV/LMV informations and content.</para>
1922 <para><emphasis role="bold">Command</emphasis></para>
1923 <screen>lfs get[dir]stripe [-v] <replaceable>filename</replaceable></screen>
1924 <para><emphasis role="bold">List foreign layout information
1926 <para>Suppose we already have a foreign file
1927 <literal>/mnt/lustre/file</literal>, created by the following command:</para>
1928 <screen># lfs setstripe --foreign=none --flags=0xda08 --mode=0640 \
1929 --xattr=foo@bar /mnt/lustre/file</screen>
1930 <para>The full foreign layout informations can be listed using the
1931 following command:</para>
1932 <screen># lfs getstripe -v /mnt/lustre/file
1934 lfm_magic: 0x0BD70BD0
1937 lfm_flags: 0x0000DA08
1940 <note><para>As you can see the <literal>lfm_length</literal> field
1941 value is the characters number in the variable length
1942 <literal>lfm_value</literal> field.</para></note>
1945 <title><literal>lfs find</literal></title>
1946 <para><literal>lfs find</literal> commands can be used to search for
1947 all the foreign files/directories or those that match the given
1948 selection paremeters.</para>
1950 [[!] --foreign[=<foreign_type>]</screen>
1951 <para>The <literal>--foreign[=<foreign_type>]</literal> option
1952 has been added to specify that all [!,but not] files and/or directories
1953 with a foreign layout [and [!,but not] of
1954 <literal><foreign_type></literal>] will be retrieved.</para>
1955 <para><emphasis role="bold">Example</emphasis></para>
1956 <screen># lfs setstripe --foreign=none --xattr=foo@bar /mnt/lustre/file
1957 # touch /mnt/lustre/file2
1959 # lfs find --foreign /mnt/lustre/*
1962 # lfs find ! --foreign /mnt/lustre/*
1965 # lfs find --foreign=none /mnt/lustre/*
1966 /mnt/lustre/file</screen>
1970 <section xml:id="file_striping.managing_free_space">
1972 <primary>space</primary>
1973 <secondary>free space</secondary>
1974 </indexterm><indexterm>
1975 <primary>striping</primary>
1976 <secondary>round-robin algorithm</secondary>
1977 </indexterm><indexterm>
1978 <primary>striping</primary>
1979 <secondary>weighted algorithm</secondary>
1980 </indexterm><indexterm>
1981 <primary>round-robin algorithm</primary>
1982 </indexterm><indexterm>
1983 <primary>weighted algorithm</primary>
1984 </indexterm>Managing Free Space</title>
1985 <para>To optimize file system performance, the MDT assigns file stripes to OSTs based on two
1986 allocation algorithms. The <emphasis role="italic">round-robin</emphasis> allocator gives
1987 preference to location (spreading out stripes across OSSs to increase network bandwidth
1988 utilization) and the weighted allocator gives preference to available space (balancing loads
1989 across OSTs). Threshold and weighting factors for these two algorithms can be adjusted by the
1990 user. The MDT reserves 0.1 percent of total OST space and 32 inodes for each OST. The MDT
1991 stops object allocation for the OST if available space is less than reserved or the OST has
1992 fewer than 32 free inodes. The MDT starts object allocation when available space is twice
1993 as big as the reserved space and the OST has more than 64 free inodes. Note, clients
1994 could append existing files no matter what object allocation state is.</para>
1995 <para condition="l29"> The reserved space for each OST can be adjusted by the user. Use the
1996 <literal>lctl set_param</literal> command, for example the next command reserve 1GB space
1998 <screen>lctl set_param -P osp.*.reserved_mb_low=1024</screen></para>
1999 <para>This section describes how to check available free space on disks
2000 and how free space is allocated. It then describes how to set the
2001 threshold and weighting factors for the allocation algorithms.</para>
2002 <section xml:id="file_striping.checking_free_space">
2003 <title>Checking File System Free Space</title>
2004 <para>Free space is an important consideration in assigning file stripes.
2005 The <literal>lfs df</literal> command can be used to show available
2006 disk space on the mounted Lustre file system and space consumption per
2007 OST. If multiple Lustre file systems are mounted, a path may be
2008 specified, but is not required. Options to the <literal>lfs df</literal>
2009 command are shown below.</para>
2010 <informaltable frame="all">
2012 <colspec colname="c1" colwidth="50*"/>
2013 <colspec colname="c2" colwidth="50*"/>
2017 <para><emphasis role="bold">Option</emphasis></para>
2020 <para><emphasis role="bold">Description</emphasis></para>
2028 <literal>-h</literal>, <literal>--human-readable</literal>
2032 <para> Displays sizes in human readable format (for example: 1K,
2033 234M, 5G) using base-2 (binary) values (i.e. 1G = 1024M).</para>
2039 <literal>-H</literal>, <literal>--si</literal>
2043 <para>Like <literal>-h</literal>, this displays counts in human
2044 readable format, but using base-10 (decimal) values
2045 (i.e. 1G = 1000M).</para>
2050 <para> <literal role="bold">-i, --inodes</literal></para>
2053 <para> Lists inodes instead of block usage.</para>
2058 <para> <literal role="bold">-l, --lazy</literal></para>
2061 <para>Do not attempt to contact any OST or MDT not currently
2062 connected to the client. This avoids blocking the
2063 <literal>lfs df</literal> output if a target is offline or
2064 unreachable, and only returns the space on OSTs that can
2065 currently be accessed.</para>
2070 <para> <literal role="bold">-p, --pool</literal></para>
2073 <para>Limit the usage to report only OSTs that are in the
2074 specified <replaceable>pool</replaceable>. If multiple
2075 Lustre filesystems are mounted, list the OSTs in
2076 <replaceable>pool</replaceable> for each filesystem, or
2077 limit the display to only a pool for a specific filesystem
2078 if <replaceable>fsname.pool</replaceable> is given.
2079 Specifying both <replaceable>fsname</replaceable> and
2080 <replaceable>pool</replaceable> is equivalent to providing
2081 a specific mountpoint.
2088 <literal>-v</literal>, <literal>--verbose</literal>
2092 <para>Display verbose status of MDTs and OSTs. This may
2093 include one or more optional flags at the end of each line.
2101 <literal>lfs df</literal> may also report additional target status
2102 as the last column in the display, if there are issues with that target.
2103 Target states include:
2107 <literal>D</literal>: OST/MDT is <literal>Degraded</literal>.
2108 The target has a failed drive in the RAID device, or is
2109 undergoing RAID reconstruction. This state is marked on
2110 the server automatically for ZFS targets via
2111 <literal>zed</literal>, or a (user-supplied) script that
2112 monitors the target device and sets
2113 "<literal>lctl set_param obdfilter.<replaceable>target</replaceable>.degraded=1</literal>"
2114 on the OST. This target will be avoided for new
2115 allocations, but will still be used to read existing files
2116 located there or if there are not enough non-degraded OSTs
2117 to make up a widely-striped file.
2120 <literal>R</literal>: OST/MDT is <literal>Read-only</literal>.
2121 The target filesystem is marked read-only due to filesystem
2122 corruption detected by ldiskfs or ZFS. No modifications
2123 are allowed on this OST, and it needs to be unmounted and
2124 <literal>e2fsck</literal> or <literal>zpool scrub</literal>
2125 run to repair the underlying filesystem.
2128 <literal>N</literal>: OST/MDT is <literal>No-precreate</literal>.
2129 The target is configured to deny object precreation set by
2130 "<literal>lctl set_param obdfilter.<replaceable>target</replaceable>.no_precreate=1</literal>"
2131 parameter or the "<literal>-o no_precreate</literal>" mount option.
2132 This may be done to add an OST to the filesystem without allowing
2133 objects to be allocated on it yet, or for other reasons.
2136 <literal>S</literal>: OST/MDT is out of <literal>Space</literal>.
2137 The target filesystem has less than the minimum required
2138 free space and will not be used for new object allocations
2139 until it has more free space.
2142 <literal>I</literal>: OST/MDT is out of <literal>Inodes</literal>.
2143 The target filesystem has less than the minimum required
2144 free inodes and will not be used for new object allocations
2145 until it has more free inodes.
2148 <literal>f</literal>: OST/MDT is on <literal>flash</literal>.
2149 The target filesystem is using a flash (non-rotational)
2150 storage device. This is normally detected from the
2151 underlying Linux block device, but can be set manually
2152 with "<literal>lctl set_param osd-*.*.nonrotational=1</literal>
2153 on the respective OSTs. This lower-case status is only
2154 shown in conjunction with the <literal>-v</literal> option,
2155 since it is not an error condition.
2159 <para>The <literal>df -i</literal> and <literal>lfs df -i</literal>
2160 commands show the <emphasis role="italic">minimum</emphasis> number
2161 of inodes that can be created in the file system at the current time.
2162 If the total number of objects available across all of the OSTs is
2163 smaller than those available on the MDT(s), taking into account the
2164 default file striping, then <literal>df -i</literal> will also
2165 report a smaller number of inodes than could be created. Running
2166 <literal>lfs df -i</literal> will report the actual number of inodes
2167 that are free on each target.
2169 <para>For ZFS file systems, the number of inodes that can be created
2170 is dynamic and depends on the free space in the file system. The
2171 Free and Total inode counts reported for a ZFS file system are only
2172 an estimate based on the current usage for each target. The Used
2173 inode count is the actual number of inodes used by the file system.
2176 <para><emphasis role="bold">Examples</emphasis></para>
2177 <screen>client$ lfs df
2178 UUID 1K-blocks Used Available Use% Mounted on
2179 testfs-OST0000_UUID 9174328 1020024 8154304 11% /mnt/lustre[MDT:0]
2180 testfs-OST0000_UUID 94181368 56330708 37850660 59% /mnt/lustre[OST:0]
2181 testfs-OST0001_UUID 94181368 56385748 37795620 59% /mnt/lustre[OST:1]
2182 testfs-OST0002_UUID 94181368 54352012 39829356 57% /mnt/lustre[OST:2]
2183 filesystem summary: 282544104 167068468 39829356 57% /mnt/lustre
2185 [client1] $ lfs df -hv
2186 UUID bytes Used Available Use% Mounted on
2187 testfs-MDT0000_UUID 8.7G 996.1M 7.8G 11% /mnt/lustre[MDT:0]
2188 testfs-OST0000_UUID 89.8G 53.7G 36.1G 59% /mnt/lustre[OST:0] f
2189 testfs-OST0001_UUID 89.8G 53.8G 36.0G 59% /mnt/lustre[OST:1] f
2190 testfs-OST0002_UUID 89.8G 51.8G 38.0G 57% /mnt/lustre[OST:2] f
2191 filesystem summary: 269.5G 159.3G 110.1G 59% /mnt/lustre
2193 [client1] $ lfs df -iH
2194 UUID Inodes IUsed IFree IUse% Mounted on
2195 testfs-MDT0000_UUID 2.21M 41.9k 2.17M 1% /mnt/lustre[MDT:0]
2196 testfs-OST0000_UUID 737.3k 12.1k 725.1k 1% /mnt/lustre[OST:0]
2197 testfs-OST0001_UUID 737.3k 12.2k 725.0k 1% /mnt/lustre[OST:1]
2198 testfs-OST0002_UUID 737.3k 12.2k 725.0k 1% /mnt/lustre[OST:2]
2199 filesystem summary: 2.21M 41.9k 2.17M 1% /mnt/lustre[OST:2]
2202 <section remap="h3">
2204 <primary>striping</primary>
2205 <secondary>allocations</secondary>
2206 </indexterm> Stripe Allocation Methods</title>
2207 <para>Two stripe allocation methods are provided:</para>
2210 <para><emphasis role="bold">Round-robin allocator</emphasis> - When the OSTs have
2211 approximately the same amount of free space, the round-robin allocator alternates
2212 stripes between OSTs on different OSSs, so the OST used for stripe 0 of each file is
2213 evenly distributed among OSTs, regardless of the stripe count. In a simple example with
2214 eight OSTs numbered 0-7, objects would be allocated like this:</para>
2216 <screen>File 1: OST1, OST2, OST3, OST4
2217 File 2: OST5, OST6, OST7
2218 File 3: OST0, OST1, OST2, OST3, OST4, OST5
2219 File 4: OST6, OST7, OST0</screen>
2221 <para>Here are several more sample round-robin stripe orders (each letter represents a
2222 different OST on a single OSS):</para>
2223 <informaltable frame="none">
2225 <colspec colname="c1" colwidth="50*"/>
2226 <colspec colname="c2" colwidth="50*"/>
2230 <para> 3: AAA</para>
2233 <para> One 3-OST OSS</para>
2238 <para> 3x3: ABABAB</para>
2241 <para> Two 3-OST OSSs</para>
2246 <para> 3x4: BBABABA</para>
2249 <para> One 3-OST OSS (A) and one 4-OST OSS (B)</para>
2254 <para> 3x5: BBABBABA</para>
2257 <para> One 3-OST OSS (A) and one 5-OST OSS (B)</para>
2262 <para> 3x3x3: ABCABCABC</para>
2265 <para> Three 3-OST OSSs</para>
2273 <para><emphasis role="bold">Weighted allocator</emphasis> - When the free space difference
2274 between the OSTs becomes significant, the weighting algorithm is used to influence OST
2275 ordering based on size (amount of free space available on each OST) and location
2276 (stripes evenly distributed across OSTs). The weighted allocator fills the emptier OSTs
2277 faster, but uses a weighted random algorithm, so the OST with the most free space is not
2278 necessarily chosen each time.</para>
2281 <para>The allocation method is determined by the amount of free-space
2282 imbalance on the OSTs. When free space is relatively balanced across
2283 OSTs, the faster round-robin allocator is used, which maximizes network
2284 balancing. The weighted allocator is used when any two OSTs are out of
2285 balance by more than the specified threshold (17% by default). The
2286 threshold between the two allocation methods is defined by the
2287 <literal>qos_threshold_rr</literal> parameter. </para>
2288 <para>To temporarily set the <literal>qos_threshold_rr</literal> to
2289 <literal>25</literal>, enter the folowing on each MDS:
2290 <screen>mds# lctl set_param lod.<replaceable>fsname</replaceable>*.qos_threshold_rr=25</screen></para>
2292 <section remap="h3">
2294 <primary>space</primary>
2295 <secondary>location weighting</secondary>
2296 </indexterm>Adjusting the Weighting Between Free Space and Location</title>
2297 <para>The weighting priority used by the weighted allocator is set by the
2298 the <literal>qos_prio_free</literal> parameter.
2299 Increasing the value of <literal>qos_prio_free</literal> puts more
2300 weighting on the amount of free space available on each OST and less
2301 on how stripes are distributed across OSTs. The default value is
2302 <literal>91</literal> (percent). When the free space priority is set to
2303 <literal>100</literal> (percent), weighting is based entirely on free space and location
2304 is no longer used by the striping algorithm. </para>
2305 <para>To permanently change the allocator weighting to <literal>100</literal>, enter this command on the
2307 <screen>lctl conf_param <replaceable>fsname</replaceable>-MDT0000-*.lod.qos_prio_free=100</screen>
2310 <para>When <literal>qos_prio_free</literal> is set to <literal>100</literal>, a weighted
2311 random algorithm is still used to assign stripes, so, for example, if OST2 has twice as
2312 much free space as OST1, OST2 is twice as likely to be used, but it is not guaranteed to
2317 <section xml:id="wide_striping">
2319 <primary>striping</primary>
2320 <secondary>wide striping</secondary>
2321 </indexterm><indexterm>
2322 <primary>wide striping</primary>
2323 </indexterm>Lustre Striping Internals</title>
2324 <para>Individual files can only be striped over a finite number of OSTs,
2325 based on the maximum size of the attributes that can be stored on the MDT.
2326 If the MDT is ldiskfs-based without the <literal>ea_inode</literal>
2327 feature, a file can be striped across at most 160 OSTs. With ZFS-based
2328 MDTs, or if the <literal>ea_inode</literal> feature is enabled for an
2329 ldiskfs-based MDT, a file can be striped across up to 2000 OSTs.
2331 <para>Lustre inodes use an extended attribute to record on which OST each
2332 object is located, and the identifier each object on that OST. The size of
2333 the extended attribute is a function of the number of stripes.</para>
2334 <para>If using an ldiskfs-based MDT, the maximum number of OSTs over which
2335 files can be striped can been raised to 2000 by enabling the
2336 <literal>ea_inode</literal> feature on the MDT:
2337 <screen>tune2fs -O ea_inode /dev/<replaceable>mdtdev</replaceable></screen>
2339 <note condition='l2D'><para>Since Lustre 2.13 the
2340 <literal>ea_inode</literal> feature is enabled by default on all newly
2341 formatted ldiskfs MDT filesystems.</para></note>
2342 <note><para>The maximum stripe count for a single file does not limit the
2343 maximum number of OSTs that are in the filesystem as a whole, only the
2344 maximum possible size and maximum aggregate bandwidth for the file.
2349 vim:expandtab:shiftwidth=2:tabstop=8:
git://git.whamcloud.com / doc / manual.git / blob