1 <?xml version='1.0' encoding='UTF-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="managingstripingfreespace">
5 <title xml:id="managingstripingfreespace.title">Managing File Layout (Striping) and Free
7 <para>This chapter describes file layout (striping) and I/O options, and includes the following
11 <para><xref linkend="dbdoclet.50438209_79324"/></para>
14 <para><xref linkend="dbdoclet.50438209_48033"/></para>
17 <para><xref linkend="dbdoclet.50438209_78664"/></para>
20 <para><xref linkend="dbdoclet.50438209_44776"/></para>
23 <para><xref linkend="dbdoclet.50438209_10424"/></para>
26 <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/></para>
29 <section xml:id="dbdoclet.50438209_79324">
32 <primary>space</primary>
35 <primary>striping</primary>
36 <secondary>how it works</secondary>
39 <primary>striping</primary>
43 <primary>space</primary>
44 <secondary>striping</secondary>
45 </indexterm>How Lustre File System Striping Works</title>
46 <para>In a Lustre file system, the MDS allocates objects to OSTs using either a round-robin
47 algorithm or a weighted algorithm. When the amount of free space is well balanced (i.e., by
48 default, when the free space across OSTs differs by less than 17%), the round-robin algorithm
49 is used to select the next OST to which a stripe is to be written. Periodically, the MDS
50 adjusts the striping layout to eliminate some degenerated cases in which applications that
51 create very regular file layouts (striping patterns) preferentially use a particular OST in
53 <para> Normally the usage of OSTs is well balanced. However, if users create a small number of
54 exceptionally large files or incorrectly specify striping parameters, imbalanced OST usage may
55 result. When the free space across OSTs differs by more than a specific amount (17% by
56 default), the MDS then uses weighted random allocations with a preference for allocating
57 objects on OSTs with more free space. (This can reduce I/O performance until space usage is
58 rebalanced again.) For a more detailed description of how striping is allocated, see <xref
59 linkend="dbdoclet.50438209_10424"/>.</para>
60 <para>Files can only be striped over a finite number of OSTs, based on the
61 maximum size of the attributes that can be stored on the MDT. If the MDT
62 is ldiskfs-based without the <literal>ea_inode</literal> feature, a file
63 can be striped across at most 160 OSTs. With a ZFS-based MDT, or if the
64 <literal>ea_inode</literal> feature is enabled for an ldiskfs-based MDT,
65 a file can be striped across up to 2000 OSTs. For more information, see
66 <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/>.
69 <section xml:id="dbdoclet.50438209_48033">
71 <primary>file layout</primary>
72 <secondary>See striping</secondary>
73 </indexterm><indexterm>
74 <primary>striping</primary>
75 <secondary>considerations</secondary>
78 <primary>space</primary>
79 <secondary>considerations</secondary>
80 </indexterm> Lustre File Layout (Striping) Considerations</title>
81 <para>Whether you should set up file striping and what parameter values you select depends on
82 your needs. A good rule of thumb is to stripe over as few objects as will meet those needs and
84 <para>Some reasons for using striping include:</para>
87 <para><emphasis role="bold">Providing high-bandwidth access.</emphasis> Many applications
88 require high-bandwidth access to a single file, which may be more bandwidth than can be
89 provided by a single OSS. Examples are a scientific application that writes to a single
90 file from hundreds of nodes, or a binary executable that is loaded by many nodes when an
91 application starts.</para>
92 <para>In cases like these, a file can be striped over as many OSSs as it takes to achieve
93 the required peak aggregate bandwidth for that file. Striping across a larger number of
94 OSSs should only be used when the file size is very large and/or is accessed by many nodes
95 at a time. Currently, Lustre files can be striped across up to 2000 OSTs</para>
98 <para><emphasis role="bold">Improving performance when OSS bandwidth is exceeded.</emphasis>
99 Striping across many OSSs can improve performance if the aggregate client bandwidth
100 exceeds the server bandwidth and the application reads and writes data fast enough to take
101 advantage of the additional OSS bandwidth. The largest useful stripe count is bounded by
102 the I/O rate of the clients/jobs divided by the performance per OSS.</para>
105 <para condition="l2D"><emphasis role="bold">Matching stripes to I/O
106 pattern.</emphasis>When writing to a single file from multiple nodes,
107 having more than one client writing to a stripe can lead to issues
108 with lock exchange, where clients contend over writing to that stripe,
109 even if their I/Os do not overlap. This can be avoided if I/O can be
110 stripe aligned so that each stripe is accessed by only one client.
111 Since Lustre 2.13, the 'overstriping' feature is available, allowing more
112 than stripe per OST. This is particularly helpful for the case where
113 thread count exceeds OST count, making it possible to match stripe count
114 to thread count even in this case.</para>
117 <para><emphasis role="bold">Providing space for very large files.</emphasis> Striping is
118 useful when a single OST does not have enough free space to hold the entire file.</para>
121 <para>Some reasons to minimize or avoid striping:</para>
124 <para><emphasis role="bold">Increased overhead.</emphasis> Striping results in more locks
125 and extra network operations during common operations such as <literal>stat</literal> and
126 <literal>unlink</literal>. Even when these operations are performed in parallel, one
127 network operation takes less time than 100 operations.</para>
128 <para>Increased overhead also results from server contention. Consider a cluster with 100
129 clients and 100 OSSs, each with one OST. If each file has exactly one object and the load
130 is distributed evenly, there is no contention and the disks on each server can manage
131 sequential I/O. If each file has 100 objects, then the clients all compete with one
132 another for the attention of the servers, and the disks on each node seek in 100 different
133 directions resulting in needless contention.</para>
136 <para><emphasis role="bold">Increased risk.</emphasis> When files are striped across all
137 servers and one of the servers breaks down, a small part of each striped file is lost. By
138 comparison, if each file has exactly one stripe, fewer files are lost, but they are lost
139 in their entirety. Many users would prefer to lose some of their files entirely than all
140 of their files partially.</para>
144 <title><indexterm><primary>striping</primary><secondary>size</secondary></indexterm>
145 Choosing a Stripe Size</title>
146 <para>Choosing a stripe size is a balancing act, but reasonable defaults are described below.
147 The stripe size has no effect on a single-stripe file.</para>
150 <para><emphasis role="bold">The stripe size must be a multiple of the page
151 size.</emphasis> Lustre software tools enforce a multiple of 64 KB (the maximum page
152 size on ia64 and PPC64 nodes) so that users on platforms with smaller pages do not
153 accidentally create files that might cause problems for ia64 clients.</para>
156 <para><emphasis role="bold">The smallest recommended stripe size is 512 KB.</emphasis>
157 Although you can create files with a stripe size of 64 KB, the smallest practical stripe
158 size is 512 KB because the Lustre file system sends 1MB chunks over the network.
159 Choosing a smaller stripe size may result in inefficient I/O to the disks and reduced
163 <para><emphasis role="bold">A good stripe size for sequential I/O using high-speed
164 networks is between 1 MB and 4 MB.</emphasis> In most situations, stripe sizes larger
165 than 4 MB may result in longer lock hold times and contention during shared file
169 <para><emphasis role="bold">The maximum stripe size is 4 GB.</emphasis> Using a large
170 stripe size can improve performance when accessing very large files. It allows each
171 client to have exclusive access to its own part of a file. However, a large stripe size
172 can be counterproductive in cases where it does not match your I/O pattern.</para>
175 <para><emphasis role="bold">Choose a stripe pattern that takes into account the write
176 patterns of your application.</emphasis> Writes that cross an object boundary are
177 slightly less efficient than writes that go entirely to one server. If the file is
178 written in a consistent and aligned way, make the stripe size a multiple of the
179 <literal>write()</literal> size.</para>
184 <section xml:id="dbdoclet.50438209_78664">
186 <primary>striping</primary>
187 <secondary>configuration</secondary>
188 </indexterm>Setting the File Layout/Striping Configuration (<literal>lfs
189 setstripe</literal>)</title>
190 <para>Use the <literal>lfs setstripe</literal> command to create new files with a specific file layout (stripe pattern) configuration.</para>
191 <screen>lfs setstripe [--size|-s stripe_size] [--stripe-count|-c stripe_count] [--overstripe-count|-C stripe_count] \
192 [--index|-i start_ost] [--pool|-p pool_name] <replaceable>filename|dirname</replaceable> </screen>
193 <para><emphasis role="bold">
194 <literal>stripe_size</literal>
197 <para>The <literal>stripe_size</literal> indicates how much data to write to one OST before
198 moving to the next OST. The default <literal>stripe_size</literal> is 1 MB. Passing a
199 <literal>stripe_size</literal> of 0 causes the default stripe size to be used. Otherwise,
200 the <literal>stripe_size</literal> value must be a multiple of 64 KB.</para>
201 <para><emphasis role="bold">
202 <literal>stripe_count (--stripe-count, --overstripe-count)</literal>
205 <para>The <literal>stripe_count</literal> indicates how many stripes to use.
206 The default <literal>stripe_count</literal> value is 1. Setting
207 <literal>stripe_count</literal> to 0 causes the default stripe count to be
208 used. Setting <literal>stripe_count</literal> to -1 means stripe over all
209 available OSTs (full OSTs are skipped). When --overstripe-count is used,
210 per OST if necessary.</para>
211 <para><emphasis role="bold">
212 <literal>start_ost</literal>
215 <para>The start OST is the first OST to which files are written. The default value for
216 <literal>start_ost</literal> is -1, which allows the MDS to choose the starting index. This
217 setting is strongly recommended, as it allows space and load balancing to be done by the MDS
218 as needed. If the value of <literal>start_ost</literal> is set to a value other than -1, the
219 file starts on the specified OST index. OST index numbering starts at 0.</para>
221 <para>If the specified OST is inactive or in a degraded mode, the MDS will silently choose
222 another target.</para>
225 <para>If you pass a <literal>start_ost</literal> value of 0 and a
226 <literal>stripe_count</literal> value of <emphasis>1</emphasis>, all files are written to
227 OST 0, until space is exhausted. <emphasis role="italic">This is probably not what you meant
228 to do.</emphasis> If you only want to adjust the stripe count and keep the other
229 parameters at their default settings, do not specify any of the other parameters:</para>
230 <para><screen>client# lfs setstripe -c <replaceable>stripe_count</replaceable> <replaceable>filename</replaceable></screen></para>
232 <para><emphasis role="bold">
233 <literal>pool_name</literal>
236 <para>The <literal>pool_name</literal> specifies the OST pool to which the file will be written.
237 This allows limiting the OSTs used to a subset of all OSTs in the file system. For more
238 details about using OST pools, see <link xl:href="ManagingFileSystemIO.html#50438211_75549"
239 >Creating and Managing OST Pools</link>.</para>
241 <title>Specifying a File Layout (Striping Pattern) for a Single File</title>
242 <para>It is possible to specify the file layout when a new file is created using the command <literal>lfs setstripe</literal>. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an <literal>lfs setstripe</literal> command fails if the file already exists.</para>
243 <section xml:id="dbdoclet.50438209_60155">
244 <title>Setting the Stripe Size</title>
245 <para>The command to create a new file with a specified stripe size is similar to:</para>
246 <screen>[client]# lfs setstripe -s 4M /mnt/lustre/new_file</screen>
247 <para>This example command creates the new file <literal>/mnt/lustre/new_file</literal> with a stripe size of 4 MB.</para>
248 <para>Now, when the file is created, the new stripe setting creates the file on a single OST with a stripe size of 4M:</para>
249 <screen> [client]# lfs getstripe /mnt/lustre/new_file
252 lmm_stripe_size: 4194304
256 obdidx objid objid group
257 1 690550 0xa8976 0 </screen>
258 <para>In this example, the stripe size is 4 MB.</para>
261 <title><indexterm><primary>striping</primary><secondary>count</secondary></indexterm>
262 Setting the Stripe Count</title>
263 <para>The command below creates a new file with a stripe count of <literal>-1</literal> to
264 specify striping over all available OSTs:</para>
265 <screen>[client]# lfs setstripe -c -1 /mnt/lustre/full_stripe</screen>
266 <para>The example below indicates that the file <literal>full_stripe</literal> is striped
267 over all six active OSTs in the configuration:</para>
268 <screen>[client]# lfs getstripe /mnt/lustre/full_stripe
269 /mnt/lustre/full_stripe
270 obdidx objid objid group
277 <para> This is in contrast to the output in <xref linkend="dbdoclet.50438209_60155"/>, which
278 shows only a single object for the file.</para>
283 <primary>striping</primary>
284 <secondary>per directory</secondary>
285 </indexterm>Setting the Striping Layout for a Directory</title>
286 <para>In a directory, the <literal>lfs setstripe</literal> command sets a default striping
287 configuration for files created in the directory. The usage is the same as <literal>lfs
288 setstripe</literal> for a regular file, except that the directory must exist prior to
289 setting the default striping configuration. If a file is created in a directory with a
290 default stripe configuration (without otherwise specifying striping), the Lustre file system
291 uses those striping parameters instead of the file system default for the new file.</para>
292 <para>To change the striping pattern for a sub-directory, create a directory with desired file
293 layout as described above. Sub-directories inherit the file layout of the root/parent
298 <primary>striping</primary>
299 <secondary>per file system</secondary>
300 </indexterm>Setting the Striping Layout for a File System</title>
301 <para>Setting the striping specification on the <literal>root</literal> directory determines
302 the striping for all new files created in the file system unless an overriding striping
303 specification takes precedence (such as a striping layout specified by the application, or
304 set using <literal>lfs setstripe</literal>, or specified for the parent directory).</para>
306 <para>The striping settings for a <literal>root</literal> directory are, by default, applied
307 to any new child directories created in the root directory, unless striping settings have
308 been specified for the child directory.</para>
313 <primary>striping</primary>
314 <secondary>on specific OST</secondary>
315 </indexterm>Creating a File on a Specific OST</title>
316 <para>You can use <literal>lfs setstripe</literal> to create a file on a specific OST. In the
317 following example, the file <literal>file1</literal> is created on the first OST (OST index
319 <screen>$ lfs setstripe --stripe-count 1 --index 0 file1
320 $ dd if=/dev/zero of=file1 count=1 bs=100M
324 $ lfs getstripe file1
327 lmm_stripe_size: 1048576
331 obdidx objid objid group
332 0 37364 0x91f4 0</screen>
335 <section xml:id="dbdoclet.50438209_44776">
336 <title><indexterm><primary>striping</primary><secondary>getting information</secondary></indexterm>Retrieving File Layout/Striping Information (<literal>getstripe</literal>)</title>
337 <para>The <literal>lfs getstripe</literal> command is used to display information that shows
338 over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along
339 with the OST index and object ID for each stripe in the file. For directories, the default
340 settings for files created in that directory are displayed.</para>
342 <title>Displaying the Current Stripe Size</title>
343 <para>To see the current stripe size for a Lustre file or directory, use the <literal>lfs
344 getstripe</literal> command. For example, to view information for a directory, enter a
345 command similar to:</para>
346 <screen>[client]# lfs getstripe /mnt/lustre </screen>
347 <para>This command produces output similar to:</para>
349 (Default) stripe_count: 1 stripe_size: 1M stripe_offset: -1</screen>
350 <para>In this example, the default stripe count is <literal>1</literal> (data blocks are
351 striped over a single OST), the default stripe size is 1 MB, and the objects are created
352 over all available OSTs.</para>
353 <para>To view information for a file, enter a command similar to:</para>
354 <screen>$ lfs getstripe /mnt/lustre/foo
357 lmm_stripe_size: 1048576
361 obdidx objid objid group
362 2 835487 m0xcbf9f 0 </screen>
363 <para>In this example, the file is located on <literal>obdidx 2</literal>, which corresponds
364 to the OST <literal>lustre-OST0002</literal>. To see which node is serving that OST, run:
365 <screen>$ lctl get_param osc.lustre-OST0002-osc.ost_conn_uuid
366 osc.lustre-OST0002-osc.ost_conn_uuid=192.168.20.1@tcp</screen></para>
369 <title>Inspecting the File Tree</title>
370 <para>To inspect an entire tree of files, use the <literal>lfs find</literal> command:</para>
371 <screen>lfs find [--recursive | -r] <replaceable>file|directory</replaceable> ...</screen>
375 <primary>striping</primary>
376 <secondary>remote directories</secondary>
377 </indexterm>Locating the MDT for a remote directory</title>
378 <para>Lustre can be configured with multiple MDTs in the same file
379 system. Each directory and file could be located on a different MDT.
380 To identify which MDT a given subdirectory is located, pass the
381 <literal>getstripe [--mdt-index|-M]</literal> parameter to
382 <literal>lfs</literal>. An example of this command is provided in
383 the section <xref linkend="lustremaint.rmremotedir"/>.</para>
386 <section xml:id="pfl" condition='l2A'>
388 <primary>striping</primary>
389 <secondary>PFL</secondary>
390 </indexterm>Progressive File Layout(PFL)</title>
391 <para>The Lustre Progressive File Layout (PFL) feature simplifies the use
392 of Lustre so that users can expect reasonable performance for a variety of
393 normal file IO patterns without the need to explicitly understand their IO
394 model or Lustre usage details in advance. In particular, users do not
395 necessarily need to know the size or concurrency of output files in
396 advance of their creation and explicitly specify an optimal layout for
397 each file in order to achieve good performance for both highly concurrent
398 shared-single-large-file IO or parallel IO to many smaller per-process
400 <para>The layout of a PFL file is stored on disk as <literal>composite
401 layout</literal>. A PFL file is essentially an array of
402 <literal>sub-layout components</literal>, with each sub-layout component
403 being a plain layout covering different and non-overlapped extents of
404 the file. For PFL files, the file layout is composed of a series of
405 components, therefore it's possible that there are some file extents are
406 not described by any components.</para>
407 <para>An example of how data blocks of PFL files are mapped to OST objects
408 of components is shown in the following PFL object mapping diagram:</para>
409 <figure xml:id="managinglayout.fig.pfl">
410 <title>PFL object mapping diagram</title>
413 <imagedata scalefit="1" width="100%"
414 fileref="figures/PFL_object_mapping_diagram.png" />
417 <phrase>PFL object mapping diagram</phrase>
421 <para>The PFL file in <xref linkend="managinglayout.fig.pfl"/> has 3
422 components and shows the mapping for the blocks of a 2055MB file.
423 The stripe size for the first two components is 1MB, while the stripe size
424 for the third component is 4MB. The stripe count is increasing for each
425 successive component. The first component only has two 1MB blocks and the
426 single object has a size of 2MB. The second component holds the next 254MB
427 of the file spread over 4 separate OST objects in RAID-0, each one will
428 have a size of 256MB / 4 objects = 64MB per object. Note the first two
429 objects <literal>obj 2,0</literal> and <literal>obj 2,1</literal>
430 have a 1MB hole at the start where the data is stored in the first
431 component. The final component holds the next 1800MB spread over 32 OST
432 objects. There is a 256MB / 32 = 8MB hole at the start each one for the
433 data stored in the first two components. Each object will be
434 2048MB / 32 objects = 64MB per object, except the
435 <literal>obj 3,0</literal> that holds an extra 4MB chunk and
436 <literal>obj 3,1</literal> that holds an extra 3MB chunk. If more data
437 was written to the file, only the objects in component 3 would increase
439 <para>When a file range with defined but not instantiated component is
440 accessed, clients will send a Layout Intent RPC to the MDT, and the MDT
441 would instantiate the objects of the components covering that range.
443 <para>Next, some commands for user to operate PFL files are introduced and
444 some examples of possible composite layout are illustrated as well.
445 Lustre provides commands
446 <literal>lfs setstripe</literal> and <literal>lfs migrate</literal> for
447 users to operate PFL files. <literal>lfs setstripe</literal> commands
448 are used to create PFL files, add or delete components to or from an
449 existing composite file; <literal>lfs migrate</literal> commands are used
450 to re-layout the data in existing files using the new layout parameter by
451 copying the data from the existing OST(s) to the new OST(s). Also,
452 as introduced in the previous sections, <literal>lfs getstripe</literal>
453 commands can be used to list the striping/component information for a
454 given PFL file, and <literal>lfs find</literal> commands can be used to
455 search the directory tree rooted at the given directory or file name for
456 the files that match the given PFL component parameters.</para>
457 <note><para>Using PFL files requires both the client and server to
458 understand the PFL file layout, which isn't available for Lustre 2.9 and
459 earlier. And it will not prevent older clients from accessing non-PFL
460 files in the filesystem.</para></note>
462 <title><literal>lfs setstripe</literal></title>
463 <para><literal>lfs setstripe</literal> commands are used to create PFL
464 files, add or delete components to or from an existing composite file.
465 (Suppose we have 8 OSTs in the following examples and stripe size is 1MB
468 <title>Create a PFL file</title>
469 <para><emphasis role="bold">Command</emphasis></para>
470 <screen>lfs setstripe
471 [--component-end|-E end1] [STRIPE_OPTIONS]
472 [--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable></screen>
473 <para>The <literal>-E</literal> option is used to specify the end offset
474 (in bytes or using a suffix “kMGTP”, e.g. 256M) of each component, and
475 it also indicates the following <literal>STRIPE_OPTIONS</literal> are
476 for this component. Each component defines the stripe pattern of the
477 file in the range of [start, end). The first component must start from
478 offset 0 and all components must be adjacent with each other, no holes
479 are allowed, so each extent will start at the end of previous extent.
480 A <literal>-1</literal> end offset or <literal>eof</literal> indicates
481 this is the last component extending to the end of file.</para>
482 <para><emphasis role="bold">Example</emphasis></para>
483 <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 -E -1 -c -1 -i 4 \
484 /mnt/testfs/create_comp</screen>
485 <para>This command creates a file with composite layout illustrated in
486 the following figure. The first component has 1 stripe and covers
487 [0, 4M), the second component has 4 stripes and covers [4M, 64M), and
488 the last component stripes start at OST4, cross over all available
489 OSTs and covers [64M, EOF).</para>
490 <figure xml:id="managinglayout.fig.pfl_create">
491 <title>Example: create a composite file</title>
494 <imagedata scalefit="1" depth="2.75in" align="center"
495 fileref="figures/PFL_createfile.png" />
498 <phrase>Example: create a composite file</phrase>
502 <para>The composite layout can be output by the following command:</para>
503 <screen>$ lfs getstripe /mnt/testfs/create_comp
504 /mnt/testfs/create_comp
509 lcme_extent.e_start: 0
510 lcme_extent.e_end: 4194304
512 lmm_stripe_size: 1048576
517 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
521 lcme_extent.e_start: 4194304
522 lcme_extent.e_end: 67108864
524 lmm_stripe_size: 1048576
527 lmm_stripe_offset: -1
530 lcme_extent.e_start: 67108864
531 lcme_extent.e_end: EOF
533 lmm_stripe_size: 1048576
536 lmm_stripe_offset: 4</screen>
537 <note><para>Only the first component’s OST objects of the PFL file are
538 instantiated when the layout is being set. Other instantiation is
539 delayed to later write/truncate operations.</para></note>
540 <para>If we write 128M data to this PFL file, the second and third
541 components will be instantiated:</para>
542 <screen>$ dd if=/dev/zero of=/mnt/testfs/create_comp bs=1M count=128
543 $ lfs getstripe /mnt/testfs/create_comp
544 /mnt/testfs/create_comp
549 lcme_extent.e_start: 0
550 lcme_extent.e_end: 4194304
552 lmm_stripe_size: 1048576
557 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
561 lcme_extent.e_start: 4194304
562 lcme_extent.e_end: 67108864
564 lmm_stripe_size: 1048576
569 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
570 - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
571 - 2: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }
572 - 3: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
576 lcme_extent.e_start: 67108864
577 lcme_extent.e_end: EOF
579 lmm_stripe_size: 1048576
584 - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x3:0x0] }
585 - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
586 - 2: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
587 - 3: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
588 - 4: { l_ost_idx: 0, l_fid: [0x100000000:0x3:0x0] }
589 - 5: { l_ost_idx: 1, l_fid: [0x100010000:0x3:0x0] }
590 - 6: { l_ost_idx: 2, l_fid: [0x100020000:0x3:0x0] }
591 - 7: { l_ost_idx: 3, l_fid: [0x100030000:0x3:0x0] }</screen>
594 <title>Add component(s) to an existing composite file</title>
595 <para><emphasis role="bold">Command</emphasis></para>
596 <screen>lfs setstripe --component-add
597 [--component-end|-E end1] [STRIPE_OPTIONS]
598 [--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable></screen>
599 <para>The option <literal>--component-add</literal> is used to add
600 components to an existing composite file. The extent start of
601 the first component to be added is equal to the extent end of last
602 component in the existing file, and all components to be added must
603 be adjacent with each other.</para>
604 <note><para>If the last existing component is specified by
605 <literal>-E -1</literal> or <literal>-E eof</literal>, which covers
606 to the end of the file, it must be deleted before a new one is added.
608 <para><emphasis role="bold">Example</emphasis></para>
609 <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 /mnt/testfs/add_comp
610 $ lfs setstripe --component-add -E -1 -c 4 -o 6-7,0,5 \
611 /mnt/testfs/add_comp</screen>
612 <para>This command adds a new component which starts from the end of
613 the last existing component to the end of file. The layout of this
614 example is illustrated in
615 <xref linkend="managinglayout.fig.pfl_addcomp"/>. The last component
616 stripes across 4 OSTs in sequence OST6, OST7, OST0 and OST5, covers
618 <figure xml:id="managinglayout.fig.pfl_addcomp">
619 <title>Example: add a component to an existing composite file</title>
622 <imagedata scalefit="1" depth="2.75in" align="center"
623 fileref="figures/PFL_addcomp.png" />
626 <phrase>Example: add a component to an existing composite file
631 <para>The layout can be printed out by the following command:</para>
632 <screen>$ lfs getstripe /mnt/testfs/add_comp
638 lcme_extent.e_start: 0
639 lcme_extent.e_end: 4194304
641 lmm_stripe_size: 1048576
646 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
650 lcme_extent.e_start: 4194304
651 lcme_extent.e_end: 67108864
653 lmm_stripe_size: 1048576
658 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
659 - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
660 - 2: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }
661 - 3: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
665 lcme_extent.e_start: 67108864
666 lcme_extent.e_end: EOF
668 lmm_stripe_size: 1048576
671 lmm_stripe_offset: -1</screen>
672 <para>The component ID "lcme_id" changes as layout generation
673 changes. It is not necessarily sequential and does not imply ordering
674 of individual components.</para>
675 <note><para>Similar to specifying a full-file composite layout at file
676 creation time, <literal>--component-add</literal> won't instantiate
677 OST objects, the instantiation is delayed to later write/truncate
678 operations. For example, after writing beyond the 64MB start of the
679 file's last component, the new component has had objects allocated:
681 <screen>$ lfs getstripe -I5 /mnt/testfs/add_comp
687 lcme_extent.e_start: 67108864
688 lcme_extent.e_end: EOF
690 lmm_stripe_size: 1048576
695 - 0: { l_ost_idx: 6, l_fid: [0x100060000:0x4:0x0] }
696 - 1: { l_ost_idx: 7, l_fid: [0x100070000:0x4:0x0] }
697 - 2: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
698 - 3: { l_ost_idx: 5, l_fid: [0x100050000:0x4:0x0] }</screen>
701 <title>Delete component(s) from an existing file</title>
702 <para><emphasis role="bold">Command</emphasis></para>
703 <screen>lfs setstripe --component-del
704 [--component-id|-I comp_id | --component-flags comp_flags]
705 <replaceable>filename</replaceable></screen>
706 <para>The option <literal>--component-del</literal> is used to remove
707 the component(s) specified by component ID or flags from an existing
708 file. This operation will result in any data stored in the deleted
709 component will be lost.</para>
710 <para>The ID specified by <literal>-I</literal> option is the numerical
711 unique ID of the component, which can be obtained by command
712 <literal>lfs getstripe -I</literal> command, and the flag specified by
713 <literal>--component-flags</literal> option is a certain type of
714 components, which can be obtained by command
715 <literal>lfs getstripe --component-flags</literal>. For now, we only
716 have two flags <literal>init</literal> and <literal>^init</literal>
717 for instantiated and un-instantiated components respectively.</para>
718 <note><para>Deletion must start with the last component because hole is
719 not allowed.</para></note>
720 <para><emphasis role="bold">Example</emphasis></para>
721 <screen>$ lfs getstripe -I /mnt/testfs/del_comp
725 $ lfs setstripe --component-del -I 5 /mnt/testfs/del_comp</screen>
726 <para>This example deletes the component with ID 5 from file
727 <literal>/mnt/testfs/del_comp</literal>. If we still use the last
728 example, the final result is illustrated in
729 <xref linkend="managinglayout.fig.pfl_delcomp"/>.</para>
730 <figure xml:id="managinglayout.fig.pfl_delcomp">
731 <title>Example: delete a component from an existing file</title>
734 <imagedata scalefit="1" depth="2.75in" align="center"
735 fileref="figures/PFL_delcomp.png" />
738 <phrase>Example: delete a component from an existing file</phrase>
742 <para>If you try to delete a non-last component, you will see the
743 following error:</para>
744 <screen>$ lfs setstripe -component-del -I 2 /mnt/testfs/del_comp
745 Delete component 0x2 from /mnt/testfs/del_comp failed. Invalid argument
746 error: setstripe: delete component of file '/mnt/testfs/del_comp' failed: Invalid argument</screen>
749 <title>Set default PFL layout to an existing directory</title>
750 <para>Similar to create a PFL file, you can set default PFL layout to
751 an existing directory. After that, all the files created will inherit
752 this layout by default.</para>
753 <para><emphasis role="bold">Command</emphasis></para>
754 <screen>lfs setstripe
755 [--component-end|-E end1] [STRIPE_OPTIONS]
756 [--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>dirname</replaceable></screen>
757 <para><emphasis role="bold">Example</emphasis></para>
759 $ mkdir /mnt/testfs/pfldir
760 $ lfs setstripe -E 256M -c 1 -E 16G -c 4 -E -1 -S 4M -c -1 /mnt/testfs/pfldir
762 <para>When you run <literal>lfs getstripe</literal>, you will see:
765 $ lfs getstripe /mnt/testfs/pfldir
771 lcme_extent.e_start: 0
772 lcme_extent.e_end: 268435456
773 stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
776 lcme_extent.e_start: 268435456
777 lcme_extent.e_end: 17179869184
778 stripe_count: 4 stripe_size: 1048576 stripe_offset: -1
781 lcme_extent.e_start: 17179869184
782 lcme_extent.e_end: EOF
783 stripe_count: -1 stripe_size: 4194304 stripe_offset: -1
785 <para>If you create a file under <literal>/mnt/testfs/pfldir</literal>,
786 the layout of that file will inherit the layout from its parent
789 $ touch /mnt/testfs/pfldir/pflfile
790 $ lfs getstripe /mnt/testfs/pfldir/pflfile
791 /mnt/testfs/pfldir/pflfile
796 lcme_extent.e_start: 0
797 lcme_extent.e_end: 268435456
799 lmm_stripe_size: 1048576
804 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0xa:0x0] }
808 lcme_extent.e_start: 268435456
809 lcme_extent.e_end: 17179869184
811 lmm_stripe_size: 1048576
814 lmm_stripe_offset: -1
818 lcme_extent.e_start: 17179869184
819 lcme_extent.e_end: EOF
821 lmm_stripe_size: 4194304
824 lmm_stripe_offset: -1
827 <literal>lfs setstripe --component-add/del</literal> can't be run
828 on a directory, because default layout in directory is likea config,
829 which can be arbitrarily changed by <literal>lfs setstripe</literal>,
830 while layout in file may have data (OST objects) attached. If you want
831 to delete default layout in a directory, run
832 <literal>lfs setstripe -d <replaceable>dirname</replaceable></literal>
833 to return the directory to the filesystem-wide defaults, like:
835 $ lfs setstripe -d /mnt/testfs/pfldir
836 $ lfs getstripe -d /mnt/testfs/pfldir
838 stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
839 /mnt/testfs/pfldir/commonfile
841 lmm_stripe_size: 1048576
845 obdidx objid objid group
852 <title><literal>lfs migrate</literal></title>
853 <para><literal>lfs migrate</literal> commands are used to re-layout the
854 data in the existing files with the new layout parameter by copying the
855 data from the existing OST(s) to the new OST(s).</para>
856 <para><emphasis role="bold">Command</emphasis></para>
857 <screen>lfs migrate [--component-end|-E comp_end] [STRIPE_OPTIONS] ...
858 <replaceable>filename</replaceable></screen>
859 <para>The difference between <literal>migrate</literal> and
860 <literal>setstripe</literal> is that <literal>migrate</literal> is to
861 re-layout the data in the existing files, while
862 <literal>setstripe</literal> is to create new files with the specified
864 <para><emphasis role="bold">Example</emphasis></para>
865 <para><emphasis role="bold">Case1. Migrate a normal one to a composite
866 layout</emphasis></para>
867 <screen>$ lfs setstripe -c 1 -S 128K /mnt/testfs/norm_to_2comp
868 $ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
869 $ lfs getstripe /mnt/testfs/norm_to_2comp --yaml
870 /mnt/testfs/norm_to_comp
872 lmm_stripe_size: 131072
878 l_fid: 0x100070000:0x2:0x0
879 $ lfs migrate -E 1M -S 512K -c 1 -E -1 -S 1M -c 2 \
880 /mnt/testfs/norm_to_2comp</screen>
881 <para>In this example, a 5MB size file with 1 stripe and 128K stripe size
882 is migrated to a composite layout file with 2 components, illustrated in
883 <xref linkend="managinglayout.fig.pfl_norm_to_comp"/>.</para>
884 <figure xml:id="managinglayout.fig.pfl_norm_to_comp">
885 <title>Example: migrate normal to composite</title>
888 <imagedata scalefit="1" depth="2.75in" align="center"
889 fileref="figures/PFL_norm_to_comp.png" />
892 <phrase>Example: migrate normal to composite</phrase>
896 <para>The stripe information after migration is like:</para>
897 <screen>$ lfs getstripe /mnt/testfs/norm_to_2comp
898 /mnt/testfs/norm_to_2comp
903 lcme_extent.e_start: 0
904 lcme_extent.e_end: 1048576
906 lmm_stripe_size: 524288
911 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
915 lcme_extent.e_start: 1048576
916 lcme_extent.e_end: EOF
918 lmm_stripe_size: 1048576
923 - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
924 - 1: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }</screen>
925 <para><emphasis role="bold">Case2. Migrate a composite layout to another
926 composite layout</emphasis></para>
927 <screen>$ lfs setstripe -E 1M -S 512K -c 1 -E -1 -S 1M -c 2 \
928 /mnt/testfs/2comp_to_3comp
929 $ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
930 $ lfs migrate -E 1M -S 1M -c 2 -E 4M -S 1M -c 2 -E -1 -S 3M -c 3 \
931 /mnt/testfs/2comp_to_3comp</screen>
932 <para>In this example, a composite layout file with 2 components is
933 migrated a composite layout file with 3 components. If we still use
934 the example in case1, the migration process is illustrated in
935 <xref linkend="managinglayout.fig.pfl_comp_to_comp"/>.</para>
936 <figure xml:id="managinglayout.fig.pfl_comp_to_comp">
937 <title>Example: migrate composite to composite</title>
940 <imagedata scalefit="1" depth="2.75in" align="center"
941 fileref="figures/PFL_comp_to_comp.png" />
944 <phrase>Example: migrate composite to composite</phrase>
948 <para>The stripe information is like:</para>
949 <screen>$ lfs getstripe /mnt/testfs/2comp_to_3comp
950 /mnt/testfs/2comp_to_3comp
955 lcme_extent.e_start: 0
956 lcme_extent.e_end: 1048576
958 lmm_stripe_size: 1048576
963 - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
964 - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
968 lcme_extent.e_start: 1048576
969 lcme_extent.e_end: 4194304
971 lmm_stripe_size: 1048576
976 - 0: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
977 - 1: { l_ost_idx: 7, l_fid: [0x100070000:0x3:0x0] }
981 lcme_extent.e_start: 4194304
982 lcme_extent.e_end: EOF
984 lmm_stripe_size: 3145728
989 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x3:0x0] }
990 - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
991 - 2: { l_ost_idx: 2, l_fid: [0x100020000:0x3:0x0] }</screen>
992 <para><emphasis role="bold">Case3. Migrate a composite layout to a
993 normal one</emphasis></para>
994 <screen>$ lfs migrate -E 1M -S 1M -c 2 -E 4M -S 1M -c 2 -E -1 -S 3M -c 3 \
995 /mnt/testfs/3comp_to_norm
996 $ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
997 $ lfs migrate -c 2 -S 2M /mnt/testfs/3comp_to_normal</screen>
998 <para>In this example, a composite file with 3 components is migrated to
999 a normal file with 2 stripes and 2M stripe size. If we still use the
1000 example in Case2, the migration process is illustrated in
1001 <xref linkend="managinglayout.fig.pfl_comp_to_norm"/>.</para>
1002 <figure xml:id="managinglayout.fig.pfl_comp_to_norm">
1003 <title>Example: migrate composite to normal</title>
1006 <imagedata scalefit="1" depth="2.75in" align="center"
1007 fileref="figures/PFL_comp_to_norm.png" />
1010 <phrase>Example: migrate composite to normal</phrase>
1014 <para>The stripe information is like:</para>
1015 <screen>$ lfs getstripe /mnt/testfs/3comp_to_norm --yaml
1016 /mnt/testfs/3comp_to_norm
1018 lmm_stripe_size: 2097152
1021 lmm_stripe_offset: 4
1024 l_fid: 0x100040000:0x3:0x0
1026 l_fid: 0x100050000:0x3:0x0</screen>
1028 <section remap="h3">
1029 <title><literal>lfs getstripe</literal></title>
1030 <para><literal>lfs getstripe</literal> commands can be used to list the
1031 striping/component information for a given PFL file. Here, only those
1032 parameters new for PFL files are shown.</para>
1033 <para><emphasis role="bold">Command</emphasis></para>
1034 <screen>lfs getstripe
1035 [--component-id|-I [comp_id]]
1036 [--component-flags [comp_flags]]
1038 [--component-start [+-][N][kMGTPE]]
1039 [--component-end|-E [+-][N][kMGTPE]]
1040 <replaceable>dirname|filename</replaceable></screen>
1041 <para><emphasis role="bold">Example</emphasis></para>
1042 <para>Suppose we already have a composite file
1043 <literal>/mnt/testfs/3comp</literal>, created by the following
1045 <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 -E -1 -c -1 -i 4 \
1046 /mnt/testfs/3comp</screen>
1047 <para>And write some data</para>
1048 <screen>$ dd if=/dev/zero of=/mnt/testfs/3comp bs=1M count=5</screen>
1049 <para><emphasis role="bold">Case1. List component ID and its related
1050 information</emphasis></para>
1053 <para>List all the components ID</para>
1054 <screen>$ lfs getstripe -I /mnt/testfs/3comp
1060 <para>List the detailed striping information of component ID=2</para>
1061 <screen>$ lfs getstripe -I2 /mnt/testfs/3comp
1067 lcme_extent.e_start: 4194304
1068 lcme_extent.e_end: 67108864
1070 lmm_stripe_size: 1048576
1073 lmm_stripe_offset: 5
1075 - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
1076 - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
1077 - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
1078 - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
1081 <para>List the stripe offset and stripe count of component ID=2</para>
1082 <screen>$ lfs getstripe -I2 -i -c /mnt/testfs/3comp
1084 lmm_stripe_offset: 5</screen>
1087 <para><emphasis role="bold">Case2. List the component which contains the
1088 specified flag</emphasis></para>
1091 <para>List the flag of each component</para>
1092 <screen>$ lfs getstripe -component-flag -I /mnt/testfs/3comp
1098 lcme_flags: 0</screen>
1101 <para>List component(s) who is not instantiated</para>
1102 <screen>$ lfs getstripe --component-flags=^init /mnt/testfs/3comp
1108 lcme_extent.e_start: 67108864
1109 lcme_extent.e_end: EOF
1110 lmm_stripe_count: -1
1111 lmm_stripe_size: 1048576
1114 lmm_stripe_offset: 4</screen>
1117 <para><emphasis role="bold">Case3. List the total number of all the
1118 component(s)</emphasis></para>
1121 <para>List the total number of all the components</para>
1122 <screen>$ lfs getstripe --component-count /mnt/testfs/3comp
1126 <para><emphasis role="bold">Case4. List the component with the specified
1127 extent start or end positions</emphasis></para>
1130 <para>List the start position in bytes of each component</para>
1131 <screen>$ lfs getstripe --component-start /mnt/testfs/3comp
1137 <para>List the start position in bytes of component ID=3</para>
1138 <screen>$ lfs getstripe --component-start -I3 /mnt/testfs/3comp
1142 <para>List the component with start = 64M</para>
1143 <screen>$ lfs getstripe --component-start=64M /mnt/testfs/3comp
1149 lcme_extent.e_start: 67108864
1150 lcme_extent.e_end: EOF
1151 lmm_stripe_count: -1
1152 lmm_stripe_size: 1048576
1155 lmm_stripe_offset: 4</screen>
1158 <para>List the component(s) with start > 5M</para>
1159 <screen>$ lfs getstripe --component-start=+5M /mnt/testfs/3comp
1165 lcme_extent.e_start: 67108864
1166 lcme_extent.e_end: EOF
1167 lmm_stripe_count: -1
1168 lmm_stripe_size: 1048576
1171 lmm_stripe_offset: 4</screen>
1174 <para>List the component(s) with start < 5M</para>
1175 <screen>$ lfs getstripe --component-start=-5M /mnt/testfs/3comp
1181 lcme_extent.e_start: 0
1182 lcme_extent.e_end: 4194304
1184 lmm_stripe_size: 1048576
1187 lmm_stripe_offset: 4
1189 - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
1193 lcme_extent.e_start: 4194304
1194 lcme_extent.e_end: 67108864
1196 lmm_stripe_size: 1048576
1199 lmm_stripe_offset: 5
1201 - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
1202 - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
1203 - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
1204 - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
1207 <para>List the component(s) with start > 3M and end < 70M</para>
1208 <screen>$ lfs getstripe --component-start=+3M --component-end=-70M \
1215 lcme_extent.e_start: 4194304
1216 lcme_extent.e_end: 67108864
1218 lmm_stripe_size: 1048576
1221 lmm_stripe_offset: 5
1223 - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
1224 - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
1225 - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
1226 - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
1230 <section remap="h3">
1231 <title><literal>lfs find</literal></title>
1232 <para><literal>lfs find</literal> commands can be used to search the
1233 directory tree rooted at the given directory or file name for the files
1234 that match the given PFL component parameters. Here, only those
1235 parameters new for PFL files are shown. Their usages are similar to
1236 <literal>lfs getstripe</literal> commands.</para>
1237 <para><emphasis role="bold">Command</emphasis></para>
1238 <screen>lfs find <replaceable>directory|filename</replaceable>
1239 [[!] --component-count [+-=]<replaceable>comp_cnt</replaceable>]
1240 [[!] --component-start [+-=]<replaceable>N</replaceable>[kMGTPE]]
1241 [[!] --component-end|-E [+-=]<replaceable>N</replaceable>[kMGTPE]]
1242 [[!] --component-flags=<replaceable>comp_flags</replaceable>]</screen>
1243 <note><para>If you use <literal>--component-xxx</literal> options, only
1244 the composite files will be searched; but if you use
1245 <literal>! --component-xxx</literal> options, all the files will be
1246 searched.</para></note>
1247 <para><emphasis role="bold">Example</emphasis></para>
1248 <para>We use the following directory and composite files to show how
1249 <literal>lfs find</literal> works.</para>
1250 <screen>$ mkdir /mnt/testfs/testdir
1251 $ lfs setstripe -E 1M -E 10M -E eof /mnt/testfs/testdir/3comp
1252 $ lfs setstripe -E 4M -E 20M -E 30M -E eof /mnt/testfs/testdir/4comp
1253 $ mkdir -p /mnt/testfs/testdir/dir_3comp
1254 $ lfs setstripe -E 6M -E 30M -E eof /mnt/testfs/testdir/dir_3comp
1255 $ lfs setstripe -E 8M -E eof /mnt/testfs/testdir/dir_3comp/2comp
1256 $ lfs setstripe -c 1 /mnt/testfs/testdir/dir_3comp/commnfile</screen>
1257 <para><emphasis role="bold">Case1. Find the files that match the specified
1258 component count condition</emphasis></para>
1259 <para>Find the files under directory /mnt/testfs/testdir whose number of
1260 components is not equal to 3.</para>
1261 <screen>$ lfs find /mnt/testfs/testdir ! --component-count=3
1263 /mnt/testfs/testdir/4comp
1264 /mnt/testfs/testdir/dir_3comp/2comp
1265 /mnt/testfs/testdir/dir_3comp/commonfile</screen>
1266 <para><emphasis role="bold">Case2. Find the files/dirs that match the
1267 specified component start/end condition</emphasis></para>
1268 <para>Find the file(s) under directory /mnt/testfs/testdir with component
1269 start = 4M and end < 70M</para>
1270 <screen>$ lfs find /mnt/testfs/testdir --component-start=4M -E -30M
1271 /mnt/testfs/testdir/4comp</screen>
1272 <para><emphasis role="bold">Case3. Find the files/dirs that match the
1273 specified component flag condition</emphasis></para>
1274 <para>Find the file(s) under directory /mnt/testfs/testdir whose component
1275 flags contain <literal>init</literal></para>
1276 <screen>$ lfs find /mnt/testfs/testdir --component-flag=init
1277 /mnt/testfs/testdir/3comp
1278 /mnt/testfs/testdir/4comp
1279 /mnt/testfs/testdir/dir_3comp/2comp</screen>
1280 <note><para>Since <literal>lfs find</literal> uses
1281 "<literal>!</literal>" to do negative search, we don’t support
1282 flag <literal>^init</literal> here.</para></note>
1286 <section xml:id="striping.sel" condition='l2D'>
1288 <indexterm><primary>striping</primary><secondary>SEL</secondary>
1289 </indexterm>Self-Extending Layout (SEL)</title>
1290 <para>The Lustre Self-Extending Layout (SEL) feature is an extension of the
1291 <xref linkend="pfl"/> feature, which allows the MDS to change the defined
1292 PFL layout dynamically. With this feature, the MDS monitors the used space
1293 on OSTs and swaps the OSTs for the current file when they are low on space.
1294 This avoids <literal>ENOSPC</literal> problems for SEL files when
1295 applications are writing to them.</para>
1296 <para>Whereas PFL delays the instantiation of some components until an IO
1297 operation occurs on this region, SEL allows splitting such non-instantiated
1298 components in two parts: an “extendable” component and an “extension”
1299 component. The extendable component is a regular PFL component, covering
1300 just a part of the region, which is small originally. The extension (or SEL)
1301 component is a new component type which is always non-instantiated and
1302 unassigned, covering the other part of the region. When a write reaches this
1303 unassigned space, and the client calls the MDS to have it instantiated, the
1304 MDS makes a decision as to whether to grant additional space to the extendable
1305 component. The granted region moves from the head of the extension
1306 component to the tail of the extendable component, thus the extendable
1307 component grows and the SEL one is shortened. Therefore, it allows the file
1308 to continue on the same OSTs, or in the case where space is low on one of
1309 the current OSTs, to modify the layout to switch to a new component on new
1310 OSTs. In particular, it lets IO automatically spill over to a large HDD OST
1311 pool once a small SSD OST pool is getting low on space.</para>
1312 <para>The default extension policy modifies the layout in the following
1314 <orderedlist numeration="arabic">
1316 <para>Extension: continue on the same OSTs – used when not low on space
1317 on any of the OSTs of the current component; a particular extent is
1318 granted to the extendable component.</para>
1321 <para>Spill over: switch to next component OSTs – it is used only for
1322 not the last component when <emphasis>at least one</emphasis>
1323 of the current OSTs is low on space; the whole region of the SEL
1324 component moves to the next component and the SEL component is removed
1328 <para>Repeating: create a new component with the same layout but on
1329 free OSTs – it is used only for the last component when <emphasis>
1330 at least one</emphasis> of the current OSTs is low on space; a new
1331 component has the same layout but instantiated on different OSTs (from
1332 the same pool) which have enough space.</para>
1335 <para>Forced extension: continue with the current component OSTs despite
1336 the low on space condition – it is used only for the last component when
1337 a repeating attempt detected low on space condition as well - spillover
1338 is impossible and there is no sense in the repeating.</para>
1341 <note><para>The SEL feature does not require clients to understand the SEL
1342 format of already created files, only the MDS support is needed which is
1343 introduced in Lustre 2.13. However, old clients will have some limitations
1344 as the Lustre tools will not support it.</para></note>
1346 <title><literal>lfs setstripe</literal></title>
1347 <para>The <literal>lfs setstripe</literal> command is used to create files
1348 with composite layouts, as well as add or delete components to or from an
1349 existing file. It is extended to support SEL components.</para>
1351 <title>Create a SEL file</title>
1352 <para><emphasis role="bold">Command</emphasis></para>
1353 <screen>lfs setstripe
1354 [--component-end|-E end1] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable>
1357 --extension-size, --ext-size, -z <ext_size></screen>
1358 <para>The <literal>-z</literal> option is added to specify the size of
1359 the region which is granted to the extendable component on each
1360 iteration. While declaring any component, this option turns the declared
1361 component to a pair of components: extendable and extension ones.</para>
1362 <para><emphasis role="bold">Example</emphasis></para>
1363 <para>The following command creates 2 pairs of extendable and
1364 extension components:
1365 <screen># lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/file</screen>
1366 <figure xml:id="managinglayout.fig.sel_createfile">
1367 <title>Example: create a SEL file</title>
1370 <imagedata scalefit="1" depth="0.8in" align="center"
1371 fileref="figures/SEL_Createfile.png" />
1374 <phrase>Example: create a SEL file</phrase>
1379 <note><para>As usual, only the first PFL component is instantiated at
1380 the creation time, thus it is immediately extended to the extension
1381 size (64M for the first component), whereas the third component is left
1382 zero-length.</para></note>
1383 <screen># lfs getstripe /mnt/lustre/file
1391 lcme_extent.e_start: 0
1392 lcme_extent.e_end: 67108864
1394 lmm_stripe_size: 1048576
1397 lmm_stripe_offset: 0
1399 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1403 lcme_flags: extension
1404 lcme_extent.e_start: 67108864
1405 lcme_extent.e_end: 1073741824
1407 lmm_extension_size: 67108864
1410 lmm_stripe_offset: -1
1415 lcme_extent.e_start: 1073741824
1416 lcme_extent.e_end: 1073741824
1418 lmm_stripe_size: 1048576
1421 lmm_stripe_offset: -1
1425 lcme_flags: extension
1426 lcme_extent.e_start: 1073741824
1427 lcme_extent.e_end: EOF
1429 lmm_extension_size: 268435456
1432 lmm_stripe_offset: -1</screen>
1435 <title>Create a SEL layout template</title>
1436 <para>Similar to PFL, it is possible to set a SEL layout template to
1437 a directory. After that, all the files created under it will inherit this
1438 layout by default.</para>
1439 <screen># lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/dir
1440 # ./lustre/utils/lfs getstripe /mnt/lustre/dir
1448 lcme_extent.e_start: 0
1449 lcme_extent.e_end: 67108864
1450 stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1
1454 lcme_flags: extension
1455 lcme_extent.e_start: 67108864
1456 lcme_extent.e_end: 1073741824
1457 stripe_count: 1 extension_size: 67108864 pattern: raid0 stripe_offset: -1
1462 lcme_extent.e_start: 1073741824
1463 lcme_extent.e_end: 1073741824
1464 stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1
1468 lcme_flags: extension
1469 lcme_extent.e_start: 1073741824
1470 lcme_extent.e_end: EOF
1471 stripe_count: 1 extension_size: 268435456 pattern: raid0 stripe_offset: -1
1476 <title><literal>lfs getstripe</literal></title>
1477 <para><literal>lfs getstripe</literal> commands can be used to list the
1478 striping/component information for a given SEL file. Here, only those parameters
1479 new for SEL files are shown.</para>
1480 <para><emphasis role="bold">Command</emphasis></para>
1481 <screen>lfs getstripe
1482 [--extension-size|--ext-size|-z] <replaceable>filename</replaceable></screen>
1483 <para>The <literal>-z</literal> option is added to print the extension
1484 size in bytes. For composite files this is the extension size of the
1485 first extension component. If a particular component is identified by
1486 other options (<literal>--component-id, --component-start</literal>,
1487 etc...), this component extension size is printed.</para>
1488 <para><emphasis role="bold">Example 1: List a SEL component information
1490 <para>Suppose we already have a composite file
1491 <literal>/mnt/lustre/file</literal>, created by the following command:</para>
1492 <screen># lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/file</screen>
1493 <para>The 2nd component could be listed with the following command:</para>
1494 <screen># lfs getstripe -I2 /mnt/lustre/file
1501 lcme_flags: extension
1502 lcme_extent.e_start: 67108864
1503 lcme_extent.e_end: 1073741824
1505 lmm_extension_size: 67108864
1508 lmm_stripe_offset: -1
1510 <note><para>As you can see the SEL components are marked by the <literal>
1511 extension</literal> flag and <literal>lmm_extension_size</literal> field
1512 keeps the specified extension size.</para></note>
1513 <para><emphasis role="bold">Example 2: List the extension size</emphasis></para>
1514 <para>Having the same file as in the above example, the extension size of
1515 the second component could be listed with:</para>
1516 <screen># lfs getstripe -z -I2 /mnt/lustre/file
1518 <para><emphasis role="bold">Example 3: Extension</emphasis></para>
1519 <para>Having the same file as in the above example, suppose there is a
1520 write which crosses the end of the first component (64M), and then another
1521 write another write which crosses the end of the first component (128M) again,
1522 the layout changes as following:</para>
1523 <figure xml:id="managinglayout.fig.sel_extension">
1524 <title>Example: an extension of a SEL file</title>
1527 <imagedata scalefit="1" depth="3.5in" align="center"
1528 fileref="figures/SEL_extension.png" />
1531 <phrase>Example: an extension of a SEL file</phrase>
1535 <para>The layout can be printed out by the following command:</para>
1536 <screen># lfs getstripe /mnt/lustre/file
1544 lcme_extent.e_start: 0
1545 lcme_extent.e_end: 201326592
1547 lmm_stripe_size: 1048576
1550 lmm_stripe_offset: 0
1552 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1556 lcme_flags: extension
1557 lcme_extent.e_start: 201326592
1558 lcme_extent.e_end: 1073741824
1560 lmm_extension_size: 67108864
1563 lmm_stripe_offset: -1
1568 lcme_extent.e_start: 1073741824
1569 lcme_extent.e_end: 1073741824
1571 lmm_stripe_size: 1048576
1574 lmm_stripe_offset: -1
1578 lcme_flags: extension
1579 lcme_extent.e_start: 1073741824
1580 lcme_extent.e_end: EOF
1582 lmm_extension_size: 268435456
1585 lmm_stripe_offset: -1</screen>
1586 <para><emphasis role="bold">Example 4: Spillover</emphasis></para>
1587 <para>In case where <literal>OST0</literal> is low on space and an IO
1588 happens to a SEL component, a spillover happens: the full region of the
1589 SEL component is added to the next component, e.g. in the example above
1590 the next layout modification will look like:</para>
1591 <figure xml:id="managinglayout.fig.sel_spillover">
1592 <title>Example: a spillover in a SEL file</title>
1595 <imagedata scalefit="1" depth="2.25in" align="center"
1596 fileref="figures/SEL_spillover.png" />
1599 <phrase>Example: a spillover in a SEL file</phrase>
1603 <note><para>Despite the fact the third component was [1G, 1G] originally,
1604 while it is not instantiated, instead of getting extended backward, it is
1605 moved backward to the start of the previous SEL component (192M) and
1606 extended on its extension size (256M) from that position, thus it becomes
1607 <literal>[192M, 448M]</literal>.</para></note>
1608 <screen># lfs getstripe /mnt/lustre/file
1616 lcme_extent.e_start: 0
1617 lcme_extent.e_end: 201326592
1619 lmm_stripe_size: 1048576
1622 lmm_stripe_offset: 0
1624 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1629 lcme_extent.e_start: 201326592
1630 lcme_extent.e_end: 469762048
1632 lmm_stripe_size: 1048576
1635 lmm_stripe_offset: 1
1637 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x8:0x0] }
1641 lcme_flags: extension
1642 lcme_extent.e_start: 469762048
1643 lcme_extent.e_end: EOF
1645 lmm_extension_size: 268435456
1648 lmm_stripe_offset: -1</screen>
1649 <para><emphasis role="bold">Example 5: Repeating</emphasis></para>
1650 <para>Suppose in the example above, <literal>OST0</literal> got
1651 enough free space back but <literal>OST1</literal> is low on space,
1652 the following write to the last SEL component leads to a new component
1653 allocation before the SEL component, which repeats the previous
1654 component layout but instantiated on free OSTs:</para>
1655 <figure xml:id="managinglayout.fig.sel_repeat">
1656 <title>Example: repeat a SEL component</title>
1659 <imagedata scalefit="1" depth="2.25in" align="center"
1660 fileref="figures/SEL_repeating.png" />
1663 <phrase>Example: repeat a SEL component
1668 <screen># lfs getstripe /mnt/lustre/file
1676 lcme_extent.e_start: 0
1677 lcme_extent.e_end: 201326592
1679 lmm_stripe_size: 1048576
1682 lmm_stripe_offset: 0
1684 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1689 lcme_extent.e_start: 201326592
1690 lcme_extent.e_end: 469762048
1692 lmm_stripe_size: 1048576
1695 lmm_stripe_offset: 1
1697 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x8:0x0] }
1702 lcme_extent.e_start: 469762048
1703 lcme_extent.e_end: 738197504
1705 lmm_stripe_size: 1048576
1707 lmm_layout_gen: 65535
1708 lmm_stripe_offset: 0
1710 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6:0x0] }
1714 lcme_flags: extension
1715 lcme_extent.e_start: 738197504
1716 lcme_extent.e_end: EOF
1718 lmm_extension_size: 268435456
1721 lmm_stripe_offset: -1</screen>
1722 <para><emphasis role="bold">Example 6: Forced extension</emphasis></para>
1723 <para>Suppose in the example above, both <literal>OST0</literal> and
1724 <literal>OST1</literal> are low on space, the following write to the
1725 last SEL component will behave as an extension as there is no sense to
1727 <figure xml:id="managinglayout.fig.pfl_forced">
1728 <title>Example: forced extension in a SEL file</title>
1731 <imagedata scalefit="1" depth="2.25in" align="center"
1732 fileref="figures/SEL_forced.png" />
1735 <phrase>Example: forced extension in a SEL file.
1740 <screen># lfs getstripe /mnt/lustre/file
1748 lcme_extent.e_start: 0
1749 lcme_extent.e_end: 201326592
1751 lmm_stripe_size: 1048576
1754 lmm_stripe_offset: 0
1756 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
1761 lcme_extent.e_start: 201326592
1762 lcme_extent.e_end: 469762048
1764 lmm_stripe_size: 1048576
1767 lmm_stripe_offset: 1
1769 - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x8:0x0] }
1774 lcme_extent.e_start: 469762048
1775 lcme_extent.e_end: 1006632960
1777 lmm_stripe_size: 1048576
1779 lmm_layout_gen: 65535
1780 lmm_stripe_offset: 0
1782 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6:0x0] }
1786 lcme_flags: extension
1787 lcme_extent.e_start: 1006632960
1788 lcme_extent.e_end: EOF
1790 lmm_extension_size: 268435456
1793 lmm_stripe_offset: -1</screen>
1796 <title><literal>lfs find</literal></title>
1797 <para><literal>lfs find</literal> commands can be used to search for
1798 the files that match the given SEL component paremeters. Here, only
1799 those parameters new for the SEL files are shown.</para>
1801 [[!] --extension-size|--ext-size|-z [+-]ext-size[KMG]
1802 [[!] --component-flags=extension]</screen>
1803 <para>The <literal>-z</literal> option is added to specify the extension
1804 size to search for. The files which have any component with the
1805 extension size matched the given criteria are printed out. As always
1806 “+” and “-“ signs are allowed to specify the least and the most size.
1808 <para>A new <literal>extension</literal> component flag is added. Only
1809 files which have at least one SEL component are printed.</para>
1810 <note><para>The negative search for flags searches the files which
1811 <emphasis role="strong">have</emphasis> a non-SEL component (not files
1812 which <emphasis role="strong">do not have</emphasis> any SEL component).
1814 <para><emphasis role="bold">Example</emphasis></para>
1815 <screen># lfs setstripe --extension-size 64M -c 1 -E -1 /mnt/lustre/file
1817 # lfs find --comp-flags extension /mnt/lustre/*
1820 # lfs find ! --comp-flags extension /mnt/lustre/*
1823 # lfs find -z 64M /mnt/lustre/*
1826 # lfs find -z +64M /mnt/lustre/*
1828 # lfs find -z -64M /mnt/lustre/*
1830 # lfs find -z +63M /mnt/lustre/*
1833 # lfs find -z -65M /mnt/lustre/*
1836 # lfs find -z 65M /mnt/lustre/*
1838 # lfs find ! -z 64M /mnt/lustre/*
1840 # lfs find ! -z +64M /mnt/lustre/*
1843 # lfs find ! -z -64M /mnt/lustre/*
1846 # lfs find ! -z +63M /mnt/lustre/*
1848 # lfs find ! -z -65M /mnt/lustre/*
1850 # lfs find ! -z 65M /mnt/lustre/*
1851 /mnt/lustre/file</screen>
1855 <section xml:id="foreign_layout" condition='l2D'>
1857 <indexterm><primary>striping</primary><secondary>Foreign</secondary>
1858 </indexterm>Foreign Layout</title>
1859 <para>The Lustre Foreign Layout feature is an extension of both the
1860 LOV and LMV formats which allows the creation of empty files and directories
1861 with the necessary specifications to point to corresponding objects outside
1862 from Lustre namespace.</para>
1863 <para>The new LOV/LMV foreign internal format can be represented as:</para>
1864 <figure xml:id="managinglayout.fig.foreign_format">
1865 <title>LOV/LMV foreign format</title>
1868 <imagedata scalefit="1" width="100%"
1869 fileref="figures/Foreign_Format.png" />
1872 <phrase>LOV/LMV foreign format</phrase>
1877 <title><literal>lfs set[dir]stripe</literal></title>
1878 <para>The <literal>lfs set[dir]stripe</literal> commands are used to
1879 create files or directories with foreign layouts, by calling the
1880 corresponding API, itself invoking the appropriate ioctl().</para>
1882 <title>Create a Foreign file/dir</title>
1883 <para><emphasis role="bold">Command</emphasis></para>
1884 <screen>lfs set[dir]stripe \
1885 --foreign[=<foreign_type>] --xattr|-x <layout_string> \
1886 [--flags <hex_bitmask>] [--mode <mode_bits>] \
1887 <replaceable>{file,dir}name</replaceable></screen>
1888 <para>Both the <literal>--foreign</literal> and
1889 <literal>--xattr|-x</literal> options are mandatory.
1890 The <literal><foreign_type></literal> (default is "none", meaning
1891 no special behavior), and both <literal>--flags</literal> and
1892 <literal>--mode</literal> (default is 0666) options are optional.</para>
1893 <para><emphasis role="bold">Example</emphasis></para>
1894 <para>The following command creates a foreign file of "none" type and
1895 with "foo@bar" LOV content and specific mode and flags:
1896 <screen># lfs setstripe --foreign=none --flags=0xda08 --mode=0640 \
1897 --xattr=foo@bar /mnt/lustre/file</screen>
1898 <figure xml:id="managinglayout.fig.foreign_createfile">
1899 <title>Example: create a foreign file</title>
1902 <imagedata scalefit="1" width="100%" align="center"
1903 fileref="figures/Foreign_Createfile.png" />
1906 <phrase>Example: create a foreign file</phrase>
1914 <title><literal>lfs get[dir]stripe</literal></title>
1915 <para><literal>lfs get[dir]stripe</literal> commands can be used to
1916 retrieve foreign LOV/LMV informations and content.</para>
1917 <para><emphasis role="bold">Command</emphasis></para>
1918 <screen>lfs get[dir]stripe [-v] <replaceable>filename</replaceable></screen>
1919 <para><emphasis role="bold">List foreign layout information
1921 <para>Suppose we already have a foreign file
1922 <literal>/mnt/lustre/file</literal>, created by the following command:</para>
1923 <screen># lfs setstripe --foreign=none --flags=0xda08 --mode=0640 \
1924 --xattr=foo@bar /mnt/lustre/file</screen>
1925 <para>The full foreign layout informations can be listed using the
1926 following command:</para>
1927 <screen># lfs getstripe -v /mnt/lustre/file
1929 lfm_magic: 0x0BD70BD0
1932 lfm_flags: 0x0000DA08
1935 <note><para>As you can see the <literal>lfm_length</literal> field
1936 value is the characters number in the variable length
1937 <literal>lfm_value</literal> field.</para></note>
1940 <title><literal>lfs find</literal></title>
1941 <para><literal>lfs find</literal> commands can be used to search for
1942 all the foreign files/directories or those that match the given
1943 selection paremeters.</para>
1945 [[!] --foreign[=<foreign_type>]</screen>
1946 <para>The <literal>--foreign[=<foreign_type>]</literal> option
1947 has been added to specify that all [!,but not] files and/or directories
1948 with a foreign layout [and [!,but not] of
1949 <literal><foreign_type></literal>] will be retrieved.</para>
1950 <para><emphasis role="bold">Example</emphasis></para>
1951 <screen># lfs setstripe --foreign=none --xattr=foo@bar /mnt/lustre/file
1952 # touch /mnt/lustre/file2
1954 # lfs find --foreign /mnt/lustre/*
1957 # lfs find ! --foreign /mnt/lustre/*
1960 # lfs find --foreign=none /mnt/lustre/*
1961 /mnt/lustre/file</screen>
1965 <section xml:id="dbdoclet.50438209_10424">
1967 <primary>space</primary>
1968 <secondary>free space</secondary>
1969 </indexterm><indexterm>
1970 <primary>striping</primary>
1971 <secondary>round-robin algorithm</secondary>
1972 </indexterm><indexterm>
1973 <primary>striping</primary>
1974 <secondary>weighted algorithm</secondary>
1975 </indexterm><indexterm>
1976 <primary>round-robin algorithm</primary>
1977 </indexterm><indexterm>
1978 <primary>weighted algorithm</primary>
1979 </indexterm>Managing Free Space</title>
1980 <para>To optimize file system performance, the MDT assigns file stripes to OSTs based on two
1981 allocation algorithms. The <emphasis role="italic">round-robin</emphasis> allocator gives
1982 preference to location (spreading out stripes across OSSs to increase network bandwidth
1983 utilization) and the weighted allocator gives preference to available space (balancing loads
1984 across OSTs). Threshold and weighting factors for these two algorithms can be adjusted by the
1985 user. The MDT reserves 0.1 percent of total OST space and 32 inodes for each OST. The MDT
1986 stops object allocation for the OST if available space is less than reserved or the OST has
1987 fewer than 32 free inodes. The MDT starts object allocation when available space is twice
1988 as big as the reserved space and the OST has more than 64 free inodes. Note, clients
1989 could append existing files no matter what object allocation state is.</para>
1990 <para condition="l29"> The reserved space for each OST can be adjusted by the user. Use the
1991 <literal>lctl set_param</literal> command, for example the next command reserve 1GB space
1993 <screen>lctl set_param -P osp.*.reserved_mb_low=1024</screen></para>
1994 <para>This section describes how to check available free space on disks and how free space is
1995 allocated. It then describes how to set the threshold and weighting factors for the allocation
1997 <section xml:id="dbdoclet.checking_free_space">
1998 <title>Checking File System Free Space</title>
1999 <para>Free space is an important consideration in assigning file stripes. The <literal>lfs
2000 df</literal> command can be used to show available disk space on the mounted Lustre file
2001 system and space consumption per OST. If multiple Lustre file systems are mounted, a path
2002 may be specified, but is not required. Options to the <literal>lfs df</literal> command are
2004 <informaltable frame="all">
2006 <colspec colname="c1" colwidth="50*"/>
2007 <colspec colname="c2" colwidth="50*"/>
2011 <para><emphasis role="bold">Option</emphasis></para>
2014 <para><emphasis role="bold">Description</emphasis></para>
2021 <para> <literal>-h</literal></para>
2024 <para> Displays sizes in human readable format (for example: 1K, 234M, 5G).</para>
2029 <para> <literal role="bold">-i, --inodes</literal></para>
2032 <para> Lists inodes instead of block usage.</para>
2039 <para>The <literal>df -i</literal> and <literal>lfs df -i</literal> commands show the
2040 <emphasis role="italic">minimum</emphasis> number of inodes that can be created in the
2041 file system at the current time. If the total number of objects available across all of
2042 the OSTs is smaller than those available on the MDT(s), taking into account the default
2043 file striping, then <literal>df -i</literal> will also report a smaller number of inodes
2044 than could be created. Running <literal>lfs df -i</literal> will report the actual number
2045 of inodes that are free on each target.</para>
2046 <para>For ZFS file systems, the number of inodes that can be created is dynamic and depends
2047 on the free space in the file system. The Free and Total inode counts reported for a ZFS
2048 file system are only an estimate based on the current usage for each target. The Used
2049 inode count is the actual number of inodes used by the file system.</para>
2051 <para><emphasis role="bold">Examples</emphasis></para>
2052 <screen>[client1] $ lfs df
2053 UUID 1K-blockS Used Available Use% Mounted on
2054 mds-lustre-0_UUID 9174328 1020024 8154304 11% /mnt/lustre[MDT:0]
2055 ost-lustre-0_UUID 94181368 56330708 37850660 59% /mnt/lustre[OST:0]
2056 ost-lustre-1_UUID 94181368 56385748 37795620 59% /mnt/lustre[OST:1]
2057 ost-lustre-2_UUID 94181368 54352012 39829356 57% /mnt/lustre[OST:2]
2058 filesystem summary: 282544104 167068468 39829356 57% /mnt/lustre
2060 [client1] $ lfs df -h
2061 UUID bytes Used Available Use% Mounted on
2062 mds-lustre-0_UUID 8.7G 996.1M 7.8G 11% /mnt/lustre[MDT:0]
2063 ost-lustre-0_UUID 89.8G 53.7G 36.1G 59% /mnt/lustre[OST:0]
2064 ost-lustre-1_UUID 89.8G 53.8G 36.0G 59% /mnt/lustre[OST:1]
2065 ost-lustre-2_UUID 89.8G 51.8G 38.0G 57% /mnt/lustre[OST:2]
2066 filesystem summary: 269.5G 159.3G 110.1G 59% /mnt/lustre
2068 [client1] $ lfs df -i
2069 UUID Inodes IUsed IFree IUse% Mounted on
2070 mds-lustre-0_UUID 2211572 41924 2169648 1% /mnt/lustre[MDT:0]
2071 ost-lustre-0_UUID 737280 12183 725097 1% /mnt/lustre[OST:0]
2072 ost-lustre-1_UUID 737280 12232 725048 1% /mnt/lustre[OST:1]
2073 ost-lustre-2_UUID 737280 12214 725066 1% /mnt/lustre[OST:2]
2074 filesystem summary: 2211572 41924 2169648 1% /mnt/lustre[OST:2]</screen>
2076 <section remap="h3">
2078 <primary>striping</primary>
2079 <secondary>allocations</secondary>
2080 </indexterm> Stripe Allocation Methods</title>
2081 <para>Two stripe allocation methods are provided:</para>
2084 <para><emphasis role="bold">Round-robin allocator</emphasis> - When the OSTs have
2085 approximately the same amount of free space, the round-robin allocator alternates
2086 stripes between OSTs on different OSSs, so the OST used for stripe 0 of each file is
2087 evenly distributed among OSTs, regardless of the stripe count. In a simple example with
2088 eight OSTs numbered 0-7, objects would be allocated like this:</para>
2090 <screen>File 1: OST1, OST2, OST3, OST4
2091 File 2: OST5, OST6, OST7
2092 File 3: OST0, OST1, OST2, OST3, OST4, OST5
2093 File 4: OST6, OST7, OST0</screen>
2095 <para>Here are several more sample round-robin stripe orders (each letter represents a
2096 different OST on a single OSS):</para>
2097 <informaltable frame="none">
2099 <colspec colname="c1" colwidth="50*"/>
2100 <colspec colname="c2" colwidth="50*"/>
2104 <para> 3: AAA</para>
2107 <para> One 3-OST OSS</para>
2112 <para> 3x3: ABABAB</para>
2115 <para> Two 3-OST OSSs</para>
2120 <para> 3x4: BBABABA</para>
2123 <para> One 3-OST OSS (A) and one 4-OST OSS (B)</para>
2128 <para> 3x5: BBABBABA</para>
2131 <para> One 3-OST OSS (A) and one 5-OST OSS (B)</para>
2136 <para> 3x3x3: ABCABCABC</para>
2139 <para> Three 3-OST OSSs</para>
2147 <para><emphasis role="bold">Weighted allocator</emphasis> - When the free space difference
2148 between the OSTs becomes significant, the weighting algorithm is used to influence OST
2149 ordering based on size (amount of free space available on each OST) and location
2150 (stripes evenly distributed across OSTs). The weighted allocator fills the emptier OSTs
2151 faster, but uses a weighted random algorithm, so the OST with the most free space is not
2152 necessarily chosen each time.</para>
2155 <para>The allocation method is determined by the amount of free-space
2156 imbalance on the OSTs. When free space is relatively balanced across
2157 OSTs, the faster round-robin allocator is used, which maximizes network
2158 balancing. The weighted allocator is used when any two OSTs are out of
2159 balance by more than the specified threshold (17% by default). The
2160 threshold between the two allocation methods is defined by the
2161 <literal>qos_threshold_rr</literal> parameter. </para>
2162 <para>To temporarily set the <literal>qos_threshold_rr</literal> to
2163 <literal>25</literal>, enter the folowing on each MDS:
2164 <screen>mds# lctl set_param lod.<replaceable>fsname</replaceable>*.qos_threshold_rr=25</screen></para>
2166 <section remap="h3">
2168 <primary>space</primary>
2169 <secondary>location weighting</secondary>
2170 </indexterm>Adjusting the Weighting Between Free Space and Location</title>
2171 <para>The weighting priority used by the weighted allocator is set by the
2172 the <literal>qos_prio_free</literal> parameter.
2173 Increasing the value of <literal>qos_prio_free</literal> puts more
2174 weighting on the amount of free space available on each OST and less
2175 on how stripes are distributed across OSTs. The default value is
2176 <literal>91</literal> (percent). When the free space priority is set to
2177 <literal>100</literal> (percent), weighting is based entirely on free space and location
2178 is no longer used by the striping algorithm. </para>
2179 <para>To permanently change the allocator weighting to <literal>100</literal>, enter this command on the
2181 <screen>lctl conf_param <replaceable>fsname</replaceable>-MDT0000-*.lod.qos_prio_free=100</screen>
2184 <para>When <literal>qos_prio_free</literal> is set to <literal>100</literal>, a weighted
2185 random algorithm is still used to assign stripes, so, for example, if OST2 has twice as
2186 much free space as OST1, OST2 is twice as likely to be used, but it is not guaranteed to
2191 <section xml:id="wide_striping">
2193 <primary>striping</primary>
2194 <secondary>wide striping</secondary>
2195 </indexterm><indexterm>
2196 <primary>wide striping</primary>
2197 </indexterm>Lustre Striping Internals</title>
2198 <para>Individual files can only be striped over a finite number of OSTs,
2199 based on the maximum size of the attributes that can be stored on the MDT.
2200 If the MDT is ldiskfs-based without the <literal>ea_inode</literal>
2201 feature, a file can be striped across at most 160 OSTs. With ZFS-based
2202 MDTs, or if the <literal>ea_inode</literal> feature is enabled for an
2203 ldiskfs-based MDT, a file can be striped across up to 2000 OSTs.
2205 <para>Lustre inodes use an extended attribute to record on which OST each
2206 object is located, and the identifier each object on that OST. The size of
2207 the extended attribute is a function of the number of stripes.</para>
2208 <para>If using an ldiskfs-based MDT, the maximum number of OSTs over which
2209 files can be striped can been raised to 2000 by enabling the
2210 <literal>ea_inode</literal> feature on the MDT:
2211 <screen>tune2fs -O ea_inode /dev/<replaceable>mdtdev</replaceable></screen>
2213 <note condition='l2D'><para>Since Lustre 2.13 the
2214 <literal>ea_inode</literal> feature is enabled by default on all newly
2215 formatted ldiskfs MDT filesystems.</para></note>
2216 <note><para>The maximum stripe count for a single file does not limit the
2217 maximum number of OSTs that are in the filesystem as a whole, only the
2218 maximum possible size and maximum aggregate bandwidth for the file.
2223 vim:expandtab:shiftwidth=2:tabstop=8: