-<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingstripingfreespace">
+<?xml version='1.0' encoding='UTF-8'?>
+<chapter xmlns="http://docbook.org/ns/docbook"
+ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
+ xml:id="managingstripingfreespace">
<title xml:id="managingstripingfreespace.title">Managing File Layout (Striping) and Free
Space</title>
<para>This chapter describes file layout (striping) and I/O options, and includes the following
sections:</para>
<itemizedlist>
<listitem>
- <para><xref linkend="dbdoclet.50438209_79324"/></para>
+ <para><xref linkend="file_striping.how_it_works"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438209_48033"/></para>
+ <para><xref linkend="file_striping.considerations"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438209_78664"/></para>
+ <para><xref linkend="file_striping.lfs_setstripe"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438209_44776"/></para>
+ <para><xref linkend="file_striping.lfs_getstripe"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438209_10424"/></para>
+ <para><xref linkend="file_striping.managing_free_space"/></para>
</listitem>
<listitem>
- <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_syy_gcl_qk"/></para>
+ <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/></para>
</listitem>
</itemizedlist>
- <section xml:id="dbdoclet.50438209_79324">
+ <section xml:id="file_striping.how_it_works">
<title>
<indexterm>
<primary>space</primary>
default), the MDS then uses weighted random allocations with a preference for allocating
objects on OSTs with more free space. (This can reduce I/O performance until space usage is
rebalanced again.) For a more detailed description of how striping is allocated, see <xref
- linkend="dbdoclet.50438209_10424"/>.</para>
- <para condition="l22">Files can only be striped over a finite number of OSTs. Prior to the
- Lustre 2.2 release, the maximum number of OSTs that a file could be striped across was limited
- to 160. As of the Lustre 2.2 release, the maximum number of OSTs is 2000. For more
- information, see <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_syy_gcl_qk"
- />.</para>
+ linkend="file_striping.managing_free_space"/>.</para>
+ <para>Files can only be striped over a finite number of OSTs, based on the
+ maximum size of the attributes that can be stored on the MDT. If the MDT
+ is ldiskfs-based without the <literal>ea_inode</literal> feature, a file
+ can be striped across at most 160 OSTs. With a ZFS-based MDT, or if the
+ <literal>ea_inode</literal> feature is enabled for an ldiskfs-based MDT
+ (the default since Lustre 2.13.0),
+ a file can be striped across up to 2000 OSTs. For more information, see
+ <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/>.
+ </para>
</section>
- <section xml:id="dbdoclet.50438209_48033">
+ <section xml:id="file_striping.considerations">
<title><indexterm>
<primary>file layout</primary>
<secondary>See striping</secondary>
<para>In cases like these, a file can be striped over as many OSSs as it takes to achieve
the required peak aggregate bandwidth for that file. Striping across a larger number of
OSSs should only be used when the file size is very large and/or is accessed by many nodes
- at a time. Currently, Lustre files can be striped across up to 2000 OSTs, the maximum
- stripe count for an <literal>ldiskfs</literal> file system.</para>
+ at a time. Currently, Lustre files can be striped across up to 2000 OSTs</para>
</listitem>
<listitem>
<para><emphasis role="bold">Improving performance when OSS bandwidth is exceeded.</emphasis>
the I/O rate of the clients/jobs divided by the performance per OSS.</para>
</listitem>
<listitem>
+ <para condition="l2D"><emphasis role="bold">Matching stripes to I/O
+ pattern.</emphasis>When writing to a single file from multiple nodes,
+ having more than one client writing to a stripe can lead to issues
+ with lock exchange, where clients contend over writing to that stripe,
+ even if their I/Os do not overlap. This can be avoided if I/O can be
+ stripe aligned so that each stripe is accessed by only one client.
+ Since Lustre 2.13, the 'overstriping' feature is available, allowing more
+ than one stripe per OST. This is particularly helpful for the case where
+ thread count exceeds OST count, making it possible to match stripe count
+ to thread count even in this case.</para>
+ </listitem>
+ <listitem>
<para><emphasis role="bold">Providing space for very large files.</emphasis> Striping is
useful when a single OST does not have enough free space to hold the entire file.</para>
</listitem>
</itemizedlist>
</section>
</section>
- <section xml:id="dbdoclet.50438209_78664">
+ <section xml:id="file_striping.lfs_setstripe">
<title><indexterm>
<primary>striping</primary>
<secondary>configuration</secondary>
</indexterm>Setting the File Layout/Striping Configuration (<literal>lfs
setstripe</literal>)</title>
<para>Use the <literal>lfs setstripe</literal> command to create new files with a specific file layout (stripe pattern) configuration.</para>
- <screen>lfs setstripe [--size|-s stripe_size] [--count|-c stripe_count] \
+ <screen>lfs setstripe [--size|-s stripe_size] [--stripe-count|-c stripe_count] [--overstripe-count|-C stripe_count] \
[--index|-i start_ost] [--pool|-p pool_name] <replaceable>filename|dirname</replaceable> </screen>
<para><emphasis role="bold">
<literal>stripe_size</literal>
<literal>stripe_size</literal> of 0 causes the default stripe size to be used. Otherwise,
the <literal>stripe_size</literal> value must be a multiple of 64 KB.</para>
<para><emphasis role="bold">
- <literal>stripe_count</literal>
+ <literal>stripe_count (--stripe-count, --overstripe-count)</literal>
</emphasis>
- </para>
- <para>The <literal>stripe_count</literal> indicates how many OSTs to use. The default <literal>stripe_count</literal> value is 1. Setting <literal>stripe_count</literal> to 0 causes the default stripe count to be used. Setting <literal>stripe_count</literal> to -1 means stripe over all available OSTs (full OSTs are skipped).</para>
+ </para>
+ <para>The <literal>stripe_count</literal> indicates how many stripes to use.
+ The default <literal>stripe_count</literal> value is 1. Setting
+ <literal>stripe_count</literal> to 0 causes the default stripe count to be
+ used. Setting <literal>stripe_count</literal> to -1 means stripe over all
+ available OSTs (full OSTs are skipped). When --overstripe-count is used,
+ per OST if necessary.</para>
<para><emphasis role="bold">
<literal>start_ost</literal>
</emphasis>
<literal>pool_name</literal>
</emphasis>
</para>
- <para>The <literal>pool_name</literal> specifies the OST pool to which the file will be written.
- This allows limiting the OSTs used to a subset of all OSTs in the file system. For more
- details about using OST pools, see <link xl:href="ManagingFileSystemIO.html#50438211_75549"
- >Creating and Managing OST Pools</link>.</para>
+ <para>The <literal>pool_name</literal> specifies the OST pool to which the
+ file will be written. This allows limiting the OSTs used to a subset of
+ all OSTs in the file system. For more details about using OST pools, see
+ <link xl:href="managingfilesystemio.managing_ost_pools">
+ Creating and Managing OST Pools
+ </link>.</para>
<section remap="h3">
<title>Specifying a File Layout (Striping Pattern) for a Single File</title>
<para>It is possible to specify the file layout when a new file is created using the command <literal>lfs setstripe</literal>. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an <literal>lfs setstripe</literal> command fails if the file already exists.</para>
- <section xml:id="dbdoclet.50438209_60155">
+ <section xml:id="file_striping.stripe_size">
<title>Setting the Stripe Size</title>
<para>The command to create a new file with a specified stripe size is similar to:</para>
<screen>[client]# lfs setstripe -s 4M /mnt/lustre/new_file</screen>
<para>This example command creates the new file <literal>/mnt/lustre/new_file</literal> with a stripe size of 4 MB.</para>
- <para>Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:</para>
+ <para>Now, when the file is created, the new stripe setting creates the file on a single OST with a stripe size of 4M:</para>
<screen> [client]# lfs getstripe /mnt/lustre/new_file
/mnt/lustre/4mb_file
lmm_stripe_count: 1
lmm_stripe_size: 4194304
+lmm_pattern: 1
+lmm_layout_gen: 0
lmm_stripe_offset: 1
obdidx objid objid group
1 690550 0xa8976 0 </screen>
<para>The command below creates a new file with a stripe count of <literal>-1</literal> to
specify striping over all available OSTs:</para>
<screen>[client]# lfs setstripe -c -1 /mnt/lustre/full_stripe</screen>
- <para>The example below indicates that the file <literal>full_stripe</literal> is striped
+ <para>The example below indicates that the file
+ <literal>full_stripe</literal> is striped
over all six active OSTs in the configuration:</para>
<screen>[client]# lfs getstripe /mnt/lustre/full_stripe
/mnt/lustre/full_stripe
3 5 0x5 0
4 4 0x4 0
5 2 0x2 0</screen>
- <para> This is in contrast to the output in <xref linkend="dbdoclet.50438209_60155"/>, which
- shows only a single object for the file.</para>
+ <para> This is in contrast to the output in
+ <xref linkend="file_striping.stripe_size"/>,
+ which shows only a single object for the file.</para>
</section>
</section>
<section remap="h3">
<section remap="h3">
<title><indexterm>
<primary>striping</primary>
+ <secondary>stripe count limit</secondary>
+ </indexterm>Per File System Stripe Count Limit</title>
+ <para>Sometime there are many OSTs in a filesystem, but it is not always
+ desirable to stripe file to across all OSTs, even if the given
+ <literal>stripe_count=-1</literal> (unlimited).
+ In this case, the per-filesystem tunable parameter
+ <literal>lod.*.max_stripecount</literal> can be used to limit the real
+ stripe count of file to a lower number than the OST count.
+ If <literal>lod.*.max_stripecount</literal> is not 0, and the file
+ <literal>stripe_count=-1</literal>, the real stripe count will be
+ the minimum of the OST count and <literal>max_stripecount</literal>. If
+ <literal>lod.*.max_stripecount=0</literal>, or an explicit stripe count
+ is given for the file, it is ignored.</para>
+ <para>To set <literal>max_stripecount</literal>, on all MDSes of
+ file system, run:
+ <screen>
+mgs# lctl set_param -P lod.$fsname-MDTxxxx-mdtlov.max_stripecount=<N>
+ </screen>
+ </para>
+ <para>To check <literal>max_stripecount</literal>, run:
+ <screen>
+mds# lctl get_param lod.$fsname-MDTxxxx-mdtlov.max_stripecount
+ </screen>
+ </para>
+ <para>To reset <literal>max_stripecount</literal>, run:
+ <screen>
+mgs# lctl set_param -P -d lod.$fsname-MDTxxxx-mdtlov.max_stripecount
+ </screen>
+ </para>
+ </section>
+
+ <section remap="h3">
+ <title><indexterm>
+ <primary>striping</primary>
<secondary>on specific OST</secondary>
</indexterm>Creating a File on a Specific OST</title>
<para>You can use <literal>lfs setstripe</literal> to create a file on a specific OST. In the
following example, the file <literal>file1</literal> is created on the first OST (OST index
is 0).</para>
- <screen>$ lfs setstripe --count 1 --index 0 file1
+ <screen>$ lfs setstripe --stripe-count 1 --index 0 file1
$ dd if=/dev/zero of=file1 count=1 bs=100M
1+0 records in
1+0 records out
$ lfs getstripe file1
/mnt/testfs/file1
-/mnt/testfs/file1
lmm_stripe_count: 1
lmm_stripe_size: 1048576
-lmm_stripe_offset: 0
- obdidx objid objid group
+lmm_pattern: 1
+lmm_layout_gen: 0
+lmm_stripe_offset: 0
+ obdidx objid objid group
0 37364 0x91f4 0</screen>
</section>
</section>
- <section xml:id="dbdoclet.50438209_44776">
+ <section xml:id="file_striping.lfs_getstripe">
<title><indexterm><primary>striping</primary><secondary>getting information</secondary></indexterm>Retrieving File Layout/Striping Information (<literal>getstripe</literal>)</title>
<para>The <literal>lfs getstripe</literal> command is used to display information that shows
over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along
command similar to:</para>
<screen>[client]# lfs getstripe /mnt/lustre </screen>
<para>This command produces output similar to:</para>
- <screen>/mnt/lustre
+ <screen>/mnt/lustre
(Default) stripe_count: 1 stripe_size: 1M stripe_offset: -1</screen>
<para>In this example, the default stripe count is <literal>1</literal> (data blocks are
striped over a single OST), the default stripe size is 1 MB, and the objects are created
<para>To view information for a file, enter a command similar to:</para>
<screen>$ lfs getstripe /mnt/lustre/foo
/mnt/lustre/foo
+lmm_stripe_count: 1
+lmm_stripe_size: 1048576
+lmm_pattern: 1
+lmm_layout_gen: 0
+lmm_stripe_offset: 0
obdidx objid objid group
2 835487 m0xcbf9f 0 </screen>
<para>In this example, the file is located on <literal>obdidx 2</literal>, which corresponds
<primary>striping</primary>
<secondary>remote directories</secondary>
</indexterm>Locating the MDT for a remote directory</title>
- <para condition="l24">Lustre release 2.4 can be configured with multiple MDTs in the same
- file system. Each sub-directory can have a different MDT. To identify on which MDT a given
- subdirectory is located, pass the <literal>getstripe -M</literal> parameters to
- <literal>lfs</literal>. An example of this command is provided in the section <xref
- linkend="dbdoclet.rmremotedir"/>.</para>
+ <para>Lustre can be configured with multiple MDTs in the same file
+ system. Each directory and file could be located on a different MDT.
+ To identify which MDT a given subdirectory is located, pass the
+ <literal>getstripe [--mdt-index|-M]</literal> parameter to
+ <literal>lfs</literal>. An example of this command is provided in
+ the section <xref linkend="lustremaint.rmremotedir"/>.</para>
</section>
</section>
- <section xml:id="dbdoclet.50438209_10424">
+ <section xml:id="pfl" condition='l2A'>
+ <title><indexterm>
+ <primary>striping</primary>
+ <secondary>PFL</secondary>
+ </indexterm>Progressive File Layout(PFL)</title>
+ <para>The Lustre Progressive File Layout (PFL) feature simplifies the use
+ of Lustre so that users can expect reasonable performance for a variety of
+ normal file IO patterns without the need to explicitly understand their IO
+ model or Lustre usage details in advance. In particular, users do not
+ necessarily need to know the size or concurrency of output files in
+ advance of their creation and explicitly specify an optimal layout for
+ each file in order to achieve good performance for both highly concurrent
+ shared-single-large-file IO or parallel IO to many smaller per-process
+ files. </para>
+ <para>The layout of a PFL file is stored on disk as <literal>composite
+ layout</literal>. A PFL file is essentially an array of
+ <literal>sub-layout components</literal>, with each sub-layout component
+ being a plain layout covering different and non-overlapped extents of
+ the file. For PFL files, the file layout is composed of a series of
+ components, therefore it's possible that there are some file extents are
+ not described by any components.</para>
+ <para>An example of how data blocks of PFL files are mapped to OST objects
+ of components is shown in the following PFL object mapping diagram:</para>
+ <figure xml:id="managinglayout.fig.pfl">
+ <title>PFL object mapping diagram</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" width="100%"
+ fileref="figures/PFL_object_mapping_diagram.png" />
+ </imageobject>
+ <textobject>
+ <phrase>PFL object mapping diagram</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The PFL file in <xref linkend="managinglayout.fig.pfl"/> has 3
+ components and shows the mapping for the blocks of a 2055MB file.
+ The stripe size for the first two components is 1MB, while the stripe size
+ for the third component is 4MB. The stripe count is increasing for each
+ successive component. The first component only has two 1MB blocks and the
+ single object has a size of 2MB. The second component holds the next 254MB
+ of the file spread over 4 separate OST objects in RAID-0, each one will
+ have a size of 256MB / 4 objects = 64MB per object. Note the first two
+ objects <literal>obj 2,0</literal> and <literal>obj 2,1</literal>
+ have a 1MB hole at the start where the data is stored in the first
+ component. The final component holds the next 1800MB spread over 32 OST
+ objects. There is a 256MB / 32 = 8MB hole at the start each one for the
+ data stored in the first two components. Each object will be
+ 2048MB / 32 objects = 64MB per object, except the
+ <literal>obj 3,0</literal> that holds an extra 4MB chunk and
+ <literal>obj 3,1</literal> that holds an extra 3MB chunk. If more data
+ was written to the file, only the objects in component 3 would increase
+ in size.</para>
+ <para>When a file range with defined but not instantiated component is
+ accessed, clients will send a Layout Intent RPC to the MDT, and the MDT
+ would instantiate the objects of the components covering that range.
+ </para>
+ <para>Next, some commands for user to operate PFL files are introduced and
+ some examples of possible composite layout are illustrated as well.
+ Lustre provides commands
+ <literal>lfs setstripe</literal> and <literal>lfs migrate</literal> for
+ users to operate PFL files. <literal>lfs setstripe</literal> commands
+ are used to create PFL files, add or delete components to or from an
+ existing composite file; <literal>lfs migrate</literal> commands are used
+ to re-layout the data in existing files using the new layout parameter by
+ copying the data from the existing OST(s) to the new OST(s). Also,
+ as introduced in the previous sections, <literal>lfs getstripe</literal>
+ commands can be used to list the striping/component information for a
+ given PFL file, and <literal>lfs find</literal> commands can be used to
+ search the directory tree rooted at the given directory or file name for
+ the files that match the given PFL component parameters.</para>
+ <note><para>Using PFL files requires both the client and server to
+ understand the PFL file layout, which isn't available for Lustre 2.9 and
+ earlier. And it will not prevent older clients from accessing non-PFL
+ files in the filesystem.</para></note>
+ <section remap="h3">
+ <title><literal>lfs setstripe</literal></title>
+ <para><literal>lfs setstripe</literal> commands are used to create PFL
+ files, add or delete components to or from an existing composite file.
+ (Suppose we have 8 OSTs in the following examples and stripe size is 1MB
+ by default.)</para>
+ <section remap="h4">
+ <title>Create a PFL file</title>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs setstripe
+[--component-end|-E end1] [STRIPE_OPTIONS]
+[--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable></screen>
+ <para>The <literal>-E</literal> option is used to specify the end offset
+ (in bytes or using a suffix “kMGTP”, e.g. 256M) of each component, and
+ it also indicates the following <literal>STRIPE_OPTIONS</literal> are
+ for this component. Each component defines the stripe pattern of the
+ file in the range of [start, end). The first component must start from
+ offset 0 and all components must be adjacent with each other, no holes
+ are allowed, so each extent will start at the end of previous extent.
+ A <literal>-1</literal> end offset or <literal>eof</literal> indicates
+ this is the last component extending to the end of file.</para>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 -E -1 -c -1 -i 4 \
+/mnt/testfs/create_comp</screen>
+ <para>This command creates a file with composite layout illustrated in
+ the following figure. The first component has 1 stripe and covers
+ [0, 4M), the second component has 4 stripes and covers [4M, 64M), and
+ the last component stripes start at OST4, cross over all available
+ OSTs and covers [64M, EOF).</para>
+ <figure xml:id="managinglayout.fig.pfl_create">
+ <title>Example: create a composite file</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.75in" align="center"
+ fileref="figures/PFL_createfile.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: create a composite file</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The composite layout can be output by the following command:</para>
+ <screen>$ lfs getstripe /mnt/testfs/create_comp
+/mnt/testfs/create_comp
+ lcm_layout_gen: 3
+ lcm_entry_count: 3
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 4194304
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
+
+ lcme_id: 2
+ lcme_flags: 0
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1
+ lcme_id: 3
+ lcme_flags: 0
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: -1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 4</screen>
+ <note><para>Only the first component’s OST objects of the PFL file are
+ instantiated when the layout is being set. Other instantiation is
+ delayed to later write/truncate operations.</para></note>
+ <para>If we write 128M data to this PFL file, the second and third
+ components will be instantiated:</para>
+ <screen>$ dd if=/dev/zero of=/mnt/testfs/create_comp bs=1M count=128
+$ lfs getstripe /mnt/testfs/create_comp
+/mnt/testfs/create_comp
+ lcm_layout_gen: 5
+ lcm_entry_count: 3
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 4194304
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
+
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 1
+ lmm_objects:
+ - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
+ - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
+ - 2: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }
+ - 3: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
+
+ lcme_id: 3
+ lcme_flags: init
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 8
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 4
+ lmm_objects:
+ - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x3:0x0] }
+ - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
+ - 2: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
+ - 3: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
+ - 4: { l_ost_idx: 0, l_fid: [0x100000000:0x3:0x0] }
+ - 5: { l_ost_idx: 1, l_fid: [0x100010000:0x3:0x0] }
+ - 6: { l_ost_idx: 2, l_fid: [0x100020000:0x3:0x0] }
+ - 7: { l_ost_idx: 3, l_fid: [0x100030000:0x3:0x0] }</screen>
+ </section>
+ <section remap="h4">
+ <title>Add component(s) to an existing composite file</title>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs setstripe --component-add
+[--component-end|-E end1] [STRIPE_OPTIONS]
+[--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable></screen>
+ <para>The option <literal>--component-add</literal> is used to add
+ components to an existing composite file. The extent start of
+ the first component to be added is equal to the extent end of last
+ component in the existing file, and all components to be added must
+ be adjacent with each other.</para>
+ <note><para>If the last existing component is specified by
+ <literal>-E -1</literal> or <literal>-E eof</literal>, which covers
+ to the end of the file, it must be deleted before a new one is added.
+ </para></note>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 /mnt/testfs/add_comp
+$ lfs setstripe --component-add -E -1 -c 4 -o 6-7,0,5 \
+/mnt/testfs/add_comp</screen>
+ <para>This command adds a new component which starts from the end of
+ the last existing component to the end of file. The layout of this
+ example is illustrated in
+ <xref linkend="managinglayout.fig.pfl_addcomp"/>. The last component
+ stripes across 4 OSTs in sequence OST6, OST7, OST0 and OST5, covers
+ [64M, EOF).</para>
+ <figure xml:id="managinglayout.fig.pfl_addcomp">
+ <title>Example: add a component to an existing composite file</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.75in" align="center"
+ fileref="figures/PFL_addcomp.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: add a component to an existing composite file
+ </phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The layout can be printed out by the following command:</para>
+ <screen>$ lfs getstripe /mnt/testfs/add_comp
+/mnt/testfs/add_comp
+ lcm_layout_gen: 5
+ lcm_entry_count: 3
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 4194304
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
+
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 1
+ lmm_objects:
+ - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
+ - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
+ - 2: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }
+ - 3: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
+
+ lcme_id: 5
+ lcme_flags: 0
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1</screen>
+ <para>The component ID "lcme_id" changes as layout generation
+ changes. It is not necessarily sequential and does not imply ordering
+ of individual components.</para>
+ <note><para>Similar to specifying a full-file composite layout at file
+ creation time, <literal>--component-add</literal> won't instantiate
+ OST objects, the instantiation is delayed to later write/truncate
+ operations. For example, after writing beyond the 64MB start of the
+ file's last component, the new component has had objects allocated:
+ </para></note>
+ <screen>$ lfs getstripe -I5 /mnt/testfs/add_comp
+/mnt/testfs/add_comp
+ lcm_layout_gen: 6
+ lcm_entry_count: 3
+ lcme_id: 5
+ lcme_flags: init
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 6
+ lmm_objects:
+ - 0: { l_ost_idx: 6, l_fid: [0x100060000:0x4:0x0] }
+ - 1: { l_ost_idx: 7, l_fid: [0x100070000:0x4:0x0] }
+ - 2: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
+ - 3: { l_ost_idx: 5, l_fid: [0x100050000:0x4:0x0] }</screen>
+ </section>
+ <section remap="h4">
+ <title>Delete component(s) from an existing file</title>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs setstripe --component-del
+[--component-id|-I comp_id | --component-flags comp_flags]
+<replaceable>filename</replaceable></screen>
+ <para>The option <literal>--component-del</literal> is used to remove
+ the component(s) specified by component ID or flags from an existing
+ file. Any data stored in the deleted component will be lost after
+ this operation.</para>
+ <para>The ID specified by <literal>-I</literal> option is the numerical
+ unique ID of the component, which can be obtained by command
+ <literal>lfs getstripe -I</literal> command, and the flag specified by
+ <literal>--component-flags</literal> option is a certain type of
+ components, which can be obtained by command
+ <literal>lfs getstripe --component-flags</literal>. For now, we only
+ have two flags <literal>init</literal> and <literal>^init</literal>
+ for instantiated and un-instantiated components respectively.</para>
+ <note><para>Deletion must start with the last component because creation
+ of a hole in the middle of a file layout is not allowed.</para></note>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <screen>$ lfs getstripe -I /mnt/testfs/del_comp
+1
+2
+5
+$ lfs setstripe --component-del -I 5 /mnt/testfs/del_comp</screen>
+ <para>This example deletes the component with ID 5 from file
+ <literal>/mnt/testfs/del_comp</literal>. If we still use the last
+ example, the final result is illustrated in
+ <xref linkend="managinglayout.fig.pfl_delcomp"/>.</para>
+ <figure xml:id="managinglayout.fig.pfl_delcomp">
+ <title>Example: delete a component from an existing file</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.75in" align="center"
+ fileref="figures/PFL_delcomp.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: delete a component from an existing file</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>If you try to delete a non-last component, you will see the
+ following error:</para>
+ <screen>$ lfs setstripe -component-del -I 2 /mnt/testfs/del_comp
+Delete component 0x2 from /mnt/testfs/del_comp failed. Invalid argument
+error: setstripe: delete component of file '/mnt/testfs/del_comp' failed: Invalid argument</screen>
+ </section>
+ <section remap="h4">
+ <title>Set default PFL layout to an existing directory</title>
+ <para>Similar to create a PFL file, you can set default PFL layout to
+ an existing directory. After that, all the files created will inherit
+ this layout by default.</para>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs setstripe
+[--component-end|-E end1] [STRIPE_OPTIONS]
+[--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>dirname</replaceable></screen>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <screen>
+$ mkdir /mnt/testfs/pfldir
+$ lfs setstripe -E 256M -c 1 -E 16G -c 4 -E -1 -S 4M -c -1 /mnt/testfs/pfldir
+</screen>
+ <para>When you run <literal>lfs getstripe</literal>, you will see:
+ </para>
+ <screen>
+$ lfs getstripe /mnt/testfs/pfldir
+/mnt/testfs/pfldir
+ lcm_layout_gen: 0
+ lcm_entry_count: 3
+ lcme_id: N/A
+ lcme_flags: 0
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 268435456
+ stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
+ lcme_id: N/A
+ lcme_flags: 0
+ lcme_extent.e_start: 268435456
+ lcme_extent.e_end: 17179869184
+ stripe_count: 4 stripe_size: 1048576 stripe_offset: -1
+ lcme_id: N/A
+ lcme_flags: 0
+ lcme_extent.e_start: 17179869184
+ lcme_extent.e_end: EOF
+ stripe_count: -1 stripe_size: 4194304 stripe_offset: -1
+</screen>
+ <para>If you create a file under <literal>/mnt/testfs/pfldir</literal>,
+ the layout of that file will inherit the layout from its parent
+ directory:</para>
+ <screen>
+$ touch /mnt/testfs/pfldir/pflfile
+$ lfs getstripe /mnt/testfs/pfldir/pflfile
+/mnt/testfs/pfldir/pflfile
+ lcm_layout_gen: 2
+ lcm_entry_count: 3
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 268435456
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 1
+ lmm_objects:
+ - 0: { l_ost_idx: 1, l_fid: [0x100010000:0xa:0x0] }
+
+ lcme_id: 2
+ lcme_flags: 0
+ lcme_extent.e_start: 268435456
+ lcme_extent.e_end: 17179869184
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1
+
+ lcme_id: 3
+ lcme_flags: 0
+ lcme_extent.e_start: 17179869184
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: -1
+ lmm_stripe_size: 4194304
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1
+</screen>
+ <note><para>
+ <literal>lfs setstripe --component-add/del</literal> can't be run
+ on a directory, because the default layout in directory is like a config,
+ which can be arbitrarily changed by <literal>lfs setstripe</literal>,
+ while the layout of a file may have data (OST objects) attached.
+ If you want to delete the default layout in a directory, run
+ <literal>lfs setstripe -d <replaceable>dirname</replaceable></literal>
+ to return the directory to the filesystem-wide defaults, like:
+ <screen>
+$ lfs setstripe -d /mnt/testfs/pfldir
+$ lfs getstripe -d /mnt/testfs/pfldir
+/mnt/testfs/pfldir
+stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
+/mnt/testfs/pfldir/commonfile
+lmm_stripe_count: 1
+lmm_stripe_size: 1048576
+lmm_pattern: 1
+lmm_layout_gen: 0
+lmm_stripe_offset: 0
+ obdidx objid objid group
+ 2 9 0x9 0
+</screen>
+ </para></note>
+ </section>
+ </section>
+ <section remap="h3">
+ <title><literal>lfs migrate</literal></title>
+ <para><literal>lfs migrate</literal> commands are used to re-layout the
+ data in the existing files with the new layout parameter by copying the
+ data from the existing OST(s) to the new OST(s).</para>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs migrate [--component-end|-E comp_end] [STRIPE_OPTIONS] ...
+<replaceable>filename</replaceable></screen>
+ <para>The difference between <literal>migrate</literal> and
+ <literal>setstripe</literal> is that <literal>migrate</literal> is to
+ re-layout the data in the existing files, while
+ <literal>setstripe</literal> is to create new files with the specified
+ layout.</para>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <para><emphasis role="bold">Case1. Migrate a normal one to a composite
+ layout</emphasis></para>
+ <screen>$ lfs setstripe -c 1 -S 128K /mnt/testfs/norm_to_2comp
+$ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
+$ lfs getstripe /mnt/testfs/norm_to_2comp --yaml
+/mnt/testfs/norm_to_comp
+lmm_stripe_count: 1
+lmm_stripe_size: 131072
+lmm_pattern: 1
+lmm_layout_gen: 0
+lmm_stripe_offset: 7
+lmm_objects:
+ - l_ost_idx: 7
+ l_fid: 0x100070000:0x2:0x0
+$ lfs migrate -E 1M -S 512K -c 1 -E -1 -S 1M -c 2 \
+/mnt/testfs/norm_to_2comp</screen>
+ <para>In this example, a 5MB size file with 1 stripe and 128K stripe size
+ is migrated to a composite layout file with 2 components, illustrated in
+ <xref linkend="managinglayout.fig.pfl_norm_to_comp"/>.</para>
+ <figure xml:id="managinglayout.fig.pfl_norm_to_comp">
+ <title>Example: migrate normal to composite</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.75in" align="center"
+ fileref="figures/PFL_norm_to_comp.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: migrate normal to composite</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The stripe information after migration is like:</para>
+ <screen>$ lfs getstripe /mnt/testfs/norm_to_2comp
+/mnt/testfs/norm_to_2comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 2
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 1048576
+ lmm_stripe_count: 1
+ lmm_stripe_size: 524288
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
+
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 1048576
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 2
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 2
+ lmm_objects:
+ - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
+ - 1: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }</screen>
+ <para><emphasis role="bold">Case2. Migrate a composite layout to another
+ composite layout</emphasis></para>
+ <screen>$ lfs setstripe -E 1M -S 512K -c 1 -E -1 -S 1M -c 2 \
+/mnt/testfs/2comp_to_3comp
+$ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
+$ lfs migrate -E 1M -S 1M -c 2 -E 4M -S 1M -c 2 -E -1 -S 3M -c 3 \
+/mnt/testfs/2comp_to_3comp</screen>
+ <para>In this example, a composite layout file with 2 components is
+ migrated a composite layout file with 3 components. If we still use
+ the example in case1, the migration process is illustrated in
+ <xref linkend="managinglayout.fig.pfl_comp_to_comp"/>.</para>
+ <figure xml:id="managinglayout.fig.pfl_comp_to_comp">
+ <title>Example: migrate composite to composite</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.75in" align="center"
+ fileref="figures/PFL_comp_to_comp.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: migrate composite to composite</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The stripe information is like:</para>
+ <screen>$ lfs getstripe /mnt/testfs/2comp_to_3comp
+/mnt/testfs/2comp_to_3comp
+ lcm_layout_gen: 6
+ lcm_entry_count: 3
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 1048576
+ lmm_stripe_count: 2
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 4
+ lmm_objects:
+ - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
+ - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
+
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 1048576
+ lcme_extent.e_end: 4194304
+ lmm_stripe_count: 2
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 6
+ lmm_objects:
+ - 0: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
+ - 1: { l_ost_idx: 7, l_fid: [0x100070000:0x3:0x0] }
+
+ lcme_id: 3
+ lcme_flags: init
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 3
+ lmm_stripe_size: 3145728
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x3:0x0] }
+ - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
+ - 2: { l_ost_idx: 2, l_fid: [0x100020000:0x3:0x0] }</screen>
+ <para><emphasis role="bold">Case3. Migrate a composite layout to a
+ normal one</emphasis></para>
+ <screen>$ lfs migrate -E 1M -S 1M -c 2 -E 4M -S 1M -c 2 -E -1 -S 3M -c 3 \
+/mnt/testfs/3comp_to_norm
+$ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
+$ lfs migrate -c 2 -S 2M /mnt/testfs/3comp_to_normal</screen>
+ <para>In this example, a composite file with 3 components is migrated to
+ a normal file with 2 stripes and 2M stripe size. If we still use the
+ example in Case2, the migration process is illustrated in
+ <xref linkend="managinglayout.fig.pfl_comp_to_norm"/>.</para>
+ <figure xml:id="managinglayout.fig.pfl_comp_to_norm">
+ <title>Example: migrate composite to normal</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.75in" align="center"
+ fileref="figures/PFL_comp_to_norm.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: migrate composite to normal</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The stripe information is like:</para>
+ <screen>$ lfs getstripe /mnt/testfs/3comp_to_norm --yaml
+/mnt/testfs/3comp_to_norm
+lmm_stripe_count: 2
+lmm_stripe_size: 2097152
+lmm_pattern: 1
+lmm_layout_gen: 7
+lmm_stripe_offset: 4
+lmm_objects:
+ - l_ost_idx: 4
+ l_fid: 0x100040000:0x3:0x0
+ - l_ost_idx: 5
+ l_fid: 0x100050000:0x3:0x0</screen>
+ </section>
+ <section remap="h3">
+ <title><literal>lfs getstripe</literal></title>
+ <para><literal>lfs getstripe</literal> commands can be used to list the
+ striping/component information for a given PFL file. Here, only those
+ parameters new for PFL files are shown.</para>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs getstripe
+[--component-id|-I [comp_id]]
+[--component-flags [comp_flags]]
+[--component-count]
+[--component-start [+-][N][kMGTPE]]
+[--component-end|-E [+-][N][kMGTPE]]
+<replaceable>dirname|filename</replaceable></screen>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <para>Suppose we already have a composite file
+ <literal>/mnt/testfs/3comp</literal>, created by the following
+ command:</para>
+ <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 -E -1 -c -1 -i 4 \
+/mnt/testfs/3comp</screen>
+ <para>And write some data</para>
+ <screen>$ dd if=/dev/zero of=/mnt/testfs/3comp bs=1M count=5</screen>
+ <para><emphasis role="bold">Case1. List component ID and its related
+ information</emphasis></para>
+ <itemizedlist>
+ <listitem>
+ <para>List all the components ID</para>
+ <screen>$ lfs getstripe -I /mnt/testfs/3comp
+1
+2
+3</screen>
+ </listitem>
+ <listitem>
+ <para>List the detailed striping information of component ID=2</para>
+ <screen>$ lfs getstripe -I2 /mnt/testfs/3comp
+/mnt/testfs/3comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 3
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 5
+ lmm_objects:
+ - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
+ - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
+ - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
+ - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
+ </listitem>
+ <listitem>
+ <para>List the stripe offset and stripe count of component ID=2</para>
+ <screen>$ lfs getstripe -I2 -i -c /mnt/testfs/3comp
+ lmm_stripe_count: 4
+ lmm_stripe_offset: 5</screen>
+ </listitem>
+ </itemizedlist>
+ <para><emphasis role="bold">Case2. List the component which contains the
+ specified flag</emphasis></para>
+ <itemizedlist>
+ <listitem>
+ <para>List the flag of each component</para>
+ <screen>$ lfs getstripe -component-flag -I /mnt/testfs/3comp
+ lcme_id: 1
+ lcme_flags: init
+ lcme_id: 2
+ lcme_flags: init
+ lcme_id: 3
+ lcme_flags: 0</screen>
+ </listitem>
+ <listitem>
+ <para>List component(s) who is not instantiated</para>
+ <screen>$ lfs getstripe --component-flags=^init /mnt/testfs/3comp
+/mnt/testfs/3comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 3
+ lcme_id: 3
+ lcme_flags: 0
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: -1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 4
+ lmm_stripe_offset: 4</screen>
+ </listitem>
+ </itemizedlist>
+ <para><emphasis role="bold">Case3. List the total number of all the
+ component(s)</emphasis></para>
+ <itemizedlist>
+ <listitem>
+ <para>List the total number of all the components</para>
+ <screen>$ lfs getstripe --component-count /mnt/testfs/3comp
+3</screen>
+ </listitem>
+ </itemizedlist>
+ <para><emphasis role="bold">Case4. List the component with the specified
+ extent start or end positions</emphasis></para>
+ <itemizedlist>
+ <listitem>
+ <para>List the start position in bytes of each component</para>
+ <screen>$ lfs getstripe --component-start /mnt/testfs/3comp
+0
+4194304
+67108864</screen>
+ </listitem>
+ <listitem>
+ <para>List the start position in bytes of component ID=3</para>
+ <screen>$ lfs getstripe --component-start -I3 /mnt/testfs/3comp
+67108864</screen>
+ </listitem>
+ <listitem>
+ <para>List the component with start = 64M</para>
+ <screen>$ lfs getstripe --component-start=64M /mnt/testfs/3comp
+/mnt/testfs/3comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 3
+ lcme_id: 3
+ lcme_flags: 0
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: -1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 4
+ lmm_stripe_offset: 4</screen>
+ </listitem>
+ <listitem>
+ <para>List the component(s) with start > 5M</para>
+ <screen>$ lfs getstripe --component-start=+5M /mnt/testfs/3comp
+/mnt/testfs/3comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 3
+ lcme_id: 3
+ lcme_flags: 0
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: -1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 4
+ lmm_stripe_offset: 4</screen>
+ </listitem>
+ <listitem>
+ <para>List the component(s) with start < 5M</para>
+ <screen>$ lfs getstripe --component-start=-5M /mnt/testfs/3comp
+/mnt/testfs/3comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 3
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 4194304
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 4
+ lmm_objects:
+ - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
+
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 5
+ lmm_objects:
+ - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
+ - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
+ - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
+ - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
+ </listitem>
+ <listitem>
+ <para>List the component(s) with start > 3M and end < 70M</para>
+ <screen>$ lfs getstripe --component-start=+3M --component-end=-70M \
+/mnt/testfs/3comp
+/mnt/testfs/3comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 3
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 5
+ lmm_objects:
+ - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
+ - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
+ - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
+ - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section remap="h3">
+ <title><literal>lfs find</literal></title>
+ <para><literal>lfs find</literal> commands can be used to search the
+ directory tree rooted at the given directory or file name for the files
+ that match the given PFL component parameters. Here, only those
+ parameters new for PFL files are shown. Their usages are similar to
+ <literal>lfs getstripe</literal> commands.</para>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs find <replaceable>directory|filename</replaceable>
+[[!] --component-count [+-=]<replaceable>comp_cnt</replaceable>]
+[[!] --component-start [+-=]<replaceable>N</replaceable>[kMGTPE]]
+[[!] --component-end|-E [+-=]<replaceable>N</replaceable>[kMGTPE]]
+[[!] --component-flags=<replaceable>comp_flags</replaceable>]</screen>
+ <note><para>If you use <literal>--component-xxx</literal> options, only
+ the composite files will be searched; but if you use
+ <literal>! --component-xxx</literal> options, all the files will be
+ searched.</para></note>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <para>We use the following directory and composite files to show how
+ <literal>lfs find</literal> works.</para>
+ <screen>$ mkdir /mnt/testfs/testdir
+$ lfs setstripe -E 1M -E 10M -E eof /mnt/testfs/testdir/3comp
+$ lfs setstripe -E 4M -E 20M -E 30M -E eof /mnt/testfs/testdir/4comp
+$ mkdir -p /mnt/testfs/testdir/dir_3comp
+$ lfs setstripe -E 6M -E 30M -E eof /mnt/testfs/testdir/dir_3comp
+$ lfs setstripe -E 8M -E eof /mnt/testfs/testdir/dir_3comp/2comp
+$ lfs setstripe -c 1 /mnt/testfs/testdir/dir_3comp/commnfile</screen>
+ <para><emphasis role="bold">Case1. Find the files that match the specified
+ component count condition</emphasis></para>
+ <para>Find the files under directory /mnt/testfs/testdir whose number of
+ components is not equal to 3.</para>
+ <screen>$ lfs find /mnt/testfs/testdir ! --component-count=3
+/mnt/testfs/testdir
+/mnt/testfs/testdir/4comp
+/mnt/testfs/testdir/dir_3comp/2comp
+/mnt/testfs/testdir/dir_3comp/commonfile</screen>
+ <para><emphasis role="bold">Case2. Find the files/dirs that match the
+ specified component start/end condition</emphasis></para>
+ <para>Find the file(s) under directory /mnt/testfs/testdir with component
+ start = 4M and end < 70M</para>
+ <screen>$ lfs find /mnt/testfs/testdir --component-start=4M -E -30M
+/mnt/testfs/testdir/4comp</screen>
+ <para><emphasis role="bold">Case3. Find the files/dirs that match the
+ specified component flag condition</emphasis></para>
+ <para>Find the file(s) under directory /mnt/testfs/testdir whose component
+ flags contain <literal>init</literal></para>
+ <screen>$ lfs find /mnt/testfs/testdir --component-flag=init
+/mnt/testfs/testdir/3comp
+/mnt/testfs/testdir/4comp
+/mnt/testfs/testdir/dir_3comp/2comp</screen>
+ <note><para>Since <literal>lfs find</literal> uses
+ "<literal>!</literal>" to do negative search, we don’t support
+ flag <literal>^init</literal> here.</para></note>
+ </section>
+ </section>
+
+ <section xml:id="striping.sel" condition='l2D'>
+ <title>
+ <indexterm><primary>striping</primary><secondary>SEL</secondary>
+ </indexterm>Self-Extending Layout (SEL)</title>
+ <para>The Lustre Self-Extending Layout (SEL) feature is an extension of the
+ <xref linkend="pfl"/> feature, which allows the MDS to change the defined
+ PFL layout dynamically. With this feature, the MDS monitors the used space
+ on OSTs and swaps the OSTs for the current file when they are low on space.
+ This avoids <literal>ENOSPC</literal> problems for SEL files when
+ applications are writing to them.</para>
+ <para>Whereas PFL delays the instantiation of some components until an IO
+ operation occurs on this region, SEL allows splitting such non-instantiated
+ components in two parts: an “extendable” component and an “extension”
+ component. The extendable component is a regular PFL component, covering
+ just a part of the region, which is small originally. The extension (or SEL)
+ component is a new component type which is always non-instantiated and
+ unassigned, covering the other part of the region. When a write reaches this
+ unassigned space, and the client calls the MDS to have it instantiated, the
+ MDS makes a decision as to whether to grant additional space to the extendable
+ component. The granted region moves from the head of the extension
+ component to the tail of the extendable component, thus the extendable
+ component grows and the SEL one is shortened. Therefore, it allows the file
+ to continue on the same OSTs, or in the case where space is low on one of
+ the current OSTs, to modify the layout to switch to a new component on new
+ OSTs. In particular, it lets IO automatically spill over to a large HDD OST
+ pool once a small SSD OST pool is getting low on space.</para>
+ <para>The default extension policy modifies the layout in the following
+ ways:</para>
+ <orderedlist numeration="arabic">
+ <listitem>
+ <para>Extension: continue on the same OSTs – used when not low on space
+ on any of the OSTs of the current component; a particular extent is
+ granted to the extendable component.</para>
+ </listitem>
+ <listitem>
+ <para>Spill over: switch to next component OSTs – it is used only for
+ not the last component when <emphasis>at least one</emphasis>
+ of the current OSTs is low on space; the whole region of the SEL
+ component moves to the next component and the SEL component is removed
+ in its turn.</para>
+ </listitem>
+ <listitem>
+ <para>Repeating: create a new component with the same layout but on
+ free OSTs – it is used only for the last component when <emphasis>
+ at least one</emphasis> of the current OSTs is low on space; a new
+ component has the same layout but instantiated on different OSTs (from
+ the same pool) which have enough space.</para>
+ </listitem>
+ <listitem>
+ <para>Forced extension: continue with the current component OSTs despite
+ the low on space condition – it is used only for the last component when
+ a repeating attempt detected low on space condition as well - spillover
+ is impossible and there is no sense in the repeating.</para>
+ </listitem>
+ </orderedlist>
+ <note><para>The SEL feature does not require clients to understand the SEL
+ format of already created files, only the MDS support is needed which is
+ introduced in Lustre 2.13. However, old clients will have some limitations
+ as the Lustre tools will not support it.</para></note>
+ <section>
+ <title><literal>lfs setstripe</literal></title>
+ <para>The <literal>lfs setstripe</literal> command is used to create files
+ with composite layouts, as well as add or delete components to or from an
+ existing file. It is extended to support SEL components.</para>
+ <section>
+ <title>Create a SEL file</title>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs setstripe
+[--component-end|-E end1] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable>
+
+STRIPE OPTIONS:
+--extension-size, --ext-size, -z <ext_size></screen>
+ <para>The <literal>-z</literal> option is added to specify the size of
+ the region which is granted to the extendable component on each
+ iteration. While declaring any component, this option turns the declared
+ component to a pair of components: extendable and extension ones.</para>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <para>The following command creates 2 pairs of extendable and
+ extension components:
+ <screen># lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/file</screen>
+ <figure xml:id="managinglayout.fig.sel_createfile">
+ <title>Example: create a SEL file</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="0.8in" align="center"
+ fileref="figures/SEL_Createfile.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: create a SEL file</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ </para>
+ <note><para>As usual, only the first PFL component is instantiated at
+ the creation time, thus it is immediately extended to the extension
+ size (64M for the first component), whereas the third component is left
+ zero-length.</para></note>
+ <screen># lfs getstripe /mnt/lustre/file
+/mnt/lustre/file
+ lcm_layout_gen: 4
+ lcm_mirror_count: 1
+ lcm_entry_count: 4
+ lcme_id: 1
+ lcme_mirror_id: 0
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
+
+ lcme_id: 2
+ lcme_mirror_id: 0
+ lcme_flags: extension
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: 1073741824
+ lmm_stripe_count: 0
+ lmm_extension_size: 67108864
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1
+
+ lcme_id: 3
+ lcme_mirror_id: 0
+ lcme_flags: 0
+ lcme_extent.e_start: 1073741824
+ lcme_extent.e_end: 1073741824
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1
+
+ lcme_id: 4
+ lcme_mirror_id: 0
+ lcme_flags: extension
+ lcme_extent.e_start: 1073741824
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 0
+ lmm_extension_size: 268435456
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1</screen>
+ </section>
+ <section>
+ <title>Create a SEL layout template</title>
+ <para>Similar to PFL, it is possible to set a SEL layout template to
+ a directory. After that, all the files created under it will inherit this
+ layout by default.</para>
+ <screen># lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/dir
+# ./lustre/utils/lfs getstripe /mnt/lustre/dir
+/mnt/lustre/dir
+ lcm_layout_gen: 0
+ lcm_mirror_count: 1
+ lcm_entry_count: 4
+ lcme_id: N/A
+ lcme_mirror_id: N/A
+ lcme_flags: 0
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 67108864
+ stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1
+
+ lcme_id: N/A
+ lcme_mirror_id: N/A
+ lcme_flags: extension
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: 1073741824
+ stripe_count: 1 extension_size: 67108864 pattern: raid0 stripe_offset: -1
+
+ lcme_id: N/A
+ lcme_mirror_id: N/A
+ lcme_flags: 0
+ lcme_extent.e_start: 1073741824
+ lcme_extent.e_end: 1073741824
+ stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1
+
+ lcme_id: N/A
+ lcme_mirror_id: N/A
+ lcme_flags: extension
+ lcme_extent.e_start: 1073741824
+ lcme_extent.e_end: EOF
+ stripe_count: 1 extension_size: 268435456 pattern: raid0 stripe_offset: -1
+ </screen>
+ </section>
+ </section>
+ <section>
+ <title><literal>lfs getstripe</literal></title>
+ <para><literal>lfs getstripe</literal> commands can be used to list the
+ striping/component information for a given SEL file. Here, only those parameters
+ new for SEL files are shown.</para>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs getstripe
+[--extension-size|--ext-size|-z] <replaceable>filename</replaceable></screen>
+ <para>The <literal>-z</literal> option is added to print the extension
+ size in bytes. For composite files this is the extension size of the
+ first extension component. If a particular component is identified by
+ other options (<literal>--component-id, --component-start</literal>,
+ etc...), this component extension size is printed.</para>
+ <para><emphasis role="bold">Example 1: List a SEL component information
+ </emphasis></para>
+ <para>Suppose we already have a composite file
+ <literal>/mnt/lustre/file</literal>, created by the following command:</para>
+ <screen># lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/file</screen>
+ <para>The 2nd component could be listed with the following command:</para>
+ <screen># lfs getstripe -I2 /mnt/lustre/file
+/mnt/lustre/file
+ lcm_layout_gen: 4
+ lcm_mirror_count: 1
+ lcm_entry_count: 4
+ lcme_id: 2
+ lcme_mirror_id: 0
+ lcme_flags: extension
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: 1073741824
+ lmm_stripe_count: 0
+ lmm_extension_size: 67108864
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1
+ </screen>
+ <note><para>As you can see the SEL components are marked by the <literal>
+ extension</literal> flag and <literal>lmm_extension_size</literal> field
+ keeps the specified extension size.</para></note>
+ <para><emphasis role="bold">Example 2: List the extension size</emphasis></para>
+ <para>Having the same file as in the above example, the extension size of
+ the second component could be listed with:</para>
+ <screen># lfs getstripe -z -I2 /mnt/lustre/file
+67108864</screen>
+ <para><emphasis role="bold">Example 3: Extension</emphasis></para>
+ <para>Having the same file as in the above example, suppose there is a
+ write which crosses the end of the first component (64M), and then another
+ write another write which crosses the end of the first component (128M) again,
+ the layout changes as following:</para>
+ <figure xml:id="managinglayout.fig.sel_extension">
+ <title>Example: an extension of a SEL file</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="3.5in" align="center"
+ fileref="figures/SEL_extension.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: an extension of a SEL file</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The layout can be printed out by the following command:</para>
+ <screen># lfs getstripe /mnt/lustre/file
+/mnt/lustre/file
+ lcm_layout_gen: 6
+ lcm_mirror_count: 1
+ lcm_entry_count: 4
+ lcme_id: 1
+ lcme_mirror_id: 0
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 201326592
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
+
+ lcme_id: 2
+ lcme_mirror_id: 0
+ lcme_flags: extension
+ lcme_extent.e_start: 201326592
+ lcme_extent.e_end: 1073741824
+ lmm_stripe_count: 0
+ lmm_extension_size: 67108864
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1
+
+ lcme_id: 3
+ lcme_mirror_id: 0
+ lcme_flags: 0
+ lcme_extent.e_start: 1073741824
+ lcme_extent.e_end: 1073741824
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1
+
+ lcme_id: 4
+ lcme_mirror_id: 0
+ lcme_flags: extension
+ lcme_extent.e_start: 1073741824
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 0
+ lmm_extension_size: 268435456
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1</screen>
+ <para><emphasis role="bold">Example 4: Spillover</emphasis></para>
+ <para>In case where <literal>OST0</literal> is low on space and an IO
+ happens to a SEL component, a spillover happens: the full region of the
+ SEL component is added to the next component, e.g. in the example above
+ the next layout modification will look like:</para>
+ <figure xml:id="managinglayout.fig.sel_spillover">
+ <title>Example: a spillover in a SEL file</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.25in" align="center"
+ fileref="figures/SEL_spillover.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: a spillover in a SEL file</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <note><para>Despite the fact the third component was [1G, 1G] originally,
+ while it is not instantiated, instead of getting extended backward, it is
+ moved backward to the start of the previous SEL component (192M) and
+ extended on its extension size (256M) from that position, thus it becomes
+ <literal>[192M, 448M]</literal>.</para></note>
+ <screen># lfs getstripe /mnt/lustre/file
+/mnt/lustre/file
+ lcm_layout_gen: 7
+ lcm_mirror_count: 1
+ lcm_entry_count: 3
+ lcme_id: 1
+ lcme_mirror_id: 0
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 201326592
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
+
+ lcme_id: 3
+ lcme_mirror_id: 0
+ lcme_flags: init
+ lcme_extent.e_start: 201326592
+ lcme_extent.e_end: 469762048
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 1
+ lmm_objects:
+ - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x8:0x0] }
+
+ lcme_id: 4
+ lcme_mirror_id: 0
+ lcme_flags: extension
+ lcme_extent.e_start: 469762048
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 0
+ lmm_extension_size: 268435456
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1</screen>
+ <para><emphasis role="bold">Example 5: Repeating</emphasis></para>
+ <para>Suppose in the example above, <literal>OST0</literal> got
+ enough free space back but <literal>OST1</literal> is low on space,
+ the following write to the last SEL component leads to a new component
+ allocation before the SEL component, which repeats the previous
+ component layout but instantiated on free OSTs:</para>
+ <figure xml:id="managinglayout.fig.sel_repeat">
+ <title>Example: repeat a SEL component</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.25in" align="center"
+ fileref="figures/SEL_repeating.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: repeat a SEL component
+ </phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <screen># lfs getstripe /mnt/lustre/file
+/mnt/lustre/file
+ lcm_layout_gen: 9
+ lcm_mirror_count: 1
+ lcm_entry_count: 4
+ lcme_id: 1
+ lcme_mirror_id: 0
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 201326592
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
+
+ lcme_id: 3
+ lcme_mirror_id: 0
+ lcme_flags: init
+ lcme_extent.e_start: 201326592
+ lcme_extent.e_end: 469762048
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 1
+ lmm_objects:
+ - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x8:0x0] }
+
+ lcme_id: 8
+ lcme_mirror_id: 0
+ lcme_flags: init
+ lcme_extent.e_start: 469762048
+ lcme_extent.e_end: 738197504
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 65535
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6:0x0] }
+
+ lcme_id: 4
+ lcme_mirror_id: 0
+ lcme_flags: extension
+ lcme_extent.e_start: 738197504
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 0
+ lmm_extension_size: 268435456
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1</screen>
+ <para><emphasis role="bold">Example 6: Forced extension</emphasis></para>
+ <para>Suppose in the example above, both <literal>OST0</literal> and
+ <literal>OST1</literal> are low on space, the following write to the
+ last SEL component will behave as an extension as there is no sense to
+ repeat.</para>
+ <figure xml:id="managinglayout.fig.pfl_forced">
+ <title>Example: forced extension in a SEL file</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.25in" align="center"
+ fileref="figures/SEL_forced.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: forced extension in a SEL file.
+ </phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <screen># lfs getstripe /mnt/lustre/file
+/mnt/lustre/file
+ lcm_layout_gen: 11
+ lcm_mirror_count: 1
+ lcm_entry_count: 4
+ lcme_id: 1
+ lcme_mirror_id: 0
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 201326592
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
+
+ lcme_id: 3
+ lcme_mirror_id: 0
+ lcme_flags: init
+ lcme_extent.e_start: 201326592
+ lcme_extent.e_end: 469762048
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 1
+ lmm_objects:
+ - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x8:0x0] }
+
+ lcme_id: 8
+ lcme_mirror_id: 0
+ lcme_flags: init
+ lcme_extent.e_start: 469762048
+ lcme_extent.e_end: 1006632960
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: raid0
+ lmm_layout_gen: 65535
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6:0x0] }
+
+ lcme_id: 4
+ lcme_mirror_id: 0
+ lcme_flags: extension
+ lcme_extent.e_start: 1006632960
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 0
+ lmm_extension_size: 268435456
+ lmm_pattern: raid0
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1</screen>
+ </section>
+ <section>
+ <title><literal>lfs find</literal></title>
+ <para><literal>lfs find</literal> commands can be used to search for
+ the files that match the given SEL component paremeters. Here, only
+ those parameters new for the SEL files are shown.</para>
+ <screen>lfs find
+[[!] --extension-size|--ext-size|-z [+-]ext-size[KMG]
+[[!] --component-flags=extension]</screen>
+ <para>The <literal>-z</literal> option is added to specify the extension
+ size to search for. The files which have any component with the
+ extension size matched the given criteria are printed out. As always
+ “+” and “-“ signs are allowed to specify the least and the most size.
+ </para>
+ <para>A new <literal>extension</literal> component flag is added. Only
+ files which have at least one SEL component are printed.</para>
+ <note><para>The negative search for flags searches the files which
+ <emphasis role="strong">have</emphasis> a non-SEL component (not files
+ which <emphasis role="strong">do not have</emphasis> any SEL component).
+ </para></note>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <screen># lfs setstripe --extension-size 64M -c 1 -E -1 /mnt/lustre/file
+
+# lfs find --comp-flags extension /mnt/lustre/*
+/mnt/lustre/file
+
+# lfs find ! --comp-flags extension /mnt/lustre/*
+/mnt/lustre/file
+
+# lfs find -z 64M /mnt/lustre/*
+/mnt/lustre/file
+
+# lfs find -z +64M /mnt/lustre/*
+
+# lfs find -z -64M /mnt/lustre/*
+
+# lfs find -z +63M /mnt/lustre/*
+/mnt/lustre/file
+
+# lfs find -z -65M /mnt/lustre/*
+/mnt/lustre/file
+
+# lfs find -z 65M /mnt/lustre/*
+
+# lfs find ! -z 64M /mnt/lustre/*
+
+# lfs find ! -z +64M /mnt/lustre/*
+/mnt/lustre/file
+
+# lfs find ! -z -64M /mnt/lustre/*
+/mnt/lustre/file
+
+# lfs find ! -z +63M /mnt/lustre/*
+
+# lfs find ! -z -65M /mnt/lustre/*
+
+# lfs find ! -z 65M /mnt/lustre/*
+/mnt/lustre/file</screen>
+ </section>
+ </section>
+
+ <section xml:id="foreign_layout" condition='l2D'>
+ <title>
+ <indexterm><primary>striping</primary><secondary>Foreign</secondary>
+ </indexterm>Foreign Layout</title>
+ <para>The Lustre Foreign Layout feature is an extension of both the
+ LOV and LMV formats which allows the creation of empty files and directories
+ with the necessary specifications to point to corresponding objects outside
+ from Lustre namespace.</para>
+ <para>The new LOV/LMV foreign internal format can be represented as:</para>
+ <figure xml:id="managinglayout.fig.foreign_format">
+ <title>LOV/LMV foreign format</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" width="100%"
+ fileref="figures/Foreign_Format.png" />
+ </imageobject>
+ <textobject>
+ <phrase>LOV/LMV foreign format</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <section>
+ <title><literal>lfs set[dir]stripe</literal></title>
+ <para>The <literal>lfs set[dir]stripe</literal> commands are used to
+ create files or directories with foreign layouts, by calling the
+ corresponding API, itself invoking the appropriate ioctl().</para>
+ <section>
+ <title>Create a Foreign file/dir</title>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs set[dir]stripe \
+--foreign[=<foreign_type>] --xattr|-x <layout_string> \
+[--flags <hex_bitmask>] [--mode <mode_bits>] \
+<replaceable>{file,dir}name</replaceable></screen>
+ <para>Both the <literal>--foreign</literal> and
+ <literal>--xattr|-x</literal> options are mandatory.
+ The <literal><foreign_type></literal> (default is "none", meaning
+ no special behavior), and both <literal>--flags</literal> and
+ <literal>--mode</literal> (default is 0666) options are optional.</para>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <para>The following command creates a foreign file of "none" type and
+ with "foo@bar" LOV content and specific mode and flags:
+ <screen># lfs setstripe --foreign=none --flags=0xda08 --mode=0640 \
+--xattr=foo@bar /mnt/lustre/file</screen>
+ <figure xml:id="managinglayout.fig.foreign_createfile">
+ <title>Example: create a foreign file</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" width="100%" align="center"
+ fileref="figures/Foreign_Createfile.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: create a foreign file</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ </para>
+ </section>
+ </section>
+ <section>
+ <title><literal>lfs get[dir]stripe</literal></title>
+ <para><literal>lfs get[dir]stripe</literal> commands can be used to
+ retrieve foreign LOV/LMV informations and content.</para>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs get[dir]stripe [-v] <replaceable>filename</replaceable></screen>
+ <para><emphasis role="bold">List foreign layout information
+ </emphasis></para>
+ <para>Suppose we already have a foreign file
+ <literal>/mnt/lustre/file</literal>, created by the following command:</para>
+ <screen># lfs setstripe --foreign=none --flags=0xda08 --mode=0640 \
+--xattr=foo@bar /mnt/lustre/file</screen>
+ <para>The full foreign layout informations can be listed using the
+ following command:</para>
+ <screen># lfs getstripe -v /mnt/lustre/file
+/mnt/lustre/file
+ lfm_magic: 0x0BD70BD0
+ lfm_length: 7
+ lfm_type: none
+ lfm_flags: 0x0000DA08
+ lfm_value: foo@bar
+ </screen>
+ <note><para>As you can see the <literal>lfm_length</literal> field
+ value is the characters number in the variable length
+ <literal>lfm_value</literal> field.</para></note>
+ </section>
+ <section>
+ <title><literal>lfs find</literal></title>
+ <para><literal>lfs find</literal> commands can be used to search for
+ all the foreign files/directories or those that match the given
+ selection paremeters.</para>
+ <screen>lfs find
+[[!] --foreign[=<foreign_type>]</screen>
+ <para>The <literal>--foreign[=<foreign_type>]</literal> option
+ has been added to specify that all [!,but not] files and/or directories
+ with a foreign layout [and [!,but not] of
+ <literal><foreign_type></literal>] will be retrieved.</para>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <screen># lfs setstripe --foreign=none --xattr=foo@bar /mnt/lustre/file
+# touch /mnt/lustre/file2
+
+# lfs find --foreign /mnt/lustre/*
+/mnt/lustre/file
+
+# lfs find ! --foreign /mnt/lustre/*
+/mnt/lustre/file2
+
+# lfs find --foreign=none /mnt/lustre/*
+/mnt/lustre/file</screen>
+ </section>
+ </section>
+
+ <section xml:id="file_striping.managing_free_space">
<title><indexterm>
<primary>space</primary>
<secondary>free space</secondary>
preference to location (spreading out stripes across OSSs to increase network bandwidth
utilization) and the weighted allocator gives preference to available space (balancing loads
across OSTs). Threshold and weighting factors for these two algorithms can be adjusted by the
- user. This section describes how to check available free space on disks and how free space is
- allocated. It then describes how to set the threshold and weighting factors for the allocation
- algorithms.</para>
- <section xml:id="dbdoclet.50438209_35838">
+ user. The MDT reserves 0.1 percent of total OST space and 32 inodes for each OST. The MDT
+ stops object allocation for the OST if available space is less than reserved or the OST has
+ fewer than 32 free inodes. The MDT starts object allocation when available space is twice
+ as big as the reserved space and the OST has more than 64 free inodes. Note, clients
+ could append existing files no matter what object allocation state is.</para>
+ <para condition="l29"> The reserved space for each OST can be adjusted by the user. Use the
+ <literal>lctl set_param</literal> command, for example the next command reserve 1GB space
+ for all OSTs.
+ <screen>lctl set_param -P osp.*.reserved_mb_low=1024</screen></para>
+ <para>This section describes how to check available free space on disks
+ and how free space is allocated. It then describes how to set the
+ threshold and weighting factors for the allocation algorithms.</para>
+ <section xml:id="file_striping.checking_free_space">
<title>Checking File System Free Space</title>
- <para>Free space is an important consideration in assigning file stripes. The <literal>lfs
- df</literal> command can be used to show available disk space on the mounted Lustre file
- system and space consumption per OST. If multiple Lustre file systems are mounted, a path
- may be specified, but is not required. Options to the <literal>lfs df</literal> command are
- shown below.</para>
+ <para>Free space is an important consideration in assigning file stripes.
+ The <literal>lfs df</literal> command can be used to show available
+ disk space on the mounted Lustre file system and space consumption per
+ OST. If multiple Lustre file systems are mounted, a path may be
+ specified, but is not required. Options to the <literal>lfs df</literal>
+ command are shown below.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<tbody>
<row>
<entry>
- <para> <literal>-h</literal></para>
+ <para>
+ <literal>-h</literal>, <literal>--human-readable</literal>
+ </para>
+ </entry>
+ <entry>
+ <para> Displays sizes in human readable format (for example: 1K,
+ 234M, 5G) using base-2 (binary) values (i.e. 1G = 1024M).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <literal>-H</literal>, <literal>--si</literal>
+ </para>
</entry>
<entry>
- <para> Displays sizes in human readable format (for example: 1K, 234M, 5G).</para>
+ <para>Like <literal>-h</literal>, this displays counts in human
+ readable format, but using base-10 (decimal) values
+ (i.e. 1G = 1000M).</para>
</entry>
</row>
<row>
<para> Lists inodes instead of block usage.</para>
</entry>
</row>
+ <row>
+ <entry>
+ <para> <literal role="bold">-l, --lazy</literal></para>
+ </entry>
+ <entry>
+ <para>Do not attempt to contact any OST or MDT not currently
+ connected to the client. This avoids blocking the
+ <literal>lfs df</literal> output if a target is offline or
+ unreachable, and only returns the space on OSTs that can
+ currently be accessed.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal role="bold">-p, --pool</literal></para>
+ </entry>
+ <entry>
+ <para>Limit the usage to report only OSTs that are in the
+ specified <replaceable>pool</replaceable>. If multiple
+ Lustre filesystems are mounted, list the OSTs in
+ <replaceable>pool</replaceable> for each filesystem, or
+ limit the display to only a pool for a specific filesystem
+ if <replaceable>fsname.pool</replaceable> is given.
+ Specifying both <replaceable>fsname</replaceable> and
+ <replaceable>pool</replaceable> is equivalent to providing
+ a specific mountpoint.
+ </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <literal>-v</literal>, <literal>--verbose</literal>
+ </para>
+ </entry>
+ <entry>
+ <para>Display verbose status of MDTs and OSTs. This may
+ include one or more optional flags at the end of each line.
+ </para>
+ </entry>
+ </row>
</tbody>
</tgroup>
</informaltable>
+ <para>
+ <literal>lfs df</literal> may also report additional target status
+ as the last column in the display, if there are issues with that target.
+ Target states include:
+ </para>
+ <itemizedlist>
+ <listitem><para>
+ <literal>D</literal>: OST/MDT is <literal>Degraded</literal>.
+ The target has a failed drive in the RAID device, or is
+ undergoing RAID reconstruction. This state is marked on
+ the server automatically for ZFS targets via
+ <literal>zed</literal>, or a (user-supplied) script that
+ monitors the target device and sets
+ "<literal>lctl set_param obdfilter.<replaceable>target</replaceable>.degraded=1</literal>"
+ on the OST. This target will be avoided for new
+ allocations, but will still be used to read existing files
+ located there or if there are not enough non-degraded OSTs
+ to make up a widely-striped file.
+ </para></listitem>
+ <listitem><para>
+ <literal>R</literal>: OST/MDT is <literal>Read-only</literal>.
+ The target filesystem is marked read-only due to filesystem
+ corruption detected by ldiskfs or ZFS. No modifications
+ are allowed on this OST, and it needs to be unmounted and
+ <literal>e2fsck</literal> or <literal>zpool scrub</literal>
+ run to repair the underlying filesystem.
+ </para></listitem>
+ <listitem><para>
+ <literal>N</literal>: OST/MDT is <literal>No-precreate</literal>.
+ The target is configured to deny object precreation set by
+ "<literal>lctl set_param obdfilter.<replaceable>target</replaceable>.no_precreate=1</literal>"
+ parameter or the "<literal>-o no_precreate</literal>" mount option.
+ This may be done to add an OST to the filesystem without allowing
+ objects to be allocated on it yet, or for other reasons.
+ </para></listitem>
+ <listitem><para>
+ <literal>S</literal>: OST/MDT is out of <literal>Space</literal>.
+ The target filesystem has less than the minimum required
+ free space and will not be used for new object allocations
+ until it has more free space.
+ </para></listitem>
+ <listitem><para>
+ <literal>I</literal>: OST/MDT is out of <literal>Inodes</literal>.
+ The target filesystem has less than the minimum required
+ free inodes and will not be used for new object allocations
+ until it has more free inodes.
+ </para></listitem>
+ <listitem><para>
+ <literal>f</literal>: OST/MDT is on <literal>flash</literal>.
+ The target filesystem is using a flash (non-rotational)
+ storage device. This is normally detected from the
+ underlying Linux block device, but can be set manually
+ with "<literal>lctl set_param osd-*.*.nonrotational=1</literal>
+ on the respective OSTs. This lower-case status is only
+ shown in conjunction with the <literal>-v</literal> option,
+ since it is not an error condition.
+ </para></listitem>
+ </itemizedlist>
<note>
- <para>The <literal>df -i</literal> and <literal>lfs df -i</literal> commands show the
- <emphasis role="italic">minimum</emphasis> number of inodes that can be created in the
- file system at the current time. If the total number of objects available across all of
- the OSTs is smaller than those available on the MDT(s), taking into account the default
- file striping, then <literal>df -i</literal> will also report a smaller number of inodes
- than could be created. Running <literal>lfs df -i</literal> will report the actual number
- of inodes that are free on each target.</para>
- <para>For ZFS file systems, the number of inodes that can be created is dynamic and depends
- on the free space in the file system. The Free and Total inode counts reported for a ZFS
- file system are only an estimate based on the current usage for each target. The Used
- inode count is the actual number of inodes used by the file system.</para>
+ <para>The <literal>df -i</literal> and <literal>lfs df -i</literal>
+ commands show the <emphasis role="italic">minimum</emphasis> number
+ of inodes that can be created in the file system at the current time.
+ If the total number of objects available across all of the OSTs is
+ smaller than those available on the MDT(s), taking into account the
+ default file striping, then <literal>df -i</literal> will also
+ report a smaller number of inodes than could be created. Running
+ <literal>lfs df -i</literal> will report the actual number of inodes
+ that are free on each target.
+ </para>
+ <para>For ZFS file systems, the number of inodes that can be created
+ is dynamic and depends on the free space in the file system. The
+ Free and Total inode counts reported for a ZFS file system are only
+ an estimate based on the current usage for each target. The Used
+ inode count is the actual number of inodes used by the file system.
+ </para>
</note>
<para><emphasis role="bold">Examples</emphasis></para>
- <screen>[client1] $ lfs df
-UUID 1K-blockS Used Available Use% Mounted on
-mds-lustre-0_UUID 9174328 1020024 8154304 11% /mnt/lustre[MDT:0]
-ost-lustre-0_UUID 94181368 56330708 37850660 59% /mnt/lustre[OST:0]
-ost-lustre-1_UUID 94181368 56385748 37795620 59% /mnt/lustre[OST:1]
-ost-lustre-2_UUID 94181368 54352012 39829356 57% /mnt/lustre[OST:2]
-filesystem summary: 282544104 167068468 39829356 57% /mnt/lustre
-
-[client1] $ lfs df -h
-UUID bytes Used Available Use% Mounted on
-mds-lustre-0_UUID 8.7G 996.1M 7.8G 11% /mnt/lustre[MDT:0]
-ost-lustre-0_UUID 89.8G 53.7G 36.1G 59% /mnt/lustre[OST:0]
-ost-lustre-1_UUID 89.8G 53.8G 36.0G 59% /mnt/lustre[OST:1]
-ost-lustre-2_UUID 89.8G 51.8G 38.0G 57% /mnt/lustre[OST:2]
-filesystem summary: 269.5G 159.3G 110.1G 59% /mnt/lustre
-
-[client1] $ lfs df -i
-UUID Inodes IUsed IFree IUse% Mounted on
-mds-lustre-0_UUID 2211572 41924 2169648 1% /mnt/lustre[MDT:0]
-ost-lustre-0_UUID 737280 12183 725097 1% /mnt/lustre[OST:0]
-ost-lustre-1_UUID 737280 12232 725048 1% /mnt/lustre[OST:1]
-ost-lustre-2_UUID 737280 12214 725066 1% /mnt/lustre[OST:2]
-filesystem summary: 2211572 41924 2169648 1% /mnt/lustre[OST:2]</screen>
+ <screen>client$ lfs df
+UUID 1K-blocks Used Available Use% Mounted on
+testfs-OST0000_UUID 9174328 1020024 8154304 11% /mnt/lustre[MDT:0]
+testfs-OST0000_UUID 94181368 56330708 37850660 59% /mnt/lustre[OST:0]
+testfs-OST0001_UUID 94181368 56385748 37795620 59% /mnt/lustre[OST:1]
+testfs-OST0002_UUID 94181368 54352012 39829356 57% /mnt/lustre[OST:2]
+filesystem summary: 282544104 167068468 39829356 57% /mnt/lustre
+
+[client1] $ lfs df -hv
+UUID bytes Used Available Use% Mounted on
+testfs-MDT0000_UUID 8.7G 996.1M 7.8G 11% /mnt/lustre[MDT:0]
+testfs-OST0000_UUID 89.8G 53.7G 36.1G 59% /mnt/lustre[OST:0] f
+testfs-OST0001_UUID 89.8G 53.8G 36.0G 59% /mnt/lustre[OST:1] f
+testfs-OST0002_UUID 89.8G 51.8G 38.0G 57% /mnt/lustre[OST:2] f
+filesystem summary: 269.5G 159.3G 110.1G 59% /mnt/lustre
+
+[client1] $ lfs df -iH
+UUID Inodes IUsed IFree IUse% Mounted on
+testfs-MDT0000_UUID 2.21M 41.9k 2.17M 1% /mnt/lustre[MDT:0]
+testfs-OST0000_UUID 737.3k 12.1k 725.1k 1% /mnt/lustre[OST:0]
+testfs-OST0001_UUID 737.3k 12.2k 725.0k 1% /mnt/lustre[OST:1]
+testfs-OST0002_UUID 737.3k 12.2k 725.0k 1% /mnt/lustre[OST:2]
+filesystem summary: 2.21M 41.9k 2.17M 1% /mnt/lustre[OST:2]
+</screen>
</section>
<section remap="h3">
<title><indexterm>
necessarily chosen each time.</para>
</listitem>
</itemizedlist>
- <para>The allocation method is determined by the amount of free-space imbalance on the OSTs.
- When free space is relatively balanced across OSTs, the faster round-robin allocator is
- used, which maximizes network balancing. The weighted allocator is used when any two OSTs
- are out of balance by more than the specified threshold (17% by default). The threshold
- between the two allocation methods is defined in the file
- <literal>/proc/fs/<replaceable>fsname</replaceable>/lov/<replaceable>fsname</replaceable>-mdtlov/qos_threshold_rr</literal>. </para>
- <para>To set the <literal>qos_threshold_r</literal> to <literal>25</literal>, enter this
- command on the
- MGS:<screen>lctl set_param lov.<replaceable>fsname</replaceable>-mdtlov.quos_threshold_rr=25</screen></para>
+ <para>The allocation method is determined by the amount of free-space
+ imbalance on the OSTs. When free space is relatively balanced across
+ OSTs, the faster round-robin allocator is used, which maximizes network
+ balancing. The weighted allocator is used when any two OSTs are out of
+ balance by more than the specified threshold (17% by default). The
+ threshold between the two allocation methods is defined by the
+ <literal>qos_threshold_rr</literal> parameter. </para>
+ <para>To temporarily set the <literal>qos_threshold_rr</literal> to
+ <literal>25</literal>, enter the folowing on each MDS:
+ <screen>mds# lctl set_param lod.<replaceable>fsname</replaceable>*.qos_threshold_rr=25</screen></para>
</section>
<section remap="h3">
<title><indexterm>
<primary>space</primary>
<secondary>location weighting</secondary>
</indexterm>Adjusting the Weighting Between Free Space and Location</title>
- <para>The weighting priority used by the weighted allocator is set in the file
- <literal>/proc/fs/<replaceable>fsname</replaceable>/lov/<replaceable>fsname</replaceable>-mdtlov/qos_prio_free</literal>.
- Increasing the value of <literal>qos_prio_free</literal> puts more weighting on the amount
- of free space available on each OST and less on how stripes are distributed across OSTs. The
- default value is <literal>91</literal> (percent). When the free space priority is set to
+ <para>The weighting priority used by the weighted allocator is set by the
+ the <literal>qos_prio_free</literal> parameter.
+ Increasing the value of <literal>qos_prio_free</literal> puts more
+ weighting on the amount of free space available on each OST and less
+ on how stripes are distributed across OSTs. The default value is
+ <literal>91</literal> (percent). When the free space priority is set to
<literal>100</literal> (percent), weighting is based entirely on free space and location
- is no longer used by the striping algorthm. </para>
- <para>To change the allocator weighting to <literal>100</literal>, enter this command on the
+ is no longer used by the striping algorithm. </para>
+ <para>To permanently change the allocator weighting to <literal>100</literal>, enter this command on the
MGS:</para>
- <screen>lctl conf_param <replaceable>fsname</replaceable>-MDT0000.lov.qos_prio_free=100</screen>
+ <screen>lctl conf_param <replaceable>fsname</replaceable>-MDT0000-*.lod.qos_prio_free=100</screen>
<para> .</para>
<note>
<para>When <literal>qos_prio_free</literal> is set to <literal>100</literal>, a weighted
</note>
</section>
</section>
- <section xml:id="section_syy_gcl_qk">
+ <section xml:id="wide_striping">
<title><indexterm>
<primary>striping</primary>
<secondary>wide striping</secondary>
</indexterm><indexterm>
<primary>wide striping</primary>
</indexterm>Lustre Striping Internals</title>
- <para>For Lustre releases prior to Lustre release 2.2, files can be striped across a maximum of
- 160 OSTs. Lustre inodes use an extended attribute to record the location of each object (the
- object ID and the number of the OST on which it is stored). The size of the extended attribute
- limits the maximum stripe count to 160 objects.</para>
- <para condition="l22">In Lustre release 2.2 and subsequent releases, the maximum number of OSTs
- over which files can be striped has been raised to 2000 by allocating a new block on which to
- store the extended attribute that holds the object information. This feature, known as "wide
- striping," only allocates the additional extended attribute data block if the file is striped
- with a stripe count greater than 160. The file layout (object ID, OST number) is stored on the
- new data block with a pointer to this block stored in the original Lustre inode for the file.
- For files smaller than 160 objects, the Lustre inode is used to store the file layout.</para>
+ <para>Individual files can only be striped over a finite number of OSTs,
+ based on the maximum size of the attributes that can be stored on the MDT.
+ If the MDT is ldiskfs-based without the <literal>ea_inode</literal>
+ feature, a file can be striped across at most 160 OSTs. With ZFS-based
+ MDTs, or if the <literal>ea_inode</literal> feature is enabled for an
+ ldiskfs-based MDT, a file can be striped across up to 2000 OSTs.
+ </para>
+ <para>Lustre inodes use an extended attribute to record on which OST each
+ object is located, and the identifier each object on that OST. The size of
+ the extended attribute is a function of the number of stripes.</para>
+ <para>If using an ldiskfs-based MDT, the maximum number of OSTs over which
+ files can be striped can been raised to 2000 by enabling the
+ <literal>ea_inode</literal> feature on the MDT:
+ <screen>tune2fs -O ea_inode /dev/<replaceable>mdtdev</replaceable></screen>
+ </para>
+ <note condition='l2D'><para>Since Lustre 2.13 the
+ <literal>ea_inode</literal> feature is enabled by default on all newly
+ formatted ldiskfs MDT filesystems.</para></note>
+ <note><para>The maximum stripe count for a single file does not limit the
+ maximum number of OSTs that are in the filesystem as a whole, only the
+ maximum possible size and maximum aggregate bandwidth for the file.
+ </para></note>
</section>
</chapter>
+<!--
+ vim:expandtab:shiftwidth=2:tabstop=8:
+ -->
git://git.whamcloud.com / doc / manual.git / blobdiff