-<?xml version="1.0" encoding="UTF-8"?>
-<article version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink">
- <info>
- <title>Managing File Striping and Free Space</title>
- </info>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
-
-
- <tbody>
- <row>
- <entry align="left"><para>Lustre 2.0 Operations Manual</para></entry>
- <entry align="right" valign="top"><para><link xl:href="index.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/toc01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/toc01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="BackupAndRestore.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/prev01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/prev01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="ManagingFileSystemIO.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/next01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/next01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="ix.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/index01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/index01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link></para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><link xl:href=""/></para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
-
- <tbody>
- <row>
- <entry align="right"><para><anchor xml:id="dbdoclet.50438209_pgfId-874" xreflabel=""/>C H A P T E R 18<anchor xml:id="dbdoclet.50438209_21279" xreflabel=""/></para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
-
- <tbody>
- <row>
- <entry align="right"><para><anchor xml:id="dbdoclet.50438209_pgfId-5529" xreflabel=""/><anchor xml:id="dbdoclet.50438209_66186" xreflabel=""/>Managing File Striping and Free Space</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291802" xreflabel=""/>This chapter describes file striping and I/O options, and includes the following sections:</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291806" xreflabel=""/><link xl:href="ManagingStripingFreeSpace.html#50438209_79324">How Lustre Striping Works</link></para>
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingstripingfreespace">
+ <title xml:id="managingstripingfreespace.title">Managing File Layout (Striping) and Free
+ Space</title>
+ <para>This chapter describes file layout (striping) and I/O options, and includes the following
+ sections:</para>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438209_79324"/></para>
</listitem>
-<listitem>
- <para> </para>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438209_48033"/></para>
</listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296869" xreflabel=""/><link xl:href="ManagingStripingFreeSpace.html#50438209_48033">Lustre File Striping Considerations</link></para>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438209_78664"/></para>
</listitem>
-<listitem>
- <para> </para>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438209_44776"/></para>
</listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291810" xreflabel=""/><link xl:href="ManagingStripingFreeSpace.html#50438209_78664">Setting the File Layout/Striping Configuration (lfs setstripe)</link></para>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438209_10424"/></para>
</listitem>
-<listitem>
- <para> </para>
+ <listitem>
+ <para><xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/></para>
</listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297090" xreflabel=""/><link xl:href="ManagingStripingFreeSpace.html#50438209_44776">Retrieving File Layout/Striping Information (getstripe)</link></para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1293633" xreflabel=""/><link xl:href="ManagingStripingFreeSpace.html#50438209_10424">Managing Free Space</link></para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-</itemizedlist>
- <section remap="h2">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1291833" xreflabel=""/></title>
- <section remap="h2">
- <title>18.1 <anchor xml:id="dbdoclet.50438209_79324" xreflabel=""/>How Lustre Striping Works</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296776" xreflabel=""/>Lustre uses a round-robin algorithm for selecting the next OST to which a stripe is to be written. Normally the usage of OSTs is well balanced. However, if users create a small number of exceptionally large files or incorrectly specify striping parameters, imbalanced OST usage may result.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296839" xreflabel=""/>The MDS allocates objects on seqential OSTs. Periodically, it will adjust the striping layout to eliminate some degenerated cases where applications that create very regular file layouts (striping patterns) would preferentially use a particular OST in the sequence.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297552" xreflabel=""/>Stripes are written to sequential OSTs until free space across the OSTs differs by more than 20%. The MDS will then use weighted random allocations with a preference for allocating objects on OSTs with more free space. This can reduce I/O performance until space usage is rebalanced to within 20% again.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296783" xreflabel=""/>For a more detailed description of stripe assignments, see <link xl:href="ManagingStripingFreeSpace.html#50438209_10424">Managing Free Space</link>.</para>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438209_79324">
+ <title>
+ <indexterm>
+ <primary>space</primary>
+ </indexterm>
+ <indexterm>
+ <primary>striping</primary>
+ <secondary>how it works</secondary>
+ </indexterm>
+ <indexterm>
+ <primary>striping</primary>
+ <see>space</see>
+ </indexterm>
+ <indexterm>
+ <primary>space</primary>
+ <secondary>striping</secondary>
+ </indexterm>How Lustre File System Striping Works</title>
+ <para>In a Lustre file system, the MDS allocates objects to OSTs using either a round-robin
+ algorithm or a weighted algorithm. When the amount of free space is well balanced (i.e., by
+ default, when the free space across OSTs differs by less than 17%), the round-robin algorithm
+ is used to select the next OST to which a stripe is to be written. Periodically, the MDS
+ adjusts the striping layout to eliminate some degenerated cases in which applications that
+ create very regular file layouts (striping patterns) preferentially use a particular OST in
+ the sequence.</para>
+ <para> Normally the usage of OSTs is well balanced. However, if users create a small number of
+ exceptionally large files or incorrectly specify striping parameters, imbalanced OST usage may
+ result. When the free space across OSTs differs by more than a specific amount (17% by
+ default), the MDS then uses weighted random allocations with a preference for allocating
+ objects on OSTs with more free space. (This can reduce I/O performance until space usage is
+ rebalanced again.) For a more detailed description of how striping is allocated, see <xref
+ linkend="dbdoclet.50438209_10424"/>.</para>
+ <para>Files can only be striped over a finite number of OSTs, based on the
+ maximum size of the attributes that can be stored on the MDT. If the MDT
+ is ldiskfs-based without the <literal>ea_inode</literal> feature, a file
+ can be striped across at most 160 OSTs. With a ZFS-based MDT, or if the
+ <literal>ea_inode</literal> feature is enabled for an ldiskfs-based MDT,
+ a file can be striped across up to 2000 OSTs. For more information, see
+ <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping"/>.
+ </para>
+ </section>
+ <section xml:id="dbdoclet.50438209_48033">
+ <title><indexterm>
+ <primary>file layout</primary>
+ <secondary>See striping</secondary>
+ </indexterm><indexterm>
+ <primary>striping</primary>
+ <secondary>considerations</secondary>
+ </indexterm>
+ <indexterm>
+ <primary>space</primary>
+ <secondary>considerations</secondary>
+ </indexterm> Lustre File Layout (Striping) Considerations</title>
+ <para>Whether you should set up file striping and what parameter values you select depends on
+ your needs. A good rule of thumb is to stripe over as few objects as will meet those needs and
+ no more.</para>
+ <para>Some reasons for using striping include:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">Providing high-bandwidth access.</emphasis> Many applications
+ require high-bandwidth access to a single file, which may be more bandwidth than can be
+ provided by a single OSS. Examples are a scientific application that writes to a single
+ file from hundreds of nodes, or a binary executable that is loaded by many nodes when an
+ application starts.</para>
+ <para>In cases like these, a file can be striped over as many OSSs as it takes to achieve
+ the required peak aggregate bandwidth for that file. Striping across a larger number of
+ OSSs should only be used when the file size is very large and/or is accessed by many nodes
+ at a time. Currently, Lustre files can be striped across up to 2000 OSTs, the maximum
+ stripe count for an <literal>ldiskfs</literal> file system.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Improving performance when OSS bandwidth is exceeded.</emphasis>
+ Striping across many OSSs can improve performance if the aggregate client bandwidth
+ exceeds the server bandwidth and the application reads and writes data fast enough to take
+ advantage of the additional OSS bandwidth. The largest useful stripe count is bounded by
+ the I/O rate of the clients/jobs divided by the performance per OSS.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Providing space for very large files.</emphasis> Striping is
+ useful when a single OST does not have enough free space to hold the entire file.</para>
+ </listitem>
+ </itemizedlist>
+ <para>Some reasons to minimize or avoid striping:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">Increased overhead.</emphasis> Striping results in more locks
+ and extra network operations during common operations such as <literal>stat</literal> and
+ <literal>unlink</literal>. Even when these operations are performed in parallel, one
+ network operation takes less time than 100 operations.</para>
+ <para>Increased overhead also results from server contention. Consider a cluster with 100
+ clients and 100 OSSs, each with one OST. If each file has exactly one object and the load
+ is distributed evenly, there is no contention and the disks on each server can manage
+ sequential I/O. If each file has 100 objects, then the clients all compete with one
+ another for the attention of the servers, and the disks on each node seek in 100 different
+ directions resulting in needless contention.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Increased risk.</emphasis> When files are striped across all
+ servers and one of the servers breaks down, a small part of each striped file is lost. By
+ comparison, if each file has exactly one stripe, fewer files are lost, but they are lost
+ in their entirety. Many users would prefer to lose some of their files entirely than all
+ of their files partially.</para>
+ </listitem>
+ </itemizedlist>
+ <section remap="h3">
+ <title><indexterm><primary>striping</primary><secondary>size</secondary></indexterm>
+ Choosing a Stripe Size</title>
+ <para>Choosing a stripe size is a balancing act, but reasonable defaults are described below.
+ The stripe size has no effect on a single-stripe file.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">The stripe size must be a multiple of the page
+ size.</emphasis> Lustre software tools enforce a multiple of 64 KB (the maximum page
+ size on ia64 and PPC64 nodes) so that users on platforms with smaller pages do not
+ accidentally create files that might cause problems for ia64 clients.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">The smallest recommended stripe size is 512 KB.</emphasis>
+ Although you can create files with a stripe size of 64 KB, the smallest practical stripe
+ size is 512 KB because the Lustre file system sends 1MB chunks over the network.
+ Choosing a smaller stripe size may result in inefficient I/O to the disks and reduced
+ performance.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">A good stripe size for sequential I/O using high-speed
+ networks is between 1 MB and 4 MB.</emphasis> In most situations, stripe sizes larger
+ than 4 MB may result in longer lock hold times and contention during shared file
+ access.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">The maximum stripe size is 4 GB.</emphasis> Using a large
+ stripe size can improve performance when accessing very large files. It allows each
+ client to have exclusive access to its own part of a file. However, a large stripe size
+ can be counterproductive in cases where it does not match your I/O pattern.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Choose a stripe pattern that takes into account the write
+ patterns of your application.</emphasis> Writes that cross an object boundary are
+ slightly less efficient than writes that go entirely to one server. If the file is
+ written in a consistent and aligned way, make the stripe size a multiple of the
+ <literal>write()</literal> size.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438209_78664">
+ <title><indexterm>
+ <primary>striping</primary>
+ <secondary>configuration</secondary>
+ </indexterm>Setting the File Layout/Striping Configuration (<literal>lfs
+ setstripe</literal>)</title>
+ <para>Use the <literal>lfs setstripe</literal> command to create new files with a specific file layout (stripe pattern) configuration.</para>
+ <screen>lfs setstripe [--size|-s stripe_size] [--count|-c stripe_count] \
+[--index|-i start_ost] [--pool|-p pool_name] <replaceable>filename|dirname</replaceable> </screen>
+ <para><emphasis role="bold">
+ <literal>stripe_size</literal>
+ </emphasis>
+ </para>
+ <para>The <literal>stripe_size</literal> indicates how much data to write to one OST before
+ moving to the next OST. The default <literal>stripe_size</literal> is 1 MB. Passing a
+ <literal>stripe_size</literal> of 0 causes the default stripe size to be used. Otherwise,
+ the <literal>stripe_size</literal> value must be a multiple of 64 KB.</para>
+ <para><emphasis role="bold">
+ <literal>stripe_count</literal>
+ </emphasis>
+ </para>
+ <para>The <literal>stripe_count</literal> indicates how many OSTs to use. The default <literal>stripe_count</literal> value is 1. Setting <literal>stripe_count</literal> to 0 causes the default stripe count to be used. Setting <literal>stripe_count</literal> to -1 means stripe over all available OSTs (full OSTs are skipped).</para>
+ <para><emphasis role="bold">
+ <literal>start_ost</literal>
+ </emphasis>
+ </para>
+ <para>The start OST is the first OST to which files are written. The default value for
+ <literal>start_ost</literal> is -1, which allows the MDS to choose the starting index. This
+ setting is strongly recommended, as it allows space and load balancing to be done by the MDS
+ as needed. If the value of <literal>start_ost</literal> is set to a value other than -1, the
+ file starts on the specified OST index. OST index numbering starts at 0.</para>
+ <note>
+ <para>If the specified OST is inactive or in a degraded mode, the MDS will silently choose
+ another target.</para>
+ </note>
+ <note>
+ <para>If you pass a <literal>start_ost</literal> value of 0 and a
+ <literal>stripe_count</literal> value of <emphasis>1</emphasis>, all files are written to
+ OST 0, until space is exhausted. <emphasis role="italic">This is probably not what you meant
+ to do.</emphasis> If you only want to adjust the stripe count and keep the other
+ parameters at their default settings, do not specify any of the other parameters:</para>
+ <para><screen>client# lfs setstripe -c <replaceable>stripe_count</replaceable> <replaceable>filename</replaceable></screen></para>
+ </note>
+ <para><emphasis role="bold">
+ <literal>pool_name</literal>
+ </emphasis>
+ </para>
+ <para>The <literal>pool_name</literal> specifies the OST pool to which the file will be written.
+ This allows limiting the OSTs used to a subset of all OSTs in the file system. For more
+ details about using OST pools, see <link xl:href="ManagingFileSystemIO.html#50438211_75549"
+ >Creating and Managing OST Pools</link>.</para>
+ <section remap="h3">
+ <title>Specifying a File Layout (Striping Pattern) for a Single File</title>
+ <para>It is possible to specify the file layout when a new file is created using the command <literal>lfs setstripe</literal>. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an <literal>lfs setstripe</literal> command fails if the file already exists.</para>
+ <section xml:id="dbdoclet.50438209_60155">
+ <title>Setting the Stripe Size</title>
+ <para>The command to create a new file with a specified stripe size is similar to:</para>
+ <screen>[client]# lfs setstripe -s 4M /mnt/lustre/new_file</screen>
+ <para>This example command creates the new file <literal>/mnt/lustre/new_file</literal> with a stripe size of 4 MB.</para>
+ <para>Now, when the file is created, the new stripe setting creates the file on a single OST with a stripe size of 4M:</para>
+ <screen> [client]# lfs getstripe /mnt/lustre/new_file
+/mnt/lustre/4mb_file
+lmm_stripe_count: 1
+lmm_stripe_size: 4194304
+lmm_pattern: 1
+lmm_layout_gen: 0
+lmm_stripe_offset: 1
+obdidx objid objid group
+1 690550 0xa8976 0 </screen>
+ <para>In this example, the stripe size is 4 MB.</para>
+ </section>
+ <section remap="h4">
+ <title><indexterm><primary>striping</primary><secondary>count</secondary></indexterm>
+ Setting the Stripe Count</title>
+ <para>The command below creates a new file with a stripe count of <literal>-1</literal> to
+ specify striping over all available OSTs:</para>
+ <screen>[client]# lfs setstripe -c -1 /mnt/lustre/full_stripe</screen>
+ <para>The example below indicates that the file <literal>full_stripe</literal> is striped
+ over all six active OSTs in the configuration:</para>
+ <screen>[client]# lfs getstripe /mnt/lustre/full_stripe
+/mnt/lustre/full_stripe
+ obdidx objid objid group
+ 0 8 0x8 0
+ 1 4 0x4 0
+ 2 5 0x5 0
+ 3 5 0x5 0
+ 4 4 0x4 0
+ 5 2 0x2 0</screen>
+ <para> This is in contrast to the output in <xref linkend="dbdoclet.50438209_60155"/>, which
+ shows only a single object for the file.</para>
+ </section>
+ </section>
+ <section remap="h3">
+ <title><indexterm>
+ <primary>striping</primary>
+ <secondary>per directory</secondary>
+ </indexterm>Setting the Striping Layout for a Directory</title>
+ <para>In a directory, the <literal>lfs setstripe</literal> command sets a default striping
+ configuration for files created in the directory. The usage is the same as <literal>lfs
+ setstripe</literal> for a regular file, except that the directory must exist prior to
+ setting the default striping configuration. If a file is created in a directory with a
+ default stripe configuration (without otherwise specifying striping), the Lustre file system
+ uses those striping parameters instead of the file system default for the new file.</para>
+ <para>To change the striping pattern for a sub-directory, create a directory with desired file
+ layout as described above. Sub-directories inherit the file layout of the root/parent
+ directory.</para>
+ </section>
+ <section remap="h3">
+ <title><indexterm>
+ <primary>striping</primary>
+ <secondary>per file system</secondary>
+ </indexterm>Setting the Striping Layout for a File System</title>
+ <para>Setting the striping specification on the <literal>root</literal> directory determines
+ the striping for all new files created in the file system unless an overriding striping
+ specification takes precedence (such as a striping layout specified by the application, or
+ set using <literal>lfs setstripe</literal>, or specified for the parent directory).</para>
+ <note>
+ <para>The striping settings for a <literal>root</literal> directory are, by default, applied
+ to any new child directories created in the root directory, unless striping settings have
+ been specified for the child directory.</para>
+ </note>
+ </section>
+ <section remap="h3">
+ <title><indexterm>
+ <primary>striping</primary>
+ <secondary>on specific OST</secondary>
+ </indexterm>Creating a File on a Specific OST</title>
+ <para>You can use <literal>lfs setstripe</literal> to create a file on a specific OST. In the
+ following example, the file <literal>file1</literal> is created on the first OST (OST index
+ is 0).</para>
+ <screen>$ lfs setstripe --count 1 --index 0 file1
+$ dd if=/dev/zero of=file1 count=1 bs=100M
+1+0 records in
+1+0 records out
+
+$ lfs getstripe file1
+/mnt/testfs/file1
+lmm_stripe_count: 1
+lmm_stripe_size: 1048576
+lmm_pattern: 1
+lmm_layout_gen: 0
+lmm_stripe_offset: 0
+ obdidx objid objid group
+ 0 37364 0x91f4 0</screen>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438209_44776">
+ <title><indexterm><primary>striping</primary><secondary>getting information</secondary></indexterm>Retrieving File Layout/Striping Information (<literal>getstripe</literal>)</title>
+ <para>The <literal>lfs getstripe</literal> command is used to display information that shows
+ over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along
+ with the OST index and object ID for each stripe in the file. For directories, the default
+ settings for files created in that directory are displayed.</para>
+ <section remap="h3">
+ <title>Displaying the Current Stripe Size</title>
+ <para>To see the current stripe size for a Lustre file or directory, use the <literal>lfs
+ getstripe</literal> command. For example, to view information for a directory, enter a
+ command similar to:</para>
+ <screen>[client]# lfs getstripe /mnt/lustre </screen>
+ <para>This command produces output similar to:</para>
+ <screen>/mnt/lustre
+(Default) stripe_count: 1 stripe_size: 1M stripe_offset: -1</screen>
+ <para>In this example, the default stripe count is <literal>1</literal> (data blocks are
+ striped over a single OST), the default stripe size is 1 MB, and the objects are created
+ over all available OSTs.</para>
+ <para>To view information for a file, enter a command similar to:</para>
+ <screen>$ lfs getstripe /mnt/lustre/foo
+/mnt/lustre/foo
+lmm_stripe_count: 1
+lmm_stripe_size: 1048576
+lmm_pattern: 1
+lmm_layout_gen: 0
+lmm_stripe_offset: 0
+ obdidx objid objid group
+ 2 835487 m0xcbf9f 0 </screen>
+ <para>In this example, the file is located on <literal>obdidx 2</literal>, which corresponds
+ to the OST <literal>lustre-OST0002</literal>. To see which node is serving that OST, run:
+ <screen>$ lctl get_param osc.lustre-OST0002-osc.ost_conn_uuid
+osc.lustre-OST0002-osc.ost_conn_uuid=192.168.20.1@tcp</screen></para>
+ </section>
+ <section remap="h3">
+ <title>Inspecting the File Tree</title>
+ <para>To inspect an entire tree of files, use the <literal>lfs find</literal> command:</para>
+ <screen>lfs find [--recursive | -r] <replaceable>file|directory</replaceable> ...</screen>
</section>
- <section remap="h2">
- <title>18.2 Lustre <anchor xml:id="dbdoclet.50438209_48033" xreflabel=""/>File <anchor xml:id="dbdoclet.50438209_marker-1291832" xreflabel=""/>Striping Considerations</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296641" xreflabel=""/>Whether you should set up file striping and what parameter values you select depends on your need. A good rule of thumb is to stripe over as few objects as will meet those needs and no more.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296653" xreflabel=""/>Some reasons for using striping include:</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296634" xreflabel=""/><emphasis role="bold">Providing high-bandwidth access</emphasis> - Many applications require high-bandwidth access to a single file - more bandwidth than can be provided by a single OSS. For example, scientific applications that write to a single file from hundreds of nodes, or a binary executable that is loaded by many nodes when an application starts.</para>
+ <section>
+ <title><indexterm>
+ <primary>striping</primary>
+ <secondary>remote directories</secondary>
+ </indexterm>Locating the MDT for a remote directory</title>
+ <para condition="l24">Lustre software release 2.4 can be configured with
+ multiple MDTs in the same file system. Each sub-directory can have a
+ different MDT. To identify on which MDT a given subdirectory is
+ located, pass the <literal>getstripe [--mdt-index|-M]</literal>
+ parameters to <literal>lfs</literal>. An example of this command is
+ provided in the section <xref linkend="dbdoclet.rmremotedir"/>.</para>
+ </section>
+ </section>
+ <section xml:id="pfl" condition='l2A'>
+ <title><indexterm>
+ <primary>striping</primary>
+ <secondary>PFL</secondary>
+ </indexterm>Progressive File Layout(PFL)</title>
+ <para>The Lustre Progressive File Layout (PFL) feature simplifies the use
+ of Lustre so that users can expect reasonable performance for a variety of
+ normal file IO patterns without the need to explicitly understand their IO
+ model or Lustre usage details in advance. In particular, users do not
+ necessarily need to know the size or concurrency of output files in
+ advance of their creation and explicitly specify an optimal layout for
+ each file in order to achieve good performance for both highly concurrent
+ shared-single-large-file IO or parallel IO to many smaller per-process
+ files. </para>
+ <para>The layout of a PFL file is stored on disk as <literal>composite
+ layout</literal>. A PFL file is essentially an array of
+ <literal>sub-layout components</literal>, with each sub-layout component
+ being a plain layout covering different and non-overlapped extents of
+ the file. For PFL files, the file layout is composed of a series of
+ components, therefore it's possible that there are some file extents are
+ not described by any components.</para>
+ <para>An example of how data blocks of PFL files are mapped to OST objects
+ of components is shown in the following PFL object mapping diagram:</para>
+ <figure xml:id="managinglayout.fig.pfl">
+ <title>PFL object mapping diagram</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" width="100%"
+ fileref="figures/PFL_object_mapping_diagram.png" />
+ </imageobject>
+ <textobject>
+ <phrase>PFL object mapping diagram</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The PFL file in <xref linkend="managinglayout.fig.pfl"/> has 3
+ components and shows the mapping for the blocks of a 2055MB file.
+ The stripe size for the first two components is 1MB, while the stripe size
+ for the third component is 4MB. The stripe count is increasing for each
+ successive component. The first component only has two 1MB blocks and the
+ single object has a size of 2MB. The second component holds the next 254MB
+ of the file spread over 4 separate OST objects in RAID-0, each one will
+ have a size of 256MB / 4 objects = 64MB per object. Note the first two
+ objects <literal>obj 2,0</literal> and <literal>obj 2,1</literal>
+ have a 1MB hole at the start where the data is stored in the first
+ component. The final component holds the next 1800MB spread over 32 OST
+ objects. There is a 256MB / 32 = 8MB hole at the start each one for the
+ data stored in the first two components. Each object will be
+ 2048MB / 32 objects = 64MB per object, except the
+ <literal>obj 3,0</literal> that holds an extra 4MB chunk and
+ <literal>obj 3,1</literal> that holds an extra 3MB chunk. If more data
+ was written to the file, only the objects in component 3 would increase
+ in size.</para>
+ <para>When a file range with defined but not instantiated component is
+ accessed, clients will send a Layout Intent RPC to the MDT, and the MDT
+ would instantiate the objects of the components covering that range.
+ </para>
+ <para>Next, some commands for user to operate PFL files are introduced and
+ some examples of possible composite layout are illustrated as well.
+ Lustre provides commands
+ <literal>lfs setstripe</literal> and <literal>lfs migrate</literal> for
+ users to operate PFL files. <literal>lfs setstripe</literal> commands
+ are used to create PFL files, add or delete components to or from an
+ existing composite file; <literal>lfs migrate</literal> commands are used
+ to re-layout the data in existing files using the new layout parameter by
+ copying the data from the existing OST(s) to the new OST(s). Also,
+ as introduced in the previous sections, <literal>lfs getstripe</literal>
+ commands can be used to list the striping/component information for a
+ given PFL file, and <literal>lfs find</literal> commands can be used to
+ search the directory tree rooted at the given directory or file name for
+ the files that match the given PFL component parameters.</para>
+ <note><para>Using PFL files requires both the client and server to
+ understand the PFL file layout, which isn't available for Lustre 2.9 and
+ earlier. And it will not prevent older clients from accessing non-PFL
+ files in the filesystem.</para></note>
+ <section remap="h3">
+ <title><literal>lfs setstripe</literal></title>
+ <para><literal>lfs setstripe</literal> commands are used to create PFL
+ files, add or delete components to or from an existing composite file.
+ (Suppose we have 8 OSTs in the following examples and stripe size is 1MB
+ by default.)</para>
+ <section remap="h4">
+ <title>Create a PFL file</title>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs setstripe
+[--component-end|-E end1] [STRIPE_OPTIONS]
+[--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable></screen>
+ <para>The <literal>-E</literal> option is used to specify the end offset
+ (in bytes or using a suffix “kMGTP”, e.g. 256M) of each component, and
+ it also indicates the following <literal>STRIPE_OPTIONS</literal> are
+ for this component. Each component defines the stripe pattern of the
+ file in the range of [start, end). The first component must start from
+ offset 0 and all components must be adjacent with each other, no holes
+ are allowed, so each extent will start at the end of previous extent.
+ A <literal>-1</literal> end offset or <literal>eof</literal> indicates
+ this is the last component extending to the end of file.</para>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 -E -1 -c -1 -i 4 \
+/mnt/testfs/create_comp</screen>
+ <para>This command creates a file with composite layout illustrated in
+ the following figure. The first component has 1 stripe and covers
+ [0, 4M), the second component has 4 stripes and covers [4M, 64M), and
+ the last component stripes start at OST4, cross over all available
+ OSTs and covers [64M, EOF).</para>
+ <figure xml:id="managinglayout.fig.pfl_create">
+ <title>Example: create a composite file</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.75in" align="center"
+ fileref="figures/PFL_createfile.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: create a composite file</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The composite layout can be output by the following command:</para>
+ <screen>$ lfs getstripe /mnt/testfs/create_comp
+/mnt/testfs/create_comp
+ lcm_layout_gen: 3
+ lcm_entry_count: 3
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 4194304
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
+
+ lcme_id: 2
+ lcme_flags: 0
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1
+ lcme_id: 3
+ lcme_flags: 0
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: -1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 4</screen>
+ <note><para>Only the first component’s OST objects of the PFL file are
+ instantiated when the layout is being set. Other instantiation is
+ delayed to later write/truncate operations.</para></note>
+ <para>If we write 128M data to this PFL file, the second and third
+ components will be instantiated:</para>
+ <screen>$ dd if=/dev/zero of=/mnt/testfs/create_comp bs=1M count=128
+$ lfs getstripe /mnt/testfs/create_comp
+/mnt/testfs/create_comp
+ lcm_layout_gen: 5
+ lcm_entry_count: 3
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 4194304
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
+
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 1
+ lmm_objects:
+ - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
+ - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
+ - 2: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }
+ - 3: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
+
+ lcme_id: 3
+ lcme_flags: init
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 8
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 4
+ lmm_objects:
+ - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x3:0x0] }
+ - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
+ - 2: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
+ - 3: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
+ - 4: { l_ost_idx: 0, l_fid: [0x100000000:0x3:0x0] }
+ - 5: { l_ost_idx: 1, l_fid: [0x100010000:0x3:0x0] }
+ - 6: { l_ost_idx: 2, l_fid: [0x100020000:0x3:0x0] }
+ - 7: { l_ost_idx: 3, l_fid: [0x100030000:0x3:0x0] }</screen>
+ </section>
+ <section remap="h4">
+ <title>Add component(s) to an existing composite file</title>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs setstripe --component-add
+[--component-end|-E end1] [STRIPE_OPTIONS]
+[--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>filename</replaceable></screen>
+ <para>The option <literal>--component-add</literal> is used to add
+ components to an existing composite file. The extent start of
+ the first component to be added is equal to the extent end of last
+ component in the existing file, and all components to be added must
+ be adjacent with each other.</para>
+ <note><para>If the last existing component is specified by
+ <literal>-E -1</literal> or <literal>-E eof</literal>, which covers
+ to the end of the file, it must be deleted before a new one is added.
+ </para></note>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 /mnt/testfs/add_comp
+$ lfs setstripe --component-add -E -1 -c 4 -o 6-7,0,5 \
+/mnt/testfs/add_comp</screen>
+ <para>This command adds a new component which starts from the end of
+ the last existing component to the end of file. The layout of this
+ example is illustrated in
+ <xref linkend="managinglayout.fig.pfl_addcomp"/>. The last component
+ stripes across 4 OSTs in sequence OST6, OST7, OST0 and OST5, covers
+ [64M, EOF).</para>
+ <figure xml:id="managinglayout.fig.pfl_addcomp">
+ <title>Example: add a component to an existing composite file</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.75in" align="center"
+ fileref="figures/PFL_addcomp.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: add a component to an existing composite file
+ </phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The layout can be printed out by the following command:</para>
+ <screen>$ lfs getstripe /mnt/testfs/add_comp
+/mnt/testfs/add_comp
+ lcm_layout_gen: 5
+ lcm_entry_count: 3
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 4194304
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
+
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 1
+ lmm_objects:
+ - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
+ - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
+ - 2: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }
+ - 3: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
+
+ lcme_id: 5
+ lcme_flags: 0
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: -1</screen>
+ <para>The component ID "lcme_id" changes as layout generation
+ changes. It is not necessarily sequential and does not imply ordering
+ of individual components.</para>
+ <note><para>Similar to specifying a full-file composite layout at file
+ creation time, <literal>--component-add</literal> won't instantiate
+ OST objects, the instantiation is delayed to later write/truncate
+ operations. For example, after writing beyond the 64MB start of the
+ file's last component, the new component has had objects allocated:
+ </para></note>
+ <screen>$ lfs getstripe -I5 /mnt/testfs/add_comp
+/mnt/testfs/add_comp
+ lcm_layout_gen: 6
+ lcm_entry_count: 3
+ lcme_id: 5
+ lcme_flags: init
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 6
+ lmm_objects:
+ - 0: { l_ost_idx: 6, l_fid: [0x100060000:0x4:0x0] }
+ - 1: { l_ost_idx: 7, l_fid: [0x100070000:0x4:0x0] }
+ - 2: { l_ost_idx: 0, l_fid: [0x100000000:0x5:0x0] }
+ - 3: { l_ost_idx: 5, l_fid: [0x100050000:0x4:0x0] }</screen>
+ </section>
+ <section remap="h4">
+ <title>Delete component(s) from an existing file</title>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs setstripe --component-del
+[--component-id|-I comp_id | --component-flags comp_flags]
+<replaceable>filename</replaceable></screen>
+ <para>The option <literal>--component-del</literal> is used to remove
+ the component(s) specified by component ID or flags from an existing
+ file. This operation will result in any data stored in the deleted
+ component will be lost.</para>
+ <para>The ID specified by <literal>-I</literal> option is the numerical
+ unique ID of the component, which can be obtained by command
+ <literal>lfs getstripe -I</literal> command, and the flag specified by
+ <literal>--component-flags</literal> option is a certain type of
+ components, which can be obtained by command
+ <literal>lfs getstripe --component-flags</literal>. For now, we only
+ have two flags <literal>init</literal> and <literal>^init</literal>
+ for instantiated and un-instantiated components respectively.</para>
+ <note><para>Deletion must start with the last component because hole is
+ not allowed.</para></note>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <screen>$ lfs getstripe -I /mnt/testfs/del_comp
+1
+2
+5
+$ lfs setstripe --component-del -I 5 /mnt/testfs/del_comp</screen>
+ <para>This example deletes the component with ID 5 from file
+ <literal>/mnt/testfs/del_comp</literal>. If we still use the last
+ example, the final result is illustrated in
+ <xref linkend="managinglayout.fig.pfl_delcomp"/>.</para>
+ <figure xml:id="managinglayout.fig.pfl_delcomp">
+ <title>Example: delete a component from an existing file</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.75in" align="center"
+ fileref="figures/PFL_delcomp.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: delete a component from an existing file</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>If you try to delete a non-last component, you will see the
+ following error:</para>
+ <screen>$ lfs setstripe -component-del -I 2 /mnt/testfs/del_comp
+Delete component 0x2 from /mnt/testfs/del_comp failed. Invalid argument
+error: setstripe: delete component of file '/mnt/testfs/del_comp' failed: Invalid argument</screen>
+ </section>
+ <section remap="h4">
+ <title>Set default PFL layout to an existing directory</title>
+ <para>Similar to create a PFL file, you can set default PFL layout to
+ an existing directory. After that, all the files created will inherit
+ this layout by default.</para>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs setstripe
+[--component-end|-E end1] [STRIPE_OPTIONS]
+[--component-end|-E end2] [STRIPE_OPTIONS] ... <replaceable>dirname</replaceable></screen>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <screen>$ mkdir /mnt/testfs/pfldir
+$ touch /mnt/testfs/pfldir/commonfile
+$ lfs setstripe -E 64M -c 2 -i 0 -E -1 -c 4 -i 0 /mnt/testfs/pfldir</screen>
+ <para>When you run <literal>lfs getstripe</literal>, you will see:
+ </para>
+ <screen>$ lfs getstripe /mnt/testfs/pfldir
+/mnt/testfs/pfldir
+ lcm_layout_gen: 0
+ lcm_entry_count: 2
+ lcme_id: N/A
+ lcme_flags: 0
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 67108864
+ stripe_count: 2 stripe_size: 1048576 stripe_offset: 0
+ lcme_id: N/A
+ lcme_flags: 0
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ stripe_count: 4 stripe_size: 1048576 stripe_offset: 0
+/mnt/testfs/pfldir/commonfile
+lmm_stripe_count: 1
+lmm_stripe_size: 1048576
+lmm_pattern: 1
+lmm_layout_gen: 0
+lmm_stripe_offset: 0
+ obdidx objid objid group
+ 0 9 0x9 0</screen>
+ <para>If you create a file under <literal>/mnt/testfs/pfldir</literal>,
+ the layout of that file will inherit 2 components from its parent
+ directory:</para>
+ <screen>$ touch /mnt/testfs/pfldir/pflfile
+$ lfs getstripe /mnt/testfs/pfldir/pflfile
+/mnt/testfs/pfldir/pflfile
+ lcm_layout_gen: 2
+ lcm_entry_count: 2
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 2
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0xa:0x0] }
+ - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x9:0x0] }
+
+ lcme_id: 2
+ lcme_flags: 0
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0</screen>
+ <note><para>
+ <literal>lfs setstripe --component-add/del</literal> can't be run
+ on a directory, because default layout in directory is likea config,
+ which can be arbitrarily changed by <literal>lfs setstripe</literal>,
+ while layout in file may have data (OST objects) attached. If you want
+ to delete default layout in a directory, please run <literal>lfs
+ setstripe -d <replaceable>dirname</replaceable></literal>, like :
+ <screen>$ lfs setstripe -d /mnt/testfs/pfldir
+$ lfs getstripe -d /mnt/testfs/pfldir
+/mnt/testfs/pfldir
+stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
+/mnt/testfs/pfldir/commonfile
+lmm_stripe_count: 1
+lmm_stripe_size: 1048576
+lmm_pattern: 1
+lmm_layout_gen: 0
+lmm_stripe_offset: 0
+ obdidx objid objid group
+ 0 9 0x9 0</screen>
+ </para></note>
+ </section>
+ </section>
+ <section remap="h3">
+ <title><literal>lfs migrate</literal></title>
+ <para><literal>lfs migrate</literal> commands are used to re-layout the
+ data in the existing files with the new layout parameter by copying the
+ data from the existing OST(s) to the new OST(s).</para>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs migrate [--component-end|-E comp_end] [STRIPE_OPTIONS] ...
+<replaceable>filename</replaceable></screen>
+ <para>The difference between <literal>migrate</literal> and
+ <literal>setstripe</literal> is that <literal>migrate</literal> is to
+ re-layout the data in the existing files, while
+ <literal>setstripe</literal> is to create new files with the specified
+ layout.</para>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <para><emphasis role="bold">Case1. Migrate a normal one to a composite
+ layout</emphasis></para>
+ <screen>$ lfs setstripe -c 1 -S 128K /mnt/testfs/norm_to_2comp
+$ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
+$ lfs getstripe /mnt/testfs/norm_to_2comp --yaml
+/mnt/testfs/norm_to_comp
+lmm_stripe_count: 1
+lmm_stripe_size: 131072
+lmm_pattern: 1
+lmm_layout_gen: 0
+lmm_stripe_offset: 7
+lmm_objects:
+ - l_ost_idx: 7
+ l_fid: 0x100070000:0x2:0x0
+$ lfs migrate -E 1M -S 512K -c 1 -E -1 -S 1M -c 2 \
+/mnt/testfs/norm_to_2comp</screen>
+ <para>In this example, a 5MB size file with 1 stripe and 128K stripe size
+ is migrated to a composite layout file with 2 components, illustrated in
+ <xref linkend="managinglayout.fig.pfl_norm_to_comp"/>.</para>
+ <figure xml:id="managinglayout.fig.pfl_norm_to_comp">
+ <title>Example: migrate normal to composite</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.75in" align="center"
+ fileref="figures/PFL_norm_to_comp.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: migrate normal to composite</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The stripe information after migration is like:</para>
+ <screen>$ lfs getstripe /mnt/testfs/norm_to_2comp
+/mnt/testfs/norm_to_2comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 2
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 1048576
+ lmm_stripe_count: 1
+ lmm_stripe_size: 524288
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
+
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 1048576
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 2
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 2
+ lmm_objects:
+ - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
+ - 1: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }</screen>
+ <para><emphasis role="bold">Case2. Migrate a composite layout to another
+ composite layout</emphasis></para>
+ <screen>$ lfs setstripe -E 1M -S 512K -c 1 -E -1 -S 1M -c 2 \
+/mnt/testfs/2comp_to_3comp
+$ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
+$ lfs migrate -E 1M -S 1M -c 2 -E 4M -S 1M -c 2 -E -1 -S 3M -c 3 \
+/mnt/testfs/2comp_to_3comp</screen>
+ <para>In this example, a composite layout file with 2 components is
+ migrated a composite layout file with 3 components. If we still use
+ the example in case1, the migration process is illustrated in
+ <xref linkend="managinglayout.fig.pfl_comp_to_comp"/>.</para>
+ <figure xml:id="managinglayout.fig.pfl_comp_to_comp">
+ <title>Example: migrate composite to composite</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.75in" align="center"
+ fileref="figures/PFL_comp_to_comp.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: migrate composite to composite</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The stripe information is like:</para>
+ <screen>$ lfs getstripe /mnt/testfs/2comp_to_3comp
+/mnt/testfs/2comp_to_3comp
+ lcm_layout_gen: 6
+ lcm_entry_count: 3
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 1048576
+ lmm_stripe_count: 2
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 4
+ lmm_objects:
+ - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
+ - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
+
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 1048576
+ lcme_extent.e_end: 4194304
+ lmm_stripe_count: 2
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 6
+ lmm_objects:
+ - 0: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
+ - 1: { l_ost_idx: 7, l_fid: [0x100070000:0x3:0x0] }
+
+ lcme_id: 3
+ lcme_flags: init
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: 3
+ lmm_stripe_size: 3145728
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 0
+ lmm_objects:
+ - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x3:0x0] }
+ - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
+ - 2: { l_ost_idx: 2, l_fid: [0x100020000:0x3:0x0] }</screen>
+ <para><emphasis role="bold">Case3. Migrate a composite layout to a
+ normal one</emphasis></para>
+ <screen>$ lfs migrate -E 1M -S 1M -c 2 -E 4M -S 1M -c 2 -E -1 -S 3M -c 3 \
+/mnt/testfs/3comp_to_norm
+$ dd if=/dev/urandom of=/mnt/testfs/norm_to_2comp bs=1M count=5
+$ lfs migrate -c 2 -S 2M /mnt/testfs/3comp_to_normal</screen>
+ <para>In this example, a composite file with 3 components is migrated to
+ a normal file with 2 stripes and 2M stripe size. If we still use the
+ example in Case2, the migration process is illustrated in
+ <xref linkend="managinglayout.fig.pfl_comp_to_norm"/>.</para>
+ <figure xml:id="managinglayout.fig.pfl_comp_to_norm">
+ <title>Example: migrate composite to normal</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" depth="2.75in" align="center"
+ fileref="figures/PFL_comp_to_norm.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Example: migrate composite to normal</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
+ <para>The stripe information is like:</para>
+ <screen>$ lfs getstripe /mnt/testfs/3comp_to_norm --yaml
+/mnt/testfs/3comp_to_norm
+lmm_stripe_count: 2
+lmm_stripe_size: 2097152
+lmm_pattern: 1
+lmm_layout_gen: 7
+lmm_stripe_offset: 4
+lmm_objects:
+ - l_ost_idx: 4
+ l_fid: 0x100040000:0x3:0x0
+ - l_ost_idx: 5
+ l_fid: 0x100050000:0x3:0x0</screen>
+ </section>
+ <section remap="h3">
+ <title><literal>lfs getstripe</literal></title>
+ <para><literal>lfs getstripe</literal> commands can be used to list the
+ striping/component information for a given PFL file. Here, only those
+ parameters new for PFL files are shown.</para>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs getstripe
+[--component-id|-I [comp_id]]
+[--component-flags [comp_flags]]
+[--component-count]
+[--component-start [+-][N][kMGTPE]]
+[--component-end|-E [+-][N][kMGTPE]]
+<replaceable>dirname|filename</replaceable></screen>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <para>Suppose we already have a composite file
+ <literal>/mnt/testfs/3comp</literal>, created by the following
+ command:</para>
+ <screen>$ lfs setstripe -E 4M -c 1 -E 64M -c 4 -E -1 -c -1 -i 4 \
+/mnt/testfs/3comp</screen>
+ <para>And write some data</para>
+ <screen>$ dd if=/dev/zero of=/mnt/testfs/3comp bs=1M count=5</screen>
+ <para><emphasis role="bold">Case1. List component ID and its related
+ information</emphasis></para>
+ <itemizedlist>
+ <listitem>
+ <para>List all the components ID</para>
+ <screen>$ lfs getstripe -I /mnt/testfs/3comp
+1
+2
+3</screen>
</listitem>
-<listitem>
- <para> </para>
+ <listitem>
+ <para>List the detailed striping information of component ID=2</para>
+ <screen>$ lfs getstripe -I2 /mnt/testfs/3comp
+/mnt/testfs/3comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 3
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 5
+ lmm_objects:
+ - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
+ - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
+ - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
+ - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
</listitem>
-</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1294264" xreflabel=""/>In cases like these, a file can be striped over as many OSSs as it takes to achieve the required peak aggregate bandwidth for that file. Striping across a larger number of OSSs should only be used when the file size is very large and/or is accessed by many nodes at a time. Currently, Lustre files can be striped across up to 160 OSSs, the maximum stripe count for an ldiskfs file system.</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1294265" xreflabel=""/><emphasis role="bold">Improving performance when OSS bandwidth is exceeded</emphasis> - Striping across many OSSs can improve performance if the aggregate client bandwidth exceeds the server bandwidth and the application reads and writes data fast enough to take advantage of the additional OSS bandwidth. The largest useful stripe count is bounded by the I/O rate of the clients/jobs divided by the performance per OSS.</para>
+ <listitem>
+ <para>List the stripe offset and stripe count of component ID=2</para>
+ <screen>$ lfs getstripe -I2 -i -c /mnt/testfs/3comp
+ lmm_stripe_count: 4
+ lmm_stripe_offset: 5</screen>
</listitem>
-<listitem>
- <para> </para>
+ </itemizedlist>
+ <para><emphasis role="bold">Case2. List the component which contains the
+ specified flag</emphasis></para>
+ <itemizedlist>
+ <listitem>
+ <para>List the flag of each component</para>
+ <screen>$ lfs getstripe -component-flag -I /mnt/testfs/3comp
+ lcme_id: 1
+ lcme_flags: init
+ lcme_id: 2
+ lcme_flags: init
+ lcme_id: 3
+ lcme_flags: 0</screen>
</listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1294262" xreflabel=""/><emphasis role="bold">Providing space for very large files.</emphasis> Striping is also useful when a single OST does not have enough free space to hold the entire file.</para>
+ <listitem>
+ <para>List component(s) who is not instantiated</para>
+ <screen>$ lfs getstripe --component-flags=^init /mnt/testfs/3comp
+/mnt/testfs/3comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 3
+ lcme_id: 3
+ lcme_flags: 0
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: -1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 4
+ lmm_stripe_offset: 4</screen>
</listitem>
-<listitem>
- <para> </para>
+ </itemizedlist>
+ <para><emphasis role="bold">Case3. List the total number of all the
+ component(s)</emphasis></para>
+ <itemizedlist>
+ <listitem>
+ <para>List the total number of all the components</para>
+ <screen>$ lfs getstripe --component-count /mnt/testfs/3comp
+3</screen>
</listitem>
-</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296431" xreflabel=""/>Some reasons to minimize or avoid striping:</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296706" xreflabel=""/><emphasis role="bold">Increased overhead</emphasis> - Striping results in more locks and extra network operations during common operations such as stat and unlink. Even when these operations are performed in parallel, one network operation takes less time than 100 operations.</para>
+ </itemizedlist>
+ <para><emphasis role="bold">Case4. List the component with the specified
+ extent start or end positions</emphasis></para>
+ <itemizedlist>
+ <listitem>
+ <para>List the start position in bytes of each component</para>
+ <screen>$ lfs getstripe --component-start /mnt/testfs/3comp
+0
+4194304
+67108864</screen>
</listitem>
-<listitem>
- <para> </para>
+ <listitem>
+ <para>List the start position in bytes of component ID=3</para>
+ <screen>$ lfs getstripe --component-start -I3 /mnt/testfs/3comp
+67108864</screen>
</listitem>
-</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291856" xreflabel=""/>Increased overhead also results from server contention. Consider a cluster with 100 clients and 100 OSSs, each with one OST. If each file has exactly one object and the load is distributed evenly, there is no contention and the disks on each server can manage sequential I/O. If each file has 100 objects, then the clients all compete with one another for the attention of the servers, and the disks on each node seek in 100 different directions. In this case, there is needless contention.</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291857" xreflabel=""/><emphasis role="bold">Increased risk</emphasis> - When a file is striped across all servers and one of the servers breaks down, a small part of each striped file is lost. By comparison, if each file has exactly one stripe, you lose fewer files, but you lose them in their entirety. Many users would prefer to lose some of their files entirely than all of their files partially.</para>
+ <listitem>
+ <para>List the component with start = 64M</para>
+ <screen>$ lfs getstripe --component-start=64M /mnt/testfs/3comp
+/mnt/testfs/3comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 3
+ lcme_id: 3
+ lcme_flags: 0
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: -1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 4
+ lmm_stripe_offset: 4</screen>
</listitem>
-<listitem>
- <para> </para>
+ <listitem>
+ <para>List the component(s) with start > 5M</para>
+ <screen>$ lfs getstripe --component-start=+5M /mnt/testfs/3comp
+/mnt/testfs/3comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 3
+ lcme_id: 3
+ lcme_flags: 0
+ lcme_extent.e_start: 67108864
+ lcme_extent.e_end: EOF
+ lmm_stripe_count: -1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 4
+ lmm_stripe_offset: 4</screen>
</listitem>
-</itemizedlist>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1291860" xreflabel=""/>18.2.1 Choosing a Stripe <anchor xml:id="dbdoclet.50438209_marker-1291859" xreflabel=""/>Size</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296612" xreflabel=""/>Choosing a stripe size is a small balancing act, but there are reasonable defaults.</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296443" xreflabel=""/><emphasis role="bold">The stripe size must be a multiple of the page size.</emphasis> Lustre’s tools enforce a multiple of 64 KB (the maximum page size on ia64 and PPC64 nodes) so that users on platforms with smaller pages do not accidentally create files that might cause problems for ia64 clients.</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291862" xreflabel=""/><emphasis role="bold">The smallest recommended stripe size is 512 KB.</emphasis> Although you can create files with a stripe size of 64 KB, the smallest practical stripe size is 512 KB because Lustre sends 1MB chunks over the network. Choosing a smaller stripe size may result in inefficient I/O to the disks and reduced performance.</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291864" xreflabel=""/><emphasis role="bold">A good stripe size for sequential I/O using high-speed networks is between 1 MB and 4 MB.</emphasis> In most situations, stripe sizes larger than 4 MB may result in longer lock hold times and contention on shared file access.</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297206" xreflabel=""/><emphasis role="bold">The maximum stripe size is 4GB.</emphasis> Using a large stripe size can improve performance when accessing very large files. It allows each client to have exclusive access to its own part of a file. However, it can be counterproductive in some cases if it does not match your I/O pattern.</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291865" xreflabel=""/><emphasis role="bold">Choose a stripe pattern that takes into account your application’s write patterns.</emphasis> Writes that cross an object boundary are slightly less efficient than writes that go entirely to one server. If the file is written in a very consistent and aligned way, make the stripe size a multiple of the write() size.</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1291866" xreflabel=""/><emphasis role="bold">The choice of stripe size has no effect on a single-stripe file.</emphasis></para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-</itemizedlist>
- </section>
+ <listitem>
+ <para>List the component(s) with start < 5M</para>
+ <screen>$ lfs getstripe --component-start=-5M /mnt/testfs/3comp
+/mnt/testfs/3comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 3
+ lcme_id: 1
+ lcme_flags: init
+ lcme_extent.e_start: 0
+ lcme_extent.e_end: 4194304
+ lmm_stripe_count: 1
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 4
+ lmm_objects:
+ - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x2:0x0] }
+
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 5
+ lmm_objects:
+ - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
+ - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
+ - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
+ - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
+ </listitem>
+ <listitem>
+ <para>List the component(s) with start > 3M and end < 70M</para>
+ <screen>$ lfs getstripe --component-start=+3M --component-end=-70M \
+/mnt/testfs/3comp
+/mnt/testfs/3comp
+ lcm_layout_gen: 4
+ lcm_entry_count: 3
+ lcme_id: 2
+ lcme_flags: init
+ lcme_extent.e_start: 4194304
+ lcme_extent.e_end: 67108864
+ lmm_stripe_count: 4
+ lmm_stripe_size: 1048576
+ lmm_pattern: 1
+ lmm_layout_gen: 0
+ lmm_stripe_offset: 5
+ lmm_objects:
+ - 0: { l_ost_idx: 5, l_fid: [0x100050000:0x2:0x0] }
+ - 1: { l_ost_idx: 6, l_fid: [0x100060000:0x2:0x0] }
+ - 2: { l_ost_idx: 7, l_fid: [0x100070000:0x2:0x0] }
+ - 3: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }</screen>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section remap="h3">
+ <title><literal>lfs find</literal></title>
+ <para><literal>lfs find</literal> commands can be used to search the
+ directory tree rooted at the given directory or file name for the files
+ that match the given PFL component parameters. Here, only those
+ parameters new for PFL files are shown. Their usages are similar to
+ <literal>lfs getstripe</literal> commands.</para>
+ <para><emphasis role="bold">Command</emphasis></para>
+ <screen>lfs find <replaceable>directory|filename</replaceable>
+[[!] --component-count [+-=]<replaceable>comp_cnt</replaceable>]
+[[!] --component-start [+-=]<replaceable>N</replaceable>[kMGTPE]]
+[[!] --component-end|-E [+-=]<replaceable>N</replaceable>[kMGTPE]]
+[[!] --component-flags=<replaceable>comp_flags</replaceable>]</screen>
+ <note><para>If you use <literal>--component-xxx</literal> options, only
+ the composite files will be searched; but if you use
+ <literal>! --component-xxx</literal> options, all the files will be
+ searched.</para></note>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <para>We use the following directory and composite files to show how
+ <literal>lfs find</literal> works.</para>
+ <screen>$ mkdir /mnt/testfs/testdir
+$ lfs setstripe -E 1M -E 10M -E eof /mnt/testfs/testdir/3comp
+$ lfs setstripe -E 4M -E 20M -E 30M -E eof /mnt/testfs/testdir/4comp
+$ mkdir -p /mnt/testfs/testdir/dir_3comp
+$ lfs setstripe -E 6M -E 30M -E eof /mnt/testfs/testdir/dir_3comp
+$ lfs setstripe -E 8M -E eof /mnt/testfs/testdir/dir_3comp/2comp
+$ lfs setstripe -c 1 /mnt/testfs/testdir/dir_3comp/commnfile</screen>
+ <para><emphasis role="bold">Case1. Find the files that match the specified
+ component count condition</emphasis></para>
+ <para>Find the files under directory /mnt/testfs/testdir whose number of
+ components is not equal to 3.</para>
+ <screen>$ lfs find /mnt/testfs/testdir ! --component-count=3
+/mnt/testfs/testdir
+/mnt/testfs/testdir/4comp
+/mnt/testfs/testdir/dir_3comp/2comp
+/mnt/testfs/testdir/dir_3comp/commonfile</screen>
+ <para><emphasis role="bold">Case2. Find the files/dirs that match the
+ specified component start/end condition</emphasis></para>
+ <para>Find the file(s) under directory /mnt/testfs/testdir with component
+ start = 4M and end < 70M</para>
+ <screen>$ lfs find /mnt/testfs/testdir --component-start=4M -E -30M
+/mnt/testfs/testdir/4comp</screen>
+ <para><emphasis role="bold">Case3. Find the files/dirs that match the
+ specified component flag condition</emphasis></para>
+ <para>Find the file(s) under directory /mnt/testfs/testdir whose component
+ flags contain <literal>init</literal></para>
+ <screen>$ lfs find /mnt/testfs/testdir --component-flag=init
+/mnt/testfs/testdir/3comp
+/mnt/testfs/testdir/4comp
+/mnt/testfs/testdir/dir_3comp/2comp</screen>
+ <note><para>Since <literal>lfs find</literal> uses
+ "<literal>!</literal>" to do negative search, we don’t support
+ flag <literal>^init</literal> here.</para></note>
</section>
- <section remap="h2">
- <title>18.3 <anchor xml:id="dbdoclet.50438209_78664" xreflabel=""/>Setting the File Layout/Striping Configuration (lfs setstripe)</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297878" xreflabel=""/>Use the lfs setstripe command to create new files with a specific file layout (stripe pattern) configuration.</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1297879" xreflabel=""/>lfs setstripe [--size|-s stripe_size] [--count|-c stripe_count]
-<anchor xml:id="dbdoclet.50438209_pgfId-1297880" xreflabel=""/>[--index|-i start_ost] [--pool|-p pool_name] <filename|dirname>
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297881" xreflabel=""/><emphasis role="bold">stripe_size</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297882" xreflabel=""/>The stripe size indicates how much data to write to one OST before moving to the next OST. The default stripe_size is 1 MB, and passing a stripe_size of 0 causes the default stripe size to be used. Otherwise, the stripe_size value must be a multiple of 64 KB.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297883" xreflabel=""/><emphasis role="bold">stripe_count</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297884" xreflabel=""/>The stripe count indicates how many OSTs to use. The default stripe_count value is 1. Setting stripe_count to 0 causes the default stripe count to be used. Setting stripe_count to -1 means stripe over all available OSTs (full OSTs are skipped).</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297885" xreflabel=""/><emphasis role="bold">start_ost</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298131" xreflabel=""/>The start OST is the first OST to which files are written. The default value for start_ost is -1 , which allows the MDS to choose the starting index. This setting is strongly recommended, as it allows space and load balancing to be done by the MDS as needed. Otherwise, the file starts on the specified OST index. The numbering of the OSTs starts at 0.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
+ </section>
+ <section xml:id="dbdoclet.50438209_10424">
+ <title><indexterm>
+ <primary>space</primary>
+ <secondary>free space</secondary>
+ </indexterm><indexterm>
+ <primary>striping</primary>
+ <secondary>round-robin algorithm</secondary>
+ </indexterm><indexterm>
+ <primary>striping</primary>
+ <secondary>weighted algorithm</secondary>
+ </indexterm><indexterm>
+ <primary>round-robin algorithm</primary>
+ </indexterm><indexterm>
+ <primary>weighted algorithm</primary>
+ </indexterm>Managing Free Space</title>
+ <para>To optimize file system performance, the MDT assigns file stripes to OSTs based on two
+ allocation algorithms. The <emphasis role="italic">round-robin</emphasis> allocator gives
+ preference to location (spreading out stripes across OSSs to increase network bandwidth
+ utilization) and the weighted allocator gives preference to available space (balancing loads
+ across OSTs). Threshold and weighting factors for these two algorithms can be adjusted by the
+ user. The MDT reserves 0.1 percent of total OST space and 32 inodes for each OST. The MDT
+ stops object allocation for the OST if available space is less than reserved or the OST has
+ fewer than 32 free inodes. The MDT starts object allocation when available space is twice
+ as big as the reserved space and the OST has more than 64 free inodes. Note, clients
+ could append existing files no matter what object allocation state is.</para>
+ <para condition="l29"> The reserved space for each OST can be adjusted by the user. Use the
+ <literal>lctl set_param</literal> command, for example the next command reserve 1GB space
+ for all OSTs.
+ <screen>lctl set_param -P osp.*.reserved_mb_low=1024</screen></para>
+ <para>This section describes how to check available free space on disks and how free space is
+ allocated. It then describes how to set the threshold and weighting factors for the allocation
+ algorithms.</para>
+ <section xml:id="dbdoclet.50438209_35838">
+ <title>Checking File System Free Space</title>
+ <para>Free space is an important consideration in assigning file stripes. The <literal>lfs
+ df</literal> command can be used to show available disk space on the mounted Lustre file
+ system and space consumption per OST. If multiple Lustre file systems are mounted, a path
+ may be specified, but is not required. Options to the <literal>lfs df</literal> command are
+ shown below.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
<tbody>
<row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438209_pgfId-1297890" xreflabel=""/>If you pass a start_ost value of 0 and a stripe_count value of <emphasis>1</emphasis>, all files are written to OST 0, until space is exhausted. This is probably not what you meant to do. If you only want to adjust the stripe count and keep the other parameters at their default settings, do not specify any of the other parameters:</para><para>lfs setstripe -c <stripe_count> <file></para></entry>
+ <entry>
+ <para> <literal>-h</literal></para>
+ </entry>
+ <entry>
+ <para> Displays sizes in human readable format (for example: 1K, 234M, 5G).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal role="bold">-i, --inodes</literal></para>
+ </entry>
+ <entry>
+ <para> Lists inodes instead of block usage.</para>
+ </entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297973" xreflabel=""/><emphasis role="bold">pool_name</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297976" xreflabel=""/>Specify the OST pool on which the file will be written. This allows limiting the OSTs used to a subset of all OSTs in the file system. For more details about using OST pools, see <link xl:href="ManagingFileSystemIO.html#50438211_75549">Creating and Managing OST Pools</link>.</para>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1297896" xreflabel=""/>18.3.1 <anchor xml:id="dbdoclet.50438209_48100" xreflabel=""/>Using a Specific Striping Pattern/File Layout for a Single File</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297948" xreflabel=""/>It is possible to specify the file layout when a new file is created using the command lfssetstripe. This allows users to override the file system default parameters to tune the file layout more optimally for their application. Execution of an lfssetstripe command fails if the file already exists.</para>
- <section remap="h4">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1298004" xreflabel=""/>18.3.1.1 <anchor xml:id="dbdoclet.50438209_60155" xreflabel=""/>Setting the Stripe Size</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298005" xreflabel=""/>The command to create a new file with a specified stripe size is similar to:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1298006" xreflabel=""/>[client]# lfs setstripe -s 4M /mnt/lustre/new_file
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298007" xreflabel=""/>This example command creates the new file /mnt/lustre/new_file with a stripe size of 4 MB.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298232" xreflabel=""/>Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1298011" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438209_pgfId-1298012" xreflabel=""/>[client]# lfs getstripe /mnt/lustre/new_file
-<anchor xml:id="dbdoclet.50438209_pgfId-1298013" xreflabel=""/>/mnt/lustre/4mb_file
-<anchor xml:id="dbdoclet.50438209_pgfId-1298014" xreflabel=""/>lmm_stripe_count: 1
-<anchor xml:id="dbdoclet.50438209_pgfId-1298015" xreflabel=""/>lmm_stripe_size: 4194304
-<anchor xml:id="dbdoclet.50438209_pgfId-1298016" xreflabel=""/>lmm_stripe_offset: 1
-<anchor xml:id="dbdoclet.50438209_pgfId-1298017" xreflabel=""/>obdidx objid objid group
-<anchor xml:id="dbdoclet.50438209_pgfId-1298018" xreflabel=""/>1 690550 0xa8976 0
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298019" xreflabel=""/>As can be seen, the stripe size is 4 MB.</para>
- </section>
- <section remap="h4">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1298020" xreflabel=""/>18.3.1.2 Setting the Stripe Count</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298021" xreflabel=""/>The command below creates a new file with a stripe count of -1 to specify striping over all available OSTs:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1298022" xreflabel=""/>[client]# lfs setstripe -c -1 /mnt/lustre/full_stripe
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298028" xreflabel=""/>The example below indicates that the file full_stripe is striped over all six active OSTs in the configuration:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1298029" xreflabel=""/>[client]# lfs getstripe /mnt/lustre/full_stripe
-<anchor xml:id="dbdoclet.50438209_pgfId-1298030" xreflabel=""/>/mnt/lustre/full_stripe
-<anchor xml:id="dbdoclet.50438209_pgfId-1298031" xreflabel=""/>obdidx objid objid group
-<anchor xml:id="dbdoclet.50438209_pgfId-1298032" xreflabel=""/>0 8 0x8 0
-<anchor xml:id="dbdoclet.50438209_pgfId-1298033" xreflabel=""/>1 4 0x4 0
-<anchor xml:id="dbdoclet.50438209_pgfId-1298034" xreflabel=""/>2 5 0x5 0
-<anchor xml:id="dbdoclet.50438209_pgfId-1298035" xreflabel=""/>3 5 0x5 0
-<anchor xml:id="dbdoclet.50438209_pgfId-1298036" xreflabel=""/>4 4 0x4 0
-<anchor xml:id="dbdoclet.50438209_pgfId-1298037" xreflabel=""/>5 2 0x2 0
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1298040" xreflabel=""/> This is in contrast to the output in <link xl:href="ManagingStripingFreeSpace.html#50438209_60155">Setting the Stripe Size</link> which shows only a single object for the file.</para>
- </section>
- </section>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1297942" xreflabel=""/>18.3.2 Changing Striping for a Directory</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297943" xreflabel=""/>In a directory, the lfssetstripe command sets a default striping configuration for files created in the directory. The usage is the same as lfssetstripe for a regular file, except that the directory must exist prior to setting the default striping configuration. If a file is created in a directory with a default stripe configuration (without otherwise specifying striping), Lustre uses those striping parameters instead of the file system default for the new file.</para>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297944" xreflabel=""/>To change the striping pattern (file layout) for a sub-directory, create a directory with desired file layout as described above. Sub-directories inherit the file layout of the root/parent directory.</para>
- </section>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1298250" xreflabel=""/>18.3.3 Changing Striping for a File System</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297945" xreflabel=""/>Change the striping on the file system root will change the striping for all newly created files that would otherwise have a striping parameter from the parent directory or explicitly on the command line.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438209_pgfId-1298262" xreflabel=""/>Striping of new files and sub-directories is done per the striping parameter settings of the root directory. Once you set striping on the root directory, then, by default, it applies to any new child directories created in that root directory (unless they have their own striping settings).</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1297901" xreflabel=""/>18.3.4 Creating a File on a Specific OST</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297902" xreflabel=""/>You can use lfs setstripe to create a file on a specific OST. In the following example, the file "bob" will be created on the first OST (id 0).</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1297903" xreflabel=""/>$ lfs setstripe --count 1 --index 0 bob
-<anchor xml:id="dbdoclet.50438209_pgfId-1297904" xreflabel=""/>$ dd if=/dev/zero of=bob count=1 bs=100M
-<anchor xml:id="dbdoclet.50438209_pgfId-1297905" xreflabel=""/>1+0 records in
-<anchor xml:id="dbdoclet.50438209_pgfId-1297906" xreflabel=""/>1+0 records out
-<anchor xml:id="dbdoclet.50438209_pgfId-1297907" xreflabel=""/>$ lfs getstripe bob
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297908" xreflabel=""/>OBDS:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1297909" xreflabel=""/>0: home-OST0000_UUID ACTIVE
-<anchor xml:id="dbdoclet.50438209_pgfId-1297910" xreflabel=""/>[...]
-<anchor xml:id="dbdoclet.50438209_pgfId-1297911" xreflabel=""/>bob
-<anchor xml:id="dbdoclet.50438209_pgfId-1297912" xreflabel=""/> obdidx objid objid group
-<anchor xml:id="dbdoclet.50438209_pgfId-1297913" xreflabel=""/> 0 33459243 0x1fe8c2b 0
-</screen>
- </section>
- </section>
- <section remap="h2">
- <title>18.4 <anchor xml:id="dbdoclet.50438209_44776" xreflabel=""/>Retrieving File Layout/Striping Information (getstripe)</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297836" xreflabel=""/>The lfsgetstripe command is used to display information that shows over which OSTs a file is distributed. For each OST, the index and UUID is displayed, along with the OST index and object ID for each stripe in the file. For directories, the default settings for files created in that directory are printed.</para>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1297837" xreflabel=""/>18.4.1 Displaying the Current Stripe Size</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297838" xreflabel=""/>To see the current stripe size, use the lfsgetstripe command on a Lustre file or directory. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295575" xreflabel=""/>[client]# lfs getstripe /mnt/lustre
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296462" xreflabel=""/>This command produces output similar to this:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295583" xreflabel=""/>/mnt/lustre
-<anchor xml:id="dbdoclet.50438209_pgfId-1295584" xreflabel=""/>(Default) stripe_count: 1 stripe_size: 1M stripe_offset: -1
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1295585" xreflabel=""/>In this example, the default stripe count is 1 (data blocks are striped over a single OSTs), the default stripe size is 1 MB, and objects are created over all available OSTs.</para>
- </section>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1297424" xreflabel=""/>18.4.2 Inspecting the File Tree</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1296903" xreflabel=""/>To inspect an entire tree of files, use the lfs find command:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295699" xreflabel=""/>lfs find [--recursive | -r] <file or directory> ...
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1295700" xreflabel=""/>You can also use ls -l /proc/<emphasis><pid></emphasis>/fd/ to find open files using Lustre. For example:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295701" xreflabel=""/>$ lfs getstripe $(readlink /proc/$(pidof cat)/fd/1)
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297751" xreflabel=""/>Typical output is:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295707" xreflabel=""/>/mnt/lustre/foo
-<anchor xml:id="dbdoclet.50438209_pgfId-1295708" xreflabel=""/>obdidx objid objid \
-group
-<anchor xml:id="dbdoclet.50438209_pgfId-1297465" xreflabel=""/>2 835487 0xcbf9f 0
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297466" xreflabel=""/>In this example, the file lives on obdidx 2, which is lustre-OST0002. To see which node is serving that OST, run:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1297482" xreflabel=""/>$ lctl get_param osc.lustre-OST0002-osc.ost_conn_uuid
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1297761" xreflabel=""/>Typical output is:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295712" xreflabel=""/>osc.lustre-OST0002-osc.ost_conn_uuid=192.168.20.1@tcp
-</screen>
- </section>
+ <note>
+ <para>The <literal>df -i</literal> and <literal>lfs df -i</literal> commands show the
+ <emphasis role="italic">minimum</emphasis> number of inodes that can be created in the
+ file system at the current time. If the total number of objects available across all of
+ the OSTs is smaller than those available on the MDT(s), taking into account the default
+ file striping, then <literal>df -i</literal> will also report a smaller number of inodes
+ than could be created. Running <literal>lfs df -i</literal> will report the actual number
+ of inodes that are free on each target.</para>
+ <para>For ZFS file systems, the number of inodes that can be created is dynamic and depends
+ on the free space in the file system. The Free and Total inode counts reported for a ZFS
+ file system are only an estimate based on the current usage for each target. The Used
+ inode count is the actual number of inodes used by the file system.</para>
+ </note>
+ <para><emphasis role="bold">Examples</emphasis></para>
+ <screen>[client1] $ lfs df
+UUID 1K-blockS Used Available Use% Mounted on
+mds-lustre-0_UUID 9174328 1020024 8154304 11% /mnt/lustre[MDT:0]
+ost-lustre-0_UUID 94181368 56330708 37850660 59% /mnt/lustre[OST:0]
+ost-lustre-1_UUID 94181368 56385748 37795620 59% /mnt/lustre[OST:1]
+ost-lustre-2_UUID 94181368 54352012 39829356 57% /mnt/lustre[OST:2]
+filesystem summary: 282544104 167068468 39829356 57% /mnt/lustre
+
+[client1] $ lfs df -h
+UUID bytes Used Available Use% Mounted on
+mds-lustre-0_UUID 8.7G 996.1M 7.8G 11% /mnt/lustre[MDT:0]
+ost-lustre-0_UUID 89.8G 53.7G 36.1G 59% /mnt/lustre[OST:0]
+ost-lustre-1_UUID 89.8G 53.8G 36.0G 59% /mnt/lustre[OST:1]
+ost-lustre-2_UUID 89.8G 51.8G 38.0G 57% /mnt/lustre[OST:2]
+filesystem summary: 269.5G 159.3G 110.1G 59% /mnt/lustre
+
+[client1] $ lfs df -i
+UUID Inodes IUsed IFree IUse% Mounted on
+mds-lustre-0_UUID 2211572 41924 2169648 1% /mnt/lustre[MDT:0]
+ost-lustre-0_UUID 737280 12183 725097 1% /mnt/lustre[OST:0]
+ost-lustre-1_UUID 737280 12232 725048 1% /mnt/lustre[OST:1]
+ost-lustre-2_UUID 737280 12214 725066 1% /mnt/lustre[OST:2]
+filesystem summary: 2211572 41924 2169648 1% /mnt/lustre[OST:2]</screen>
</section>
- <section remap="h2">
- <title>18.5 <anchor xml:id="dbdoclet.50438209_10424" xreflabel=""/>Managing Free <anchor xml:id="dbdoclet.50438209_marker-1295520" xreflabel=""/>Space</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1293927" xreflabel=""/>The MDT assigns file stripes to OSTs based on location (which OSS) and size considerations (free space) to optimize file system performance. Emptier OSTs are preferentially selected for stripes, and stripes are preferentially spread out between OSSs to increase network bandwidth utilization. The weighting factor between these two optimizations can be adjusted by the user.</para>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1293929" xreflabel=""/>18.5.1 <anchor xml:id="dbdoclet.50438209_35838" xreflabel=""/>Checking File System Free Space</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1293930" xreflabel=""/>Free space is an important consideration in assigning file stripes. The lfsdf command shows available disk space on the mounted Lustre file system and space consumption per OST. If multiple Lustre file systems are mounted, a path may be specified, but is not required.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438209_pgfId-1293933" xreflabel=""/>Option</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438209_pgfId-1293935" xreflabel=""/>Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293937" xreflabel=""/><emphasis role="bold">-h</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293939" xreflabel=""/>Human-readable print sizes in human readable format (for example: 1K, 234M, 5G).</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293941" xreflabel=""/><emphasis role="bold">-i, --inodes</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293943" xreflabel=""/>Lists inodes instead of block usage.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438209_pgfId-1293944" xreflabel=""/>The df-i and lfsdf-i commands show the minimum number of inodes that can be created in the file system. Depending on the configuration, it may be possible to create more inodes than initially reported by df-i. Later, df-i operations will show the current, estimated free inode count.</para><para> If the underlying file system has fewer free blocks than inodes, then the total inode count for the file system reports only as many inodes as there are free blocks. This is done because Lustre may need to store an external attribute for each new inode, and it is better to report a free inode count that is the guaranteed, minimum number of inodes that can be created.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1293952" xreflabel=""/><emphasis role="bold">Examples</emphasis></para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1293953" xreflabel=""/>[lin-cli1] $ lfs df
-<anchor xml:id="dbdoclet.50438209_pgfId-1293954" xreflabel=""/>UUID 1K-blockS Used \
-Available Use% Mounted on
-<anchor xml:id="dbdoclet.50438209_pgfId-1293955" xreflabel=""/>mds-lustre-0_UUID 9174328 1020024 8154304 \
- 11% /mnt/lustre[MDT:0]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293956" xreflabel=""/>ost-lustre-0_UUID 94181368 56330708 37850660\
- 59% /mnt/lustre[OST:0]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293957" xreflabel=""/>ost-lustre-1_UUID 94181368 56385748 37795620\
- 59% /mnt/lustre[OST:1]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293958" xreflabel=""/>ost-lustre-2_UUID 94181368 54352012 39829356\
- 57% /mnt/lustre[OST:2]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293959" xreflabel=""/>filesystem summary: 282544104 167068468 \
-39829356 57% /mnt/lustre
-<anchor xml:id="dbdoclet.50438209_pgfId-1293960" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438209_pgfId-1293961" xreflabel=""/>[lin-cli1] $ lfs df -h
-<anchor xml:id="dbdoclet.50438209_pgfId-1293962" xreflabel=""/>UUID bytes Used \
-Available Use% Mounted on
-<anchor xml:id="dbdoclet.50438209_pgfId-1293963" xreflabel=""/>mds-lustre-0_UUID 8.7G 996.1M 7.8G \
- 11% /mnt/lustre[MDT:0]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293964" xreflabel=""/>ost-lustre-0_UUID 89.8G 53.7G 36.1G \
- 59% /mnt/lustre[OST:0]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293965" xreflabel=""/>ost-lustre-1_UUID 89.8G 53.8G 36.0G \
- 59% /mnt/lustre[OST:1]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293966" xreflabel=""/>ost-lustre-2_UUID 89.8G 51.8G 38.0G \
- 57% /mnt/lustre[OST:2]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293967" xreflabel=""/>filesystem summary: 269.5G 159.3G \
-110.1G 59% /mnt/lustre
-<anchor xml:id="dbdoclet.50438209_pgfId-1293968" xreflabel=""/>
-<anchor xml:id="dbdoclet.50438209_pgfId-1293969" xreflabel=""/>[lin-cli1] $ lfs df -i
-<anchor xml:id="dbdoclet.50438209_pgfId-1293970" xreflabel=""/>UUID Inodes IUsed \
-IFree IUse% Mounted on
-<anchor xml:id="dbdoclet.50438209_pgfId-1293971" xreflabel=""/>mds-lustre-0_UUID 2211572 41924 2169648 \
- 1% /mnt/lustre[MDT:0]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293972" xreflabel=""/>ost-lustre-0_UUID 737280 12183 725097 \
- 1% /mnt/lustre[OST:0]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293973" xreflabel=""/>ost-lustre-1_UUID 737280 12232 725048 \
- 1% /mnt/lustre[OST:1]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293974" xreflabel=""/>ost-lustre-2_UUID 737280 12214 725066 \
- 1% /mnt/lustre[OST:2]
-<anchor xml:id="dbdoclet.50438209_pgfId-1293975" xreflabel=""/>filesystem summary: 2211572 41924 \
-2169648 1% /mnt/lustre[OST:2]
-</screen>
- </section>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1293986" xreflabel=""/>18.5.2 Using Stripe Allocations</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1295862" xreflabel=""/>Two stripe allocation methods are provided: <emphasis>round-robin</emphasis> and <emphasis>weighted</emphasis>. By default, the allocation method is determined by the amount of free-space imbalance on the OSTs. The weighted allocator is used when any two OSTs are imbalanced by more than 20%. Otherwise, the faster round-robin allocator is used. (The round-robin order maximizes network balancing.)</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1293989" xreflabel=""/><emphasis role="bold">Round-robin allocator</emphasis><anchor xml:id="dbdoclet.50438209_marker-1293988" xreflabel=""/> - When OSTs have approximately the same amount of free space (within 20%), an efficient round-robin allocator is used. The round-robin allocator alternates stripes between OSTs on different OSSs, so the OST used for stripe 0 of each file is evenly distributed among OSTs, regardless of the stripe count. Here are several sample round-robin stripe orders (each letter represents a different OST on a single OSS):</para>
- </listitem>
-<listitem>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <tbody>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293993" xreflabel=""/>3: AAA</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293995" xreflabel=""/>One 3-OST OSS</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293997" xreflabel=""/>3x3: ABABAB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1293999" xreflabel=""/>Two 3-OST OSSs</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1294001" xreflabel=""/>3x4: BBABABA</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1294003" xreflabel=""/>One 3-OST OSS (A) and one 4-OST OSS (B)</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1294005" xreflabel=""/>3x5: BBABBABA</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1294007" xreflabel=""/>One 3-OST OSS (A) and one 5-OST OSS (B)</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1294009" xreflabel=""/>3x3x3: ABCABCABC</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438209_pgfId-1294011" xreflabel=""/>Three 3-OST OSSs</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1294021" xreflabel=""/><emphasis role="bold">Weighted allocator</emphasis><anchor xml:id="dbdoclet.50438209_marker-1294020" xreflabel=""/> - When the free space difference between the OSTs is significant (by default, 20% of the free space), then a weighting algorithm is used to influence OST ordering based on size and location. Note that these are weightings for a random algorithm, so the OST with the most free space is not necessarily chosen each time. On average, the weighted allocator fills the emptier OSTs faster.</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-</itemizedlist>
- </section>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438209_pgfId-1294034" xreflabel=""/>18.5.3 Adjusting the <anchor xml:id="dbdoclet.50438209_marker-1294033" xreflabel=""/>Weighting Between Free Space and Location</title>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1295878" xreflabel=""/>The weighting priority can be adjusted in the /proc file /proc/fs/lustre/lov/lustre-mdtlov/qos_prio_free proc. The default value is 90%. Use this command on the MGS to permanently change this weighting:</para>
- <screen><anchor xml:id="dbdoclet.50438209_pgfId-1295879" xreflabel=""/>lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90
-</screen>
- <para><anchor xml:id="dbdoclet.50438209_pgfId-1295880" xreflabel=""/>Increasing this value puts more weighting on free space. When the free space priority is set to 100%, then location is no longer used in stripe-ordering calculations and weighting is based entirely on free space.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438209_pgfId-1295881" xreflabel=""/>Setting the priority to 100% means that OSS distribution does not count in the weighting, but the stripe assignment is still done via a weighting. For example, if OST2 has twice as much free space as OST1, then OST2 is twice as likely to be used, but it is not guaranteed to be used.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <!--
-Begin SiteCatalyst code version: G.5.
--->
- <!--
-End SiteCatalyst code version: G.5.
--->
+ <section remap="h3">
+ <title><indexterm>
+ <primary>striping</primary>
+ <secondary>allocations</secondary>
+ </indexterm> Stripe Allocation Methods</title>
+ <para>Two stripe allocation methods are provided:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">Round-robin allocator</emphasis> - When the OSTs have
+ approximately the same amount of free space, the round-robin allocator alternates
+ stripes between OSTs on different OSSs, so the OST used for stripe 0 of each file is
+ evenly distributed among OSTs, regardless of the stripe count. In a simple example with
+ eight OSTs numbered 0-7, objects would be allocated like this:</para>
+ <para>
+ <screen>File 1: OST1, OST2, OST3, OST4
+File 2: OST5, OST6, OST7
+File 3: OST0, OST1, OST2, OST3, OST4, OST5
+File 4: OST6, OST7, OST0</screen>
+ </para>
+ <para>Here are several more sample round-robin stripe orders (each letter represents a
+ different OST on a single OSS):</para>
<informaltable frame="none">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
-
-
-
- <tbody>
- <row>
- <entry align="left"><para>Lustre 2.0 Operations Manual</para></entry>
- <entry align="right"><para>821-2076-10</para></entry>
- <entry align="right" valign="top"><para><link xl:href="index.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/toc01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/toc01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="BackupAndRestore.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/prev01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/prev01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="ManagingFileSystemIO.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/next01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/next01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link><link xl:href="ix.html"><inlinemediaobject><imageobject role="html">
- <imagedata contentdepth="26" contentwidth="30" fileref="./shared/index01.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/index01.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></link></para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><link xl:href=""/></para>
- <para><link xl:href="copyright.html">Copyright</link> © 2011, Oracle and/or its affiliates. All rights reserved.</para>
- </section>
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <tbody>
+ <row>
+ <entry>
+ <para> 3: AAA</para>
+ </entry>
+ <entry>
+ <para> One 3-OST OSS</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> 3x3: ABABAB</para>
+ </entry>
+ <entry>
+ <para> Two 3-OST OSSs</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> 3x4: BBABABA</para>
+ </entry>
+ <entry>
+ <para> One 3-OST OSS (A) and one 4-OST OSS (B)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> 3x5: BBABBABA</para>
+ </entry>
+ <entry>
+ <para> One 3-OST OSS (A) and one 5-OST OSS (B)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> 3x3x3: ABCABCABC</para>
+ </entry>
+ <entry>
+ <para> Three 3-OST OSSs</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Weighted allocator</emphasis> - When the free space difference
+ between the OSTs becomes significant, the weighting algorithm is used to influence OST
+ ordering based on size (amount of free space available on each OST) and location
+ (stripes evenly distributed across OSTs). The weighted allocator fills the emptier OSTs
+ faster, but uses a weighted random algorithm, so the OST with the most free space is not
+ necessarily chosen each time.</para>
+ </listitem>
+ </itemizedlist>
+ <para>The allocation method is determined by the amount of free-space imbalance on the OSTs.
+ When free space is relatively balanced across OSTs, the faster round-robin allocator is
+ used, which maximizes network balancing. The weighted allocator is used when any two OSTs
+ are out of balance by more than the specified threshold (17% by default). The threshold
+ between the two allocation methods is defined in the file
+ <literal>/proc/fs/<replaceable>fsname</replaceable>/lov/<replaceable>fsname</replaceable>-mdtlov/qos_threshold_rr</literal>. </para>
+ <para>To set the <literal>qos_threshold_r</literal> to <literal>25</literal>, enter this
+ command on the
+ MGS:<screen>lctl set_param lov.<replaceable>fsname</replaceable>-mdtlov.qos_threshold_rr=25</screen></para>
+ </section>
+ <section remap="h3">
+ <title><indexterm>
+ <primary>space</primary>
+ <secondary>location weighting</secondary>
+ </indexterm>Adjusting the Weighting Between Free Space and Location</title>
+ <para>The weighting priority used by the weighted allocator is set in the file
+ <literal>/proc/fs/<replaceable>fsname</replaceable>/lov/<replaceable>fsname</replaceable>-mdtlov/qos_prio_free</literal>.
+ Increasing the value of <literal>qos_prio_free</literal> puts more weighting on the amount
+ of free space available on each OST and less on how stripes are distributed across OSTs. The
+ default value is <literal>91</literal> (percent). When the free space priority is set to
+ <literal>100</literal> (percent), weighting is based entirely on free space and location
+ is no longer used by the striping algorithm. </para>
+ <para>To change the allocator weighting to <literal>100</literal>, enter this command on the
+ MGS:</para>
+ <screen>lctl conf_param <replaceable>fsname</replaceable>-MDT0000.lov.qos_prio_free=100</screen>
+ <para> .</para>
+ <note>
+ <para>When <literal>qos_prio_free</literal> is set to <literal>100</literal>, a weighted
+ random algorithm is still used to assign stripes, so, for example, if OST2 has twice as
+ much free space as OST1, OST2 is twice as likely to be used, but it is not guaranteed to
+ be used.</para>
+ </note>
</section>
</section>
-</article>
+ <section xml:id="wide_striping">
+ <title><indexterm>
+ <primary>striping</primary>
+ <secondary>wide striping</secondary>
+ </indexterm><indexterm>
+ <primary>wide striping</primary>
+ </indexterm>Lustre Striping Internals</title>
+ <para>Individual files can only be striped over a finite number of OSTs,
+ based on the maximum size of the attributes that can be stored on the MDT.
+ If the MDT is ldiskfs-based without the <literal>ea_inode</literal>
+ feature, a file can be striped across at most 160 OSTs. With ZFS-based
+ MDTs, or if the <literal>ea_inode</literal> feature is enabled for an
+ ldiskfs-based MDT, a file can be striped across up to 2000 OSTs.
+ </para>
+ <para>Lustre inodes use an extended attribute to record on which OST each
+ object is located, and the identifier each object on that OST. The size of
+ the extended attribute is a function of the number of stripes.</para>
+ <para>If using an ldiskfs-based MDT, the maximum number of OSTs over which
+ files can be striped can been raised to 2000 by enabling the
+ <literal>ea_inode</literal> feature on the MDT:
+ <screen>tune2fs -O ea_inode /dev/<replaceable>mdtdev</replaceable></screen>
+ </para>
+ <note><para>The maximum stripe count for a single file does not limit the
+ maximum number of OSTs that are in the filesystem as a whole, only the
+ maximum possible size and maximum aggregate bandwidth for the file.
+ </para></note>
+ </section>
+</chapter>
git://git.whamcloud.com / doc / manual.git / blobdiff