<para>Several third-party utilities are must be installed on servers:</para>
<itemizedlist>
<listitem>
- <para><emphasis role="bold">e2fsprogs</emphasis> : Lustre requires a recent version of e2fsprogs that understands extents. Use <literal>e2fsprogs-1.41-90-wc1</literal> or later, available at:</para>
+ <para><emphasis role="bold">e2fsprogs</emphasis> : Lustre requires a recent version of e2fsprogs that understands extents and large xattr. Use <literal>e2fsprogs-1.41.90.wc4</literal> or later, available at:</para>
<para><link xl:href="http://build.whamcloud.com/">http://build.whamcloud.com/</link></para>
<para>A quilt patchset of all changes to the vanilla e2fsprogs is available in <literal>e2fsprogs-{version}-patches.tgz</literal>.</para>
<note>
<itemizedlist>
<listitem>
<para><emphasis role="bold">Providing high-bandwidth access</emphasis> - Many applications require high-bandwidth access to a single file - more bandwidth than can be provided by a single OSS. For example, scientific applications that write to a single file from hundreds of nodes, or a binary executable that is loaded by many nodes when an application starts.</para>
- <para>In cases like these, a file can be striped over as many OSSs as it takes to achieve the required peak aggregate bandwidth for that file. Striping across a larger number of OSSs should only be used when the file size is very large and/or is accessed by many nodes at a time. Currently, Lustre files can be striped across up to 160 OSSs, the maximum stripe count for an ldiskfs file system.</para>
+ <para>In cases like these, a file can be striped over as many OSSs as it takes to achieve the required peak aggregate bandwidth for that file. Striping across a larger number of OSSs should only be used when the file size is very large and/or is accessed by many nodes at a time. Currently, Lustre files can be striped across up to 2000 OSTs, the maximum stripe count for an ldiskfs file system.</para>
</listitem>
<listitem>
<para><emphasis role="bold">Improving performance when OSS bandwidth is exceeded</emphasis> - Striping across many OSSs can improve performance if the aggregate client bandwidth exceeds the server bandwidth and the application reads and writes data fast enough to take advantage of the additional OSS bandwidth. The largest useful stripe count is bounded by the I/O rate of the clients/jobs divided by the performance per OSS.</para>
<indexterm><primary>space</primary><secondary>determining requirements</secondary></indexterm>
Determining Space Requirements</title>
<para>The desired performance characteristics of the backing file systems on the MDT and OSTs are independent of one another. The size of the MDT backing file system depends on the number of inodes needed in the total Lustre file system, while the aggregate OST space depends on the total amount of data stored on the file system.</para>
- <para>Each time a file is created on a Lustre file system, it consumes one inode on the MDT and one inode for each OST object over which the file is striped. Normally, each file's stripe count is based on the system-wide default stripe count. However, this can be changed for individual files using the lfssetstripe option. For more details, see <xref linkend="managingstripingfreespace"/>.</para>
+ <para>Each time a file is created on a Lustre file system, it consumes one inode on the MDT and one inode for each OST object over which the file is striped. Normally, each file's stripe count is based on the system-wide default stripe count. However, this can be changed for individual files using the <literal>lfs setstripe</literal> option. For more details, see <xref linkend="managingstripingfreespace"/>.</para>
<para>In a Lustre ldiskfs file system, all the inodes are allocated on the MDT and OSTs when the file system is first formatted. The total number of inodes on a formatted MDT or OST cannot be easily changed, although it is possible to add OSTs with additional space and corresponding inodes. Thus, the number of inodes created at format time should be generous enough to anticipate future expansion.</para>
<para>When the file system is in use and a file is created, the metadata associated with that file is stored in one of the pre-allocated inodes and does not consume any of the free space used to store file data.</para>
<note>
<para> Maximum stripe count</para>
</entry>
<entry>
- <para> 160</para>
+ <para> 2000</para>
</entry>
<entry>
- <para>This limit is hard-coded, but is near the upper limit imposed by the underlying ldiskfs file system.</para>
+ <para>This limit is imposed by the size of the layout that needs to be stored on disk and sent in RPC requests, but is not a hard limit of the protocol.</para>
</entry>
</row>
<row>
<para> 16 TB</para>
</entry>
<entry>
- <para>The amount of data that can be stored in a single object. An object corresponds to a stripe. The ldiskfs limit of 16 TB for a single object applies. Lustre allows files to consist of up to 160 stripes, each of 16 TB. </para>
+ <para>The amount of data that can be stored in a single object. An object corresponds to a stripe. The ldiskfs limit of 16 TB for a single object applies. Lustre allows files to consist of up to 2000 stripes, each of 16 TB. </para>
</entry>
</row>
<row>
<entry>
<para> 16 TB on 32-bit systems</para>
<para> </para>
- <para> 2.5 PB on 64-bit systems</para>
+ <para> 31.25 PB on 64-bit systems</para>
</entry>
<entry>
<para>Individual files have a hard limit of nearly 16 TB on 32-bit systems imposed by the kernel memory subsystem. On 64-bit systems this limit does not exist. Hence, files can be 64-bits in size. Lustre imposes an additional size limit of up to the number of stripes, where each stripe is 16 TB.</para>
- <para>A single file can have a maximum of 160 stripes, which gives an upper single file limit of 2.5 PB for 64-bit systems. The actual amount of data that can be stored in a file depends upon the amount of free space in each OST on which the file is striped.</para>
+ <para>A single file can have a maximum of 2000 stripes, which gives an upper single file limit of 31.25 PB for 64-bit systems. The actual amount of data that can be stored in a file depends upon the amount of free space in each OST on which the file is striped.</para>
</entry>
</row>
<row>
</tgroup>
</table>
<para> </para>
+ <note>
+ <para>Versions of Lustre prior to 2.2 had a maximum stripe count for a single file was limited to 160 OSTs. In Lustre version 2.2, the large xattr feature which supports up to 2000 OSTs is disabled by default at mkfs.lustre time. In order to enable this feature, the "-O large_xattr" option needs to be set on MDT either with --mkfsoptions at format time or via tune2fs.</para>
+ </note>
</section>
</section>
<section xml:id="dbdoclet.50438256_26456">
</textobject>
</mediaobject>
</figure>
- <para>The maximum file size is not limited by the size of a single target. Lustre can stripe files across multiple objects (up to 160), and each object can be up to 2 TB in size. This leads to a maximum file size of 320 TB. (Note that Lustre itself can support files up to 2^64 bytes depending on the backing storage used by OSTs.)</para>
- <para>Athough a single file can only be striped over 160 objects, Lustre file systems can have thousands of OSTs. The I/O bandwidth to access a single file is the aggregated I/O bandwidth to the objects in a file, which can be as much as a bandwidth of up to 160 servers. On systems with more than 160 OSTs, clients can do I/O using multiple files to utilize the full file system bandwidth.</para>
+ <para>The maximum file size is not limited by the size of a single target. Lustre can stripe files across multiple objects (up to 2000), and each object can be up to 16 TB in size with ldiskfs. This leads to a maximum file size of 31.25 PB. (Note that Lustre itself can support files up to 2^64 bytes depending on the backing storage used by OSTs.)</para>
+ <note>
+ <para>Versions of Lustre prior to 2.2 had a maximum stripe count for a single file was limited to 160 OSTs.</para>
+ </note>
+ <para>Athough a single file can only be striped over 2000 objects, Lustre file systems can have thousands of OSTs. The I/O bandwidth to access a single file is the aggregated I/O bandwidth to the objects in a file, which can be as much as a bandwidth of up to 2000 servers. On systems with more than 2000 OSTs, clients can do I/O using multiple files to utilize the full file system bandwidth.</para>
<para>For more information about striping, see <xref linkend="managingstripingfreespace"/>.</para>
</section>
</section>
git://git.whamcloud.com / doc / manual.git / commitdiff