- <table frame="all" xml:id="settinguplustresystem.tab2">
- <title>File and file system limits</title>
- <tgroup cols="3">
- <colspec colname="c1" colwidth="3*"/>
- <colspec colname="c2" colwidth="2*"/>
- <colspec colname="c3" colwidth="4*"/>
- <thead>
- <row>
- <entry>
- <para><emphasis role="bold">Limit</emphasis></para>
- </entry>
- <entry>
- <para><emphasis role="bold">Value</emphasis></para>
- </entry>
- <entry>
- <para><emphasis role="bold">Description</emphasis></para>
- </entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry>
- <para> Maximum number of MDTs</para>
- </entry>
- <entry>
- <para condition='l24'>256</para>
- </entry>
- <entry>
- <para>The Lustre software release 2.3 and earlier allows a
- maximum of 1 MDT per file system, but a single MDS can host
- multiple MDTs, each one for a separate file system.</para>
- <para condition="l24">The Lustre software release 2.4 and later
- requires one MDT for the filesystem root. At least 255 more
- MDTs can be added to the filesystem and attached into
- the namespace with DNE remote or striped directories.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum number of OSTs</para>
- </entry>
- <entry>
- <para> 8150</para>
- </entry>
- <entry>
- <para>The maximum number of OSTs is a constant that can be
- changed at compile time. Lustre file systems with up to
- 4000 OSTs have been tested. Multiple OST file systems can
- be configured on a single OSS node.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum OST size</para>
- </entry>
- <entry>
- <para> 128TB (ldiskfs), 256TB (ZFS)</para>
- </entry>
- <entry>
- <para>This is not a <emphasis>hard</emphasis> limit. Larger
- OSTs are possible but today typical production systems do not
- typically go beyond the stated limit per OST because Lustre
- can add capacity and performance with additional OSTs, and
- having more OSTs improves aggregate I/O performance and
- minimizes contention.
- </para>
- <para>
- With 32-bit kernels, due to page cache limits, 16TB is the
- maximum block device size, which in turn applies to the
- size of OST. It is strongly recommended to run Lustre
- clients and servers with 64-bit kernels.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum number of clients</para>
- </entry>
- <entry>
- <para> 131072</para>
- </entry>
- <entry>
- <para>The maximum number of clients is a constant that can
- be changed at compile time. Up to 30000 clients have been
- used in production.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum size of a file system</para>
- </entry>
- <entry>
- <para> 512 PB (ldiskfs), 1EB (ZFS)</para>
- </entry>
- <entry>
- <para>Each OST can have a file system up to the
- Maximum OST size limit, and the Maximum number of OSTs
- can be combined into a single filesystem.
- </para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum stripe count</para>
- </entry>
- <entry>
- <para> 2000</para>
- </entry>
- <entry>
- <para>This limit is imposed by the size of the layout that
- needs to be stored on disk and sent in RPC requests, but is
- not a hard limit of the protocol. The number of OSTs in the
- filesystem can exceed the stripe count, but this limits the
- number of OSTs across which a single file can be striped.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum stripe size</para>
- </entry>
- <entry>
- <para> < 4 GB</para>
- </entry>
- <entry>
- <para>The amount of data written to each object before moving
- on to next object.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Minimum stripe size</para>
- </entry>
- <entry>
- <para> 64 KB</para>
- </entry>
- <entry>
- <para>Due to the 64 KB PAGE_SIZE on some 64-bit machines,
- the minimum stripe size is set to 64 KB.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum object size</para>
- </entry>
- <entry>
- <para> 16TB (ldiskfs), 256TB (ZFS)</para>
- </entry>
- <entry>
- <para>The amount of data that can be stored in a single object.
- An object corresponds to a stripe. The ldiskfs limit of 16 TB
- for a single object applies. For ZFS the limit is the size of
- the underlying OST. Files can consist of up to 2000 stripes,
- each stripe can be up to the maximum object size. </para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum <anchor xml:id="dbdoclet.50438256_marker-1290761" xreflabel=""/>file size</para>
- </entry>
- <entry>
- <para> 16 TB on 32-bit systems</para>
- <para> </para>
- <para> 31.25 PB on 64-bit ldiskfs systems, 8EB on 64-bit ZFS systems</para>
- </entry>
- <entry>
- <para>Individual files have a hard limit of nearly 16 TB on
- 32-bit systems imposed by the kernel memory subsystem. On
- 64-bit systems this limit does not exist. Hence, files can
- be 2^63 bits (8EB) in size if the backing filesystem can
- support large enough objects.</para>
- <para>A single file can have a maximum of 2000 stripes, which
- gives an upper single file limit of 31.25 PB for 64-bit
- ldiskfs systems. The actual amount of data that can be stored
- in a file depends upon the amount of free space in each OST
- on which the file is striped.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum number of files or subdirectories in a single directory</para>
- </entry>
- <entry>
- <para> 10 million files (ldiskfs), 2^48 (ZFS)</para>
- </entry>
- <entry>
- <para>The Lustre software uses the ldiskfs hashed directory
- code, which has a limit of about 10 million files, depending
- on the length of the file name. The limit on subdirectories
- is the same as the limit on regular files.</para>
- <note condition='l28'><para>Starting in the 2.8 release it is
- possible to exceed this limit by striping a single directory
- over multiple MDTs with the <literal>lfs mkdir -c</literal>
- command, which increases the single directory limit by a
- factor of the number of directory stripes used.</para></note>
- <para>Lustre file systems are tested with ten million files
- in a single directory.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum number of files in the file system</para>
- </entry>
- <entry>
- <para> 4 billion (ldiskfs), 256 trillion (ZFS)</para>
- <para condition='l24'>up to 256 times the per-MDT limit</para>
- </entry>
- <entry>
- <para>The ldiskfs filesystem imposes an upper limit of
- 4 billion inodes per filesystem. By default, the MDT
- filesystem is formatted with one inode per 2KB of space,
- meaning 512 million inodes per TB of MDT space. This can be
- increased initially at the time of MDT filesystem creation.
- For more information, see
- <xref linkend="settinguplustresystem"/>.</para>
- <para condition="l24">The ZFS filesystem
- dynamically allocates inodes and does not have a fixed ratio
- of inodes per unit of MDT space, but consumes approximately
- 4KB of space per inode, depending on the configuration.</para>
- <para condition="l24">Each additional MDT can hold up to the
- above maximum number of additional files, depending on
- available space and the distribution directories and files
- in the filesystem.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum length of a filename</para>
- </entry>
- <entry>
- <para> 255 bytes (filename)</para>
- </entry>
- <entry>
- <para>This limit is 255 bytes for a single filename, the
- same as the limit in the underlying filesystems.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum length of a pathname</para>
- </entry>
- <entry>
- <para> 4096 bytes (pathname)</para>
- </entry>
- <entry>
- <para>The Linux VFS imposes a full pathname length of 4096 bytes.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum number of open files for a Lustre file system</para>
- </entry>
- <entry>
- <para> No limit</para>
- </entry>
- <entry>
- <para>The Lustre software does not impose a maximum for the number of open files,
- but the practical limit depends on the amount of RAM on the MDS. No
- "tables" for open files exist on the MDS, as they are only linked in a
- list to a given client's export. Each client process probably has a limit of
- several thousands of open files which depends on the ulimit.</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </table>
- <para> </para>
- <note>
- <para condition="l22">In Lustre software releases prior to version 2.2,
- the maximum stripe count for a single file was limited to 160 OSTs.
- In version 2.2, the wide striping feature was added to support files
- striped over up to 2000 OSTs. In order to store the large layout for
- such files in ldiskfs, the <literal>ea_inode</literal> feature must
- be enabled on the MDT, but no similar tunable is needed for ZFS MDTs.
- This feature is disabled by default at
- <literal>mkfs.lustre</literal> time. In order to enable this feature,
- specify <literal>--mkfsoptions="-O ea_inode"</literal> at MDT format
- time, or use <literal>tune2fs -O ea_inode</literal> to enable it after
- the MDT has been formatted. Using either the deprecated
- <literal>large_xattr</literal> or preferred <literal>ea_inode</literal>
- feature name results in <literal>ea_inode</literal> being shown in
- the file system feature list.</para>
- </note>
- </section>
+ <table frame="all" xml:id="settinguplustresystem.tab2">
+ <title>File and file system limits</title>
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="3*"/>
+ <colspec colname="c2" colwidth="2*"/>
+ <colspec colname="c3" colwidth="4*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Limit</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Value</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para>Maximum number of MDTs</para>
+ </entry>
+ <entry>
+ <para condition='l24'>256</para>
+ </entry>
+ <entry>
+ <para>The Lustre software release 2.3 and earlier allows a
+ maximum of 1 MDT per file system, but a single MDS can host
+ multiple MDTs, each one for a separate file system.</para>
+ <para condition="l24">The Lustre software release 2.4 and later
+ requires one MDT for the filesystem root. At least 255 more
+ MDTs can be added to the filesystem and attached into
+ the namespace with DNE remote or striped directories.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum number of OSTs</para>
+ </entry>
+ <entry>
+ <para>8150</para>
+ </entry>
+ <entry>
+ <para>The maximum number of OSTs is a constant that can be
+ changed at compile time. Lustre file systems with up to
+ 4000 OSTs have been tested. Multiple OST file systems can
+ be configured on a single OSS node.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum OST size</para>
+ </entry>
+ <entry>
+ <para>256TiB (ldiskfs), 256TiB (ZFS)</para>
+ </entry>
+ <entry>
+ <para>This is not a <emphasis>hard</emphasis> limit. Larger
+ OSTs are possible but most production systems do not
+ typically go beyond the stated limit per OST because Lustre
+ can add capacity and performance with additional OSTs, and
+ having more OSTs improves aggregate I/O performance,
+ minimizes contention, and allows parallel recovery (e2fsck
+ for ldiskfs OSTs, scrub for ZFS OSTs).
+ </para>
+ <para>
+ With 32-bit kernels, due to page cache limits, 16TB is the
+ maximum block device size, which in turn applies to the
+ size of OST. It is strongly recommended to run Lustre
+ clients and servers with 64-bit kernels.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum number of clients</para>
+ </entry>
+ <entry>
+ <para>131072</para>
+ </entry>
+ <entry>
+ <para>The maximum number of clients is a constant that can
+ be changed at compile time. Up to 30000 clients have been
+ used in production accessing a single filesystem.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum size of a single file system</para>
+ </entry>
+ <entry>
+ <para>at least 1EiB</para>
+ </entry>
+ <entry>
+ <para>Each OST can have a file system up to the
+ Maximum OST size limit, and the Maximum number of OSTs
+ can be combined into a single filesystem.
+ </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum stripe count</para>
+ </entry>
+ <entry>
+ <para>2000</para>
+ </entry>
+ <entry>
+ <para>This limit is imposed by the size of the layout that
+ needs to be stored on disk and sent in RPC requests, but is
+ not a hard limit of the protocol. The number of OSTs in the
+ filesystem can exceed the stripe count, but this limits the
+ number of OSTs across which a single file can be striped.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum stripe size</para>
+ </entry>
+ <entry>
+ <para>< 4 GiB</para>
+ </entry>
+ <entry>
+ <para>The amount of data written to each object before moving
+ on to next object.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Minimum stripe size</para>
+ </entry>
+ <entry>
+ <para>64 KiB</para>
+ </entry>
+ <entry>
+ <para>Due to the use of 64 KiB PAGE_SIZE on some CPU
+ architectures such as ARM and POWER, the minimum stripe
+ size is 64 KiB so that a single page is not split over
+ multiple servers.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum object size</para>
+ </entry>
+ <entry>
+ <para>16TiB (ldiskfs), 256TiB (ZFS)</para>
+ </entry>
+ <entry>
+ <para>The amount of data that can be stored in a single object.
+ An object corresponds to a stripe. The ldiskfs limit of 16 TB
+ for a single object applies. For ZFS the limit is the size of
+ the underlying OST. Files can consist of up to 2000 stripes,
+ each stripe can be up to the maximum object size. </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum <anchor xml:id="dbdoclet.50438256_marker-1290761" xreflabel=""/>file size</para>
+ </entry>
+ <entry>
+ <para>16 TiB on 32-bit systems</para>
+ <para> </para>
+ <para>31.25 PiB on 64-bit ldiskfs systems,
+ 8EiB on 64-bit ZFS systems</para>
+ </entry>
+ <entry>
+ <para>Individual files have a hard limit of nearly 16 TiB on
+ 32-bit systems imposed by the kernel memory subsystem. On
+ 64-bit systems this limit does not exist. Hence, files can
+ be 2^63 bits (8EiB) in size if the backing filesystem can
+ support large enough objects.</para>
+ <para>A single file can have a maximum of 2000 stripes, which
+ gives an upper single file limit of 31.25 PiB for 64-bit
+ ldiskfs systems. The actual amount of data that can be stored
+ in a file depends upon the amount of free space in each OST
+ on which the file is striped.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum number of files or subdirectories in a single directory</para>
+ </entry>
+ <entry>
+ <para>10 million files (ldiskfs), 2^48 (ZFS)</para>
+ </entry>
+ <entry>
+ <para>The Lustre software uses the ldiskfs hashed directory
+ code, which has a limit of about 10 million files, depending
+ on the length of the file name. The limit on subdirectories
+ is the same as the limit on regular files.</para>
+ <note condition='l28'><para>Starting in the 2.8 release it is
+ possible to exceed this limit by striping a single directory
+ over multiple MDTs with the <literal>lfs mkdir -c</literal>
+ command, which increases the single directory limit by a
+ factor of the number of directory stripes used.</para></note>
+ <para>Lustre file systems are tested with ten million files
+ in a single directory.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum number of files in the file system</para>
+ </entry>
+ <entry>
+ <para>4 billion (ldiskfs), 256 trillion (ZFS)</para>
+ <para condition='l24'>up to 256 times the per-MDT limit</para>
+ </entry>
+ <entry>
+ <para>The ldiskfs filesystem imposes an upper limit of
+ 4 billion inodes per filesystem. By default, the MDT
+ filesystem is formatted with one inode per 2KB of space,
+ meaning 512 million inodes per TiB of MDT space. This can be
+ increased initially at the time of MDT filesystem creation.
+ For more information, see
+ <xref linkend="settinguplustresystem"/>.</para>
+ <para condition="l24">The ZFS filesystem dynamically allocates
+ inodes and does not have a fixed ratio of inodes per unit of MDT
+ space, but consumes approximately 4KiB of mirrored space per
+ inode, depending on the configuration.</para>
+ <para condition="l24">Each additional MDT can hold up to the
+ above maximum number of additional files, depending on
+ available space and the distribution directories and files
+ in the filesystem.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum length of a filename</para>
+ </entry>
+ <entry>
+ <para>255 bytes (filename)</para>
+ </entry>
+ <entry>
+ <para>This limit is 255 bytes for a single filename, the
+ same as the limit in the underlying filesystems.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum length of a pathname</para>
+ </entry>
+ <entry>
+ <para>4096 bytes (pathname)</para>
+ </entry>
+ <entry>
+ <para>The Linux VFS imposes a full pathname length of 4096 bytes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Maximum number of open files for a Lustre file system</para>
+ </entry>
+ <entry>
+ <para>No limit</para>
+ </entry>
+ <entry>
+ <para>The Lustre software does not impose a maximum for the number
+ of open files, but the practical limit depends on the amount of
+ RAM on the MDS. No "tables" for open files exist on the
+ MDS, as they are only linked in a list to a given client's
+ export. Each client process probably has a limit of several
+ thousands of open files which depends on the ulimit.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ <para> </para>
+ <note><para>By default for ldiskfs MDTs the maximum stripe count for a
+ <emphasis>single file</emphasis> is limited to 160 OSTs. In order to
+ increase the maximum file stripe count, use
+ <literal>--mkfsoptions="-O ea_inode"</literal> when formatting the MDT,
+ or use <literal>tune2fs -O ea_inode</literal> to enable it after the
+ MDT has been formatted.</para>
+ </note>