<?xml version='1.0' encoding='UTF-8'?>
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settinguplustresystem">
+<chapter xmlns="http://docbook.org/ns/docbook"
+ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
+ xml:id="settinguplustresystem">
<title xml:id="settinguplustresystem.title">Determining Hardware Configuration Requirements and
Formatting Options</title>
<para>This chapter describes hardware configuration requirements for a Lustre file system
<itemizedlist>
<listitem>
<para>
- <xref linkend="dbdoclet.50438256_49017"/>
+ <xref linkend="dbdoclet.storage_hardware_considerations"/>
</para>
</listitem>
<listitem>
</listitem>
<listitem>
<para>
- <xref linkend="dbdoclet.50438256_26456"/>
+ <xref linkend="dbdoclet.mds_oss_memory"/>
</para>
</listitem>
<listitem>
<para>
- <xref linkend="dbdoclet.50438256_78272"/>
+ <xref linkend="dbdoclet.network_considerations"/>
</para>
</listitem>
</itemizedlist>
- <section xml:id="dbdoclet.50438256_49017">
+ <section xml:id="dbdoclet.storage_hardware_considerations">
<title><indexterm><primary>setup</primary></indexterm>
<indexterm><primary>setup</primary><secondary>hardware</secondary></indexterm>
<indexterm><primary>design</primary><see>setup</see></indexterm>
</listitem>
</itemizedlist>
</warning>
- <para>Only servers running on 64-bit CPUs are tested and supported. 64-bit CPU clients are
- typically used for testing to match expected customer usage and avoid limitations due to the 4
- GB limit for RAM size, 1 GB low-memory limitation, and 16 TB file size limit of 32-bit CPUs.
- Also, due to kernel API limitations, performing backups of Lustre software release 2.x. file
- systems on 32-bit clients may cause backup tools to confuse files that have the same 32-bit
- inode number.</para>
+ <para>Only servers running on 64-bit CPUs are tested and supported.
+ 64-bit CPU clients are typically used for testing to match expected
+ customer usage and avoid limitations due to the 4 GB limit for RAM
+ size, 1 GB low-memory limitation, and 16 TB file size limit of 32-bit
+ CPUs. Also, due to kernel API limitations, performing backups of Lustre
+ filesystems on 32-bit clients may cause backup tools to confuse files
+ that report the same 32-bit inode number, if the backup tools depend
+ on the inode number for correct operation.</para>
<para>The storage attached to the servers typically uses RAID to provide fault tolerance and can
optionally be organized with logical volume management (LVM), which is then formatted as a
Lustre file system. Lustre OSS and MDS servers read, write and modify data in the format
a separate device.</para>
<para>The MDS can effectively utilize a lot of CPU cycles. A minimum of four processor cores are recommended. More are advisable for files systems with many clients.</para>
<note>
- <para>Lustre clients running on architectures with different endianness are supported. One limitation is that the PAGE_SIZE kernel macro on the client must be as large as the PAGE_SIZE of the server. In particular, ia64 or PPC clients with large pages (up to 64kB pages) can run with x86 servers (4kB pages). If you are running x86 clients with ia64 or PPC servers, you must compile the ia64 kernel with a 4kB PAGE_SIZE (so the server page size is not larger than the client page size). </para>
+ <para>Lustre clients running on different CPU architectures is supported.
+ One limitation is that the PAGE_SIZE kernel macro on the client must be
+ as large as the PAGE_SIZE of the server. In particular, ARM or PPC
+ clients with large pages (up to 64kB pages) can run with x86 servers
+ (4kB pages).</para>
</note>
<section remap="h3">
<title><indexterm>
<primary>setup</primary>
<secondary>MDT</secondary>
</indexterm> MGT and MDT Storage Hardware Considerations</title>
- <para>MGT storage requirements are small (less than 100 MB even in the largest Lustre file
- systems), and the data on an MGT is only accessed on a server/client mount, so disk
- performance is not a consideration. However, this data is vital for file system access, so
+ <para>MGT storage requirements are small (less than 100 MB even in the
+ largest Lustre file systems), and the data on an MGT is only accessed
+ on a server/client mount, so disk performance is not a consideration.
+ However, this data is vital for file system access, so
the MGT should be reliable storage, preferably mirrored RAID1.</para>
- <para>MDS storage is accessed in a database-like access pattern with many seeks and
- read-and-writes of small amounts of data. High throughput to MDS storage is not important.
- Storage types that provide much lower seek times, such as high-RPM SAS or SSD drives can be
- used for the MDT.</para>
- <para>For maximum performance, the MDT should be configured as RAID1 with an internal journal and two disks from different controllers.</para>
- <para>If you need a larger MDT, create multiple RAID1 devices from pairs of disks, and then make a RAID0 array of the RAID1 devices. This ensures maximum reliability because multiple disk failures only have a small chance of hitting both disks in the same RAID1 device.</para>
- <para>Doing the opposite (RAID1 of a pair of RAID0 devices) has a 50% chance that even two disk failures can cause the loss of the whole MDT device. The first failure disables an entire half of the mirror and the second failure has a 50% chance of disabling the remaining mirror.</para>
- <para condition='l24'>If multiple MDTs are going to be present in the
+ <para>MDS storage is accessed in a database-like access pattern with
+ many seeks and read-and-writes of small amounts of data.
+ Storage types that provide much lower seek times, such as SSD or NVMe
+ is strongly preferred for the MDT, and high-RPM SAS is acceptable.</para>
+ <para>For maximum performance, the MDT should be configured as RAID1 with
+ an internal journal and two disks from different controllers.</para>
+ <para>If you need a larger MDT, create multiple RAID1 devices from pairs
+ of disks, and then make a RAID0 array of the RAID1 devices. For ZFS,
+ use <literal>mirror</literal> VDEVs for the MDT. This ensures
+ maximum reliability because multiple disk failures only have a small
+ chance of hitting both disks in the same RAID1 device.</para>
+ <para>Doing the opposite (RAID1 of a pair of RAID0 devices) has a 50%
+ chance that even two disk failures can cause the loss of the whole MDT
+ device. The first failure disables an entire half of the mirror and the
+ second failure has a 50% chance of disabling the remaining mirror.</para>
+ <para>If multiple MDTs are going to be present in the
system, each MDT should be specified for the anticipated usage and load.
For details on how to add additional MDTs to the filesystem, see
- <xref linkend="dbdoclet.addingamdt"/>.</para>
- <warning condition='l24'><para>MDT0 contains the root of the Lustre file
- system. If MDT0 is unavailable for any reason, the file system cannot be
- used.</para></warning>
- <note condition='l24'><para>Using the DNE feature it is possible to
- dedicate additional MDTs to sub-directories off the file system root
- directory stored on MDT0, or arbitrarily for lower-level subdirectories.
- using the <literal>lfs mkdir -i <replaceable>mdt_index</replaceable></literal> command.
- If an MDT serving a subdirectory becomes unavailable, any subdirectories
- on that MDT and all directories beneath it will also become inaccessible.
- Configuring multiple levels of MDTs is an experimental feature for the
- 2.4 release, and is fully functional in the 2.8 release. This is
- typically useful for top-level directories to assign different users
- or projects to separate MDTs, or to distribute other large working sets
- of files to multiple MDTs.</para></note>
+ <xref linkend="lustremaint.adding_new_mdt"/>.</para>
+ <warning><para>MDT0000 contains the root of the Lustre file system. If
+ MDT0000 is unavailable for any reason, the file system cannot be used.
+ </para></warning>
+ <note><para>Using the DNE feature it is possible to dedicate additional
+ MDTs to sub-directories off the file system root directory stored on
+ MDT0000, or arbitrarily for lower-level subdirectories, using the
+ <literal>lfs mkdir -i <replaceable>mdt_index</replaceable></literal>
+ command. If an MDT serving a subdirectory becomes unavailable, any
+ subdirectories on that MDT and all directories beneath it will also
+ become inaccessible. This is typically useful for top-level directories
+ to assign different users or projects to separate MDTs, or to distribute
+ other large working sets of files to multiple MDTs.</para></note>
<note condition='l28'><para>Starting in the 2.8 release it is possible
to spread a single large directory across multiple MDTs using the DNE
striped directory feature by specifying multiple stripes (or shards)
data. This reserved space is unusable for general storage. Thus, at least
this much space will be used per OST before any file object data is saved.
</para>
- <para condition="l24">With a ZFS backing filesystem for the MDT or OST,
+ <para>With a ZFS backing filesystem for the MDT or OST,
the space allocation for inodes and file data is dynamic, and inodes are
allocated as needed. A minimum of 4kB of usable space (before mirroring)
is needed for each inode, exclusive of other overhead such as directories,
The size is determined by the total number of servers in the Lustre
file system cluster(s) that are managed by the MGS.</para>
</section>
- <section xml:id="dbdoclet.50438256_87676">
+ <section xml:id="dbdoclet.mdt_space_requirements">
<title><indexterm>
<primary>setup</primary>
<secondary>MDT</secondary>
</indexterm> Determining MDT Space Requirements</title>
<para>When calculating the MDT size, the important factor to consider
is the number of files to be stored in the file system, which depends on
- at least 4 KiB per inode of usable space on the MDT. Since MDTs typically
+ at least 2 KiB per inode of usable space on the MDT. Since MDTs typically
use RAID-1+0 mirroring, the total storage needed will be double this.
</para>
<para>Please note that the actual used space per MDT depends on the number
of files per directory, the number of stripes per file, whether files
have ACLs or user xattrs, and the number of hard links per file. The
storage required for Lustre file system metadata is typically 1-2
- percent of the total file system capacity depending upon file size.</para>
+ percent of the total file system capacity depending upon file size.
+ If the <xref linkend="dataonmdt"/> feature is in use for Lustre
+ 2.11 or later, MDT space should typically be 5 percent or more of the
+ total space, depending on the distribution of small files within the
+ filesystem and the <literal>lod.*.dom_stripesize</literal> limit on
+ the MDT and file layout used.</para>
+ <para>For ZFS-based MDT filesystems, the number of inodes created on
+ the MDT and OST is dynamic, so there is less need to determine the
+ number of inodes in advance, though there still needs to be some thought
+ given to the total MDT space compared to the total filesystem size.</para>
<para>For example, if the average file size is 5 MiB and you have
- 100 TiB of usable OST space, then you can calculate the minimum total
- number of inodes each for MDTs and OSTs as follows:</para>
+ 100 TiB of usable OST space, then you can calculate the
+ <emphasis>minimum</emphasis> total number of inodes for MDTs and OSTs
+ as follows:</para>
<informalexample>
<para>(500 TB * 1000000 MB/TB) / 5 MB/inode = 100M inodes</para>
</informalexample>
- <para>For details about formatting options for ldiskfs MDT and OST file
- systems, see <xref linkend="dbdoclet.ldiskfs_mdt_mkfs"/>.</para>
- <para>It is recommended that the MDT have at least twice the minimum
+ <para>It is recommended that the MDT(s) have at least twice the minimum
number of inodes to allow for future expansion and allow for an average
- file size smaller than expected. Thus, the minimum space for an ldiskfs
- MDT should be approximately:
+ file size smaller than expected. Thus, the minimum space for ldiskfs
+ MDT(s) should be approximately:
</para>
<informalexample>
<para>2 KiB/inode x 100 million inodes x 2 = 400 GiB ldiskfs MDT</para>
</informalexample>
+ <para>For details about formatting options for ldiskfs MDT and OST file
+ systems, see <xref linkend="dbdoclet.ldiskfs_mdt_mkfs"/>.</para>
<note>
- <para>If the average file size is very small, 4 KB for example, the
- MDT will use as much space for each file as the space used on the OST.
- However, this is an uncommon usage for a Lustre filesystem.</para>
+ <para>If the median file size is very small, 4 KB for example, the
+ MDT would use as much space for each file as the space used on the OST,
+ so the use of Data-on-MDT is strongly recommended in that case.
+ The MDT space per inode should be increased correspondingly to
+ account for the extra data space usage for each inode:
+ <informalexample>
+ <para>6 KiB/inode x 100 million inodes x 2 = 1200 GiB ldiskfs MDT</para>
+ </informalexample>
+ </para>
</note>
<note>
<para>If the MDT has too few inodes, this can cause the space on the
- OSTs to be inaccessible since no new files can be created. Be sure to
- determine the appropriate size of the MDT needed to support the file
- system before formatting the file system. It is possible to increase the
+ OSTs to be inaccessible since no new files can be created. In this
+ case, the <literal>lfs df -i</literal> and <literal>df -i</literal>
+ commands will limit the number of available inodes reported for the
+ filesystem to match the total number of available objects on the OSTs.
+ Be sure to determine the appropriate MDT size needed to support the
+ filesystem before formatting. It is possible to increase the
number of inodes after the file system is formatted, depending on the
storage. For ldiskfs MDT filesystems the <literal>resize2fs</literal>
tool can be used if the underlying block device is on a LVM logical
volume and the underlying logical volume size can be increased.
- For ZFS new (mirrored) VDEVs can be added to the MDT pool to increase
- the total space available for inode storage.
+ For ZFS new (mirrored) VDEVs can be added to the MDT pool to increase
+ the total space available for inode storage.
Inodes will be added approximately in proportion to space added.
- </para>
+ </para>
</note>
- <note condition='l24'>
+ <note>
<para>Note that the number of total and free inodes reported by
<literal>lfs df -i</literal> for ZFS MDTs and OSTs is estimated based
on the current average space used per inode. When a ZFS filesystem is
first formatted, this free inode estimate will be very conservative
(low) due to the high ratio of directories to regular files created for
- internal Lustre metadata storage, but this estimate will improve as
- more files are created by regular users and the average file size will
- better reflect actual site usage.
- </para>
+ internal Lustre metadata storage, but this estimate will improve as
+ more files are created by regular users and the average file size will
+ better reflect actual site usage.
+ </para>
</note>
- <note condition='l24'>
- <para>Starting in release 2.4, using the DNE remote directory feature
- it is possible to increase the total number of inodes of a Lustre
- filesystem, as well as increasing the aggregate metadata performance,
- by configuring additional MDTs into the filesystem, see
- <xref linkend="dbdoclet.addingamdt"/> for details.
+ <note>
+ <para>Using the DNE remote directory feature
+ it is possible to increase the total number of inodes of a Lustre
+ filesystem, as well as increasing the aggregate metadata performance,
+ by configuring additional MDTs into the filesystem, see
+ <xref linkend="lustremaint.adding_new_mdt"/> for details.
</para>
</note>
</section>
<para>For the OST, the amount of space taken by each object depends on
the usage pattern of the users/applications running on the system. The
Lustre software defaults to a conservative estimate for the average
- object size (between 64KB per object for 10GB OSTs, and 1MB per object
- for 16TB and larger OSTs). If you are confident that the average file
- size for your applications will be larger than this, you can specify a
- larger average file size (fewer total inodes for a given OST size) to
- reduce file system overhead and minimize file system check time.
+ object size (between 64 KiB per object for 10 GiB OSTs, and 1 MiB per
+ object for 16 TiB and larger OSTs). If you are confident that the average
+ file size for your applications will be different than this, you can
+ specify a different average file size (number of total inodes for a given
+ OST size) to reduce file system overhead and minimize file system check
+ time.
See <xref linkend="dbdoclet.ldiskfs_ost_mkfs"/> for more details.</para>
</section>
</section>
<title>
<indexterm>
<primary>ldiskfs</primary>
- <secondary>formatting options</secondary>
+ <secondary>formatting options</secondary>
</indexterm>
<indexterm>
<primary>setup</primary>
- <secondary>ldiskfs</secondary>
+ <secondary>ldiskfs</secondary>
</indexterm>
Setting ldiskfs File System Formatting Options
</title>
options include:</para>
<itemizedlist>
<listitem>
- <para><literal>flex_bg</literal> - When the flag is set to enable this
- flexible-block-groups feature, block and inode bitmaps for multiple groups are aggregated
- to minimize seeking when bitmaps are read or written and to reduce read/modify/write
- operations on typical RAID storage (with 1 MB RAID stripe widths). This flag is enabled on
- both OST and MDT file systems. On MDT file systems the <literal>flex_bg</literal> factor
- is left at the default value of 16. On OSTs, the <literal>flex_bg</literal> factor is set
- to 256 to allow all of the block or inode bitmaps in a single <literal>flex_bg</literal>
- to be read or written in a single I/O on typical RAID storage.</para>
+ <para><literal>flex_bg</literal> - When the flag is set to enable
+ this flexible-block-groups feature, block and inode bitmaps for
+ multiple groups are aggregated to minimize seeking when bitmaps
+ are read or written and to reduce read/modify/write operations
+ on typical RAID storage (with 1 MiB RAID stripe widths). This flag
+ is enabled on both OST and MDT file systems. On MDT file systems
+ the <literal>flex_bg</literal> factor is left at the default value
+ of 16. On OSTs, the <literal>flex_bg</literal> factor is set
+ to 256 to allow all of the block or inode bitmaps in a single
+ <literal>flex_bg</literal> to be read or written in a single
+ 1MiB I/O typical for RAID storage.</para>
</listitem>
<listitem>
- <para><literal>huge_file</literal> - Setting this flag allows files on OSTs to be
- larger than 2 TB in size.</para>
+ <para><literal>huge_file</literal> - Setting this flag allows
+ files on OSTs to be larger than 2 TiB in size.</para>
</listitem>
<listitem>
- <para><literal>lazy_journal_init</literal> - This extended option is enabled to
- prevent a full overwrite of the 400 MB journal that is allocated by default in a Lustre
- file system, which reduces the file system format time.</para>
+ <para><literal>lazy_journal_init</literal> - This extended option
+ is enabled to prevent a full overwrite to zero out the large
+ journal that is allocated by default in a Lustre file system
+ (up to 400 MiB for OSTs, up to 4GiB for MDTs), to reduce the
+ formatting time.</para>
</listitem>
</itemizedlist>
<para>To override the default formatting options, use arguments to
<para>The number of inodes on the MDT is determined at format time
based on the total size of the file system to be created. The default
<emphasis role="italic">bytes-per-inode</emphasis> ratio ("inode ratio")
- for an MDT is optimized at one inode for every 2048 bytes of file
- system space. It is recommended that this value not be changed for
- MDTs.</para>
+ for an ldiskfs MDT is optimized at one inode for every 2560 bytes of file
+ system space.</para>
<para>This setting takes into account the space needed for additional
ldiskfs filesystem-wide metadata, such as the journal (up to 4 GB),
bitmaps, and directories, as well as files that Lustre uses internally
to maintain cluster consistency. There is additional per-file metadata
such as file layout for files with a large number of stripes, Access
Control Lists (ACLs), and user extended attributes.</para>
- <para> It is possible to reserve less than the recommended 2048 bytes
+ <para condition="l2B"> Starting in Lustre 2.11, the <xref linkend=
+ "dataonmdt.title"/> (DoM) feature allows storing small files on the MDT
+ to take advantage of high-performance flash storage, as well as reduce
+ space and network overhead. If you are planning to use the DoM feature
+ with an ldiskfs MDT, it is recommended to <emphasis>increase</emphasis>
+ the bytes-per-inode ratio to have enough space on the MDT for small files,
+ as described below.
+ </para>
+ <para>It is possible to change the recommended default of 2560 bytes
per inode for an ldiskfs MDT when it is first formatted by adding the
<literal>--mkfsoptions="-i bytes-per-inode"</literal> option to
<literal>mkfs.lustre</literal>. Decreasing the inode ratio tunable
<literal>bytes-per-inode</literal> will create more inodes for a given
- MDT size, but will leave less space for extra per-file metadata. The
- inode ratio must always be strictly larger than the MDT inode size,
- which is 512 bytes by default. It is recommended to use an inode ratio
- at least 512 bytes larger than the inode size to ensure the MDT does
- not run out of space.</para>
- <para>The size of the inode may be changed by adding the
+ MDT size, but will leave less space for extra per-file metadata and is
+ not recommended. The inode ratio must always be strictly larger than
+ the MDT inode size, which is 1024 bytes by default. It is recommended
+ to use an inode ratio at least 1536 bytes larger than the inode size to
+ ensure the MDT does not run out of space. Increasing the inode ratio
+ with enough space for the most commonly file size (e.g. 5632 or 66560
+ bytes if 4KB or 64KB files are widely used) is recommended for DoM.</para>
+ <para>The size of the inode may be changed at format time by adding the
<literal>--stripe-count-hint=N</literal> to have
<literal>mkfs.lustre</literal> automatically calculate a reasonable
inode size based on the default stripe count that will be used by the
<literal>--mkfsoptions="-I inode-size"</literal> option. Increasing
the inode size will provide more space in the inode for a larger Lustre
file layout, ACLs, user and system extended attributes, SELinux and
- other security labels, and other internal metadata. However, if these
- features or other in-inode xattrs are not needed, the larger inode size
- will hurt metadata performance as 2x, 4x, or 8x as much data would be
+ other security labels, and other internal metadata and DoM data. However,
+ if these features or other in-inode xattrs are not needed, a larger inode
+ size may hurt metadata performance as 2x, 4x, or 8x as much data would be
read or written for each MDT inode access.
</para>
</section>
<secondary>OST</secondary>
</indexterm>Setting Formatting Options for an ldiskfs OST</title>
<para>When formatting an OST file system, it can be beneficial
- to take local file system usage into account. When doing so, try to
- reduce the number of inodes on each OST, while keeping enough margin
- for potential variations in future usage. This helps reduce the format
- and file system check time and makes more space available for data.</para>
+ to take local file system usage into account, for example by running
+ <literal>df</literal> and <literal>df -i</literal> on a current filesystem
+ to get the used bytes and used inodes respectively, then computing the
+ average bytes-per-inode value. When deciding on the ratio for a new
+ filesystem, try to avoid having too many inodes on each OST, while keeping
+ enough margin to allow for future usage of smaller files. This helps
+ reduce the format and e2fsck time and makes more space available for data.
+ </para>
<para>The table below shows the default
<emphasis role="italic">bytes-per-inode</emphasis> ratio ("inode ratio")
used for OSTs of various sizes when they are formatted.</para>
<tbody>
<row>
<entry>
- <para> over 10GB </para>
+ <para>under 10GiB </para>
</entry>
<entry>
- <para> 1 inode/16KB </para>
+ <para>1 inode/16KiB </para>
</entry>
<entry>
- <para> 640 - 655k </para>
+ <para>640 - 655k </para>
</entry>
</row>
<row>
<entry>
- <para> 10GB - 1TB </para>
+ <para>10GiB - 1TiB </para>
</entry>
<entry>
- <para> 1 inode/68kiB </para>
+ <para>1 inode/68KiB </para>
</entry>
<entry>
- <para> 153k - 15.7M </para>
+ <para>153k - 15.7M </para>
</entry>
</row>
<row>
<entry>
- <para> 1TB - 8TB </para>
+ <para>1TiB - 8TiB </para>
</entry>
<entry>
- <para> 1 inode/256kB </para>
+ <para>1 inode/256KiB </para>
</entry>
<entry>
- <para> 4.2M - 33.6M </para>
+ <para>4.2M - 33.6M </para>
</entry>
</row>
<row>
<entry>
- <para> over 8TB </para>
+ <para>over 8TiB </para>
</entry>
<entry>
- <para> 1 inode/1MB </para>
+ <para>1 inode/1MiB </para>
</entry>
<entry>
- <para> 8.4M - 134M </para>
+ <para>8.4M - 268M </para>
</entry>
</row>
</tbody>
ratio, use the <literal>--mkfsoptions="-i <replaceable>bytes-per-inode</replaceable>"</literal>
argument to <literal>mkfs.lustre</literal> to specify the expected
average (mean) size of OST objects. For example, to create an OST
- with an expected average object size of 8MB run:
+ with an expected average object size of 8 MiB run:
<screen>[oss#] mkfs.lustre --ost --mkfsoptions="-i $((8192 * 1024))" ...</screen>
</para>
<note>
- <para>OSTs formatted with ldiskfs are limited to a maximum of
- 320 million to 1 billion objects. Specifying a very small
- bytes-per-inode ratio for a large OST that causes this limit to be
- exceeded can cause either premature out-of-space errors and prevent
- the full OST space from being used, or will waste space and slow down
- e2fsck more than necessary. The default inode ratios are chosen to
- ensure that the total number of inodes remain below this limit.
- </para>
+ <para>OSTs formatted with ldiskfs should preferably have fewer than
+ 320 million objects per MDT, and up to a maximum of 4 billion inodes.
+ Specifying a very small bytes-per-inode ratio for a large OST that
+ exceeds this limit can cause either premature out-of-space errors and
+ prevent the full OST space from being used, or will waste space and
+ slow down e2fsck more than necessary. The default inode ratios are
+ chosen to ensure the total number of inodes remain below this limit.
+ </para>
</note>
<note>
<para>File system check time on OSTs is affected by a number of
- variables in addition to the number of inodes, including the size of
- the file system, the number of allocated blocks, the distribution of
- allocated blocks on the disk, disk speed, CPU speed, and the amount
- of RAM on the server. Reasonable file system check times for valid
- filesystems are 5-30 minutes per TB, but may increase significantly
- if substantial errors are detected and need to be required.</para>
+ variables in addition to the number of inodes, including the size of
+ the file system, the number of allocated blocks, the distribution of
+ allocated blocks on the disk, disk speed, CPU speed, and the amount
+ of RAM on the server. Reasonable file system check times for valid
+ filesystems are 5-30 minutes per TiB, but may increase significantly
+ if substantial errors are detected and need to be repaired.</para>
</note>
- <para>For more details about formatting MDT and OST file systems,
+ <para>For further details about optimizing MDT and OST file systems,
see <xref linkend="dbdoclet.ldiskfs_raid_opts"/>.</para>
</section>
- <section remap="h3">
- <title><indexterm>
- <primary>setup</primary>
- <secondary>limits</secondary>
- </indexterm><indexterm xmlns:xi="http://www.w3.org/2001/XInclude">
- <primary>wide striping</primary>
- </indexterm><indexterm xmlns:xi="http://www.w3.org/2001/XInclude">
- <primary>xattr</primary>
- <secondary><emphasis role="italic">See</emphasis> wide striping</secondary>
- </indexterm><indexterm>
- <primary>large_xattr</primary>
- <secondary>ea_inode</secondary>
- </indexterm><indexterm>
- <primary>wide striping</primary>
- <secondary>large_xattr</secondary>
- <tertiary>ea_inode</tertiary>
- </indexterm>File and File System Limits</title>
+ </section>
+ <section remap="h3">
+ <title><indexterm>
+ <primary>setup</primary>
+ <secondary>limits</secondary>
+ </indexterm><indexterm xmlns:xi="http://www.w3.org/2001/XInclude">
+ <primary>wide striping</primary>
+ </indexterm><indexterm xmlns:xi="http://www.w3.org/2001/XInclude">
+ <primary>xattr</primary>
+ <secondary><emphasis role="italic">See</emphasis> wide striping</secondary>
+ </indexterm><indexterm>
+ <primary>large_xattr</primary>
+ <secondary>ea_inode</secondary>
+ </indexterm><indexterm>
+ <primary>wide striping</primary>
+ <secondary>large_xattr</secondary>
+ <tertiary>ea_inode</tertiary>
+ </indexterm>File and File System Limits</title>
- <para><xref linkend="settinguplustresystem.tab2"/> describes
- current known limits of Lustre. These limits are imposed by either
- the Lustre architecture or the Linux virtual file system (VFS) and
- virtual memory subsystems. In a few cases, a limit is defined within
- the code and can be changed by re-compiling the Lustre software.
- Instructions to install from source code are beyond the scope of this
- document, and can be found elsewhere online. In these cases, the
- indicated limit was used for testing of the Lustre software. </para>
+ <para><xref linkend="settinguplustresystem.tab2"/> describes
+ current known limits of Lustre. These limits may be imposed by either
+ the Lustre architecture or the Linux virtual file system (VFS) and
+ virtual memory subsystems. In a few cases, a limit is defined within
+ the code Lustre based on tested values and could be changed by editing
+ and re-compiling the Lustre software. In these cases, the indicated
+ limit was used for testing of the Lustre software.</para>
-
<table frame="all" xml:id="settinguplustresystem.tab2">
- <title>File and file system limits</title>
- <tgroup cols="3">
- <colspec colname="c1" colwidth="3*"/>
- <colspec colname="c2" colwidth="2*"/>
- <colspec colname="c3" colwidth="4*"/>
- <thead>
- <row>
- <entry>
-
<para><emphasis role="bold">Limit</emphasis></para>
- </entry>
- <entry>
-
<para><emphasis role="bold">Value</emphasis></para>
- </entry>
- <entry>
-
<para><emphasis role="bold">Description</emphasis></para>
- </entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry>
- <para> Maximum number of MDTs</para>
- </entry>
- <entry>
- <para condition='l24'>256</para>
- </entry>
- <entry>
- <para>The Lustre software release 2.3 and earlier allows a
- maximum of 1 MDT per file system, but a single MDS can host
- multiple MDTs, each one for a separate file system.</para>
- <para condition="l24">The Lustre software release 2.4 and later
- requires one MDT for the filesystem root. At least 255 more
- MDTs can be added to the filesystem and attached into
- the namespace with DNE remote or striped directories.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum number of OSTs</para>
- </entry>
- <entry>
- <para> 8150</para>
- </entry>
- <entry>
-
<para>The maximum number of OSTs is a constant that can be
- changed at compile time. Lustre file systems with up to
- 4000 OSTs have been tested. Multiple OST file systems can
- be configured on a single OSS node.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum OST size</para>
- </entry>
- <entry>
- <para> 128TB (ldiskfs), 256TB (ZFS)</para>
- </entry>
- <entry>
-
<para>This is not a <emphasis>hard</emphasis> limit. Larger
- OSTs are possible but today typical production systems do not
- typically go beyond the stated limit per OST because Lustre
- can add capacity and performance with additional OSTs, and
- having more OSTs improves aggregate I/O performance and
- minimizes contention.
- </para>
- <para>
- With 32-bit kernels, due to page cache limits, 16TB is the
- maximum block device size, which in turn applies to the
- size of OST. It is strongly recommended to run Lustre
- clients and servers with 64-bit kernels.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum number of clients</para>
- </entry>
- <entry>
- <para> 131072</para>
- </entry>
- <entry>
- <para>The maximum number of clients is a constant that can
- be changed at compile time. Up to 30000 clients have been
- used in production.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum size of a file system</para>
- </entry>
- <entry>
- <para> 512 PB (ldiskfs), 1EB (ZFS)</para>
- </entry>
- <entry>
- <para>Each OST can have a file system up to the
- Maximum OST size limit, and the Maximum number of OSTs
- can be combined into a single filesystem.
- </para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum stripe count</para>
- </entry>
- <entry>
- <para> 2000</para>
- </entry>
- <entry>
- <para>This limit is imposed by the size of the layout that
- needs to be stored on disk and sent in RPC requests, but is
- not a hard limit of the protocol. The number of OSTs in the
- filesystem can exceed the stripe count, but this limits the
- number of OSTs across which a single file can be striped.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum stripe size</para>
- </entry>
- <entry>
- <para> < 4 GB</para>
- </entry>
- <entry>
- <para>The amount of data written to each object before moving
- on to next object.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Minimum stripe size</para>
- </entry>
- <entry>
- <para> 64 KB</para>
- </entry>
- <entry>
- <para>Due to the 64 KB PAGE_SIZE on some 64-bit machines,
- the minimum stripe size is set to 64 KB.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum object size</para>
- </entry>
- <entry>
- <para> 16TB (ldiskfs), 256TB (ZFS)</para>
- </entry>
- <entry>
- <para>The amount of data that can be stored in a single object.
- An object corresponds to a stripe. The ldiskfs limit of 16 TB
- for a single object applies. For ZFS the limit is the size of
- the underlying OST. Files can consist of up to 2000 stripes,
- each stripe can be up to the maximum object size. </para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum <anchor xml:id="dbdoclet.50438256_marker-1290761" xreflabel=""/>file size</para>
- </entry>
- <entry>
- <para> 16 TB on 32-bit systems</para>
- <para> </para>
- <para> 31.25 PB on 64-bit ldiskfs systems, 8EB on 64-bit ZFS systems</para>
- </entry>
- <entry>
- <para>Individual files have a hard limit of nearly 16 TB on
- 32-bit systems imposed by the kernel memory subsystem. On
- 64-bit systems this limit does not exist. Hence, files can
- be 2^63 bits (8EB) in size if the backing filesystem can
- support large enough objects.</para>
- <para>A single file can have a maximum of 2000 stripes, which
- gives an upper single file limit of 31.25 PB for 64-bit
- ldiskfs systems. The actual amount of data that can be stored
- in a file depends upon the amount of free space in each OST
- on which the file is striped.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum number of files or subdirectories in a single directory</para>
- </entry>
- <entry>
- <para> 10 million files (ldiskfs), 2^48 (ZFS)</para>
- </entry>
- <entry>
- <para>The Lustre software uses the ldiskfs hashed directory
- code, which has a limit of about 10 million files, depending
- on the length of the file name. The limit on subdirectories
- is the same as the limit on regular files.</para>
- <note condition='l28'><para>Starting in the 2.8 release it is
- possible to exceed this limit by striping a single directory
- over multiple MDTs with the <literal>lfs mkdir -c</literal>
- command, which increases the single directory limit by a
- factor of the number of directory stripes used.</para></note>
- <para>Lustre file systems are tested with ten million files
- in a single directory.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum number of files in the file system</para>
- </entry>
- <entry>
- <para> 4 billion (ldiskfs), 256 trillion (ZFS)</para>
- <para condition='l24'>up to 256 times the per-MDT limit</para>
- </entry>
- <entry>
- <para>The ldiskfs filesystem imposes an upper limit of
- 4 billion inodes per filesystem. By default, the MDT
- filesystem is formatted with one inode per 2KB of space,
- meaning 512 million inodes per TB of MDT space. This can be
- increased initially at the time of MDT filesystem creation.
- For more information, see
- <xref linkend="settinguplustresystem"/>.</para>
- <para condition="l24">The ZFS filesystem
- dynamically allocates inodes and does not have a fixed ratio
- of inodes per unit of MDT space, but consumes approximately
- 4KB of space per inode, depending on the configuration.</para>
- <para condition="l24">Each additional MDT can hold up to the
- above maximum number of additional files, depending on
- available space and the distribution directories and files
- in the filesystem.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum length of a filename</para>
- </entry>
- <entry>
- <para> 255 bytes (filename)</para>
- </entry>
- <entry>
- <para>This limit is 255 bytes for a single filename, the
- same as the limit in the underlying filesystems.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum length of a pathname</para>
- </entry>
- <entry>
- <para> 4096 bytes (pathname)</para>
- </entry>
- <entry>
- <para>The Linux VFS imposes a full pathname length of 4096 bytes.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Maximum number of open files for a Lustre file system</para>
- </entry>
- <entry>
- <para> No limit</para>
- </entry>
- <entry>
- <para>The Lustre software does not impose a maximum for the number of open files,
- but the practical limit depends on the amount of RAM on the MDS. No
- "tables" for open files exist on the MDS, as they are only linked in a
- list to a given client's export. Each client process probably has a limit of
- several thousands of open files which depends on the ulimit.</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </table>
- <para> </para>
- <note>
- <para condition="l22">In Lustre software releases prior to version 2.2,
- the maximum stripe count for a single file was limited to 160 OSTs.
- In version 2.2, the wide striping feature was added to support files
- striped over up to 2000 OSTs. In order to store the large layout for
- such files in ldiskfs, the <literal>ea_inode</literal> feature must
- be enabled on the MDT, but no similar tunable is needed for ZFS MDTs.
- This feature is disabled by default at
- <literal>mkfs.lustre</literal> time. In order to enable this feature,
- specify <literal>--mkfsoptions="-O ea_inode"</literal> at MDT format
- time, or use <literal>tune2fs -O ea_inode</literal> to enable it after
- the MDT has been formatted. Using either the deprecated
- <literal>large_xattr</literal> or preferred <literal>ea_inode</literal>
- feature name results in <literal>ea_inode</literal> being shown in
- the file system feature list.</para>
- </note>
- </section>
+ <table frame="all" xml:id="settinguplustresystem.tab2">
+ <title>File and file system limits</title>
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="3*"/>
+ <colspec colname="c2" colwidth="2*"/>
+ <colspec colname="c3" colwidth="4*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Limit</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Value</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_mdt_count" xreflabel=""/>Maximum number of MDTs</para>
+ </entry>
+ <entry>
+ <para>256</para>
+ </entry>
+ <entry>
+ <para>A single MDS can host one or more MDTs, either for separate
+ filesystems, or aggregated into a single namespace. Each
+ filesystem requires a separate MDT for the filesystem root
+ directory.
+ Up to 255 more MDTs can be added to the filesystem and are
+ attached into the filesystem namespace with creation of DNE
+ remote or striped directories.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_ost_count" xreflabel=""/>Maximum number of OSTs</para>
+ </entry>
+ <entry>
+ <para>8150</para>
+ </entry>
+ <entry>
+ <para>The maximum number of OSTs is a constant that can be
+ changed at compile time. Lustre file systems with up to 4000
+ OSTs have been configured in the past. Multiple OST targets
+ can be configured on a single OSS node.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_ost_size" xreflabel=""/>Maximum OST size</para>
+ </entry>
+ <entry>
+ <para>1024TiB (ldiskfs), 1024TiB (ZFS)</para>
+ </entry>
+ <entry>
+ <para>This is not a <emphasis>hard</emphasis> limit. Larger
+ OSTs are possible, but most production systems do not
+ typically go beyond the stated limit per OST because Lustre
+ can add capacity and performance with additional OSTs, and
+ having more OSTs improves aggregate I/O performance,
+ minimizes contention, and allows parallel recovery (e2fsck
+ for ldiskfs OSTs, scrub for ZFS OSTs).
+ </para>
+ <para>
+ With 32-bit kernels, due to page cache limits, 16TB is the
+ maximum block device size, which in turn applies to the
+ size of OST. It is <emphasis>strongly</emphasis> recommended
+ to run Lustre clients and servers with 64-bit kernels.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_client_count" xreflabel=""/>Maximum number of clients</para>
+ </entry>
+ <entry>
+ <para>131072</para>
+ </entry>
+ <entry>
+ <para>The maximum number of clients is a constant that can
+ be changed at compile time. Up to 30000 clients have been
+ used in production accessing a single filesystem.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_filesysem_size" xreflabel=""/>Maximum size of a single file system</para>
+ </entry>
+ <entry>
+ <para>2EiB or larger</para>
+ </entry>
+ <entry>
+ <para>Each OST can have a file system up to the "Maximum OST
+ size" limit, and the Maximum number of OSTs can be combined
+ into a single filesystem.
+ </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_stripe_count" xreflabel=""/>Maximum stripe count</para>
+ </entry>
+ <entry>
+ <para>2000</para>
+ </entry>
+ <entry>
+ <para>This limit is imposed by the size of the layout that
+ needs to be stored on disk and sent in RPC requests, but is
+ not a hard limit of the protocol. The number of OSTs in the
+ filesystem can exceed the stripe count, but this is the maximum
+ number of OSTs on which a <emphasis>single file</emphasis>
+ can be striped.</para>
+ <note condition='l2D'><para>Before 2.13, the default for ldiskfs
+ MDTs the maximum stripe count for a
+ <emphasis>single file</emphasis> is limited to 160 OSTs. In order to
+ increase the maximum file stripe count, use
+ <literal>--mkfsoptions="-O ea_inode"</literal> when formatting the MDT,
+ or use <literal>tune2fs -O ea_inode</literal> to enable it after the
+ MDT has been formatted.</para>
+ </note>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_stripe_size" xreflabel=""/>Maximum stripe size</para>
+ </entry>
+ <entry>
+ <para>< 4 GiB</para>
+ </entry>
+ <entry>
+ <para>The amount of data written to each object before moving
+ on to next object.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.min_stripe_size" xreflabel=""/>Minimum stripe size</para>
+ </entry>
+ <entry>
+ <para>64 KiB</para>
+ </entry>
+ <entry>
+ <para>Due to the use of 64 KiB PAGE_SIZE on some CPU
+ architectures such as ARM and POWER, the minimum stripe
+ size is 64 KiB so that a single page is not split over
+ multiple servers. This is also the minimum Data-on-MDT
+ component size that can be specified.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_object_size" xreflabel=""/>Maximum single object size</para>
+ </entry>
+ <entry>
+ <para>16TiB (ldiskfs), 256TiB (ZFS)</para>
+ </entry>
+ <entry>
+ <para>The amount of data that can be stored in a single object.
+ An object corresponds to a stripe. The ldiskfs limit of 16 TB
+ for a single object applies. For ZFS the limit is the size of
+ the underlying OST. Files can consist of up to 2000 stripes,
+ each stripe can be up to the maximum object size. </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_file_size" xreflabel=""/>Maximum file size</para>
+ </entry>
+ <entry>
+ <para>16 TiB on 32-bit systems</para>
+ <para> </para>
+ <para>31.25 PiB on 64-bit ldiskfs systems,
+ 8EiB on 64-bit ZFS systems</para>
+ </entry>
+ <entry>
+ <para>Individual files have a hard limit of nearly 16 TiB on
+ 32-bit systems imposed by the kernel memory subsystem. On
+ 64-bit systems this limit does not exist. Hence, files can
+ be 2^63 bits (8EiB) in size if the backing filesystem can
+ support large enough objects and/or the files are sparse.</para>
+ <para>A single file can have a maximum of 2000 stripes, which
+ gives an upper single file data capacity of 31.25 PiB for 64-bit
+ ldiskfs systems. The actual amount of data that can be stored
+ in a file depends upon the amount of free space in each OST
+ on which the file is striped.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_directory_size" xreflabel=""/>Maximum number of files or subdirectories in a single directory</para>
+ </entry>
+ <entry>
+ <para>600M-3.8B files (ldiskfs), 16T (ZFS)</para>
+ </entry>
+ <entry>
+ <para>The Lustre software uses the ldiskfs hashed directory
+ code, which has a limit of at least 600 million files, depending
+ on the length of the file name. The limit on subdirectories
+ is the same as the limit on regular files.</para>
+ <note condition='l28'><para>Starting in the 2.8 release it is
+ possible to exceed this limit by striping a single directory
+ over multiple MDTs with the <literal>lfs mkdir -c</literal>
+ command, which increases the single directory limit by a
+ factor of the number of directory stripes used.</para></note>
+ <note condition='l2E'><para>Starting in the 2.14 release, the
+ <literal>large_dir</literal> feature of ldiskfs is enabled by
+ default to allow directories with more than 10M entries. In
+ the 2.12 release, the <literal>large_dir</literal> feature was
+ present but not enabled by default.</para></note>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_file_count" xreflabel=""/>Maximum number of files in the file system</para>
+ </entry>
+ <entry>
+ <para>4 billion (ldiskfs), 256 trillion (ZFS) <emphasis>per MDT</emphasis></para>
+ </entry>
+ <entry>
+ <para>The ldiskfs filesystem imposes an upper limit of
+ 4 billion inodes per filesystem. By default, the MDT
+ filesystem is formatted with one inode per 2KB of space,
+ meaning 512 million inodes per TiB of MDT space. This can be
+ increased initially at the time of MDT filesystem creation.
+ For more information, see
+ <xref linkend="settinguplustresystem"/>.</para>
+ <para>The ZFS filesystem dynamically allocates
+ inodes and does not have a fixed ratio of inodes per unit of MDT
+ space, but consumes approximately 4KiB of mirrored space per
+ inode, depending on the configuration.</para>
+ <para>Each additional MDT can hold up to the
+ above maximum number of additional files, depending on
+ available space and the distribution directories and files
+ in the filesystem.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_filename_size" xreflabel=""/>Maximum length of a filename</para>
+ </entry>
+ <entry>
+ <para>255 bytes (filename)</para>
+ </entry>
+ <entry>
+ <para>This limit is 255 bytes for a single filename, the
+ same as the limit in the underlying filesystems.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_pathname_size" xreflabel=""/>Maximum length of a pathname</para>
+ </entry>
+ <entry>
+ <para>4096 bytes (pathname)</para>
+ </entry>
+ <entry>
+ <para>The Linux VFS imposes a full pathname length of 4096 bytes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><anchor xml:id="dbdoclet.max_open_files" xreflabel=""/>Maximum number of open files for a Lustre file system</para>
+ </entry>
+ <entry>
+ <para>No limit</para>
+ </entry>
+ <entry>
+ <para>The Lustre software does not impose a maximum for the number
+ of open files, but the practical limit depends on the amount of
+ RAM on the MDS. No "tables" for open files exist on the
+ MDS, as they are only linked in a list to a given client's
+ export. Each client process has a limit of several
+ thousands of open files which depends on its ulimit.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ <para> </para>
</section>
- <section xml:id="dbdoclet.50438256_26456">
+ <section xml:id="dbdoclet.mds_oss_memory">
<title><indexterm><primary>setup</primary><secondary>memory</secondary></indexterm>Determining Memory Requirements</title>
<para>This section describes the memory requirements for each Lustre file system component.</para>
<section remap="h3">
<para>Load placed on server</para>
</listitem>
</itemizedlist>
- <para>The amount of memory used by the MDS is a function of how many clients are on the system, and how many files they are using in their working set. This is driven, primarily, by the number of locks a client can hold at one time. The number of locks held by clients varies by load and memory availability on the server. Interactive clients can hold in excess of 10,000 locks at times. On the MDS, memory usage is approximately 2 KB per file, including the Lustre distributed lock manager (DLM) lock and kernel data structures for the files currently in use. Having file data in cache can improve metadata performance by a factor of 10x or more compared to reading it from disk.</para>
+ <para>The amount of memory used by the MDS is a function of how many clients are on
+ the system, and how many files they are using in their working set. This is driven,
+ primarily, by the number of locks a client can hold at one time. The number of locks
+ held by clients varies by load and memory availability on the server. Interactive
+ clients can hold in excess of 10,000 locks at times. On the MDS, memory usage is
+ approximately 2 KB per file, including the Lustre distributed lock manager (LDLM)
+ lock and kernel data structures for the files currently in use. Having file data
+ in cache can improve metadata performance by a factor of 10x or more compared to
+ reading it from storage.</para>
<para>MDS memory requirements include:</para>
<itemizedlist>
<listitem>
- <para><emphasis role="bold">File system metadata</emphasis> : A reasonable amount of RAM needs to be available for file system metadata. While no hard limit can be placed on the amount of file system metadata, if more RAM is available, then the disk I/O is needed less often to retrieve the metadata.</para>
+ <para><emphasis role="bold">File system metadata</emphasis>:
+ A reasonable amount of RAM needs to be available for file system metadata.
+ While no hard limit can be placed on the amount of file system metadata,
+ if more RAM is available, then the disk I/O is needed less often to retrieve
+ the metadata.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Network transport</emphasis> : If you are using TCP or other network transport that uses system memory for send/receive buffers, this memory requirement must also be taken into consideration.</para>
+ <para><emphasis role="bold">Network transport</emphasis>:
+ If you are using TCP or other network transport that uses system memory for
+ send/receive buffers, this memory requirement must also be taken into
+ consideration.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Journal size</emphasis> : By default, the journal size is 400 MB for each Lustre ldiskfs file system. This can pin up to an equal amount of RAM on the MDS node per file system.</para>
+ <para><emphasis role="bold">Journal size</emphasis>:
+ By default, the journal size is 4096 MB for each MDT ldiskfs file system.
+ This can pin up to an equal amount of RAM on the MDS node per file system.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Failover configuration</emphasis> : If the MDS node will be used for failover from another node, then the RAM for each journal should be doubled, so the backup server can handle the additional load if the primary server fails.</para>
+ <para><emphasis role="bold">Failover configuration</emphasis>:
+ If the MDS node will be used for failover from another node, then the RAM
+ for each journal should be doubled, so the backup server can handle the
+ additional load if the primary server fails.</para>
</listitem>
</itemizedlist>
<section remap="h4">
<title><indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>MDS</tertiary></indexterm>Calculating MDS Memory Requirements</title>
- <para>By default, 400 MB are used for the file system journal. Additional RAM is used for caching file data for the larger working set, which is not actively in use by clients but should be kept "hot" for improved access times. Approximately 1.5 KB per file is needed to keep a file in cache without a lock.</para>
- <para>For example, for a single MDT on an MDS with 1,000 clients, 16 interactive nodes, and a 2 million file working set (of which 400,000 files are cached on the clients):</para>
+ <para>By default, 4096 MB are used for the ldiskfs filesystem journal. Additional
+ RAM is used for caching file data for the larger working set, which is not
+ actively in use by clients but should be kept "hot" for improved
+ access times. Approximately 1.5 KB per file is needed to keep a file in cache
+ without a lock.</para>
+ <para>For example, for a single MDT on an MDS with 1,024 clients, 12 interactive
+ login nodes, and a 6 million file working set (of which 4M files are cached
+ on the clients):</para>
<informalexample>
- <para>Operating system overhead = 512 MB</para>
- <para>File system journal = 400 MB</para>
- <para>1000 * 4-core clients * 100 files/core * 2kB = 800 MB</para>
- <para>16 interactive clients * 10,000 files * 2kB = 320 MB</para>
- <para>1,600,000 file extra working set * 1.5kB/file = 2400 MB</para>
+ <para>Operating system overhead = 1024 MB</para>
+ <para>File system journal = 4096 MB</para>
+ <para>1024 * 4-core clients * 1024 files/core * 2kB = 4096 MB</para>
+ <para>12 interactive clients * 100,000 files * 2kB = 2400 MB</para>
+ <para>2M file extra working set * 1.5kB/file = 3096 MB</para>
</informalexample>
- <para>Thus, the minimum requirement for a system with this configuration is at least 4 GB of RAM. However, additional memory may significantly improve performance.</para>
- <para>For directories containing 1 million or more files, more memory may provide a significant benefit. For example, in an environment where clients randomly access one of 10 million files, having extra memory for the cache significantly improves performance.</para>
+ <para>Thus, the minimum requirement for an MDT with this configuration is at least
+ 16 GB of RAM. Additional memory may significantly improve performance.</para>
+ <para>For directories containing 1 million or more files, more memory can provide
+ a significant benefit. For example, in an environment where clients randomly
+ access one of 10 million files, having extra memory for the cache significantly
+ improves performance.</para>
</section>
</section>
<section remap="h3">
<title><indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>OSS</tertiary></indexterm>OSS Memory Requirements</title>
- <para>When planning the hardware for an OSS node, consider the memory usage of several
- components in the Lustre file system (i.e., journal, service threads, file system metadata,
- etc.). Also, consider the effect of the OSS read cache feature, which consumes memory as it
- caches data on the OSS node.</para>
- <para>In addition to the MDS memory requirements mentioned in <xref linkend="dbdoclet.50438256_87676"/>, the OSS requirements include:</para>
+ <para>When planning the hardware for an OSS node, consider the memory usage of
+ several components in the Lustre file system (i.e., journal, service threads,
+ file system metadata, etc.). Also, consider the effect of the OSS read cache
+ feature, which consumes memory as it caches data on the OSS node.</para>
+ <para>In addition to the MDS memory requirements mentioned above,
+ the OSS requirements also include:</para>
<itemizedlist>
<listitem>
- <para><emphasis role="bold">Service threads</emphasis> : The service threads on the OSS node pre-allocate a 4 MB I/O buffer for each ost_io service thread, so these buffers do not need to be allocated and freed for each I/O request.</para>
+ <para><emphasis role="bold">Service threads</emphasis>:
+ The service threads on the OSS node pre-allocate an RPC-sized MB I/O buffer
+ for each ost_io service thread, so these buffers do not need to be allocated
+ and freed for each I/O request.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">OSS read cache</emphasis> : OSS read cache provides read-only
- caching of data on an OSS, using the regular Linux page cache to store the data. Just
- like caching from a regular file system in the Linux operating system, OSS read cache
- uses as much physical memory as is available.</para>
+ <para><emphasis role="bold">OSS read cache</emphasis>:
+ OSS read cache provides read-only caching of data on an OSS, using the regular
+ Linux page cache to store the data. Just like caching from a regular file
+ system in the Linux operating system, OSS read cache uses as much physical
+ memory as is available.</para>
</listitem>
</itemizedlist>
- <para>The same calculation applies to files accessed from the OSS as for the MDS, but the load is distributed over many more OSSs nodes, so the amount of memory required for locks, inode cache, etc. listed under MDS is spread out over the OSS nodes.</para>
- <para>Because of these memory requirements, the following calculations should be taken as determining the absolute minimum RAM required in an OSS node.</para>
+ <para>The same calculation applies to files accessed from the OSS as for the MDS,
+ but the load is distributed over many more OSSs nodes, so the amount of memory
+ required for locks, inode cache, etc. listed under MDS is spread out over the
+ OSS nodes.</para>
+ <para>Because of these memory requirements, the following calculations should be
+ taken as determining the absolute minimum RAM required in an OSS node.</para>
<section remap="h4">
<title><indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>OSS</tertiary></indexterm>Calculating OSS Memory Requirements</title>
- <para>The minimum recommended RAM size for an OSS with two OSTs is computed below:</para>
+ <para>The minimum recommended RAM size for an OSS with eight OSTs is:</para>
<informalexample>
- <para>Ethernet/TCP send/receive buffers (4 MB * 512 threads) = 2048 MB</para>
- <para>400 MB journal size * 2 OST devices = 800 MB</para>
- <para>1.5 MB read/write per OST IO thread * 512 threads = 768 MB</para>
- <para>600 MB file system read cache * 2 OSTs = 1200 MB</para>
- <para>1000 * 4-core clients * 100 files/core * 2kB = 800MB</para>
- <para>16 interactive clients * 10,000 files * 2kB = 320MB</para>
- <para>1,600,000 file extra working set * 1.5kB/file = 2400MB</para>
- <para> DLM locks + file system metadata TOTAL = 3520MB</para>
- <para>Per OSS DLM locks + file system metadata = 3520MB/6 OSS = 600MB (approx.)</para>
- <para>Per OSS RAM minimum requirement = 4096MB (approx.)</para>
+ <para>Linux kernel and userspace daemon memory = 1024 MB</para>
+ <para>Network send/receive buffers (16 MB * 512 threads) = 8192 MB</para>
+ <para>1024 MB ldiskfs journal size * 8 OST devices = 8192 MB</para>
+ <para>16 MB read/write buffer per OST IO thread * 512 threads = 8192 MB</para>
+ <para>2048 MB file system read cache * 8 OSTs = 16384 MB</para>
+ <para>1024 * 4-core clients * 1024 files/core * 2kB/file = 8192 MB</para>
+ <para>12 interactive clients * 100,000 files * 2kB/file = 2400 MB</para>
+ <para>2M file extra working set * 2kB/file = 4096 MB</para>
+ <para>DLM locks + file cache TOTAL = 31072 MB</para>
+ <para>Per OSS DLM locks + file system metadata = 31072 MB/4 OSS = 7768 MB (approx.)</para>
+ <para>Per OSS RAM minimum requirement = 32 GB (approx.)</para>
</informalexample>
- <para>This consumes about 1,400 MB just for the pre-allocated buffers, and an additional 2 GB for minimal file system and kernel usage. Therefore, for a non-failover configuration, the minimum RAM would be 4 GB for an OSS node with two OSTs. Adding additional memory on the OSS will improve the performance of reading smaller, frequently-accessed files.</para>
- <para>For a failover configuration, the minimum RAM would be at least 6 GB. For 4 OSTs on each OSS in a failover configuration 10GB of RAM is reasonable. When the OSS is not handling any failed-over OSTs the extra RAM will be used as a read cache.</para>
- <para>As a reasonable rule of thumb, about 2 GB of base memory plus 1 GB per OST can be used. In failover configurations, about 2 GB per OST is needed.</para>
+ <para>This consumes about 16 GB just for pre-allocated buffers, and an
+ additional 1 GB for minimal file system and kernel usage. Therefore, for a
+ non-failover configuration, the minimum RAM would be about 32 GB for an OSS node
+ with eight OSTs. Adding additional memory on the OSS will improve the performance
+ of reading smaller, frequently-accessed files.</para>
+ <para>For a failover configuration, the minimum RAM would be at least 48 GB,
+ as some of the memory is per-node. When the OSS is not handling any failed-over
+ OSTs the extra RAM will be used as a read cache.</para>
+ <para>As a reasonable rule of thumb, about 8 GB of base memory plus 3 GB per OST
+ can be used. In failover configurations, about 6 GB per OST is needed.</para>
</section>
</section>
</section>
- <section xml:id="dbdoclet.50438256_78272">
+ <section xml:id="dbdoclet.network_considerations">
<title><indexterm>
<primary>setup</primary>
<secondary>network</secondary>
</note>
</section>
</chapter>
+<!--
+ vim:expandtab:shiftwidth=2:tabstop=8:
+ -->
git://git.whamcloud.com / doc / manual.git / blobdiff