<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settinguplustresystem">
<title xml:id="settinguplustresystem.title">Determining Hardware Configuration Requirements and
Formatting Options</title>
- <para>This chapter describes hardware configuration requirements for a Lustre* file system
+ <para>This chapter describes hardware configuration requirements for a Lustre file system
including:</para>
<itemizedlist>
<listitem>
<para>Since the block devices are accessed by only one or two server nodes, a storage area network (SAN) that is accessible from all the servers is not required. Expensive switches are not needed because point-to-point connections between the servers and the storage arrays normally provide the simplest and best attachments. (If failover capability is desired, the storage must be attached to multiple servers.)</para>
<para>For a production environment, it is preferable that the MGS have separate storage to allow future expansion to multiple file systems. However, it is possible to run the MDS and MGS on the same machine and have them share the same storage device.</para>
<para>For best performance in a production environment, dedicated clients are required. For a non-production Lustre environment or for testing, a Lustre client and server can run on the same machine. However, dedicated clients are the only supported configuration.</para>
- <para>Performance and other issues can occur when an MDS or OSS and a client are running on the same machine:</para>
+ <warning><para>Performance and recovery issues can occur if you put a client on an MDS or OSS:</para>
<itemizedlist>
<listitem>
- <para>Running the MDS and a client on the same machine can cause recovery and deadlock issues and impact the performance of other Lustre clients.</para>
+ <para>Running the OSS and a client on the same machine can cause issues with low memory and memory pressure. If the client consumes all the memory and then tries to write data to the file system, the OSS will need to allocate pages to receive data from the client but will not be able to perform this operation due to low memory. This can cause the client to hang.</para>
</listitem>
<listitem>
- <para>Running the OSS and a client on the same machine can cause issues with low memory and memory pressure. If the client consumes all the memory and then tries to write data to the file system, the OSS will need to allocate pages to receive data from the client but will not be able to perform this operation due to low memory. This can cause the client to hang.</para>
+ <para>Running the MDS and a client on the same machine can cause recovery and deadlock issues and impact the performance of other Lustre clients.</para>
</listitem>
</itemizedlist>
+ </warning>
<para>Only servers running on 64-bit CPUs are tested and supported. 64-bit CPU clients are
typically used for testing to match expected customer usage and avoid limitations due to the 4
GB limit for RAM size, 1 GB low-memory limitation, and 16 TB file size limit of 32-bit CPUs.
- Also, due to kernel API limitations, performing backups of Lustre 2.x. file systems on 32-bit
- clients may cause backup tools to confuse files that have the same 32-bit inode number.</para>
+ Also, due to kernel API limitations, performing backups of Lustre software release 2.x. file
+ systems on 32-bit clients may cause backup tools to confuse files that have the same 32-bit
+ inode number.</para>
<para>The storage attached to the servers typically uses RAID to provide fault tolerance and can
optionally be organized with logical volume management (LVM), which is then formatted as a
Lustre file system. Lustre OSS and MDS servers read, write and modify data in the format
<para>Lustre clients running on architectures with different endianness are supported. One limitation is that the PAGE_SIZE kernel macro on the client must be as large as the PAGE_SIZE of the server. In particular, ia64 or PPC clients with large pages (up to 64kB pages) can run with x86 servers (4kB pages). If you are running x86 clients with ia64 or PPC servers, you must compile the ia64 kernel with a 4kB PAGE_SIZE (so the server page size is not larger than the client page size). </para>
</note>
<section remap="h3">
- <title><indexterm><primary>setup</primary><secondary>MDT</secondary></indexterm>
- MDT Storage Hardware Considerations</title>
- <para>The data access pattern for MDS storage is a database-like access pattern with many seeks and read-and-writes of small amounts of data. High throughput to MDS storage is not important. Storage types that provide much lower seek times, such as high-RPM SAS or SSD drives can be used for the MDT.</para>
+ <title><indexterm>
+ <primary>setup</primary>
+ <secondary>MDT</secondary>
+ </indexterm> MGT and MDT Storage Hardware Considerations</title>
+ <para>MGT storage requirements are small (less than 100 MB even in the largest Lustre file
+ systems), and the data on an MGT is only accessed on a server/client mount, so disk
+ performance is not a consideration. However, this data is vital for file system access, so
+ the MGT should be reliable storage, preferably mirrored RAID1.</para>
+ <para>MDS storage is accessed in a database-like access pattern with many seeks and
+ read-and-writes of small amounts of data. High throughput to MDS storage is not important.
+ Storage types that provide much lower seek times, such as high-RPM SAS or SSD drives can be
+ used for the MDT.</para>
<para>For maximum performance, the MDT should be configured as RAID1 with an internal journal and two disks from different controllers.</para>
<para>If you need a larger MDT, create multiple RAID1 devices from pairs of disks, and then make a RAID0 array of the RAID1 devices. This ensures maximum reliability because multiple disk failures only have a small chance of hitting both disks in the same RAID1 device.</para>
<para>Doing the opposite (RAID1 of a pair of RAID0 devices) has a 50% chance that even two disk failures can cause the loss of the whole MDT device. The first failure disables an entire half of the mirror and the second failure has a 50% chance of disabling the remaining mirror.</para>
Subsequent directories may also be configured to have their own MDT. If an MDT serving a
subdirectory becomes unavailable this subdirectory and all directories beneath it will
also become unavailable. Configuring multiple levels of MDTs is an experimental feature
- for the Lustre 2.4 release.</para></note>
+ for the Lustre software release 2.4.</para></note>
</section>
<section remap="h3">
<title><indexterm><primary>setup</primary><secondary>OST</secondary></indexterm>OST Storage Hardware Considerations</title>
nearly 1 PB. If each OST uses ten 1 TB SATA disks (8 data disks plus 2 parity disks in a
RAID 6 configuration), it may be possible to get 50 MB/sec from each drive, providing up to
400 MB/sec of disk bandwidth per OST. If this system is used as storage backend with a
- system network, such as the InfiniBand* network, that provides a similar bandwidth, then
- each OSS could provide 800 MB/sec of end-to-end I/O throughput. (Although the architectural
+ system network, such as the InfiniBand network, that provides a similar bandwidth, then each
+ OSS could provide 800 MB/sec of end-to-end I/O throughput. (Although the architectural
constraints described here are simple, in practice it takes careful hardware selection,
benchmarking and integration to obtain such results.)</para>
</section>
<title><indexterm><primary>setup</primary><secondary>space</secondary></indexterm>
<indexterm><primary>space</primary><secondary>determining requirements</secondary></indexterm>
Determining Space Requirements</title>
- <para>The desired performance characteristics of the backing file systems on the MDT and OSTs are independent of one another. The size of the MDT backing file system depends on the number of inodes needed in the total Lustre file system, while the aggregate OST space depends on the total amount of data stored on the file system.</para>
+ <para>The desired performance characteristics of the backing file systems on the MDT and OSTs
+ are independent of one another. The size of the MDT backing file system depends on the number
+ of inodes needed in the total Lustre file system, while the aggregate OST space depends on the
+ total amount of data stored on the file system. If MGS data is to be stored on the MDT device
+ (co-located MGT and MDT), add 100 MB to the required size estimate for the MDT.</para>
<para>Each time a file is created on a Lustre file system, it consumes one inode on the MDT and one inode for each OST object over which the file is striped. Normally, each file's stripe count is based on the system-wide default stripe count. However, this can be changed for individual files using the <literal>lfs setstripe</literal> option. For more details, see <xref linkend="managingstripingfreespace"/>.</para>
<para>In a Lustre ldiskfs file system, all the inodes are allocated on the MDT and OSTs when the file system is first formatted. The total number of inodes on a formatted MDT or OST cannot be easily changed, although it is possible to add OSTs with additional space and corresponding inodes. Thus, the number of inodes created at format time should be generous enough to anticipate future expansion.</para>
<para>When the file system is in use and a file is created, the metadata associated with that file is stored in one of the pre-allocated inodes and does not consume any of the free space used to store file data.</para>
unusable for general storage. Thus, at least 400 MB of space is used on each OST before any
file object data is saved.</para>
</note>
+ <para condition="l24">With a ZFS backing filesystem for the MDT or OST,
+ the space allocation for inodes and file data is dynamic, and inodes are
+ allocated as needed. A minimum of 2kB of usable space (before RAID) is
+ needed for each inode, exclusive of other overhead such as directories,
+ internal log files, extended attributes, ACLs, etc.
+ Since the size of extended attributes and ACLs is highly dependent on
+ kernel versions and site-specific policies, it is best to over-estimate
+ the amount of space needed for the desired number of inodes, and any
+ excess space will be utilized to store more inodes.</para>
+ <section>
+ <title><indexterm>
+ <primary>setup</primary>
+ <secondary>MGT</secondary>
+ </indexterm>
+ <indexterm>
+ <primary>space</primary>
+ <secondary>determining MGT requirements</secondary>
+ </indexterm> Determining MGT Space Requirements</title>
+ <para>Less than 100 MB of space is required for the MGT. The size is determined by the number
+ of servers in the Lustre file system cluster(s) that are managed by the MGS.</para>
+ </section>
<section xml:id="dbdoclet.50438256_87676">
- <title><indexterm><primary>setup</primary><secondary>MDS/MDT</secondary></indexterm>
- <indexterm><primary>space</primary><secondary>determining MDS/MDT requirements</secondary></indexterm>
- Determining MDS/MDT Space Requirements</title>
- <para>When calculating the MDT size, the important factor to consider is the number of files to be stored in the file system. This determines the number of inodes needed, which drives the MDT sizing. To be on the safe side, plan for 2 KB per inode on the MDT, which is the default value. Attached storage required for Lustre metadata is typically 1-2 percent of the file system capacity depending upon file size.</para>
+ <title><indexterm>
+ <primary>setup</primary>
+ <secondary>MDT</secondary>
+ </indexterm>
+ <indexterm>
+ <primary>space</primary>
+ <secondary>determining MDT requirements</secondary>
+ </indexterm> Determining MDT Space Requirements</title>
+ <para>When calculating the MDT size, the important factor to consider is the number of files
+ to be stored in the file system. This determines the number of inodes needed, which drives
+ the MDT sizing. To be on the safe side, plan for 2 KB per inode on the MDT, which is the
+ default value. Attached storage required for Lustre file system metadata is typically 1-2
+ percent of the file system capacity depending upon file size.</para>
<para>For example, if the average file size is 5 MB and you have 100 TB of usable OST space, then you can calculate the minimum number of inodes as follows:</para>
<informalexample>
<para>(100 TB * 1024 GB/TB * 1024 MB/GB) / 5 MB/inode = 20 million inodes</para>
</note>
</section>
<section remap="h3">
- <title><indexterm><primary>setup</primary><secondary>OSS/OST</secondary></indexterm>
- <indexterm><primary>space</primary><secondary>determining OSS/OST requirements</secondary></indexterm>
- Determining OSS/OST Space Requirements</title>
+ <title><indexterm>
+ <primary>setup</primary>
+ <secondary>OST</secondary>
+ </indexterm>
+ <indexterm>
+ <primary>space</primary>
+ <secondary>determining OST requirements</secondary>
+ </indexterm> Determining OST Space Requirements</title>
<para>For the OST, the amount of space taken by each object depends on the usage pattern of
the users/applications running on the system. The Lustre software defaults to a conservative
estimate for the object size (16 KB per object). If you are confident that the average file
<para>To override the default formatting options, use arguments to
<literal>mkfs.lustre</literal> to pass formatting options to the backing file system:</para>
<screen>--mkfsoptions='backing fs options'</screen>
- <para>For other <literal>mkfs.lustre</literal> options, see the Linux* man page for
+ <para>For other <literal>mkfs.lustre</literal> options, see the Linux man page for
<literal>mke2fs(8)</literal>.</para>
<section xml:id="dbdoclet.50438256_pgfId-1293228">
<title><indexterm>
<secondary><emphasis role="italic">See</emphasis> wide striping</secondary>
</indexterm><indexterm>
<primary>large_xattr</primary>
+ <secondary>ea_inode</secondary>
</indexterm><indexterm>
<primary>wide striping</primary>
<secondary>large_xattr</secondary>
+ <tertiary>ea_inode</tertiary>
</indexterm>File and File System Limits</title>
<para><xref linkend="settinguplustresystem.tab2"/> describes file and file system size limits.
These limits are imposed by either the Lustre architecture or the Linux virtual file system
<para condition='l24'>4096</para>
</entry>
<entry>
- <para>The Lustre 2.3 release and earlier allows a maximum of 1 MDT per file system,
- but a single MDS can host multiple MDTs, each one for a separate file
+ <para>The Lustre software release 2.3 and earlier allows a maximum of 1 MDT per file
+ system, but a single MDS can host multiple MDTs, each one for a separate file
system.</para>
- <para condition="l24">The Lustre 2.4 release and later requires one MDT for the
- root. Upto 4095 additional MDTs can be added to the file system and attached into
- the namespace with remote directories.</para>
+ <para condition="l24">The Lustre software release 2.4 and later requires one MDT for
+ the filesystem root. Up to 4095 additional MDTs can be added to the file system and attached
+ into the namespace with remote directories.</para>
</entry>
</row>
<row>
<para> Maximum OST size</para>
</entry>
<entry>
- <para> 128TB </para>
+ <para> 128TB (ldiskfs), 256TB (ZFS)</para>
</entry>
<entry>
<para>This is not a <emphasis>hard</emphasis> limit. Larger OSTs are possible but
- today typical production systems do not go beyond 128TB per OST. </para>
+ today typical production systems do not go beyond the stated limit per OST. </para>
</entry>
</row>
<row>
<para> 131072</para>
</entry>
<entry>
- <para>The maximum number of clients is a constant that can be changed at compile time.</para>
+ <para>The maximum number of clients is a constant that can be changed at compile time. Up to 30000 clients have been used in production.</para>
</entry>
</row>
<row>
<para> Maximum size of a file system</para>
</entry>
<entry>
- <para> 512 PB</para>
+ <para> 512 PB (ldiskfs), 1EB (ZFS)</para>
</entry>
<entry>
- <para>Each OST or MDT on 64-bit kernel servers can have a file system up to 128 TB. On 32-bit systems, due to page cache limits, 16TB is the maximum block device size, which in turn applies to the size of OST on 32-bit kernel servers.</para>
+ <para>Each OST or MDT on 64-bit kernel servers can have a file system up to the above limit. On 32-bit systems, due to page cache limits, 16TB is the maximum block device size, which in turn applies to the size of OST on 32-bit kernel servers.</para>
<para>You can have multiple OST file systems on a single OSS node.</para>
</entry>
</row>
<row> <entry>
<para> Maximum object size</para> </entry>
<entry>
- <para> 16 TB</para>
+ <para> 16TB (ldiskfs), 256TB (ZFS)</para>
</entry>
<entry>
<para>The amount of data that can be stored in a single object. An object
- corresponds to a stripe. The ldiskfs limit of 16 TB for a single object applies.
- Files can consist of up to 2000 stripes, each 16 TB in size. </para>
+ corresponds to a stripe. The ldiskfs limit of 16 TB for a single object applies.
+ For ZFS the limit is the size of the underlying OST.
+ Files can consist of up to 2000 stripes, each stripe can contain the maximum object size. </para>
</entry>
</row>
<row>
<entry>
<para> 16 TB on 32-bit systems</para>
<para> </para>
- <para> 31.25 PB on 64-bit systems</para>
+ <para> 31.25 PB on 64-bit ldiskfs systems, 8EB on 64-bit ZFS systems</para>
</entry>
<entry>
<para>Individual files have a hard limit of nearly 16 TB on 32-bit systems imposed
by the kernel memory subsystem. On 64-bit systems this limit does not exist.
- Hence, files can be 64-bits in size. An additional size limit of up to the number
- of stripes is imposed, where each stripe is 16 TB.</para>
- <para>A single file can have a maximum of 2000 stripes, which gives an upper single file limit of 31.25 PB for 64-bit systems. The actual amount of data that can be stored in a file depends upon the amount of free space in each OST on which the file is striped.</para>
+ Hence, files can be 2^63 bits (8EB) in size if the backing filesystem can support large enough objects.</para>
+ <para>A single file can have a maximum of 2000 stripes, which gives an upper single file limit of 31.25 PB for 64-bit ldiskfs systems. The actual amount of data that can be stored in a file depends upon the amount of free space in each OST on which the file is striped.</para>
</entry>
</row>
<row>
<para> Maximum number of files or subdirectories in a single directory</para>
</entry>
<entry>
- <para> 10 million files</para>
+ <para> 10 million files (ldiskfs), 2^48 (ZFS)</para>
</entry>
<entry>
<para>The Lustre software uses the ldiskfs hashed directory code, which has a limit
<para> Maximum number of files in the file system</para>
</entry>
<entry>
- <para> 4 billion</para>
- <para condition='l24'>4096 * 4 billion</para>
+ <para> 4 billion (ldiskfs), 256 trillion (ZFS)</para>
+ <para condition='l24'>4096 times the per-MDT limit</para>
</entry>
<entry>
<para>The ldiskfs file system imposes an upper limit of 4 billion inodes. By default, the MDS file system is formatted with 2KB of space per inode, meaning 1 billion inodes per file system of 2 TB.</para>
<para>This can be increased initially, at the time of MDS file system creation. For more information, see <xref linkend="settinguplustresystem"/>.</para>
- <para condition="l24">Each additional MDT can hold up to 4 billion additional files, depending
- on available inodes and the distribution directories and files in the file
+ <para condition="l24">Each additional MDT can hold up to the above maximum number of additional files, depending
+ on available space and the distribution directories and files in the file
system.</para>
</entry>
</row>
<para> 255 bytes (filename)</para>
</entry>
<entry>
- <para>This limit is 255 bytes for a single filename, the same as in an ldiskfs file system.</para>
+ <para>This limit is 255 bytes for a single filename, the same as the limit in the underlying file systems.</para>
</entry>
</row>
<row>
<para> Maximum number of open files for a Lustre file system</para>
</entry>
<entry>
- <para> None</para>
+ <para> No limit</para>
</entry>
<entry>
<para>The Lustre software does not impose a maximum for the number of open files,
</table>
<para> </para>
<note>
- <para condition="l22">In Lustre releases prior to 2.2, the maximum stripe count for a single
- file was limited to 160 OSTs. In Lustre release 2.2, the large <literal>xattr</literal>
- feature ("wide striping") was added to support up to 2000 OSTs. This feature is disabled
- by default at <literal>mkfs.lustre</literal> time. In order to enable this feature, set
- the "<literal>-O large_xattr</literal>" option on the MDT either by using
- <literal>--mkfsoptions</literal> at format time or by using
- <literal>tune2fs</literal>.</para>
+ <para condition="l22">In Lustre software releases prior to release 2.2, the maximum stripe
+ count for a single file was limited to 160 OSTs. In Lustre software release 2.2, the large
+ <literal>xattr</literal> feature ("wide striping") was added to support up to 2000 OSTs.
+ This feature is disabled by default at <literal>mkfs.lustre</literal> time. In order to
+ enable this feature, set the "<literal>-O large_xattr</literal>" or "<literal>-O ea_inode</literal>"
+ option on the MDT either by using <literal>--mkfsoptions</literal> at format time or by using
+ <literal>tune2fs</literal>. Using either "<literal>large_xattr</literal>" or "<literal>ea_inode</literal>"
+ results in "<literal>ea_inode</literal>" in the file system feature list.</para>
</note>
</section>
</section>
<para><emphasis role="bold">Service threads</emphasis> : The service threads on the OSS node pre-allocate a 4 MB I/O buffer for each ost_io service thread, so these buffers do not need to be allocated and freed for each I/O request.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">OSS read cache</emphasis> : OSS read cache provides read-only caching of data on an OSS, using the regular Linux page cache to store the data. Just like caching from a regular file system in Linux, OSS read cache uses as much physical memory as is available.</para>
+ <para><emphasis role="bold">OSS read cache</emphasis> : OSS read cache provides read-only
+ caching of data on an OSS, using the regular Linux page cache to store the data. Just
+ like caching from a regular file system in the Linux operating system, OSS read cache
+ uses as much physical memory as is available.</para>
</listitem>
</itemizedlist>
<para>The same calculation applies to files accessed from the OSS as for the MDS, but the load is distributed over many more OSSs nodes, so the amount of memory required for locks, inode cache, etc. listed under MDS is spread out over the OSS nodes.</para>
git://git.whamcloud.com / doc / manual.git / blobdiff