<para>Lustre is currently not particularly well suited for "peer-to-peer" usage models where there are clients and servers running on the same node, each sharing a small amount of storage, due to the lack of Lustre-level data replication. In such uses, if one client/server fails, then the data stored on that node will not be accessible until the node is restarted.</para>
<section remap="h3">
<title><indexterm><primary>Lustre</primary><secondary>features</secondary></indexterm>Lustre Features</title>
- <para>Lustre runs on a variety of vendor's kernels. For more details, see <link xl:href="http://wiki.whamcloud.com/">Lustre Release Information</link> on the Whamcloud wiki.</para>
- <para>A Lustre installation can be scaled up or down with respect to the number of client nodes, disk storage and bandwidth. Scalability and performance are dependent on available disk and network bandwith and the processing power of the servers in the system. Lustre can be deployed in a wide variety of configurations that can be scaled well beyond the size and performance observed in production systems to date.</para>
+ <para>Lustre runs on a variety of vendor's kernels. For more details, see <link xl:href="http://wiki.whamcloud.com/display/PUB/Lustre+Support+Matrix">Lustre Support Matrix</link> on the Intel Lustre wiki.</para>
+ <para>A Lustre installation can be scaled up or down with respect to the number of client nodes, disk storage and bandwidth. Scalability and performance are dependent on available disk and network bandwidth and the processing power of the servers in the system. Lustre can be deployed in a wide variety of configurations that can be scaled well beyond the size and performance observed in production systems to date.</para>
<para><xref linkend="understandinglustre.tab1"/> shows the practical range of scalability and performance characteristics of the Lustre file system and some test results in production systems.</para>
<table frame="all">
<title xml:id="understandinglustre.tab1">Lustre Scalability and Performance</title>
<para> <emphasis role="bold">Client Scalability</emphasis></para>
</entry>
<entry>
- <para> 100-100,000</para>
+ <para> 100-100000</para>
</entry>
<entry>
- <para> 50,000+ clients</para>
- <para>Many installations in 10,000 to 20,000 range</para>
+ <para> 50000+ clients, many in the 10000 to 20000 range</para>
</entry>
</row>
<row>
<entry>
- <para>Client Performance</para>
+ <para><emphasis role="bold">Client Performance</emphasis></para>
</entry>
<entry>
<para> <emphasis>Single client: </emphasis></para>
</entry>
<entry>
<para> <emphasis>Single OSS:</emphasis></para>
- <para>1-8 OSTs per OSS, 128TB per OST</para>
+ <para>1-32 OSTs per OSS,</para>
+ <para>128TB per OST</para>
<para> <emphasis>OSS count:</emphasis></para>
- <para>500 OSSes, with up to 4000 OSTs, </para>
+ <para>500 OSSs, with up to 4000 OSTs</para>
+ <para/>
</entry>
<entry>
<para> <emphasis>Single OSS:</emphasis></para>
- <para>8 OSTs per OSS, 16TB per OST</para>
+ <para>8 OSTs per OSS,</para>
+ <para>16TB per OST</para>
<para> <emphasis>OSS count:</emphasis></para>
<para>450 OSSs with 1000 4TB OSTs</para>
<para>192 OSSs with 1344 8TB OSTs</para>
<para> <emphasis>Single MDS:</emphasis></para>
<para> 4 billion files</para>
<para> <emphasis>MDS count:</emphasis></para>
- <para> 1 + 1 (primary with one backup)</para>
+ <para> 1 primary + 1 backup</para>
+ <para condition='l24'>Since Lustre 2.4: up to 4096 MDSs and up to 4096 MDTs.</para>
</entry>
<entry>
<para> <emphasis>Single MDS:</emphasis></para>
<para> 750 million files</para>
<para> <emphasis>MDS count:</emphasis></para>
- <para> 1 + 1 (primary with one backup)</para>
+ <para> 1 primary + 1 backup</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">MDS Performance</emphasis></para>
</entry>
<entry>
- <para> 35000/s create, 100000/s stat metadata operations</para>
+ <para> 35000/s create operations,</para>
+ <para> 100000/s metadata stat operations</para>
</entry>
<entry>
- <para> 15000/s create, 35000/s stat metadata operations</para>
+ <para> 15000/s create operations,</para>
+ <para> 35000/s metadata stat operations</para>
</entry>
</row>
<row>
</entry>
<entry>
<para> <emphasis>Single File:</emphasis></para>
- <para>320 TB max file size</para>
+ <para>2.5 PB max file size</para>
<para> <emphasis>Aggregate:</emphasis></para>
- <para>64 PB space, 4 billion files</para>
+ <para>512 PB space, 4 billion files</para>
</entry>
<entry>
<para> <emphasis>Single File:</emphasis></para>
<para>Other Lustre features are:</para>
<itemizedlist>
<listitem>
- <para><emphasis role="bold">Performance-enhanced ext4 file system:</emphasis> Lustre uses an improved version of the ext4 journaling file system to store data and metadata. This version, called <emphasis role="italic">
- <literal>ldiskfs</literal>
- </emphasis>, has been enhanced to improve performance and provide additional functionality needed by Lustre.</para>
+ <para><emphasis role="bold">Performance-enhanced ext4 file system:</emphasis> Lustre uses an improved version of the ext4 journaling file system to store data and metadata. This version, called <emphasis role="italic"><literal>ldiskfs</literal></emphasis>, has been enhanced to improve performance and provide additional functionality needed by Lustre.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">POSIX compliance</emphasis> : The full POSIX test suite passes in an identical manner to a local ext4 filesystem, with limited exceptions on Lustre clients. In a cluster, most operations are atomic so that clients never see stale data or metadata. Lustre supports mmap() file I/O.</para>
+ <para><emphasis role="bold">POSIX compliance:</emphasis> The full POSIX test suite passes in an identical manner to a local ext4 filesystem, with limited exceptions on Lustre clients. In a cluster, most operations are atomic so that clients never see stale data or metadata. Lustre supports mmap() file I/O.</para>
</listitem>
<listitem>
<para><emphasis role="bold">High-performance heterogeneous networking:</emphasis> Lustre supports a variety of high performance, low latency networks and permits Remote Direct Memory Access (RDMA) for Infiniband (OFED) and other advanced networks for fast and efficient network transport. Multiple RDMA networks can be bridged using Lustre routing for maximum performance. Lustre also provides integrated network diagnostics.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">High-availability:</emphasis> Lustre offers active/active failover using shared storage partitions for OSS targets (OSTs) and active/passive failover using a shared storage partition for the MDS target (MDT). This allows application transparent recovery. Lustre can work with a variety of high availability (HA) managers to allow automated failover and has no single point of failure (NSPF). Multiple mount protection (MMP) provides integrated protection from errors in highly-available systems that would otherwise cause file system corruption.</para>
+ <para><emphasis role="bold">High-availability:</emphasis> Lustre offers active/active failover using shared storage partitions for OSS targets (OSTs). Lustre 2.3 and earlier offers active/passive failover using a shared storage partition for the MDS target (MDT).</para><para condition='l24'>With Lustre 2.4 or later servers and clients it is possible to configure active/active failover of multiple MDTs</para><para>This allows application transparent recovery. Lustre can work with a variety of high availability (HA) managers to allow automated failover and has no single point of failure (NSPF). Multiple mount protection (MMP) provides integrated protection from errors in highly-available systems that would otherwise cause file system corruption.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Security:</emphasis> By default TCP connections are only allowed from privileged ports. Unix group membership is verified is done on the MDS.</para>
+ <para><emphasis role="bold">Security:</emphasis> By default TCP connections are only allowed from privileged ports. Unix group membership is verified on the MDS.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Access control list (ACL), exended attributes:</emphasis> the Lustre security model follows that of a UNIX file system, enhanced with POSIX ACLs. Noteworthy additional features include root squash.</para>
+ <para><emphasis role="bold">Access control list (ACL), extended attributes:</emphasis> the Lustre security model follows that of a UNIX file system, enhanced with POSIX ACLs. Noteworthy additional features include root squash.</para>
</listitem>
<listitem>
<para><emphasis role="bold">Interoperability:</emphasis> Lustre runs on a variety of CPU architectures and mixed-endian clusters and is interoperable between successive major Lustre software releases.</para>
<para><emphasis role="bold">Object-based architecture:</emphasis> Clients are isolated from the on-disk file structure enabling upgrading of the storage architecture without affecting the client.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Byte-granular file and fine-grained metadata locking:</emphasis> Many clients can read and modify the same file or directory concurrently. The Lustre distributed lock manager (LDLM) ensures that files are coherent between all clients and servers in the filesystem. The MDT DLM manages locks on inode permissions and pathnames. Each OST has its own DLM for locks on file stripes stored theron, which scales the locking performance as the filesystem grows.</para>
+ <para><emphasis role="bold">Byte-granular file and fine-grained metadata locking:</emphasis> Many clients can read and modify the same file or directory concurrently. The Lustre distributed lock manager (LDLM) ensures that files are coherent between all clients and servers in the filesystem. The MDT LDLM manages locks on inode permissions and pathnames. Each OST has its own LDLM for locks on file stripes stored thereon, which scales the locking performance as the filesystem grows.</para>
</listitem>
<listitem>
<para><emphasis role="bold">Quotas:</emphasis> User and group quotas are available for Lustre.</para>
<para><emphasis role="bold">Metadata Server (MDS)</emphasis> - The MDS makes metadata stored in one or more MDTs available to Lustre clients. Each MDS manages the names and directories in the Lustre file system(s) and provides network request handling for one or more local MDTs.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Metadata Target (MDT</emphasis> ) - The MDT stores metadata (such as filenames, directories, permissions and file layout) on storage attached to an MDS. Each file system has one MDT. An MDT on a shared storage target can be available to multiple MDSs, although only one can access it at a time. If an active MDS fails, a standby MDS can serve the MDT and make it available to clients. This is referred to as MDS failover.</para>
+ <para><emphasis role="bold">Metadata Target (MDT</emphasis> ) - For Lustre 2.3 and earlier, each filesystem has one MDT. The MDT stores metadata (such as filenames, directories, permissions and file layout) on storage attached to an MDS. Each file system has one MDT. An MDT on a shared storage target can be available to multiple MDSs, although only one can access it at a time. If an active MDS fails, a standby MDS can serve the MDT and make it available to clients. This is referred to as MDS failover.</para>
+ <para condition='l24'>Since Lustre 2.4, multiple MDTs are supported. Each filesystem has at least one MDT. An MDT on shared storage target can be available via multiple MDSs, although only one MDS can export the MDT to the clients at one time. Two MDS machines share storage for two or more MDTs. After the failure of one MDS, the remaining MDS begins serving the MDT(s) of the failed MDS.</para>
</listitem>
<listitem>
<para><emphasis role="bold">Object Storage Servers (OSS)</emphasis> : The OSS provides file I/O service, and network request handling for one or more local OSTs. Typically, an OSS serves between 2 and 8 OSTs, up to 16 TB each. A typical configuration is an MDT on a dedicated node, two or more OSTs on each OSS node, and a client on each of a large number of compute nodes.</para>
</itemizedlist>
<para>The Lustre client software provides an interface between the Linux virtual file system and the Lustre servers. The client software includes a Management Client (MGC), a Metadata Client (MDC), and multiple Object Storage Clients (OSCs), one corresponding to each OST in the file system.</para>
<para>A logical object volume (LOV) aggregates the OSCs to provide transparent access across all the OSTs. Thus, a client with the Lustre file system mounted sees a single, coherent, synchronized namespace. Several clients can write to different parts of the same file simultaneously, while, at the same time, other clients can read from the file.</para>
- <para><xref linkend="understandinglustre.tab.storagerequire"/> provides the requirements for attached storage for each Lustre file system component and describes desirable characterics of the hardware used.</para>
+ <para><xref linkend="understandinglustre.tab.storagerequire"/> provides the requirements for attached storage for each Lustre file system component and describes desirable characteristics of the hardware used.</para>
<table frame="all">
<title xml:id="understandinglustre.tab.storagerequire"><indexterm><primary>Lustre</primary><secondary>requirements</secondary></indexterm>Storage and hardware requirements for Lustre components</title>
<tgroup cols="3">
<para> None</para>
</entry>
<entry>
- <para> Low latency, high bandwith network.</para>
+ <para> Low latency, high bandwidth network.</para>
</entry>
</row>
</tbody>
</section>
<section remap="h3">
<title><indexterm><primary>Lustre</primary><secondary>LNET</secondary></indexterm>Lustre Networking (LNET)</title>
- <para>Lustre Networking (LNET) is a custom networking API that provides the communication infrastructure that handles metadata and file I/O data for the Lustre file system servers and clients. For more information about LNET, see something <xref linkend="understandinglustrenetworking"/>.</para>
+ <para>Lustre Networking (LNET) is a custom networking API that provides the communication infrastructure that handles metadata and file I/O data for the Lustre file system servers and clients. For more information about LNET, see <xref linkend="understandinglustrenetworking"/>.</para>
</section>
<section remap="h3">
<title><indexterm><primary>Lustre</primary><secondary>cluster</secondary></indexterm>Lustre Cluster</title>
<para>The <emphasis>disk bandwidth</emphasis> equals the sum of the disk bandwidths of the storage targets (OSTs) up to the limit of the network bandwidth.</para>
</listitem>
<listitem>
- <para>The <emphasis>aggregate bandwidth</emphasis> equals the minimium of the disk bandwidth and the network bandwidth.</para>
+ <para>The <emphasis>aggregate bandwidth</emphasis> equals the minimum of the disk bandwidth and the network bandwidth.</para>
</listitem>
<listitem>
<para>The <emphasis>available file system space</emphasis> equals the sum of the available space of all the OSTs.</para>
<indexterm><primary>striping</primary><secondary>overview</secondary></indexterm>
Lustre File System and Striping</title>
<para>One of the main factors leading to the high performance of Lustre file systems is the ability to stripe data across multiple OSTs in a round-robin fashion. Users can optionally configure for each file the number of stripes, stripe size, and OSTs that are used.</para>
- <para>Striping can be used to improve performance when the aggregate bandwidth to a single file exeeds the bandwidth of a single OST. The ability to stripe is also useful when a single OST does not have anough free space to hold an entire file. For more information about benefits and drawbacks of file striping, see <xref linkend="dbdoclet.50438209_48033"/>.</para>
+ <para>Striping can be used to improve performance when the aggregate bandwidth to a single file exceeds the bandwidth of a single OST. The ability to stripe is also useful when a single OST does not have enough free space to hold an entire file. For more information about benefits and drawbacks of file striping, see <xref linkend="dbdoclet.50438209_48033"/>.</para>
<para>Striping allows segments or 'chunks' of data in a file to be stored on different OSTs, as shown in <xref linkend="understandinglustre.fig.filestripe"/>. In the Lustre file system, a RAID 0 pattern is used in which data is "striped" across a certain number of objects. The number of objects in a single file is called the <literal>stripe_count</literal>.</para>
<para>Each object contains a chunk of data from the file. When the chunk of data being written to a particular object exceeds the <literal>stripe_size</literal>, the next chunk of data in the file is stored on the next object.</para>
<para>Default values for <literal>stripe_count</literal> and <literal>stripe_size</literal> are set for the file system. The default value for <literal>stripe_count</literal> is 1 stripe for file and the default value for <literal>stripe_size</literal> is 1MB. The user may change these values on a per directory or per file basis. For more details, see <xref linkend="dbdoclet.50438209_78664"/>.</para>
</textobject>
</mediaobject>
</figure>
- <para>The maximum file size is not limited by the size of a single target. Lustre can stripe files across multiple objects (up to 160), and each object can be up to 2 TB in size. This leads to a maximum file size of 320 TB. (Note that Lustre itself can support files up to 2^64 bytes depending on the backing storage used by OSTs.)</para>
- <para>Athough a single file can only be striped over 160 objects, Lustre file systems can have thousands of OSTs. The I/O bandwidth to access a single file is the aggregated I/O bandwidth to the objects in a file, which can be as much as a bandwidth of up to 160 servers. On systems with more than 160 OSTs, clients can do I/O using multiple files to utilize the full file system bandwidth.</para>
+ <para>The maximum file size is not limited by the size of a single target. Lustre can stripe files across multiple objects (up to 2000), and each object can be up to 16 TB in size with ldiskfs. This leads to a maximum file size of 31.25 PB. (Note that Lustre itself can support files up to 2^64 bytes depending on the backing storage used by OSTs.)</para>
+ <note>
+ <para>Versions of Lustre prior to 2.2 had a maximum stripe count for a single file was limited to 160 OSTs.</para>
+ </note>
+ <para>Although a single file can only be striped over 2000 objects, Lustre file systems can have thousands of OSTs. The I/O bandwidth to access a single file is the aggregated I/O bandwidth to the objects in a file, which can be as much as a bandwidth of up to 2000 servers. On systems with more than 2000 OSTs, clients can do I/O using multiple files to utilize the full file system bandwidth.</para>
<para>For more information about striping, see <xref linkend="managingstripingfreespace"/>.</para>
</section>
</section>
git://git.whamcloud.com / doc / manual.git / blobdiff