<para>The Lustre storage architecture is used for many different kinds of
clusters. It is best known for powering many of the largest
high-performance computing (HPC) clusters worldwide, with tens of thousands
- of client systems, petabytes (PB) of storage and hundreds of gigabytes per
+ of client systems, petabytes (PiB) of storage and hundreds of gigabytes per
second (GB/sec) of I/O throughput. Many HPC sites use a Lustre file system
as a site-wide global file system, serving dozens of clusters.</para>
<para>The ability of a Lustre file system to scale capacity and performance
beyond the size and performance observed in production systems to
date.</para>
<para>
- <xref linkend="understandinglustre.tab1" />shows the practical range of
- scalability and performance characteristics of a Lustre file system and
- some test results in production systems.</para>
- <table frame="all">
- <title xml:id="understandinglustre.tab1">Lustre File System Scalability
- and Performance</title>
+ <xref linkend="understandinglustre.tab1" /> shows some of the
+ scalability and performance characteristics of a Lustre file system.
+ For a full list of Lustre file and filesystem limits see
+ <xref linkend="settinguplustresystem.tab2"/>.</para>
+ <table frame="all" xml:id="understandinglustre.tab1">
+ <title>Lustre File System Scalability and Performance</title>
<tgroup cols="3">
<colspec colname="c1" colwidth="1*" />
<colspec colname="c2" colwidth="2*" />
</entry>
<entry>
<para>
- <emphasis role="bold">Tested in Production</emphasis>
+ <emphasis role="bold">Known Production Usage</emphasis>
</para>
</entry>
</row>
<para>
<emphasis>Aggregate:</emphasis>
</para>
- <para>2.5 TB/sec I/O</para>
+ <para>10 TB/sec I/O</para>
</entry>
<entry>
<para>
<emphasis>Single client:</emphasis>
</para>
- <para>2 GB/sec I/O, 1000 metadata ops/sec</para>
+ <para>4.5 GB/sec I/O (FDR IB, OPA1),
+ 1000 metadata ops/sec</para>
<para>
<emphasis>Aggregate:</emphasis>
</para>
- <para>240 GB/sec I/O</para>
+ <para>2.5 TB/sec I/O </para>
</entry>
</row>
<row>
<para>
<emphasis>Single OSS:</emphasis>
</para>
- <para>1-32 OSTs per OSS,</para>
- <para>128TB per OST</para>
+ <para>1-32 OSTs per OSS</para>
+ <para>
+ <emphasis>Single OST:</emphasis>
+ </para>
+ <para>300M objects, 256TiB per OST (ldiskfs)</para>
+ <para>500M objects, 256TiB per OST (ZFS)</para>
<para>
<emphasis>OSS count:</emphasis>
</para>
- <para>500 OSSs, with up to 4000 OSTs</para>
+ <para>1000 OSSs, with up to 4000 OSTs</para>
</entry>
<entry>
<para>
<emphasis>Single OSS:</emphasis>
</para>
- <para>8 OSTs per OSS,</para>
- <para>16TB per OST</para>
+ <para>32x 8TiB OSTs per OSS (ldiskfs),</para>
+ <para>8x 32TiB OSTs per OSS (ldiskfs)</para>
+ <para>1x 72TiB OST per OSS (ZFS)</para>
<para>
<emphasis>OSS count:</emphasis>
</para>
- <para>450 OSSs with 1000 4TB OSTs</para>
- <para>192 OSSs with 1344 8TB OSTs</para>
+ <para>450 OSSs with 1000 4TiB OSTs</para>
+ <para>192 OSSs with 1344 8TiB OSTs</para>
+ <para>768 OSSs with 768 72TiB OSTs</para>
</entry>
</row>
<row>
<para>
<emphasis>Single OSS:</emphasis>
</para>
- <para>5 GB/sec</para>
+ <para>15 GB/sec</para>
<para>
<emphasis>Aggregate:</emphasis>
</para>
- <para>2.5 TB/sec</para>
+ <para>10 TB/sec</para>
</entry>
<entry>
<para>
<emphasis>Single OSS:</emphasis>
</para>
- <para>2.0+ GB/sec</para>
+ <para>10 GB/sec</para>
<para>
<emphasis>Aggregate:</emphasis>
</para>
- <para>240 GB/sec</para>
+ <para>2.5 TB/sec</para>
</entry>
</row>
<row>
</entry>
<entry>
<para>
+ <emphasis>Single MDS:</emphasis>
+ </para>
+ <para>1-4 MDTs per MDS</para>
+ <para>
<emphasis>Single MDT:</emphasis>
</para>
- <para>4 billion files (ldiskfs), 256 trillion files
- (ZFS)</para>
+ <para>4 billion files, 8TiB per MDT (ldiskfs)</para>
+ <para>64 billion files, 64TiB per MDT (ZFS)</para>
<para>
<emphasis>MDS count:</emphasis>
</para>
- <para>1 primary + 1 backup</para>
- <para condition="l24">Up to 4096 MDTs and up to 4096
- MDSs</para>
+ <para>1 primary + 1 standby</para>
+ <para condition="l24">256 MDSs, with up to 256 MDTs</para>
</entry>
<entry>
<para>
- <emphasis>Single MDT:</emphasis>
+ <emphasis>Single MDS:</emphasis>
</para>
- <para>1 billion files</para>
+ <para>3 billion files</para>
<para>
<emphasis>MDS count:</emphasis>
</para>
- <para>1 primary + 1 backup</para>
+ <para>7 MDS with 7 2TiB MDTs in production</para>
+ <para>256 MDS with 256 64GiB MDTs in testing</para>
</entry>
</row>
<row>
</para>
</entry>
<entry>
- <para>35000/s create operations,</para>
- <para>100000/s metadata stat operations</para>
+ <para>50000/s create operations,</para>
+ <para>200000/s metadata stat operations</para>
</entry>
<entry>
<para>15000/s create operations,</para>
- <para>35000/s metadata stat operations</para>
+ <para>50000/s metadata stat operations</para>
</entry>
</row>
<row>
<para>
<emphasis>Single File:</emphasis>
</para>
- <para>2.5 PB max file size</para>
+ <para>32 PiB max file size (ldiskfs)</para>
+ <para>2^63 bytes (ZFS)</para>
<para>
<emphasis>Aggregate:</emphasis>
</para>
- <para>512 PB space, 4 billion files</para>
+ <para>512 PiB space, 1 trillion files</para>
</entry>
<entry>
<para>
<emphasis>Single File:</emphasis>
</para>
- <para>multi-TB max file size</para>
+ <para>multi-TiB max file size</para>
<para>
<emphasis>Aggregate:</emphasis>
</para>
- <para>55 PB space, 1 billion files</para>
+ <para>55 PiB space, 8 billion files</para>
</entry>
</row>
</tbody>
performance, low latency networks and permits Remote Direct Memory
Access (RDMA) for InfiniBand
<superscript>*</superscript>(utilizing OpenFabrics Enterprise
- Distribution (OFED
- <superscript>*</superscript>) and other advanced networks for fast
+ Distribution (OFED<superscript>*</superscript>), Intel OmniPath®,
+ and other advanced networks for fast
and efficient network transport. Multiple RDMA networks can be
bridged using Lustre routing for maximum performance. The Lustre
software also includes integrated network diagnostics.</para>
<para>
<emphasis role="bold">Capacity growth:</emphasis>The size of a Lustre
file system and aggregate cluster bandwidth can be increased without
- interruption by adding a new OSS with OSTs to the cluster.</para>
+ interruption by adding new OSTs and MDTs to the cluster.</para>
</listitem>
<listitem>
<para>
- <emphasis role="bold">Controlled striping:</emphasis>The layout of
+ <emphasis role="bold">Controlled file layout:</emphasis>The layout of
files across OSTs can be configured on a per file, per directory, or
per file system basis. This allows file I/O to be tuned to specific
application requirements within a single file system. The Lustre file
<listitem>
<para>
<emphasis role="bold">NFS and CIFS export:</emphasis>Lustre files can
- be re-exported using NFS (via Linux knfsd) or CIFS (via Samba)
- enabling them to be shared with non-Linux clients, such as Microsoft
- <superscript>*</superscript>Windows
- <superscript>*</superscript>and Apple
+ be re-exported using NFS (via Linux knfsd or Ganesha) or CIFS (via
+ Samba), enabling them to be shared with non-Linux clients such as
+ Microsoft<superscript>*</superscript>Windows,
+ <superscript>*</superscript>Apple
<superscript>*</superscript>Mac OS X
- <superscript>*</superscript>.</para>
+ <superscript>*</superscript>, and others.</para>
</listitem>
<listitem>
<para>
</indexterm>Lustre Components</title>
<para>An installation of the Lustre software includes a management server
(MGS) and one or more Lustre file systems interconnected with Lustre
- networking (LNET).</para>
+ networking (LNet).</para>
<para>A basic configuration of Lustre file system components is shown in
<xref linkend="understandinglustre.fig.cluster" />.</para>
- <figure>
- <title xml:id="understandinglustre.fig.cluster">Lustre file system
- components in a basic cluster</title>
+ <figure xml:id="understandinglustre.fig.cluster">
+ <title>Lustre file system components in a basic cluster</title>
<mediaobject>
<imageobject>
<imagedata scalefit="1" width="100%"
<itemizedlist>
<listitem>
<para>
- <emphasis role="bold">Metadata Server (MDS)</emphasis>- The MDS makes
+ <emphasis role="bold">Metadata Servers (MDS)</emphasis>- The MDS makes
metadata stored in one or more MDTs available to Lustre clients. Each
MDS manages the names and directories in the Lustre file system(s)
and provides network request handling for one or more local
</listitem>
<listitem>
<para>
- <emphasis role="bold">Metadata Target (MDT</emphasis>) - For Lustre
+ <emphasis role="bold">Metadata Targets (MDT</emphasis>) - For Lustre
software release 2.3 and earlier, each file system has one MDT. The
MDT stores metadata (such as filenames, directories, permissions and
file layout) on storage attached to an MDS. Each file system has one
fails, a standby MDS can serve the MDT and make it available to
clients. This is referred to as MDS failover.</para>
<para condition="l24">Since Lustre software release 2.4, multiple
- MDTs are supported. Each file system has at least one MDT. An MDT on
- a shared storage target can be available via multiple MDSs, although
- only one MDS can export the MDT to the clients at one time. Two MDS
- machines share storage for two or more MDTs. After the failure of one
- MDS, the remaining MDS begins serving the MDT(s) of the failed
- MDS.</para>
+ MDTs are supported in the Distributed Namespace Environment (DNE).
+ In addition to the primary MDT that holds the filesystem root, it
+ is possible to add additional MDS nodes, each with their own MDTs,
+ to hold sub-directory trees of the filesystem.</para>
+ <para condition="l28">Since Lustre software release 2.8, DNE also
+ allows the filesystem to distribute files of a single directory over
+ multiple MDT nodes. A directory which is distributed across multiple
+ MDTs is known as a <emphasis>striped directory</emphasis>.</para>
</listitem>
<listitem>
<para>
<emphasis role="bold">Object Storage Servers (OSS)</emphasis>: The
OSS provides file I/O service and network request handling for one or
more local OSTs. Typically, an OSS serves between two and eight OSTs,
- up to 16 TB each. A typical configuration is an MDT on a dedicated
+ up to 16 TiB each. A typical configuration is an MDT on a dedicated
node, two or more OSTs on each OSS node, and a client on each of a
large number of compute nodes.</para>
</listitem>
Several clients can write to different parts of the same file
simultaneously, while, at the same time, other clients can read from the
file.</para>
+ <para>A logical metadata volume (LMV) aggregates the MDCs to provide
+ transparent access across all the MDTs in a similar manner as the LOV
+ does for file access. This allows the client to see the directory tree
+ on multiple MDTs as a single coherent namespace, and striped directories
+ are merged on the clients to form a single visible directory to users
+ and applications.
+ </para>
<para>
<xref linkend="understandinglustre.tab.storagerequire" />provides the
requirements for attached storage for each Lustre file system component
and describes desirable characteristics of the hardware used.</para>
- <table frame="all">
- <title xml:id="understandinglustre.tab.storagerequire">
+ <table frame="all" xml:id="understandinglustre.tab.storagerequire">
+ <title>
<indexterm>
<primary>Lustre</primary>
<secondary>requirements</secondary>
</para>
</entry>
<entry>
- <para>1-16 TB per OST, 1-8 OSTs per OSS</para>
+ <para>1-128 TiB per OST, 1-8 OSTs per OSS</para>
</entry>
<entry>
<para>Good bus bandwidth. Recommended that storage be balanced
- evenly across OSSs.</para>
+ evenly across OSSs and matched to network bandwidth.</para>
</entry>
</row>
<row>
</para>
</entry>
<entry>
- <para>None</para>
+ <para>No local storage needed</para>
</entry>
<entry>
<para>Low latency, high bandwidth network.</para>
<title>
<indexterm>
<primary>Lustre</primary>
- <secondary>LNET</secondary>
- </indexterm>Lustre Networking (LNET)</title>
- <para>Lustre Networking (LNET) is a custom networking API that provides
+ <secondary>LNet</secondary>
+ </indexterm>Lustre Networking (LNet)</title>
+ <para>Lustre Networking (LNet) is a custom networking API that provides
the communication infrastructure that handles metadata and file I/O data
for the Lustre file system servers and clients. For more information
- about LNET, see
+ about LNet, see
<xref linkend="understandinglustrenetworking" />.</para>
</section>
<section remap="h3">
OSSs enables failover capability. For more details about OSS failover,
see
<xref linkend="understandingfailover" />.</para>
- <figure>
- <title xml:id="understandinglustre.fig.lustrescale">
+ <figure xml:id="understandinglustre.fig.lustrescale">
+ <title>
<indexterm>
<primary>Lustre</primary>
<secondary>at scale</secondary>
MDTs). This change enabled future support for multiple MDTs (introduced in
Lustre software release 2.4) and ZFS (introduced in Lustre software release
2.4).</para>
- <para>Also introduced in release 2.0 is a feature call
+ <para>Also introduced in release 2.0 is an ldiskfs feature named
<emphasis role="italic">FID-in-dirent</emphasis>(also known as
<emphasis role="italic">dirdata</emphasis>) in which the FID is stored as
part of the name of the file in the parent directory. This feature
<literal>ls</literal> command executions by reducing disk I/O. The
FID-in-dirent is generated at the time the file is created.</para>
<note>
- <para>The FID-in-dirent feature is not compatible with the Lustre
- software release 1.8 format. Therefore, when an upgrade from Lustre
- software release 1.8 to a Lustre software release 2.x is performed, the
- FID-in-dirent feature is not automatically enabled. For upgrades from
- Lustre software release 1.8 to Lustre software releases 2.0 through 2.3,
- FID-in-dirent can be enabled manually but only takes effect for new
- files.</para>
+ <para>The FID-in-dirent feature is not backward compatible with the
+ release 1.8 ldiskfs disk format. Therefore, when an upgrade from
+ release 1.8 to release 2.x is performed, the FID-in-dirent feature is
+ not automatically enabled. For upgrades from release 1.8 to releases
+ 2.0 through 2.3, FID-in-dirent can be enabled manually but only takes
+ effect for new files.</para>
<para>For more information about upgrading from Lustre software release
1.8 and enabling FID-in-dirent for existing files, see
<xref xmlns:xlink="http://www.w3.org/1999/xlink"
<listitem>
<para>Verifies the linkEA entry for each and regenerates the linkEA
if it is invalid or missing. The
- <emphasis role="italic">linkEA</emphasis>consists of the file name and
+ <emphasis role="italic">linkEA</emphasis> consists of the file name and
parent FID. It is stored as an extended attribute in the file
- itself. Thus, the linkEA can be used to reconstruct the full path name of
- a file.</para>
+ itself. Thus, the linkEA can be used to reconstruct the full path name
+ of a file.</para>
</listitem>
</itemizedlist></para>
<para>Information about where file data is located on the OST(s) is stored
the OST(s) that contain the file data. If the MDT file points to one
object, all the file data is stored in that object. If the MDT file points
to more than one object, the file data is
- <emphasis role="italic">striped</emphasis>across the objects using RAID 0,
+ <emphasis role="italic">striped</emphasis> across the objects using RAID 0,
and each object is stored on a different OST. (For more information about
how striping is implemented in a Lustre file system, see
<xref linkend="dbdoclet.50438250_89922" />.</para>
<itemizedlist>
<listitem>
<para>The
- <emphasis>network bandwidth</emphasis>equals the aggregated bandwidth
+ <emphasis>network bandwidth</emphasis> equals the aggregated bandwidth
of the OSSs to the targets.</para>
</listitem>
<listitem>
<para>The
- <emphasis>disk bandwidth</emphasis>equals the sum of the disk
+ <emphasis>disk bandwidth</emphasis> equals the sum of the disk
bandwidths of the storage targets (OSTs) up to the limit of the network
bandwidth.</para>
</listitem>
<listitem>
<para>The
- <emphasis>aggregate bandwidth</emphasis>equals the minimum of the disk
+ <emphasis>aggregate bandwidth</emphasis> equals the minimum of the disk
bandwidth and the network bandwidth.</para>
</listitem>
<listitem>
<para>The
- <emphasis>available file system space</emphasis>equals the sum of the
+ <emphasis>available file system space</emphasis> equals the sum of the
available space of all the OSTs.</para>
</listitem>
</itemizedlist>
<literal>stripe_count</literal> for File B and File C is 1.</para>
<para>No space is reserved on the OST for unwritten data. File A in
<xref linkend="understandinglustre.fig.filestripe" />.</para>
- <figure>
- <title xml:id="understandinglustre.fig.filestripe">File striping on a
+ <figure xml:id="understandinglustre.fig.filestripe">
+ <title>File striping on a
Lustre file system</title>
<mediaobject>
<imageobject>
</figure>
<para>The maximum file size is not limited by the size of a single
target. In a Lustre file system, files can be striped across multiple
- objects (up to 2000), and each object can be up to 16 TB in size with
- ldiskfs, or up to 256PB with ZFS. This leads to a maximum file size of
- 31.25 PB for ldiskfs or 8EB with ZFS. Note that a Lustre file system can
- support files up to 2^63 bytes (8EB), limited only by the space available
+ objects (up to 2000), and each object can be up to 16 TiB in size with
+ ldiskfs, or up to 256PiB with ZFS. This leads to a maximum file size of
+ 31.25 PiB for ldiskfs or 8EiB with ZFS. Note that a Lustre file system can
+ support files up to 2^63 bytes (8EiB), limited only by the space available
on the OSTs.</para>
<note>
<para>Versions of the Lustre software prior to Release 2.2 limited the
git://git.whamcloud.com / doc / manual.git / blobdiff