1 <?xml version='1.0' encoding='utf-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="understandinglustre">
5 <title xml:id="understandinglustre.title">Understanding Lustre
7 <para>This chapter describes the Lustre architecture and features of the
8 Lustre file system. It includes the following sections:</para>
12 <xref linkend="understandinglustre.whatislustre" />
17 <xref linkend="understandinglustre.components" />
22 <xref linkend="understandinglustre.storageio" />
26 <section xml:id="understandinglustre.whatislustre">
29 <primary>Lustre</primary>
30 </indexterm>What a Lustre File System Is (and What It Isn't)</title>
31 <para>The Lustre architecture is a storage architecture for clusters. The
32 central component of the Lustre architecture is the Lustre file system,
33 which is supported on the Linux operating system and provides a POSIX
34 <superscript>*</superscript>standard-compliant UNIX file system
36 <para>The Lustre storage architecture is used for many different kinds of
37 clusters. It is best known for powering many of the largest
38 high-performance computing (HPC) clusters worldwide, with tens of thousands
39 of client systems, petabytes (PiB) of storage and hundreds of gigabytes per
40 second (GB/sec) of I/O throughput. Many HPC sites use a Lustre file system
41 as a site-wide global file system, serving dozens of clusters.</para>
42 <para>The ability of a Lustre file system to scale capacity and performance
43 for any need reduces the need to deploy many separate file systems, such as
44 one for each compute cluster. Storage management is simplified by avoiding
45 the need to copy data between compute clusters. In addition to aggregating
46 storage capacity of many servers, the I/O throughput is also aggregated and
47 scales with additional servers. Moreover, throughput and/or capacity can be
48 easily increased by adding servers dynamically.</para>
49 <para>While a Lustre file system can function in many work environments, it
50 is not necessarily the best choice for all applications. It is best suited
51 for uses that exceed the capacity that a single server can provide, though
52 in some use cases, a Lustre file system can perform better with a single
53 server than other file systems due to its strong locking and data
55 <para>A Lustre file system is currently not particularly well suited for
56 "peer-to-peer" usage models where clients and servers are running on the
57 same node, each sharing a small amount of storage, due to the lack of data
58 replication at the Lustre software level. In such uses, if one
59 client/server fails, then the data stored on that node will not be
60 accessible until the node is restarted.</para>
64 <primary>Lustre</primary>
65 <secondary>features</secondary>
66 </indexterm>Lustre Features</title>
67 <para>Lustre file systems run on a variety of vendor's kernels. For more
68 details, see the Lustre Test Matrix
69 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
70 linkend="preparing_installation" />.</para>
71 <para>A Lustre installation can be scaled up or down with respect to the
72 number of client nodes, disk storage and bandwidth. Scalability and
73 performance are dependent on available disk and network bandwidth and the
74 processing power of the servers in the system. A Lustre file system can
75 be deployed in a wide variety of configurations that can be scaled well
76 beyond the size and performance observed in production systems to
79 <xref linkend="understandinglustre.tab1" /> shows some of the
80 scalability and performance characteristics of a Lustre file system.
81 For a full list of Lustre file and filesystem limits see
82 <xref linkend="settinguplustresystem.tab2"/>.</para>
83 <table frame="all" xml:id="understandinglustre.tab1">
84 <title>Lustre File System Scalability and Performance</title>
86 <colspec colname="c1" colwidth="1*" />
87 <colspec colname="c2" colwidth="2*" />
88 <colspec colname="c3" colwidth="3*" />
93 <emphasis role="bold">Feature</emphasis>
98 <emphasis role="bold">Current Practical Range</emphasis>
103 <emphasis role="bold">Known Production Usage</emphasis>
112 <emphasis role="bold">Client Scalability</emphasis>
116 <para>100-100000</para>
119 <para>50000+ clients, many in the 10000 to 20000 range</para>
125 <emphasis role="bold">Client Performance</emphasis>
130 <emphasis>Single client:</emphasis>
132 <para>I/O 90% of network bandwidth</para>
134 <emphasis>Aggregate:</emphasis>
136 <para>10 TB/sec I/O</para>
140 <emphasis>Single client:</emphasis>
142 <para>4.5 GB/sec I/O (FDR IB, OPA1),
143 1000 metadata ops/sec</para>
145 <emphasis>Aggregate:</emphasis>
147 <para>2.5 TB/sec I/O </para>
153 <emphasis role="bold">OSS Scalability</emphasis>
158 <emphasis>Single OSS:</emphasis>
160 <para>1-32 OSTs per OSS</para>
162 <emphasis>Single OST:</emphasis>
164 <para>300M objects, 256TiB per OST (ldiskfs)</para>
165 <para>500M objects, 256TiB per OST (ZFS)</para>
167 <emphasis>OSS count:</emphasis>
169 <para>1000 OSSs, with up to 4000 OSTs</para>
173 <emphasis>Single OSS:</emphasis>
175 <para>32x 8TiB OSTs per OSS (ldiskfs),</para>
176 <para>8x 32TiB OSTs per OSS (ldiskfs)</para>
177 <para>1x 72TiB OST per OSS (ZFS)</para>
179 <emphasis>OSS count:</emphasis>
181 <para>450 OSSs with 1000 4TiB OSTs</para>
182 <para>192 OSSs with 1344 8TiB OSTs</para>
183 <para>768 OSSs with 768 72TiB OSTs</para>
189 <emphasis role="bold">OSS Performance</emphasis>
194 <emphasis>Single OSS:</emphasis>
196 <para>15 GB/sec</para>
198 <emphasis>Aggregate:</emphasis>
200 <para>10 TB/sec</para>
204 <emphasis>Single OSS:</emphasis>
206 <para>10 GB/sec</para>
208 <emphasis>Aggregate:</emphasis>
210 <para>2.5 TB/sec</para>
216 <emphasis role="bold">MDS Scalability</emphasis>
221 <emphasis>Single MDS:</emphasis>
223 <para>1-4 MDTs per MDS</para>
225 <emphasis>Single MDT:</emphasis>
227 <para>4 billion files, 8TiB per MDT (ldiskfs)</para>
228 <para>64 billion files, 64TiB per MDT (ZFS)</para>
230 <emphasis>MDS count:</emphasis>
232 <para>256 MDSs, with up to 256 MDTs</para>
236 <emphasis>Single MDS:</emphasis>
238 <para>3 billion files</para>
240 <emphasis>MDS count:</emphasis>
242 <para>7 MDS with 7 2TiB MDTs in production</para>
243 <para>256 MDS with 256 64GiB MDTs in testing</para>
249 <emphasis role="bold">MDS Performance</emphasis>
253 <para>50000/s create operations,</para>
254 <para>200000/s metadata stat operations</para>
257 <para>15000/s create operations,</para>
258 <para>50000/s metadata stat operations</para>
264 <emphasis role="bold">File system Scalability</emphasis>
269 <emphasis>Single File:</emphasis>
271 <para>32 PiB max file size (ldiskfs)</para>
272 <para>2^63 bytes (ZFS)</para>
274 <emphasis>Aggregate:</emphasis>
276 <para>512 PiB space, 1 trillion files</para>
280 <emphasis>Single File:</emphasis>
282 <para>multi-TiB max file size</para>
284 <emphasis>Aggregate:</emphasis>
286 <para>55 PiB space, 8 billion files</para>
292 <para>Other Lustre software features are:</para>
296 <emphasis role="bold">Performance-enhanced ext4 file
297 system:</emphasis>The Lustre file system uses an improved version of
298 the ext4 journaling file system to store data and metadata. This
300 <emphasis role="italic">
301 <literal>ldiskfs</literal>
302 </emphasis>, has been enhanced to improve performance and provide
303 additional functionality needed by the Lustre file system.</para>
306 <para>It is also possible to use ZFS as the backing filesystem for
307 Lustre for the MDT, OST, and MGS storage. This allows Lustre to
308 leverage the scalability and data integrity features of ZFS for
309 individual storage targets.</para>
313 <emphasis role="bold">POSIX standard compliance:</emphasis>The full
314 POSIX test suite passes in an identical manner to a local ext4 file
315 system, with limited exceptions on Lustre clients. In a cluster, most
316 operations are atomic so that clients never see stale data or
317 metadata. The Lustre software supports mmap() file I/O.</para>
321 <emphasis role="bold">High-performance heterogeneous
322 networking:</emphasis>The Lustre software supports a variety of high
323 performance, low latency networks and permits Remote Direct Memory
324 Access (RDMA) for InfiniBand
325 <superscript>*</superscript>(utilizing OpenFabrics Enterprise
326 Distribution (OFED<superscript>*</superscript>), Intel OmniPath®,
327 and other advanced networks for fast
328 and efficient network transport. Multiple RDMA networks can be
329 bridged using Lustre routing for maximum performance. The Lustre
330 software also includes integrated network diagnostics.</para>
334 <emphasis role="bold">High-availability:</emphasis>The Lustre file
335 system supports active/active failover using shared storage
336 partitions for OSS targets (OSTs), and for MDS targets (MDTs).
337 The Lustre file system can work
338 with a variety of high availability (HA) managers to allow automated
339 failover and has no single point of failure (NSPF). This allows
340 application transparent recovery. Multiple mount protection (MMP)
341 provides integrated protection from errors in highly-available
342 systems that would otherwise cause file system corruption.</para>
346 <emphasis role="bold">Security:</emphasis>By default TCP connections
347 are only allowed from privileged ports. UNIX group membership is
348 verified on the MDS.</para>
352 <emphasis role="bold">Access control list (ACL), extended
353 attributes:</emphasis>the Lustre security model follows that of a
354 UNIX file system, enhanced with POSIX ACLs. Noteworthy additional
355 features include root squash.</para>
359 <emphasis role="bold">Interoperability:</emphasis>The Lustre file
360 system runs on a variety of CPU architectures and mixed-endian
361 clusters and is interoperable between successive major Lustre
362 software releases.</para>
366 <emphasis role="bold">Object-based architecture:</emphasis>Clients
367 are isolated from the on-disk file structure enabling upgrading of
368 the storage architecture without affecting the client.</para>
372 <emphasis role="bold">Byte-granular file and fine-grained metadata
373 locking:</emphasis>Many clients can read and modify the same file or
374 directory concurrently. The Lustre distributed lock manager (LDLM)
375 ensures that files are coherent between all clients and servers in
376 the file system. The MDT LDLM manages locks on inode permissions and
377 pathnames. Each OST has its own LDLM for locks on file stripes stored
378 thereon, which scales the locking performance as the file system
383 <emphasis role="bold">Quotas:</emphasis>User and group quotas are
384 available for a Lustre file system.</para>
388 <emphasis role="bold">Capacity growth:</emphasis>The size of a Lustre
389 file system and aggregate cluster bandwidth can be increased without
390 interruption by adding new OSTs and MDTs to the cluster.</para>
394 <emphasis role="bold">Controlled file layout:</emphasis>The layout of
395 files across OSTs can be configured on a per file, per directory, or
396 per file system basis. This allows file I/O to be tuned to specific
397 application requirements within a single file system. The Lustre file
398 system uses RAID-0 striping and balances space usage across
403 <emphasis role="bold">Network data integrity protection:</emphasis>A
404 checksum of all data sent from the client to the OSS protects against
405 corruption during data transfer.</para>
409 <emphasis role="bold">MPI I/O:</emphasis>The Lustre architecture has
410 a dedicated MPI ADIO layer that optimizes parallel I/O to match the
411 underlying file system architecture.</para>
415 <emphasis role="bold">NFS and CIFS export:</emphasis>Lustre files can
416 be re-exported using NFS (via Linux knfsd or Ganesha) or CIFS (via
417 Samba), enabling them to be shared with non-Linux clients such as
418 Microsoft<superscript>*</superscript>Windows,
419 <superscript>*</superscript>Apple
420 <superscript>*</superscript>Mac OS X
421 <superscript>*</superscript>, and others.</para>
425 <emphasis role="bold">Disaster recovery tool:</emphasis>The Lustre
426 file system provides an online distributed file system check (LFSCK)
427 that can restore consistency between storage components in case of a
428 major file system error. A Lustre file system can operate even in the
429 presence of file system inconsistencies, and LFSCK can run while the
430 filesystem is in use, so LFSCK is not required to complete before
431 returning the file system to production.</para>
435 <emphasis role="bold">Performance monitoring:</emphasis>The Lustre
436 file system offers a variety of mechanisms to examine performance and
441 <emphasis role="bold">Open source:</emphasis>The Lustre software is
442 licensed under the GPL 2.0 license for use with the Linux operating
448 <section xml:id="understandinglustre.components">
451 <primary>Lustre</primary>
452 <secondary>components</secondary>
453 </indexterm>Lustre Components</title>
454 <para>An installation of the Lustre software includes a management server
455 (MGS) and one or more Lustre file systems interconnected with Lustre
456 networking (LNet).</para>
457 <para>A basic configuration of Lustre file system components is shown in
458 <xref linkend="understandinglustre.fig.cluster" />.</para>
459 <figure xml:id="understandinglustre.fig.cluster">
460 <title>Lustre file system components in a basic cluster</title>
463 <imagedata scalefit="1" width="100%"
464 fileref="./figures/Basic_Cluster.png" />
467 <phrase>Lustre file system components in a basic cluster</phrase>
474 <primary>Lustre</primary>
475 <secondary>MGS</secondary>
476 </indexterm>Management Server (MGS)</title>
477 <para>The MGS stores configuration information for all the Lustre file
478 systems in a cluster and provides this information to other Lustre
479 components. Each Lustre target contacts the MGS to provide information,
480 and Lustre clients contact the MGS to retrieve information.</para>
481 <para>It is preferable that the MGS have its own storage space so that it
482 can be managed independently. However, the MGS can be co-located and
483 share storage space with an MDS as shown in
484 <xref linkend="understandinglustre.fig.cluster" />.</para>
487 <title>Lustre File System Components</title>
488 <para>Each Lustre file system consists of the following
493 <emphasis role="bold">Metadata Servers (MDS)</emphasis>- The MDS makes
494 metadata stored in one or more MDTs available to Lustre clients. Each
495 MDS manages the names and directories in the Lustre file system(s)
496 and provides network request handling for one or more local
501 <emphasis role="bold">Metadata Targets (MDT</emphasis>) - Each
502 filesystem has at least one MDT, which holds the root directory. The
503 MDT stores metadata (such as filenames, directories, permissions and
504 file layout) on storage attached to an MDS. Each file system has one
505 MDT. An MDT on a shared storage target can be available to multiple
506 MDSs, although only one can access it at a time. If an active MDS
507 fails, a second MDS node can serve the MDT and make it available to
508 clients. This is referred to as MDS failover.</para>
509 <para>Multiple MDTs are supported with the Distributed Namespace
510 Environment (<xref linkend="DNE"/>).
511 In addition to the primary MDT that holds the filesystem root, it
512 is possible to add additional MDS nodes, each with their own MDTs,
513 to hold sub-directory trees of the filesystem.</para>
514 <para condition="l28">Since Lustre software release 2.8, DNE also
515 allows the filesystem to distribute files of a single directory over
516 multiple MDT nodes. A directory which is distributed across multiple
517 MDTs is known as a <emphasis><xref linkend="stripeddirectory"/></emphasis>.</para>
521 <emphasis role="bold">Object Storage Servers (OSS)</emphasis>: The
522 OSS provides file I/O service and network request handling for one or
523 more local OSTs. Typically, an OSS serves between two and eight OSTs,
524 up to 16 TiB each. A typical configuration is an MDT on a dedicated
525 node, two or more OSTs on each OSS node, and a client on each of a
526 large number of compute nodes.</para>
530 <emphasis role="bold">Object Storage Target (OST)</emphasis>: User
531 file data is stored in one or more objects, each object on a separate
532 OST in a Lustre file system. The number of objects per file is
533 configurable by the user and can be tuned to optimize performance for
534 a given workload.</para>
538 <emphasis role="bold">Lustre clients</emphasis>: Lustre clients are
539 computational, visualization or desktop nodes that are running Lustre
540 client software, allowing them to mount the Lustre file
544 <para>The Lustre client software provides an interface between the Linux
545 virtual file system and the Lustre servers. The client software includes
546 a management client (MGC), a metadata client (MDC), and multiple object
547 storage clients (OSCs), one corresponding to each OST in the file
549 <para>A logical object volume (LOV) aggregates the OSCs to provide
550 transparent access across all the OSTs. Thus, a client with the Lustre
551 file system mounted sees a single, coherent, synchronized namespace.
552 Several clients can write to different parts of the same file
553 simultaneously, while, at the same time, other clients can read from the
555 <para>A logical metadata volume (LMV) aggregates the MDCs to provide
556 transparent access across all the MDTs in a similar manner as the LOV
557 does for file access. This allows the client to see the directory tree
558 on multiple MDTs as a single coherent namespace, and striped directories
559 are merged on the clients to form a single visible directory to users
563 <xref linkend="understandinglustre.tab.storagerequire" />provides the
564 requirements for attached storage for each Lustre file system component
565 and describes desirable characteristics of the hardware used.</para>
566 <table frame="all" xml:id="understandinglustre.tab.storagerequire">
569 <primary>Lustre</primary>
570 <secondary>requirements</secondary>
571 </indexterm>Storage and hardware requirements for Lustre file system
574 <colspec colname="c1" colwidth="1*" />
575 <colspec colname="c2" colwidth="3*" />
576 <colspec colname="c3" colwidth="3*" />
581 <emphasis role="bold" />
586 <emphasis role="bold">Required attached storage</emphasis>
591 <emphasis role="bold">Desirable hardware
592 characteristics</emphasis>
601 <emphasis role="bold">MDSs</emphasis>
605 <para>1-2% of file system capacity</para>
608 <para>Adequate CPU power, plenty of memory, fast disk
615 <emphasis role="bold">OSSs</emphasis>
619 <para>1-128 TiB per OST, 1-8 OSTs per OSS</para>
622 <para>Good bus bandwidth. Recommended that storage be balanced
623 evenly across OSSs and matched to network bandwidth.</para>
629 <emphasis role="bold">Clients</emphasis>
633 <para>No local storage needed</para>
636 <para>Low latency, high bandwidth network.</para>
642 <para>For additional hardware requirements and considerations, see
643 <xref linkend="settinguplustresystem" />.</para>
648 <primary>Lustre</primary>
649 <secondary>LNet</secondary>
650 </indexterm>Lustre Networking (LNet)</title>
651 <para>Lustre Networking (LNet) is a custom networking API that provides
652 the communication infrastructure that handles metadata and file I/O data
653 for the Lustre file system servers and clients. For more information
655 <xref linkend="understandinglustrenetworking" />.</para>
660 <primary>Lustre</primary>
661 <secondary>cluster</secondary>
662 </indexterm>Lustre Cluster</title>
663 <para>At scale, a Lustre file system cluster can include hundreds of OSSs
664 and thousands of clients (see
665 <xref linkend="understandinglustre.fig.lustrescale" />). More than one
666 type of network can be used in a Lustre cluster. Shared storage between
667 OSSs enables failover capability. For more details about OSS failover,
669 <xref linkend="understandingfailover" />.</para>
670 <figure xml:id="understandinglustre.fig.lustrescale">
673 <primary>Lustre</primary>
674 <secondary>at scale</secondary>
675 </indexterm>Lustre cluster at scale</title>
678 <imagedata scalefit="1" width="100%"
679 fileref="./figures/Scaled_Cluster.png" />
682 <phrase>Lustre file system cluster at scale</phrase>
688 <section xml:id="understandinglustre.storageio">
691 <primary>Lustre</primary>
692 <secondary>storage</secondary>
695 <primary>Lustre</primary>
696 <secondary>I/O</secondary>
697 </indexterm>Lustre File System Storage and I/O</title>
698 <para>Lustre File IDentifiers (FIDs) are used internally for identifying
699 files or objects, similar to inode numbers in local filesystems. A FID
700 is a 128-bit identifier, which contains a unique 64-bit sequence number
701 (SEQ), a 32-bit object ID (OID), and a 32-bit version number. The sequence
702 number is unique across all Lustre targets in a file system (OSTs and
703 MDTs). This allows multiple MDTs and OSTs to uniquely identify objects
704 without depending on identifiers in the underlying filesystem (e.g. inode
705 numbers) that are likely to be duplicated between targets. The FID SEQ
706 number also allows mapping a FID to a particular MDT or OST.</para>
707 <para>The LFSCK file system consistency checking tool provides
708 functionality that enables FID-in-dirent for existing files. It
709 includes the following functionality:
712 <para>Verifies the FID stored with each directory entry and regenerates
713 it from the inode if it is invalid or missing.</para>
716 <para>Verifies the linkEA entry for each inode and regenerates it if
717 invalid or missing. The <emphasis role="italic">linkEA</emphasis>
718 stores of the file name and parent FID. It is stored as an extended
719 attribute in each inode. Thus, the linkEA can be used to
720 reconstruct the full path name of a file from only the FID.</para>
722 </itemizedlist></para>
723 <para>Information about where file data is located on the OST(s) is stored
724 as an extended attribute called layout EA in an MDT object identified by
725 the FID for the file (see
726 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
727 linkend="Fig1.3_LayoutEAonMDT" />). If the file is a regular file (not a
728 directory or symbol link), the MDT object points to 1-to-N OST object(s) on
729 the OST(s) that contain the file data. If the MDT file points to one
730 object, all the file data is stored in that object. If the MDT file points
731 to more than one object, the file data is
732 <emphasis role="italic">striped</emphasis> across the objects using RAID 0,
733 and each object is stored on a different OST. (For more information about
734 how striping is implemented in a Lustre file system, see
735 <xref linkend="lustre_striping" />.</para>
736 <figure xml:id="Fig1.3_LayoutEAonMDT">
737 <title>Layout EA on MDT pointing to file data on OSTs</title>
740 <imagedata scalefit="1" width="80%"
741 fileref="./figures/Metadata_File.png" />
744 <phrase>Layout EA on MDT pointing to file data on OSTs</phrase>
748 <para>When a client wants to read from or write to a file, it first fetches
749 the layout EA from the MDT object for the file. The client then uses this
750 information to perform I/O on the file, directly interacting with the OSS
751 nodes where the objects are stored.
752 <?oxy_custom_start type="oxy_content_highlight" color="255,255,0"?>
753 This process is illustrated in
754 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
755 linkend="Fig1.4_ClientReqstgData" /><?oxy_custom_end?>
757 <figure xml:id="Fig1.4_ClientReqstgData">
758 <title>Lustre client requesting file data</title>
761 <imagedata scalefit="1" width="75%"
762 fileref="./figures/File_Write.png" />
765 <phrase>Lustre client requesting file data</phrase>
769 <para>The available bandwidth of a Lustre file system is determined as
774 <emphasis>network bandwidth</emphasis> equals the aggregated bandwidth
775 of the OSSs to the targets.</para>
779 <emphasis>disk bandwidth</emphasis> equals the sum of the disk
780 bandwidths of the storage targets (OSTs) up to the limit of the network
785 <emphasis>aggregate bandwidth</emphasis> equals the minimum of the disk
786 bandwidth and the network bandwidth.</para>
790 <emphasis>available file system space</emphasis> equals the sum of the
791 available space of all the OSTs.</para>
794 <section xml:id="lustre_striping">
797 <primary>Lustre</primary>
798 <secondary>striping</secondary>
801 <primary>striping</primary>
802 <secondary>overview</secondary>
803 </indexterm>Lustre File System and Striping</title>
804 <para>One of the main factors leading to the high performance of Lustre
805 file systems is the ability to stripe data across multiple OSTs in a
806 round-robin fashion. Users can optionally configure for each file the
807 number of stripes, stripe size, and OSTs that are used.</para>
808 <para>Striping can be used to improve performance when the aggregate
809 bandwidth to a single file exceeds the bandwidth of a single OST. The
810 ability to stripe is also useful when a single OST does not have enough
811 free space to hold an entire file. For more information about benefits
812 and drawbacks of file striping, see
813 <xref linkend="file_striping.considerations" />.</para>
814 <para>Striping allows segments or 'chunks' of data in a file to be stored
815 on different OSTs, as shown in
816 <xref linkend="understandinglustre.fig.filestripe" />. In the Lustre file
817 system, a RAID 0 pattern is used in which data is "striped" across a
818 certain number of objects. The number of objects in a single file is
820 <literal>stripe_count</literal>.</para>
821 <para>Each object contains a chunk of data from the file. When the chunk
822 of data being written to a particular object exceeds the
823 <literal>stripe_size</literal>, the next chunk of data in the file is
824 stored on the next object.</para>
825 <para>Default values for
826 <literal>stripe_count</literal> and
827 <literal>stripe_size</literal> are set for the file system. The default
829 <literal>stripe_count</literal> is 1 stripe for file and the default value
831 <literal>stripe_size</literal> is 1MB. The user may change these values on
832 a per directory or per file basis. For more details, see
833 <xref linkend="file_striping.lfs_setstripe" />.</para>
835 <xref linkend="understandinglustre.fig.filestripe" />, the
836 <literal>stripe_size</literal> for File C is larger than the
837 <literal>stripe_size</literal> for File A, allowing more data to be stored
838 in a single stripe for File C. The
839 <literal>stripe_count</literal> for File A is 3, resulting in data striped
840 across three objects, while the
841 <literal>stripe_count</literal> for File B and File C is 1.</para>
842 <para>No space is reserved on the OST for unwritten data. File A in
843 <xref linkend="understandinglustre.fig.filestripe" />.</para>
844 <figure xml:id="understandinglustre.fig.filestripe">
845 <title>File striping on a
846 Lustre file system</title>
849 <imagedata scalefit="1" width="100%"
850 fileref="./figures/File_Striping.png" />
853 <phrase>File striping pattern across three OSTs for three different
854 data files. The file is sparse and missing chunk 6.</phrase>
858 <para>The maximum file size is not limited by the size of a single
859 target. In a Lustre file system, files can be striped across multiple
860 objects (up to 2000), and each object can be up to 16 TiB in size with
861 ldiskfs, or up to 256PiB with ZFS. This leads to a maximum file size of
862 31.25 PiB for ldiskfs or 8EiB with ZFS. Note that a Lustre file system can
863 support files up to 2^63 bytes (8EiB), limited only by the space available
866 <para>ldiskfs filesystems without the <literal>ea_inode</literal>
867 feature limit the maximum stripe count for a single file to 160 OSTs.
870 <para>Although a single file can only be striped over 2000 objects,
871 Lustre file systems can have thousands of OSTs. The I/O bandwidth to
872 access a single file is the aggregated I/O bandwidth to the objects in a
873 file, which can be as much as a bandwidth of up to 2000 servers. On
874 systems with more than 2000 OSTs, clients can do I/O using multiple files
875 to utilize the full file system bandwidth.</para>
876 <para>For more information about striping, see
877 <xref linkend="managingstripingfreespace" />.</para>
879 <emphasis role="bold">Extended Attributes(xattrs)</emphasis></para>
880 <para>Lustre uses lov_user_md_v1/lov_user_md_v3 data-structures to
881 maintain its file striping information under xattrs. Extended
882 attributes are created when files and directory are created. Lustre
883 uses <literal>trusted</literal> extended attributes to store its
884 parameters which are root-only accessible. The parameters are:</para>
888 <emphasis role="bold"><literal>trusted.lov</literal>:</emphasis>
889 Holds layout for a regular file, or default file layout stored
890 on a directory (also accessible as <literal>lustre.lov</literal>
896 <emphasis role="bold"><literal>trusted.lma</literal>:</emphasis>
897 Holds FID and extra state flags for current file</para>
901 <emphasis role="bold"><literal>trusted.lmv</literal>:</emphasis>
902 Holds layout for a striped directory (DNE 2), not present otherwise
907 <emphasis role="bold"><literal>trusted.link</literal>:</emphasis>
908 Holds parent directory FID + filename for each link to a file
909 (for <literal>lfs fid2path</literal>)</para>
912 <para>xattr which are stored and present in the file could be verify
914 <para><screen># getfattr -d -m - /mnt/testfs/file></screen></para>
919 vim:expandtab:shiftwidth=2:tabstop=8:textwidth=80: