X-Git-Url: https://git.whamcloud.com/?a=blobdiff_plain;f=UnderstandingLustre.xml;h=f8c6ff6f9cc5f5c4f48682ffdd5ccfb5bd680072;hb=00c99af814574fe85ae7bea886d9fffcce4d0261;hp=ffde1f780da0e3a3955aa76b1675f88ea81fdb03;hpb=1f935d0231c9f14bbe5e07c87a31bcacd0492ae6;p=doc%2Fmanual.git diff --git a/UnderstandingLustre.xml b/UnderstandingLustre.xml index ffde1f7..f8c6ff6 100644 --- a/UnderstandingLustre.xml +++ b/UnderstandingLustre.xml @@ -1,7 +1,7 @@ + xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" + xml:id="understandinglustre"> Understanding Lustre Architecture This chapter describes the Lustre architecture and features of the @@ -67,7 +67,7 @@ xml:id="understandinglustre"> Lustre file systems run on a variety of vendor's kernels. For more details, see the Lustre Test Matrix . + linkend="preparing_installation" />. A Lustre installation can be scaled up or down with respect to the number of client nodes, disk storage and bandwidth. Scalability and performance are dependent on available disk and network bandwidth and the @@ -229,8 +229,7 @@ xml:id="understandinglustre"> MDS count: - 1 primary + 1 standby - 256 MDSs, with up to 256 MDTs + 256 MDSs, with up to 256 MDTs @@ -304,11 +303,10 @@ xml:id="understandinglustre"> additional functionality needed by the Lustre file system. - With the Lustre software release 2.4 and later, - it is also possible to use ZFS as the backing filesystem for Lustre - for the MDT, OST, and MGS storage. This allows Lustre to leverage the - scalability and data integrity features of ZFS for individual storage - targets. + It is also possible to use ZFS as the backing filesystem for + Lustre for the MDT, OST, and MGS storage. This allows Lustre to + leverage the scalability and data integrity features of ZFS for + individual storage targets. @@ -335,9 +333,8 @@ xml:id="understandinglustre"> High-availability:The Lustre file system supports active/active failover using shared storage - partitions for OSS targets (OSTs). Lustre software release 2.3 and - earlier releases offer active/passive failover using a shared storage - partition for the MDS target (MDT). The Lustre file system can work + partitions for OSS targets (OSTs), and for MDS targets (MDTs). + The Lustre file system can work with a variety of high availability (HA) managers to allow automated failover and has no single point of failure (NSPF). This allows application transparent recovery. Multiple mount protection (MMP) @@ -345,13 +342,6 @@ xml:id="understandinglustre"> systems that would otherwise cause file system corruption. - With Lustre software release 2.4 or later - servers and clients it is possible to configure active/active - failover of multiple MDTs. This allows scaling the metadata - performance of Lustre filesystems with the addition of MDT storage - devices and MDS nodes. - - Security:By default TCP connections are only allowed from privileged ports. UNIX group membership is @@ -508,23 +498,23 @@ xml:id="understandinglustre"> - Metadata Targets (MDT) - For Lustre - software release 2.3 and earlier, each file system has one MDT. The + Metadata Targets (MDT) - Each + filesystem has at least one MDT, which holds the root directory. The MDT stores metadata (such as filenames, directories, permissions and file layout) on storage attached to an MDS. Each file system has one MDT. An MDT on a shared storage target can be available to multiple MDSs, although only one can access it at a time. If an active MDS - fails, a standby MDS can serve the MDT and make it available to + fails, a second MDS node can serve the MDT and make it available to clients. This is referred to as MDS failover. - Since Lustre software release 2.4, multiple - MDTs are supported in the Distributed Namespace Environment (DNE). + Multiple MDTs are supported with the Distributed Namespace + Environment (). In addition to the primary MDT that holds the filesystem root, it is possible to add additional MDS nodes, each with their own MDTs, to hold sub-directory trees of the filesystem. Since Lustre software release 2.8, DNE also allows the filesystem to distribute files of a single directory over multiple MDT nodes. A directory which is distributed across multiple - MDTs is known as a striped directory. + MDTs is known as a . @@ -705,54 +695,29 @@ xml:id="understandinglustre"> Lustre I/O Lustre File System Storage and I/O - In Lustre software release 2.0, Lustre file identifiers (FIDs) were - introduced to replace UNIX inode numbers for identifying files or objects. - A FID is a 128-bit identifier that contains a unique 64-bit sequence - number, a 32-bit object ID (OID), and a 32-bit version number. The sequence + Lustre File IDentifiers (FIDs) are used internally for identifying + files or objects, similar to inode numbers in local filesystems. A FID + is a 128-bit identifier, which contains a unique 64-bit sequence number + (SEQ), a 32-bit object ID (OID), and a 32-bit version number. The sequence number is unique across all Lustre targets in a file system (OSTs and - MDTs). This change enabled future support for multiple MDTs (introduced in - Lustre software release 2.4) and ZFS (introduced in Lustre software release - 2.4). - Also introduced in release 2.0 is an ldiskfs feature named - FID-in-dirent(also known as - dirdata) in which the FID is stored as - part of the name of the file in the parent directory. This feature - significantly improves performance for - ls command executions by reducing disk I/O. The - FID-in-dirent is generated at the time the file is created. - - The FID-in-dirent feature is not backward compatible with the - release 1.8 ldiskfs disk format. Therefore, when an upgrade from - release 1.8 to release 2.x is performed, the FID-in-dirent feature is - not automatically enabled. For upgrades from release 1.8 to releases - 2.0 through 2.3, FID-in-dirent can be enabled manually but only takes - effect for new files. - For more information about upgrading from Lustre software release - 1.8 and enabling FID-in-dirent for existing files, see - Chapter 16 “Upgrading a Lustre File - System”. - - The LFSCK file system consistency checking tool - released with Lustre software release 2.4 provides functionality that - enables FID-in-dirent for existing files. It includes the following - functionality: + MDTs). This allows multiple MDTs and OSTs to uniquely identify objects + without depending on identifiers in the underlying filesystem (e.g. inode + numbers) that are likely to be duplicated between targets. The FID SEQ + number also allows mapping a FID to a particular MDT or OST. + The LFSCK file system consistency checking tool provides + functionality that enables FID-in-dirent for existing files. It + includes the following functionality: - Generates IGIF mode FIDs for existing files from a 1.8 version - file system files. - - - Verifies the FID-in-dirent for each file and regenerates the - FID-in-dirent if it is invalid or missing. + Verifies the FID stored with each directory entry and regenerates + it from the inode if it is invalid or missing. - Verifies the linkEA entry for each and regenerates the linkEA - if it is invalid or missing. The - linkEA consists of the file name and - parent FID. It is stored as an extended attribute in the file - itself. Thus, the linkEA can be used to reconstruct the full path name - of a file. + Verifies the linkEA entry for each inode and regenerates it if + invalid or missing. The linkEA + stores of the file name and parent FID. It is stored as an extended + attribute in each inode. Thus, the linkEA can be used to + reconstruct the full path name of a file from only the FID. Information about where file data is located on the OST(s) is stored @@ -767,7 +732,7 @@ xml:id="understandinglustre"> striped across the objects using RAID 0, and each object is stored on a different OST. (For more information about how striping is implemented in a Lustre file system, see - . + .
Layout EA on MDT pointing to file data on OSTs @@ -826,7 +791,7 @@ xml:id="understandinglustre"> available space of all the OSTs. -
+
<indexterm> <primary>Lustre</primary> @@ -845,7 +810,7 @@ xml:id="understandinglustre"> ability to stripe is also useful when a single OST does not have enough free space to hold an entire file. For more information about benefits and drawbacks of file striping, see - <xref linkend="dbdoclet.50438209_48033" />.</para> + <xref linkend="file_striping.considerations" />.</para> <para>Striping allows segments or 'chunks' of data in a file to be stored on different OSTs, as shown in <xref linkend="understandinglustre.fig.filestripe" />. In the Lustre file @@ -865,7 +830,7 @@ xml:id="understandinglustre"> for <literal>stripe_size</literal> is 1MB. The user may change these values on a per directory or per file basis. For more details, see - <xref linkend="dbdoclet.50438209_78664" />.</para> + <xref linkend="file_striping.lfs_setstripe" />.</para> <para> <xref linkend="understandinglustre.fig.filestripe" />, the <literal>stripe_size</literal> for File C is larger than the @@ -898,8 +863,9 @@ xml:id="understandinglustre"> support files up to 2^63 bytes (8EiB), limited only by the space available on the OSTs.</para> <note> - <para>Versions of the Lustre software prior to Release 2.2 limited the - maximum stripe count for a single file to 160 OSTs.</para> + <para>ldiskfs filesystems without the <literal>ea_inode</literal> + feature limit the maximum stripe count for a single file to 160 OSTs. + </para> </note> <para>Although a single file can only be striped over 2000 objects, Lustre file systems can have thousands of OSTs. The I/O bandwidth to @@ -909,6 +875,46 @@ xml:id="understandinglustre"> to utilize the full file system bandwidth.</para> <para>For more information about striping, see <xref linkend="managingstripingfreespace" />.</para> + <para> + <emphasis role="bold">Extended Attributes(xattrs)</emphasis></para> + <para>Lustre uses lov_user_md_v1/lov_user_md_v3 data-structures to + maintain its file striping information under xattrs. Extended + attributes are created when files and directory are created. Lustre + uses <literal>trusted</literal> extended attributes to store its + parameters which are root-only accessible. The parameters are:</para> + <itemizedlist> + <listitem> + <para> + <emphasis role="bold"><literal>trusted.lov</literal>:</emphasis> + Holds layout for a regular file, or default file layout stored + on a directory (also accessible as <literal>lustre.lov</literal> + for non-root users). + </para> + </listitem> + <listitem> + <para> + <emphasis role="bold"><literal>trusted.lma</literal>:</emphasis> + Holds FID and extra state flags for current file</para> + </listitem> + <listitem> + <para> + <emphasis role="bold"><literal>trusted.lmv</literal>:</emphasis> + Holds layout for a striped directory (DNE 2), not present otherwise + </para> + </listitem> + <listitem> + <para> + <emphasis role="bold"><literal>trusted.link</literal>:</emphasis> + Holds parent directory FID + filename for each link to a file + (for <literal>lfs fid2path</literal>)</para> + </listitem> + </itemizedlist> + <para>xattr which are stored and present in the file could be verify + using:</para> + <para><screen># getfattr -d -m - /mnt/testfs/file></screen></para> </section> </section> </chapter> +<!-- + vim:expandtab:shiftwidth=2:tabstop=8:textwidth=80: + -->