system data after running <literal>e2fsck -fy /dev/{newdev}</literal> on
the new device, along with <literal>ll_recover_lost_found_objs</literal>
for OST devices.</para>
+ <para condition="l26">With Lustre software version 2.6 and later, there is
+ no longer a need to run <literal>ll_recover_lost_found_objs</literal> on
+ the OSTs, since the <literal>LFSCK</literal> scanning will automatically
+ move objects from <literal>lost+found</literal> back into its correct
+ location on the OST after directory corruption.</para>
</section>
<section xml:id="dbdoclet.50438207_21638">
<title><indexterm><primary>backup</primary><secondary>OST file system</secondary></indexterm><indexterm><primary>backup</primary><secondary>MDT file system</secondary></indexterm>Making a File-Level Backup of an OST or MDT File System</title>
<listitem xml:id="dbdoclet.50438267_pgfId-1290915">
<para>Create the OST. On the OSS node, run:</para>
<screen>mkfs.lustre --fsname=<replaceable>fsname</replaceable> --mgsnode=<replaceable>MGS_NID</replaceable> --ost --index=<replaceable>OST_index</replaceable> <replaceable>/dev/block_device</replaceable></screen>
- <para>When you create an OST, you are formatting a <literal>ldiskfs</literal> file system on a block storage device like you would with any local file system.</para>
+ <para>When you create an OST, you are formatting a <literal>ldiskfs</literal> or <literal>ZFS</literal> file system on a block storage device like you would with any local file system.</para>
<para>You can have as many OSTs per OSS as the hardware or drivers allow. For more information about storage and memory requirements for a Lustre file system, see <xref linkend="settinguplustresystem"/>.</para>
<para>You can only configure one OST per block device. You should create an OST that uses the raw block device and does not use partitioning.</para>
<para>You should specify the OST index number at format time in order to simplify translating the OST number in error messages or file striping to the OSS node and block device later on.</para>
relies on the backend file system to maintain per-user/group block and inode usage:</para>
<itemizedlist>
<listitem>
- <para>For ldiskfs backend, <literal>mkfs.lustre</literal> now creates empty quota files
+ <para>For ldiskfs backends, <literal>mkfs.lustre</literal> now creates empty quota files
and enables the QUOTA feature flag in the superblock which turns quota accounting on at
mount time automatically. e2fsck was also modified to fix the quota files when the QUOTA
feature flag is present.</para>
<para>Each client request processed by the server that involves any state change (metadata update, file open, write, etc., depending on server type) is assigned a transaction number by the server that is a target-unique, monotonically increasing, server-wide 64-bit integer. The transaction number for each file system-modifying request is sent back to the client along with the reply to that client request. The transaction numbers allow the client and server to unambiguously order every modification to the file system in case recovery is needed.</para>
<para>Each reply sent to a client (regardless of request type) also contains the last
committed transaction number that indicates the highest transaction number committed to the
- file system. The <literal>ldiskfs</literal> backing file system that the Lustre software
+ file system. The <literal>ldiskfs</literal> and <literal>ZFS</literal> backing file systems that the Lustre software
uses enforces the requirement that any earlier disk operation will always be committed to
disk before a later disk operation, so the last committed transaction number also reports
that any requests with a lower transaction number have been committed to disk.</para>
unusable for general storage. Thus, at least 400 MB of space is used on each OST before any
file object data is saved.</para>
</note>
+ <para condition="l24">With a ZFS backing filesystem for the MDT or OST,
+ the space allocation for inodes and file data is dynamic, and inodes are
+ allocated as needed. A minimum of 2kB of usable space (before RAID) is
+ needed for each inode, exclusive of other overhead such as directories,
+ internal log files, extended attributes, ACLs, etc.
+ Since the size of extended attributes and ACLs is highly dependent on
+ kernel versions and site-specific policies, it is best to over-estimate
+ the amount of space needed for the desired number of inodes, and any
+ excess space will be utilized to store more inodes.</para>
<section>
<title><indexterm>
<primary>setup</primary>
system, but a single MDS can host multiple MDTs, each one for a separate file
system.</para>
<para condition="l24">The Lustre software release 2.4 and later requires one MDT for
- the root. Upto 4095 additional MDTs can be added to the file system and attached
+ the filesystem root. Up to 4095 additional MDTs can be added to the file system and attached
into the namespace with remote directories.</para>
</entry>
</row>
<para> Maximum OST size</para>
</entry>
<entry>
- <para> 128TB </para>
+ <para> 128TB (ldiskfs), 256TB (ZFS)</para>
</entry>
<entry>
<para>This is not a <emphasis>hard</emphasis> limit. Larger OSTs are possible but
- today typical production systems do not go beyond 128TB per OST. </para>
+ today typical production systems do not go beyond the stated limit per OST. </para>
</entry>
</row>
<row>
<para> 131072</para>
</entry>
<entry>
- <para>The maximum number of clients is a constant that can be changed at compile time.</para>
+ <para>The maximum number of clients is a constant that can be changed at compile time. Up to 30000 clients have been used in production.</para>
</entry>
</row>
<row>
<para> Maximum size of a file system</para>
</entry>
<entry>
- <para> 512 PB</para>
+ <para> 512 PB (ldiskfs), 1EB (ZFS)</para>
</entry>
<entry>
- <para>Each OST or MDT on 64-bit kernel servers can have a file system up to 128 TB. On 32-bit systems, due to page cache limits, 16TB is the maximum block device size, which in turn applies to the size of OST on 32-bit kernel servers.</para>
+ <para>Each OST or MDT on 64-bit kernel servers can have a file system up to the above limit. On 32-bit systems, due to page cache limits, 16TB is the maximum block device size, which in turn applies to the size of OST on 32-bit kernel servers.</para>
<para>You can have multiple OST file systems on a single OSS node.</para>
</entry>
</row>
<row> <entry>
<para> Maximum object size</para> </entry>
<entry>
- <para> 16 TB</para>
+ <para> 16TB (ldiskfs), 256TB (ZFS)</para>
</entry>
<entry>
<para>The amount of data that can be stored in a single object. An object
- corresponds to a stripe. The ldiskfs limit of 16 TB for a single object applies.
- Files can consist of up to 2000 stripes, each 16 TB in size. </para>
+ corresponds to a stripe. The ldiskfs limit of 16 TB for a single object applies.
+ For ZFS the limit is the size of the underlying OST.
+ Files can consist of up to 2000 stripes, each stripe can contain the maximum object size. </para>
</entry>
</row>
<row>
<entry>
<para> 16 TB on 32-bit systems</para>
<para> </para>
- <para> 31.25 PB on 64-bit systems</para>
+ <para> 31.25 PB on 64-bit ldiskfs systems, 8EB on 64-bit ZFS systems</para>
</entry>
<entry>
<para>Individual files have a hard limit of nearly 16 TB on 32-bit systems imposed
by the kernel memory subsystem. On 64-bit systems this limit does not exist.
- Hence, files can be 64-bits in size. An additional size limit of up to the number
- of stripes is imposed, where each stripe is 16 TB.</para>
- <para>A single file can have a maximum of 2000 stripes, which gives an upper single file limit of 31.25 PB for 64-bit systems. The actual amount of data that can be stored in a file depends upon the amount of free space in each OST on which the file is striped.</para>
+ Hence, files can be 2^63 bits (8EB) in size if the backing filesystem can support large enough objects.</para>
+ <para>A single file can have a maximum of 2000 stripes, which gives an upper single file limit of 31.25 PB for 64-bit ldiskfs systems. The actual amount of data that can be stored in a file depends upon the amount of free space in each OST on which the file is striped.</para>
</entry>
</row>
<row>
<para> Maximum number of files or subdirectories in a single directory</para>
</entry>
<entry>
- <para> 10 million files</para>
+ <para> 10 million files (ldiskfs), 2^48 (ZFS)</para>
</entry>
<entry>
<para>The Lustre software uses the ldiskfs hashed directory code, which has a limit
<para> Maximum number of files in the file system</para>
</entry>
<entry>
- <para> 4 billion</para>
- <para condition='l24'>4096 * 4 billion</para>
+ <para> 4 billion (ldiskfs), 256 trillion (ZFS)</para>
+ <para condition='l24'>4096 times the per-MDT limit</para>
</entry>
<entry>
<para>The ldiskfs file system imposes an upper limit of 4 billion inodes. By default, the MDS file system is formatted with 2KB of space per inode, meaning 1 billion inodes per file system of 2 TB.</para>
<para>This can be increased initially, at the time of MDS file system creation. For more information, see <xref linkend="settinguplustresystem"/>.</para>
- <para condition="l24">Each additional MDT can hold up to 4 billion additional files, depending
- on available inodes and the distribution directories and files in the file
+ <para condition="l24">Each additional MDT can hold up to the above maximum number of additional files, depending
+ on available space and the distribution directories and files in the file
system.</para>
</entry>
</row>
<para> 255 bytes (filename)</para>
</entry>
<entry>
- <para>This limit is 255 bytes for a single filename, the same as in an ldiskfs file system.</para>
+ <para>This limit is 255 bytes for a single filename, the same as the limit in the underlying file systems.</para>
</entry>
</row>
<row>
<para> Maximum number of open files for a Lustre file system</para>
</entry>
<entry>
- <para> None</para>
+ <para> No limit</para>
</entry>
<entry>
<para>The Lustre software does not impose a maximum for the number of open files,
</section>
<section remap="h5">
<title>Description</title>
- <para>The first time Lustre writes to an object, it saves the MDS inode number and the objid as an extended attribute on the object, so in case of directory corruption of the OST, it is possible to recover the objects. Running e2fsck fixes the corrupted OST directory, but it puts all of the objects into a lost and found directory, where they are inaccessible to Lustre. Use the ll_recover_lost_found_objs utility to recover all (or at least most) objects from a lost and found directory and return them to the O/0/d* directories.</para>
- <para>To use ll_recover_lost_found_objs, mount the file system locally (using the -t ldiskfs command), run the utility and then unmount it again. The OST must not be mounted by Lustre when ll_recover_lost_found_objs is run.</para>
+ <para>The first time Lustre modifies an object, it saves the MDS inode number and the objid as an extended attribute on the object, so in case of directory corruption of the OST, it is possible to recover the objects. Running e2fsck fixes the corrupted OST directory, but it puts all of the objects into a lost and found directory, where they are inaccessible to Lustre. Use the ll_recover_lost_found_objs utility to recover all (or at least most) objects from a lost and found directory and return them to the O/0/d* directories.</para>
+ <para>To use ll_recover_lost_found_objs, mount the file system locally (using the <literal>-t ldiskfs</literal>, or <literal>-t zfs</literal> command), run the utility and then unmount it again. The OST must not be mounted by Lustre when ll_recover_lost_found_objs is run.</para>
+ <para condition="l26">This utility is not needed for 2.6 and later,
+ since the <literal>LFSCK</literal> online scanning will move objects
+ from <literal>lost+found</literal> to the proper place in the OST.</para>
</section>
<section remap="h5">
<title>Options</title>
<section xml:id="dbdoclet.50438219_90386">
<title><indexterm><primary>llog_reader</primary></indexterm>
llog_reader</title>
- <para>The llog_reader utility parses Lustre's on-disk configuration logs.</para>
+ <para>The llog_reader utility translates a Lustre configuration log into human-readable form.</para>
<section remap="h5">
<title>Synopsis</title>
<screen>llog_reader filename</screen>
<section remap="h5">
<title>Description</title>
<para>The llog_reader utility parses the binary format of Lustre's on-disk configuration logs. Llog_reader can only read logs; use tunefs.lustre to write to them.</para>
- <para>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs, then use llog_reader to dump the log file's contents, for example:</para>
+ <para>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs or zfs, then use llog_reader to dump the log file's contents, for example:</para>
<screen>mount -t ldiskfs /dev/sda /mnt/mgs
llog_reader /mnt/mgs/CONFIGS/tfs-client</screen>
<para>To examine the same log file on a running Lustre server, use the ldiskfs-enabled debugfs utility (called debug.ldiskfs on some distributions) to extract the file, for example:</para>
<para> <literal>--backfstype=<replaceable>fstype</replaceable></literal></para>
</entry>
<entry>
- <para> Forces a particular format for the backing file system (such as ext3, ldiskfs).</para>
+ <para> Forces a particular format for the backing file system such as ldiskfs (the default) or zfs.</para>
</entry>
</row>
<row>
<para>The following utilities are located in /usr/bin.</para>
<para><literal>lustre_req_history.sh</literal></para>
<para>The lustre_req_history.sh utility (run from a client), assembles as much Lustre RPC request history as possible from the local node and from the servers that were contacted, providing a better picture of the coordinated network activity.</para>
- <para><literal>llstat.sh</literal></para>
- <para>The llstat.sh utility handles a wider range of statistics files, and has command line switches to produce more graphable output.</para>
- <para><literal>plot-llstat.sh</literal></para>
- <para>The plot-llstat.sh utility plots the output from llstat.sh using gnuplot.</para>
</section>
<section remap="h3">
<title>More /proc Statistics for Application Profiling</title>
quit
EOF
</screen>
- <para>Feature Requests</para>
- <para>The loadgen utility is intended to grow into a more comprehensive test tool; feature requests are encouraged. The current feature requests include:</para>
- <itemizedlist>
- <listitem>
- <para> Locking simulation</para>
- </listitem>
- <listitem>
- <itemizedlist>
- <listitem>
- <para> Many (echo) clients cache locks for the specified resource at the same time.</para>
- </listitem>
- <listitem>
- <para> Many (echo) clients enqueue locks for the specified resource simultaneously.</para>
- </listitem>
- </itemizedlist>
- </listitem>
- <listitem>
- <para> obdsurvey functionality</para>
- </listitem>
- <listitem>
- <itemizedlist>
- <listitem>
- <para> Fold the Lustre I/O kit's obdsurvey script functionality into loadgen</para>
- </listitem>
- </itemizedlist>
- </listitem>
- </itemizedlist>
- </section>
- <section remap="h5">
- <title><indexterm><primary>llog_reader</primary></indexterm>
-llog_reader</title>
- <para>The llog_reader utility translates a Lustre configuration log into human-readable form.</para>
- </section>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>llog_reader filename
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>llog_reader parses the binary format of Lustre's on-disk configuration logs. It can only read the logs. Use tunefs.lustre to write to them.</para>
- <para>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs, then use llog_reader to dump the log file's contents. For example:</para>
- <screen>mount -t ldiskfs /dev/sda /mnt/mgs
-llog_reader /mnt/mgs/CONFIGS/tfs-client
-</screen>
- <para>To examine the same log file on a running Lustre server, use the ldiskfs-enabled debugfs utility (called debug.ldiskfs on some distributions) to extract the file. For example:</para>
- <screen>debugfs -c -R 'dump CONFIGS/tfs-client /tmp/tfs-client' /dev/sda
-llog_reader /tmp/tfs-client
-</screen>
- <caution>
- <para>Although they are stored in the CONFIGS directory, mountdata files do not use the config log format and will confuse llog_reader.</para>
- </caution>
- <para>See Also <xref linkend="dbdoclet.50438219_39574"/></para>
</section>
<section remap="h5">
<title><indexterm><primary>lr_reader</primary></indexterm>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438225_71141">
- <title><indexterm><primary>recovery</primary><secondary>corruption of backing file system</secondary></indexterm>Recovering from Errors or Corruption on a Backing File System</title>
+ <title><indexterm><primary>recovery</primary><secondary>corruption of backing ldiskfs file system</secondary></indexterm>Recovering from Errors or Corruption on a Backing ldiskfs File System</title>
<para>When an OSS, MDS, or MGS server crash occurs, it is not necessary to run e2fsck on the
file system. <literal>ldiskfs</literal> journaling ensures that the file system remains
consistent over a system crash. The backing file systems are never accessed directly
</entry>
<entry>
<para>
- <emphasis>Single MDS:</emphasis></para>
- <para> 4 billion files</para>
+ <emphasis>Single MDT:</emphasis></para>
+ <para> 4 billion files (ldiskfs), 256 trillion files (ZFS)</para>
<para>
<emphasis>MDS count:</emphasis></para>
<para> 1 primary + 1 backup</para>
- <para condition="l24"><emphasis role="italic">Since Lustre software release 2.4:
- </emphasis></para>
- <para condition="l24">Up to 4096 MDSs and up to 4096 MDTs</para>
+ <para condition="l24">Up to 4096 MDTs and up to 4096 MDSs</para>
</entry>
<entry>
<para>
- <emphasis>Single MDS:</emphasis></para>
- <para> 750 million files</para>
+ <emphasis>Single MDT:</emphasis></para>
+ <para> 1 billion files</para>
<para>
<emphasis>MDS count:</emphasis></para>
<para> 1 primary + 1 backup</para>
<para>multi-TB max file size</para>
<para>
<emphasis>Aggregate:</emphasis></para>
- <para>10 PB space, 750 million files</para>
+ <para>55 PB space, 1 billion files</para>
</entry>
</row>
</tbody>
<listitem>
<para><emphasis role="bold">Performance-enhanced ext4 file system:</emphasis> The Lustre
file system uses an improved version of the ext4 journaling file system to store data
- and metadata. This version, called <emphasis role="italic"
- ><literal>ldiskfs</literal></emphasis>, has been enhanced to improve performance and
+ and metadata. This version, called <emphasis role="italic">
+ <literal>ldiskfs</literal></emphasis>, has been enhanced to improve performance and
provide additional functionality needed by the Lustre file system.</para>
</listitem>
<listitem>
+ <para condition="l24">With the Lustre software release 2.4 and later, it is also possible to use ZFS as the backing filesystem for Lustre for the MDT, OST, and MGS storage. This allows Lustre to leverage the scalability and data integrity features of ZFS for individual storage targets.</para>
+ </listitem>
+ <listitem>
<para><emphasis role="bold">POSIX standard compliance:</emphasis> The full POSIX test
suite passes in an identical manner to a local ext4 file system, with limited exceptions
on Lustre clients. In a cluster, most operations are atomic so that clients never see
<para><emphasis role="bold">High-availability:</emphasis> The Lustre file system supports
active/active failover using shared storage partitions for OSS targets (OSTs). Lustre
software release 2.3 and earlier releases offer active/passive failover using a shared
- storage partition for the MDS target (MDT).</para>
- <para condition="l24">With Lustre software release 2.4 or later servers and clients it is
- possible to configure active/active failover of multiple MDTs. This allows application
- transparent recovery. The Lustre file system can work with a variety of high
- availability (HA) managers to allow automated failover and has no single point of
- failure (NSPF). Multiple mount protection (MMP) provides integrated protection from
+ storage partition for the MDS target (MDT). The Lustre file system can work with a variety of high
+ availability (HA) managers to allow automated failover and has no single point of failure (NSPF).
+ This allows application transparent recovery. Multiple mount protection (MMP) provides integrated protection from
errors in highly-available systems that would otherwise cause file system
corruption.</para>
</listitem>
<listitem>
+ <para condition="l24">With Lustre software release 2.4 or later
+ servers and clients it is possible to configure active/active
+ failover of multiple MDTs. This allows scaling the metadata
+ performance of Lustre filesystems with the addition of MDT storage
+ devices and MDS nodes.</para>
+ </listitem>
+ <listitem>
<para><emphasis role="bold">Security:</emphasis> By default TCP connections are only
allowed from privileged ports. UNIX group membership is verified on the MDS.</para>
</listitem>
</mediaobject>
</figure>
<para>The maximum file size is not limited by the size of a single target. In a Lustre file
- system, files can be striped across multiple objects (up to 2000), and each object can be
- up to 16 TB in size with ldiskfs. This leads to a maximum file size of 31.25 PB. (Note that
- a Lustre file system can support files up to 2^64 bytes depending on the backing storage
- used by OSTs.)</para>
+ system, files can be striped across multiple objects (up to 2000), and each object can be
+ up to 16 TB in size with ldiskfs, or up to 256PB with ZFS. This leads to a maximum file size of 31.25 PB for ldiskfs or 8EB with ZFS. Note that
+ a Lustre file system can support files up to 2^63 bytes (8EB), limited
+ only by the space available on the OSTs.</para>
<note>
<para>Versions of the Lustre software prior to Release 2.2 limited the maximum stripe count
for a single file to 160 OSTs.</para>
git://git.whamcloud.com / doc / manual.git / commitdiff