Remove references to Lustre 2.4 and earlier from the manual.
There were even places with referencs to Lustre 1.8 and 2.0.
The text still exists in earlier builds and in Git for reference.
This avoids confusing users with commands that no longer exist,
or highlighting features that are included with all releases.
Add conditions for Lustre 2.15 and 2.16 as they will be used soon.
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I0786d88fcc2543f542b7a880bf042928fd3ebbe5
Reviewed-on: https://review.whamcloud.com/39511
Tested-by: jenkins <devops@whamcloud.com>
<primary>backup</primary>
<secondary>OST and MDT</secondary>
</indexterm>Backing Up an OST or MDT</title>
- <para>If backing up an MDT, substitute <literal>mdt</literal> for
- <literal>ost</literal> in the instructions below.</para>
+ <para>The below examples show backing up an OST filesystem. When backing
+ up an MDT, substitute <literal>mdt</literal> for <literal>ost</literal>
+ in the instructions below.</para>
<orderedlist>
<listitem>
<para><emphasis role="bold">Umount the target</emphasis></para>
<screen>$ nobjhi=2 thrhi=2 size=1024 case=disk sh obdfilter-survey</screen>
</listitem>
<listitem>
-
<para>Performance measurements for write, rewrite, read etc are
+ <para>Performance measurements for write, rewrite, read etc are
provided below:</para>
- <screen># example output
+ <screen># example output
Fri Sep 25 11:14:03 EDT 2015 Obdfilter-survey for case=disk from hds1fnb6123
ost 10 sz 167772160K rsz 1024K obj 10 thr 10 write 10982.73 [ 601.97,2912.91] rewrite 15696.54 [1160.92,3450.85] read 12358.60 [ 938.96,2634.87]
...</screen>
+ <para>The file
<literal>./lustre-iokit/obdfilter-survey/README.obdfilter-survey
</literal>provides an explaination for the output as follows:</para>
- <screen>ost 10 is the total number of OSTs under test.
+ <screen>ost 10 is the total number of OSTs under test.
sz 167772160K is the total amount of data read or written (in bytes).
rsz 1024K is the record size (size of each echo_client I/O, in bytes).
obj 10 is the total number of objects over all OSTs
(<literal>nobjhi</literal>), up to two threads (
<literal>thrhi</literal>), and 1024 Mb (size) transfer size:</para>
<screen>$ nobjhi=2 thrhi=2 size=1024 targets="lustre-OST0001 \
- lustre-OST0002" sh obdfilter-survey</screen>
+ lustre-OST0002" sh obdfilter-survey</screen>
</listitem>
</orderedlist>
</section>
<literal>targets=<replaceable>hostname|ip_of_server</replaceable>
</literal>. For example:</para>
<screen>$ nobjhi=2 thrhi=2 size=1024 targets="oss0 oss1" \
- case=network sh obdfilter-survey</screen>
+ case=network sh obdfilter-survey</screen>
</listitem>
<listitem>
<para>On the server side, view the statistics at:</para>
<secondary>MDS performance</secondary></indexterm>
Testing MDS Performance (<literal>mds-survey</literal>)</title>
<para>The <literal>mds-survey</literal> script tests the local
- metadata performance using the echo_client to drive different layers of
- the MDS stack: mdd, mdt, osd (the Lustre software only supports mdd
- stack). It can be used with the following classes of operations:</para>
+ metadata performance using the <literal>echo_client</literal> to
+ drive the MDD layer of the MDS stack.
+ It can be used with the following classes of operations:</para>
<itemizedlist>
<listitem>
<para><literal>Open-create/mkdir/create</literal></para>
associated with each target and the service node on which the target is mounted. Later, when
the client attempts to access data on a target, it will try the NID for each specified service
node until it connects to the target.</para>
- <para>Previous to Lustre software release 2.0, the <literal>--failnode</literal> option to
- <literal>mkfs.lustre</literal> was used to designate a failover service node for a primary
- server for a target. When the <literal>--failnode</literal> option is used, certain
- restrictions apply:<itemizedlist>
- <listitem>
- <para>The target must be initially mounted on the primary service node, not the failover
- node designated by the <literal>--failnode</literal> option.</para>
- </listitem>
- <listitem>
- <para>If the <literal>tunefs.lustre –-writeconf</literal> option is used to erase and
- regenerate the configuration log for the file system, a target cannot be initially
- mounted on a designated failnode.</para>
- </listitem>
- <listitem>
- <para>If a <literal>--failnode</literal> option is added to a target to designate a
- failover server for the target, the target must be re-mounted on the primary node before
- the <literal>--failnode</literal> option takes effect</para>
- </listitem>
- </itemizedlist></para>
</section>
<section xml:id="section_tnq_kbr_xl">
<title>Administering Failover in a Lustre File System</title>
<literal>grpquota</literal> options to mount. Space accounting is
enabled by default and quota enforcement can be enabled/disabled on
a per-filesystem basis with <literal>lctl conf_param</literal>.</para>
+ <para condition="l28">It is worth noting that the
+ <literal>lfs quotaon</literal>, <literal>lfs quotaoff</literal>,
+ <literal>lfs quotacheck</literal> and <literal>quota_type</literal>
+ sub-commands are deprecated as of Lustre 2.4.0, and removed completely
+ in Lustre 2.8.0.</para>
</listitem>
</itemizedlist>
<caution>
to fix the quota files when the QUOTA feature flag is present. The
project quota feature is disabled by default, and
<literal>tune2fs</literal> needs to be run to enable every target
- manually.</para>
+ manually. If user, group, and project quota usage is inconsistent,
+ run <literal>e2fsck -f</literal> on all unmounted MDTs and OSTs.
+ </para>
</listitem>
<listitem>
<para>For ZFS backend, <emphasis>the project quota feature is not
</listitem>
</itemizedlist>
<note>
+ <para>To (re-)enable space usage quota on ldiskfs filesystems, run
+ <literal>tune2fs -O quota</literal> against all targets. This command
+ sets the QUOTA feature flag in the superblock and runs e2fsck internally.
+ As a result, the target must be offline to build the per-UID/GID disk
+ usage database.</para>
<para condition="l2A">Lustre filesystems formatted with a Lustre release
prior to 2.10 can be still safely upgraded to release 2.10, but will not
have project quota usage reporting functional until
to be installed on the server nodes when using the ldiskfs backend
(e2fsprogs is not needed with ZFS backend). In general, we recommend
to use the latest e2fsprogs version available on
- <link xl:href="http://downloads.whamcloud.com/e2fsprogs/">
+ <link xl:href="https://downloads.whamcloud.com/public/e2fsprogs/">
http://downloads.whamcloud.com/public/e2fsprogs/</link>.</para>
<para>The ldiskfs OSD relies on the standard Linux quota to maintain
accounting information on disk. As a consequence, the Linux kernel
</glossdef>
</glossentry>
<glossentry xml:id="DNE">
- <glossterm>Distributed namespace (DNE)</glossterm>
+ <glossterm>Distributed Namespace Environment (DNE)</glossterm>
<glossdef>
<para>A collection of metadata targets serving a single file
system namespace. Without DNE, Lustre file systems are limited to a
<glossterm>EA</glossterm>
<glossdef>
<para>Extended attribute. A small amount of data that can be retrieved
- through a name (EA or attr) associated with a particular inode. A
+ through a name (EA or xattr) associated with a particular inode. A
Lustre file system uses EAs to store striping information (indicating
the location of file data on OSTs). Examples of extended attributes are
ACLs, striping information, and the FID of the file.</para>
[154523.177714] Lustre: Unmounted testfs-client</screen></para>
</listitem>
<listitem><para>Unmount the MDT and MGT</para>
- <para>On the MGS and MDS node(s), use the <literal>umount</literal>
- command:</para>
+ <para>On the MGS and MDS node(s), run the
+ <literal>umount</literal> command:</para>
<para><literal>umount -a -t lustre</literal></para>
<para>The example below shows the unmount of the MDT and MGT for
- the <literal>testfs</literal> filesystem on a combined MGS/MDS:
- </para>
+ the <literal>testfs</literal> filesystem on a combined MGS/MDS:
+ </para>
<para><screen>[root@mds1 ~]# mount |grep lustre
/dev/sda on /mnt/mgt type lustre (ro)
/dev/sdb on /mnt/mdt type lustre (ro)
[155263.566230] Lustre: Failing over testfs-MDT0000
[155263.775355] Lustre: server umount testfs-MDT0000 complete
[155269.843862] Lustre: server umount MGS complete</screen></para>
- <para>For a seperate MGS and MDS, the same command is used, first on
+
<para>For a seperate MGS and MDS, the same command is used, first on
the MDS and then followed by the MGS.</para>
</listitem>
<listitem><para>Unmount all the OSTs</para>
</para>
<para><literal>umount -a -t lustre</literal></para>
<para>The example below shows the unmount of all OSTs for the
- <literal>testfs</literal> filesystem on server
- <literal>OSS1</literal>:
+ <literal>testfs</literal> filesystem on server
+ <literal>OSS1</literal>:
</para>
<para><screen>[root@oss1 ~]# mount |grep lustre
/dev/sda on /mnt/ost0 type lustre (ro)
</screen></para>
<para>The command output is:
<screen>
-debugfs 1.42.3.wc3 (15-Aug-2012)
+debugfs 1.45.6.wc1 (20-Mar-2020)
/dev/lustre/ost_test2: catastrophic mode - not reading inode or group bitmaps
Inode: 352365 Type: regular Mode: 0666 Flags: 0x80000
Generation: 2393149953 Version: 0x0000002a:00005f81
Extended attributes stored in inode body:
fid = "b9 da 24 00 00 00 00 00 6a fa 0d 3f 01 00 00 00 eb 5b 0b 00 00 00 0000
00 00 00 00 00 00 00 00 " (32)
- fid: objid=34976 seq=0 parent=[0x24dab9:0x3f0dfa6a:0x0] stripe=1
+ fid: objid=34976 seq=0 parent=[0x200000400:0x122:0x0] stripe=1
EXTENTS:
(0-64):4620544-4620607
</screen></para>
</listitem>
<listitem>
- <para>For Lustre software release 2.x file systems, the parent FID will
- be of the form [0x200000400:0x122:0x0] and can be resolved directly
- using the
- <literal>lfs fid2path [0x200000404:0x122:0x0]
- /mnt/lustre</literal> command on any Lustre client, and the process is
+ <para>The parent FID will be of the form
+ <literal>[0x200000400:0x122:0x0]</literal> and can be resolved directly
+ using the command <literal>lfs fid2path [0x200000404:0x122:0x0]
+ /mnt/lustre</literal> on any Lustre client, and the process is
complete.</para>
</listitem>
<listitem>
- <para>In this example the parent inode FID is an upgraded 1.x inode
- (due to the first part of the FID being below 0x200000400), the MDT
- inode number is
+ <para>In cases of an upgraded 1.x inode (if the first part of the
+ FID is below 0x200000400), the MDT inode number is
<literal>0x24dab9</literal> and generation
- <literal>0x3f0dfa6a</literal> and the pathname needs to be resolved
+ <literal>0x3f0dfa6a</literal> and the pathname can also be resolved
using
<literal>debugfs</literal>.</para>
</listitem>
</screen>
<para>Here is the command output:</para>
<screen>
-debugfs 1.42.3.wc2 (15-Aug-2012)
+debugfs 1.42.3.wc3 (15-Aug-2012)
/dev/lustre/mdt_test: catastrophic mode - not reading inode or group bitmap\
s
Inode Pathname
</section>
<section xml:id="imperativerecovery">
<title><indexterm><primary>imperative recovery</primary></indexterm>Imperative Recovery</title>
- <para>Large-scale Lustre file system implementations have historically experienced problems
- recovering in a timely manner after a server failure. This is due to the way that clients
- detect the server failure and how the servers perform their recovery. Many of the processes
- are driven by the RPC timeout, which must be scaled with system size to prevent false
- diagnosis of server death. The purpose of imperative recovery is to reduce the recovery window
- by actively informing clients of server failure. The resulting reduction in the recovery
- window will minimize target downtime and therefore increase overall system availability.
+ <para>Large-scale Lustre filesystems will experience server hardware
+ failures over their lifetime, and it is important that servers can
+ recover in a timely manner after such failures. High Availability
+ software can move storage targets over to a backup server automatically.
+ Clients can detect the server failure by RPC timeouts, which must be
+ scaled with system size to prevent false diagnosis of server death in
+ cases of heavy load. The purpose of imperative recovery is to reduce
+ the recovery window by actively informing clients of server failure.
+ The resulting reduction in the recovery window will minimize target
+ downtime and therefore increase overall system availability.</para>
+ <para>
Imperative Recovery does not remove previous recovery mechanisms, and client timeout-based
recovery actions can occur in a cluster when IR is enabled as each client can still
independently disconnect and reconnect from a target. In case of a mix of IR and non-IR
in providing the read page service. The read page service handles
file close and readdir operations.</para>
</listitem>
- <listitem>
- <para>
- <literal>mds_attr_num_threads</literal> controls the number of threads
- in providing the setattr service to clients running Lustre software
- release 1.8.</para>
- </listitem>
</itemizedlist>
</section>
</section>
to
<literal>CPT4</literal>.</para>
</listitem>
- <listitem>
- <para>
- <literal>mds_attr_num_cpts=[EXPRESSION]</literal> binds the setattr
- service threads to CPTs defined by
- <literal>EXPRESSION</literal>.</para>
- </listitem>
</itemizedlist>
-
<para>Parameters must be set before module load in the file
+ <para>Parameters must be set before module load in the file
<literal>/etc/modprobe.d/lustre.conf</literal>. For example:
<example><title>lustre.conf</title>
<screen>options lnet networks=tcp0(eth0)
<literal>ksocklnd</literal> using the
<literal>nscheds</literal> parameter. This adjusts the number of threads for
each partition, not the overall number of threads on the LND.</para>
+ <note>
+ <para>The default number of threads for
+ <literal>ko2iblnd</literal> and
+ <literal>ksocklnd</literal> are automatically set and are chosen to
+ work well across a number of typical scenarios, for systems with both
+ high and low core counts.</para>
+ </note>
<section>
<title>ko2iblnd Tuning</title>
<para>The following table outlines the ko2iblnd module parameters to be used
</section>
</section>
<section remap="h3">
- <title>Ptlrpc Thread Pool</title>
- <para>The increasing use of large SMP nodes for Lustre servers requires
- good scaling of many application threads generating large amounts of IO.
- Lustre implements a ptlrpc thread pool, so
- that multiple threads can be created to serve asynchronous RPC requests.
- The number of threads spawned is controlled at module load time using
- module options. By default one thread is spawned per CPU, with a minimum
- of 2 threads spawned irrespective of module options.</para>
+ <title>PtlRPC Client Thread Pool</title>
+ <para>The use of large SMP nodes for Lustre clients
+ requires significant parallelism within the kernel to avoid
+ cases where a single CPU would be 100% utilized and other CPUs would be
+ relativity idle. This is especially noticeable when a single thread
+ traverses a large directory.</para>
+ <para>The Lustre client implements a PtlRPC daemon thread pool, so that
+ multiple threads can be created to serve asynchronous RPC requests, even
+ if only a single userspace thread is running. The number of ptlrpcd
+ threads spawned is controlled at module load time using module options.
+ By default two service threads are spawned per CPU socket.</para>
<para>One of the issues with thread operations is the cost of moving a
thread context from one CPU to another with the resulting loss of CPU
- cache warmth. To reduce this cost, ptlrpc threads can be bound to a CPU.
+ cache warmth. To reduce this cost, PtlRPC threads can be bound to a CPU.
However, if the CPUs are busy, a bound thread may not be able to respond
quickly, as the bound CPU may be busy with other tasks and the thread
must wait to schedule.</para>
- <para>Because of these considerations, the pool of ptlrpc threads can be
+ <para>Because of these considerations, the pool of ptlrpcd threads can be
a mixture of bound and unbound threads. The system operator can balance
the thread mixture based on system size and workload.</para>
<section>
<literal>etc/modprobe.d</literal> directory, as options for the ptlrpc
module.
<screen>
-options ptlrpcd max_ptlrpcds=XXX
+options ptlrpcd ptlrpcd_per_cpt_max=XXX
</screen></para>
- <para>Sets the number of ptlrpcd threads created at module load time.
- The default if not specified is one thread per CPU, including
- hyper-threaded CPUs. The lower bound is 2 (old prlrpcd behaviour)
+ <para>Sets the number of ptlrpcd threads created per socket.
+ The default if not specified is two threads per CPU socket, including
+ hyper-threaded CPUs. The lower bound is 2 threads per socket.
<screen>
options ptlrpcd ptlrpcd_bind_policy=[1-4]
</screen></para>
<literal>ea_inode</literal> feature on the MDT:
<screen>tune2fs -O ea_inode /dev/<replaceable>mdtdev</replaceable></screen>
</para>
+ <note condition='l2D'><para>Since Lustre 2.13 the
+ <literal>ea_inode</literal> feature is enabled by default on all newly
+ formatted ldiskfs MDT filesystems.</para></note>
<note><para>The maximum stripe count for a single file does not limit the
maximum number of OSTs that are in the filesystem as a whole, only the
maximum possible size and maximum aggregate bandwidth for the file.
<link xl:href="http://wiki.lustre.org/Lustre_Manual_Changes">
http://wiki.lustre.org/Lustre_Manual_Changes</link>.</para>
- <para condition="l28">This manual covers a range of Lustre 2.x software
- releases. Features that are specific to individual releases are
- identified within the table of contents using a short hand notation
- (i.e. this paragraph is tagged for a Lustre 2.8 specific feature),
- and within the text using a distinct box. For example:</para>
+ <para condition="l25">This manual covers a range of Lustre 2.x software
+ releases, currently starting with the 2.5 release. Features specific
+ to individual releases are identified within the table of contents
+ using a shorthand notation (e.g. this paragraph is tagged as a Lustre
+ 2.5 specific feature so that it will be updated when the 2.5-specific
+ tagging is removed), and within the text using a distinct box.
+ </para>
- <note xml:id="whichversion"><title>which version?</title>
+ <note xml:id="whichversion"><title>Which version amd I running?</title>
<indexterm><primary>version</primary>
<secondary>which version of Lustre am I running?</secondary></indexterm>
<para>The current version of Lustre that is in use on the node can be found
- using the command <literal>lctl get_param version</literal>, for example:
+ using the command <literal>lctl get_param version</literal> on any Lustre
+ client or server, for example:
<screen>$ lctl get_param version
version=2.10.5
</screen>
</listitem>
</itemizedlist>
</warning>
- <para>Only servers running on 64-bit CPUs are tested and supported. 64-bit CPU clients are
- typically used for testing to match expected customer usage and avoid limitations due to the 4
- GB limit for RAM size, 1 GB low-memory limitation, and 16 TB file size limit of 32-bit CPUs.
- Also, due to kernel API limitations, performing backups of Lustre software release 2.x. file
- systems on 32-bit clients may cause backup tools to confuse files that have the same 32-bit
- inode number.</para>
+ <para>Only servers running on 64-bit CPUs are tested and supported.
+ 64-bit CPU clients are typically used for testing to match expected
+ customer usage and avoid limitations due to the 4 GB limit for RAM
+ size, 1 GB low-memory limitation, and 16 TB file size limit of 32-bit
+ CPUs. Also, due to kernel API limitations, performing backups of Lustre
+ filesystems on 32-bit clients may cause backup tools to confuse files
+ that report the same 32-bit inode number, if the backup tools depend
+ on the inode number for correct operation.</para>
<para>The storage attached to the servers typically uses RAID to provide fault tolerance and can
optionally be organized with logical volume management (LVM), which is then formatted as a
Lustre file system. Lustre OSS and MDS servers read, write and modify data in the format
<note>
<para>OSTs formatted with ldiskfs can use a maximum of approximately
320 million objects per MDT, up to a maximum of 4 billion inodes.
- Specifying a very small bytes-per-inode ratio for a large OST that
- exceeds this limit can cause either premature out-of-space errors and prevent
+ Specifying a very small bytes-per-inode ratio for a large OST that
+ exceeds this limit can cause either premature out-of-space errors and prevent
the full OST space from being used, or will waste space and slow down
e2fsck more than necessary. The default inode ratios are chosen to
ensure that the total number of inodes remain below this limit.
<para>256</para>
</entry>
<entry>
- <para>A single MDS can host
- multiple MDTs, either for separate file systems, or up to 255
- additional MDTs can be added to the filesystem and attached into
- the namespace with DNE remote or striped directories.</para>
+ <para>A single MDS can host one or more MDTs, either for separate
+ filesystems, or aggregated into a single namespace. Each
+ filesystem requires a separate MDT for the filesystem root
+ directory.
+ Up to 255 more MDTs can be added to the filesystem and are
+ attached into the filesystem namespace with creation of DNE
+ remote or striped directories.</para>
</entry>
</row>
<row>
systems that would otherwise cause file system corruption.</para>
</listitem>
<listitem>
- <para>It is possible to configure active/active failover of multiple
- MDTs. This allows scaling the metadata performance of Lustre
- filesystems with the addition of MDT storage devices and MDS nodes.
- </para>
- </listitem>
- <listitem>
<para>
<emphasis role="bold">Security:</emphasis>By default TCP connections
are only allowed from privileged ports. UNIX group membership is
<listitem>
<para>
<emphasis role="bold">Metadata Targets (MDT</emphasis>) - Each
- filesystem has at least one MDT. The
+ filesystem has at least one MDT, which holds the root directory. The
MDT stores metadata (such as filenames, directories, permissions and
file layout) on storage attached to an MDS. Each file system has one
MDT. An MDT on a shared storage target can be available to multiple
MDSs, although only one can access it at a time. If an active MDS
- fails, a standby MDS can serve the MDT and make it available to
+ fails, a second MDS node can serve the MDT and make it available to
clients. This is referred to as MDS failover.</para>
- <para>Multiple MDTs are supported in the Distributed Namespace
- Environment (DNE).
+ <para>Multiple MDTs are supported with the Distributed Namespace
+ Environment (<xref linkend="DNE"/>).
In addition to the primary MDT that holds the filesystem root, it
is possible to add additional MDS nodes, each with their own MDTs,
to hold sub-directory trees of the filesystem.</para>
<para condition="l28">Since Lustre software release 2.8, DNE also
allows the filesystem to distribute files of a single directory over
multiple MDT nodes. A directory which is distributed across multiple
- MDTs is known as a <emphasis>striped directory</emphasis>.</para>
+ MDTs is known as a <emphasis><xref linkend="stripeddirectory"/></emphasis>.</para>
</listitem>
<listitem>
<para>
<primary>Lustre</primary>
<secondary>I/O</secondary>
</indexterm>Lustre File System Storage and I/O</title>
- <para>Lustre uses file identifiers (FIDs) to replace UNIX inode numbers
- for identifying files or objects. A FID is a 128-bit identifier that
- contains a unique 64-bit sequence number, a 32-bit object ID (OID), and
- a 32-bit version number. The sequence number is unique across all Lustre
- targets in a file system (OSTs and MDTs). This allows Lustre to identify
- files on multiple MDTs independent of the underlying filesystem type.
- </para>
- <para>The LFSCK file system consistency checking tool verifies the
- consistency of file objects between MDTs and OSTs. It includes the
- following functionality:
+ <para>Lustre File IDentifiers (FIDs) are used internally for identifying
+ files or objects, similar to inode numbers in local filesystems. A FID
+ is a 128-bit identifier, which contains a unique 64-bit sequence number
+ (SEQ), a 32-bit object ID (OID), and a 32-bit version number. The sequence
+ number is unique across all Lustre targets in a file system (OSTs and
+ MDTs). This allows multiple MDTs and OSTs to uniquely identify objects
+ without depending on identifiers in the underlying filesystem (e.g. inode
+ numbers) that are likely to be duplicated between targets. The FID SEQ
+ number also allows mapping a FID to a particular MDT or OST.</para>
+ <para>The LFSCK file system consistency checking tool provides
+ functionality that enables FID-in-dirent for existing files. It
+ includes the following functionality:
<itemizedlist>
<listitem>
- <para>Verifies the FID-in-dirent for each file and regenerates the
- FID-in-dirent if it is invalid or missing.</para>
+ <para>Verifies the FID stored with each directory entry and regenerates
+ it from the inode if it is invalid or missing.</para>
</listitem>
<listitem>
- <para>Verifies the linkEA entry for each and regenerates the linkEA
- if it is invalid or missing. The
- <emphasis role="italic">linkEA</emphasis> consists of the file name and
- parent FID. It is stored as an extended attribute in the file
- itself. Thus, the linkEA can be used to reconstruct the full path name
- of a file.</para>
+ <para>Verifies the linkEA entry for each inode and regenerates it if
+ invalid or missing. The <emphasis role="italic">linkEA</emphasis>
+ stores of the file name and parent FID. It is stored as an extended
+ attribute in each inode. Thus, the linkEA can be used to
+ reconstruct the full path name of a file from only the FID.</para>
</listitem>
</itemizedlist></para>
<para>Information about where file data is located on the OST(s) is stored
31.25 PiB for ldiskfs or 8EiB with ZFS. Note that a Lustre file system can
support files up to 2^63 bytes (8EiB), limited only by the space available
on the OSTs.</para>
+ <note>
+ <para>ldiskfs filesystems without the <literal>ea_inode</literal>
+ feature limit the maximum stripe count for a single file to 160 OSTs.
+ </para>
+ </note>
<para>Although a single file can only be striped over 2000 objects,
Lustre file systems can have thousands of OSTs. The I/O bandwidth to
access a single file is the aggregated I/O bandwidth to the objects in a
xml:id="upgradinglustre">
<title xml:id="upgradinglustre.title">Upgrading a Lustre File System</title>
<para>This chapter describes interoperability between Lustre software
- releases. It also provides procedures for upgrading from Lustre software
- release 1.8 to Lustre software release 2.x , from a Lustre software release
- 2.x to a more recent Lustre software release 2.x (major release upgrade), and
- from a a Lustre software release 2.x.y to a more recent Lustre software
- release 2.x.y (minor release upgrade). It includes the following
- sections:</para>
+ releases. It also provides procedures for upgrading from older Lustre 2.x
+ software releases to a more recent 2.y Lustre release a (major release
+ upgrade), and from a Lustre software release 2.x.y to a more recent
+ Lustre software release 2.x.z (minor release upgrade). It includes the
+ following sections:</para>
<itemizedlist>
<listitem>
<para>
<listitem>
<para>All servers must be be upgraded to a Linux kernel supported by
the Lustre software. See the Lustre Release Notes for your Lustre
- version for a list of tested Linux distributions.</para>
+ version for a list of tested Linux distributions.</para>
</listitem>
<listitem>
<para>Clients to be upgraded must be running a compatible Linux
<indexterm>
<primary>ea_inode</primary>
<secondary>large_xattr</secondary>
- </indexterm>
- <indexterm>
- <primary>wide striping</primary>
- <secondary>ea_inode</secondary>
- <tertiary>large_xattr</tertiary>
</indexterm>Upgrading to Lustre Software Release 2.x (Major
Release)</title>
<para>The procedure for upgrading from a Lustre software release 2.x to a
(tested) Linux distribution and reboot.</para>
</listitem>
<listitem>
- <para>Upgrade the Linux operating system on all clients to Red Hat
- Enterprise Linux 6 or other compatible (tested) distribution and
- reboot.</para>
+ <para>Upgrade the Linux operating system on all clients to a
+ compatible (tested) distribution and reboot.</para>
</listitem>
<listitem>
<para>Download the Lustre server RPMs for your platform from the
</orderedlist>
</listitem>
<listitem>
- <para condition="l2D">Lustre allows striping a single file across up to
- 2000 OSTs. Before Lustre 2.13, the "wide striping" feature that
- allowed creating files with more than 160 stripes was not enabled by
- default. From the 2.13 release onward, the <literal>ea_inode</literal>
- feature is enabled for newly-formatted MDTs. The feature can also
- be enabled by the <literal>tune2fs</literal> command on existing MDTs:
- <screen>mds# tune2fs -O ea_inode /dev/<replaceable>mdtdev</replaceable></screen>
- </para>
- <para>For more information about wide striping, see
- <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping" />.</para>
- </listitem>
- <listitem>
+ <para>The DNE feature allows using multiple MDTs within a single
+ filesystem namespace, and each MDT can each serve one or more remote
+ sub-directories in the file system. The <literal>root</literal>
+ directory is always located on MDT0.</para>
+ <para>Note that clients running a release prior to the Lustre software
+ release 2.4 can only see the namespace hosted by MDT0 and will return an
+ IO error if an attempt is made to access a directory on another
+ MDT.</para>
<para>(Optional) To format an additional MDT, complete these steps:
<orderedlist numeration="loweralpha">
<listitem>
<para>In this example, the next available index is 1.</para>
</listitem>
<listitem>
- <para>Add the new block device as a new MDT at the next available
- index by entering (on one line):
+ <para>Format the new block device as a new MDT at the next
+ available MDT index by entering (on one line):
<screen>mds# mkfs.lustre --reformat --fsname=<replaceable>filesystem_name</replaceable> --mdt \
- --mgsnode=<replaceable>mgsnode</replaceable> --index <replaceable>1</replaceable>
+ --mgsnode=<replaceable>mgsnode</replaceable> --index <replaceable>new_mdt_index</replaceable>
<replaceable>/dev/mdt1_device</replaceable></screen></para>
</listitem>
</orderedlist></para>
</listitem>
<listitem>
- <para>(Optional) If you are upgrading before Lustre software release
+ <para>(Optional) If you are upgrading from a release before Lustre
2.10, to enable the project quota feature enter the following on every
- ldiskfs backend target:
+ ldiskfs backend target while unmounted:
<screen>tune2fs –O project /dev/<replaceable>dev</replaceable></screen>
</para>
<note><para>Enabling the <literal>project</literal> feature will prevent
the filesystem from being used by older versions of ldiskfs, so it
- should only be enabled if the project quota feature is required and/or
- after it is known that the upgraded release does not need to be
- downgraded.</para></note>
+ should only be enabled if the project quota feature is required
+ and/or after it is known that the upgraded release does not need
+ to be downgraded.</para></note>
+ </listitem>
+ <listitem>
<para>When setting up the file system, enter:
<screen>conf_param $FSNAME.quota.mdt=$QUOTA_TYPE
conf_param $FSNAME.quota.ost=$QUOTA_TYPE</screen></para>
</listitem>
<listitem>
+ <para condition="l2D">(Optional) If upgrading an ldiskfs MDT formatted
+ prior to Lustre 2.13, the "wide striping" feature that allows files
+ to have more than 160 stripes and store other large xattrs was not
+ enabled by default. This feature can be enabled on existing MDTs
+ by running the following command on all MDT devices:
+ <screen>mds# tune2fs -O ea_inode /dev/<replaceable>mdtdev</replaceable></screen>
+ </para>
+ <para>For more information about wide striping, see
+ <xref xmlns:xlink="https://www.w3.org/1999/xlink"
+ linkend="wide_striping" />.</para>
+ </listitem>
+ <listitem>
<para>Start the Lustre file system by starting the components in the
order shown in the following steps:</para>
<orderedlist numeration="loweralpha">
</listitem>
</orderedlist>
</listitem>
+ <listitem>
+ <para condition='l27'>(Optional) If you are upgrading from a release before Lustre
+ 2.7, to enable OST FIDs to also store the OST index (to improve
+ reliability of LFSCK and debug messages), <emphasis>after</emphasis>
+ the OSTs are mounted run once on each OSS:
+ <screen>oss# lctl set_param osd-ldiskfs.*.osd_index_in_idif=1</screen>
+ </para>
+ <note><para>Enabling the <literal>index_in_idif</literal> feature will
+ prevent the OST from being used by older versions of Lustre, so it
+ should only be enabled once it is known there is no need for the
+ OST to be downgraded to an earlier release.</para></note>
+ </listitem>
+ <listitem>
+ <para>If a new MDT was added to the filesystem, the new MDT must be
+ attached into the namespace by creating one or more
+ <emphasis>new</emphasis> DNE subdirectories with the
+ <literal>lfs mkdir</literal> command that use the new MDT:
+<screen>
+client# lfs mkdir -i <replaceable>new_mdt_index /testfs/new_dir</replaceable>
+</screen>
+ </para>
+ <para condition='l28'>In Lustre 2.8 and later, it is possible to
+ split a new directory across multiple MDTs by creating it with
+ multiple stripes:
+<screen>
+client# lfs mkdir -c 2 <replaceable>/testfs/new_striped_dir</replaceable>
+</screen>
+ </para>
+ <para condition='l2D'>In Lustre 2.13 and later, it is possible to set
+ the default striping on <emphasis>existing</emphasis> directories
+ so that new remote subdirectories are created on less-full MDTs:
+<screen>
+client# lfs setdirstripe -c 1 -i -1 <replaceable>/testfs/some_dir</replaceable>
+</screen>
+ </para>
+ </listitem>
</orderedlist>
<note>
<para>The mounting order described in the steps above must be followed
<literal>-v</literal> option with
<literal>lfs quota</literal>.</para>
</note>
- <para condition="l24">
- The <literal>quotacheck</literal>, <literal>quotaon</literal> and <literal>quotaoff</literal>
- sub-commands were deprecated in the Lustre 2.4 release, and removed completely in
- the Lustre 2.8 release. See <xref linkend="enabling_disk_quotas"/> for details on
+ <para condition="l28">
+ The <literal>quotacheck</literal>, <literal>quotaon</literal> and
+ <literal>quotaoff</literal> sub-commands were deprecated in the
+ Lustre 2.4 release, and removed completely in the Lustre 2.8 release.
+ See <xref linkend="enabling_disk_quotas"/> for details on
configuring and checking quotas.
</para>
</section>
<xsl:with-param name='content' select="$content"/>
</xsl:call-template>
</xsl:when>
+ <xsl:when test="@condition = 'l2E'">
+ <xsl:call-template name='textdecoration-1'>
+ <xsl:with-param name='version' select="'Introduced in Lustre 2.14'"/>
+ <xsl:with-param name='content' select="$content"/>
+ </xsl:call-template>
+ </xsl:when>
+ <xsl:when test="@condition = 'l2F'">
+ <xsl:call-template name='textdecoration-1'>
+ <xsl:with-param name='version' select="'Introduced in Lustre 2.15'"/>
+ <xsl:with-param name='content' select="$content"/>
+ </xsl:call-template>
+ </xsl:when>
+ <xsl:when test="@condition = 'l2G'">
+ <xsl:call-template name='textdecoration-1'>
+ <xsl:with-param name='version' select="'Introduced in Lustre 2.16'"/>
+ <xsl:with-param name='content' select="$content"/>
+ </xsl:call-template>
+ </xsl:when>
<xsl:when test="@condition != ''">
<xsl:call-template name='textdecoration-1'>
- <xsl:with-param name='version' select="'Introduced before Lustre 2.3'"/>
+ <xsl:with-param name='version' select="'Introduced before Lustre 2.5'"/>
<xsl:with-param name='content' select="$content"/>
</xsl:call-template>
</xsl:when>
<xsl:when test="$condition = 'l2D'">
<span class='floatright'>L 2.13 </span>
</xsl:when>
+ <xsl:when test="$condition = 'l2E'">
+ <span class='floatright'>L 2.14 </span>
+ </xsl:when>
+ <xsl:when test="$condition = 'l2F'">
+ <span class='floatright'>L 2.15 </span>
+ </xsl:when>
+ <xsl:when test="$condition = 'l2G'">
+ <span class='floatright'>L 2.16 </span>
+ </xsl:when>
<xsl:when test="$condition != ''">
<span class='floatright'>L ?.? </span>
</xsl:when>
</xsl:variable>
<xsl:variable name="versionstr">
<xsl:choose>
- <xsl:when test="@condition = 'l24'">Introduced in Lustre 2.4</xsl:when>
<xsl:when test="@condition = 'l25'">Introduced in Lustre 2.5</xsl:when>
<xsl:when test="@condition = 'l26'">Introduced in Lustre 2.6</xsl:when>
<xsl:when test="@condition = 'l27'">Introduced in Lustre 2.7</xsl:when>
<xsl:when test="@condition = 'l2E'">Introduced in Lustre 2.14</xsl:when>
<xsl:when test="@condition = 'l2F'">Introduced in Lustre 2.15</xsl:when>
<xsl:when test="@condition = 'l2G'">Introduced in Lustre 2.16</xsl:when>
- <xsl:otherwise>Documentation Error: unrecognised condition attribute</xsl:otherwise>
+ <xsl:otherwise>Introduced in before Lustre 2.5</xsl:otherwise>
</xsl:choose>
</xsl:variable>
<xsl:choose>
</xsl:variable>
<xsl:variable name="lustrecond">
<xsl:choose>
- <xsl:when test="@condition='l23'">L 2.3</xsl:when>
- <xsl:when test="@condition='l24'">L 2.4</xsl:when>
<xsl:when test="@condition='l25'">L 2.5</xsl:when>
<xsl:when test="@condition='l26'">L 2.6</xsl:when>
<xsl:when test="@condition='l27'">L 2.7</xsl:when>
<xsl:when test="@condition='l2B'">L 2.11</xsl:when>
<xsl:when test="@condition='l2C'">L 2.12</xsl:when>
<xsl:when test="@condition='l2D'">L 2.13</xsl:when>
+ <xsl:when test="@condition='l2E'">L 2.14</xsl:when>
+ <xsl:when test="@condition='l2F'">L 2.15</xsl:when>
+ <xsl:when test="@condition='l2G'">L 2.16</xsl:when>
<xsl:otherwise></xsl:otherwise>
</xsl:choose>
</xsl:variable>
git://git.whamcloud.com / doc / manual.git / commitdiff