--- /dev/null
+.DS_Store
+*.orig
+*.rej
+lustre_manual.fo
+lustre_manual.html
+lustre_manual.pdf
+lustre_manual.xhtml
expensive storage than the actual MDT device(s) since it only needs to
have good streaming read/write speed instead of high random IOPS.</para>
</note>
- <warning condition='l23'>
- <para>In Lustre software release 2.0 through 2.2, the only successful
- way to backup and restore an MDT is to do a device-level backup as is
- described in this section. File-level restore of an MDT is not possible
- before Lustre software release 2.3, as the Object Index (OI) file cannot
- be rebuilt after restore without the OI Scrub functionality.
- Since Lustre 2.3, Object Index files are automatically rebuilt at first
- mount after a restore is detected (see
- <link xl:href="http://jira.whamcloud.com/browse/LU-957">LU-957</link>),
- and file-level backup is supported (see
- <xref linkend="backup_fs_level"/>).</para>
- </warning>
<para>If hardware replacement is the reason for the backup or if a spare
storage device is available, it is possible to do a raw copy of the MDT or
OST from one block device to the other, as long as the new device is at
<para>Even in the face of hardware errors, the <literal>ldiskfs</literal>
file system is very robust and it may be possible
to recover the file system data after running
- <literal>e2fsck -fy /dev/{newdev}</literal> on the new device, along with
- <literal>ll_recover_lost_found_objs</literal> for OST devices.</para>
- <para condition="l26">With Lustre software version 2.6 and later, there is
- no longer a need to run
- <literal>ll_recover_lost_found_objs</literal> on the OSTs, since the
+ <literal>e2fsck -fy /dev/{newdev}</literal> on the new device.</para>
+ <para>With Lustre software version 2.6 and later, the
<literal>LFSCK</literal> scanning will automatically move objects from
<literal>lost+found</literal> back into its correct location on the OST
after directory corruption.</para>
it is the preferred method for migration of OST devices, especially when it
is desirable to reformat the underlying file system with different
configuration options or to reduce fragmentation.</para>
- <note><para>
- <emphasis role="bold">Prior to Lustre software release 2.3</emphasis>, the
- only successful way to perform an MDT backup and restore was to do a
- device-level backup as described in
- <xref linkend="dbdoclet.backup_device" />. The ability to do MDT
- file-level backups is not available for Lustre software release 2.0
- through 2.2, because restoration of the Object Index (OI) file does not
- return the MDT to a functioning state.</para>
- <para><emphasis role="bold">Since Lustre software release 2.3</emphasis>,
- Object Index files are automatically rebuilt at first mount after a
- restore is detected (see
- <link xl:href="http://jira.whamcloud.com/browse/LU-957">LU-957</link>),
- so file-level MDT restore is supported.</para></note>
+ <note>
+ <para>Since Lustre stores internal metadata that maps FIDs to local
+ inode numbers via the Object Index file, they need to be rebuilt at
+ first mount after a restore is detected so that file-level MDT backup
+ and restore is supported. The OI Scrub rebuilds these automatically
+ at first mount after a restore is detected, which may affect MDT
+ performance after mount until the rebuild is completed. Progress can
+ be monitored via <literal>lctl get_param osd-*.*.oi_scrub</literal>
+ on the MDS or OSS node where the target filesystem was restored.
+ </para>
+ </note>
<section xml:id="backup_fs_level.index_objects" condition="l2B">
<title>
<indexterm>
before the unmount of the target (MDT or OST) in order to be able to
restore the file system successfully. To enable index backup on the
target, execute the following command on the target server:</para>
- <screen># lctl set_param osd-zfs.${fsname}-${target}.index_backup=1</screen>
+ <screen># lctl set_param osd-*.${fsname}-${target}.index_backup=1</screen>
<para><replaceable>${target}</replaceable> is composed of the target type
(MDT or OST) plus the target index, such as <literal>MDT0000</literal>,
<literal>OST0001</literal>, and so on.</para>
<primary>backup</primary>
<secondary>OST and MDT</secondary>
</indexterm>Backing Up an OST or MDT</title>
- <para>For Lustre software release 2.3 and newer with MDT file-level backup
- support, substitute <literal>mdt</literal> for <literal>ost</literal>
- in the instructions below.</para>
+ <para>If backing up an MDT, substitute <literal>mdt</literal> for
+ <literal>ost</literal> in the instructions below.</para>
<orderedlist>
<listitem>
<para><emphasis role="bold">Umount the target</emphasis></para>
<title><indexterm><primary>benchmarking</primary>
<secondary>MDS performance</secondary></indexterm>
Testing MDS Performance (<literal>mds-survey</literal>)</title>
- <para><literal>mds-survey</literal> is available in Lustre software release
- 2.2 and beyond. The <literal>mds-survey</literal> script tests the local
- metadata performance using the echo_client to drive different layers of the
- MDS stack: mdd, mdt, osd (the Lustre software only supports mdd stack). It
- can be used with the following classes of operations:</para>
+ <para>The <literal>mds-survey</literal> script tests the local
+ metadata performance using the echo_client to drive different layers of
+ the MDS stack: mdd, mdt, osd (the Lustre software only supports mdd
+ stack). It can be used with the following classes of operations:</para>
<itemizedlist>
<listitem>
<para><literal>Open-create/mkdir/create</literal></para>
</orderedlist>
</section>
</section>
-</chapter>
\ No newline at end of file
+</chapter>
</note>
</listitem>
<listitem xml:id="dbdoclet.addmdtindex">
- <para>Optional for Lustre software release 2.4 and later.
- Add in additional MDTs.</para>
+ <para>Optionally add in additional MDTs.</para>
<screen>
mkfs.lustre --fsname=
<replaceable>fsname</replaceable> --mgsnode=
<literal>lctl</literal> commands (post-mount).</para>
</listitem>
<listitem>
-
<para>The quota feature in Lustre software is distributed
+ <para>The quota feature in Lustre software is distributed
throughout the system (as the Lustre file system is a distributed file
system). Because of this, quota setup and behavior on Lustre is
- different from local disk quotas in the following ways:</para>
+ somewhat different from local disk quotas in the following ways:</para>
<itemizedlist>
<listitem>
<para>No single point of administration: some commands must be
<listitem>
<para>Accuracy: quota information is distributed throughout the file
system and can only be accurately calculated with a quiescent file
- system.</para>
+ system in order to minimize performance overhead during normal use.
+ </para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para>Client does not set the
<literal>usrquota</literal> or
- <literal>grpquota</literal> options to mount. As of Lustre software
- release 2.4, space accounting is always enabled by default and quota
- enforcement can be enabled/disabled on a per-file system basis with
- <literal>lctl conf_param</literal>. It is worth noting that both
- <literal>lfs quotaon</literal> and
- <literal>quota_type</literal> are deprecated as of Lustre software
- release 2.4.0.</para>
+ <literal>grpquota</literal> options to mount. Space accounting is
+ enabled by default and quota enforcement can be enabled/disabled on
+ a per-filesystem basis with <literal>lctl conf_param</literal>.</para>
</listitem>
</itemizedlist>
<caution>
responsible for management and enforcement. The back-end file
system is responsible for resource usage and accounting. Because of
this, it is necessary to begin enabling quotas by enabling quotas on the
- back-end disk system. Because quota setup is dependent on the Lustre
- software version in use, you may first need to run
- <literal>lctl get_param version</literal> to identify
- <xref linkend="whichversion"/> you are currently using.
+ back-end disk system.
</para>
- <section remap="h3" condition="l24" xml:id="enabling_disk_quota_after24">
- <title>Enabling Disk Quotas (Lustre Software Release 2.4 and
- later)</title>
<caution>
<para>Quota setup is orchestrated by the MGS and <emphasis>all setup
commands in this section must be run directly on the MGS</emphasis>.
enables the QUOTA feature flag in the superblock which turns quota
accounting on at mount time automatically. e2fsck was also modified
to fix the quota files when the QUOTA feature flag is present. The
- project quota feature is disabled by default, and
+ project quota feature is disabled by default, and
<literal>tune2fs</literal> needs to be run to enable every target
manually.</para>
</listitem>
</listitem>
</itemizedlist>
<note>
- <para>Lustre file systems formatted with a Lustre release prior to 2.4.0
- can be still safely upgraded to release 2.4.0, but will not have
- functional space usage report until
- <literal>tunefs.lustre --quota</literal> is run against all targets. This
- command sets the QUOTA feature flag in the superblock and runs e2fsck (as
- a result, the target must be offline) to build the per-UID/GID disk usage
- database.</para>
<para condition="l2A">Lustre filesystems formatted with a Lustre release
prior to 2.10 can be still safely upgraded to release 2.10, but will not
have project quota usage reporting functional until
considerations.</para>
</note>
<caution>
- <para>Lustre software release 2.4 and later requires a version of
- e2fsprogs that supports quota (i.e. newer or equal to 1.42.13.wc5,
- 1.42.13.wc6 or newer is needed for project quota support) to be
- installed on the server nodes using ldiskfs backend (e2fsprogs is not
- needed with ZFS backend). In general, we recommend to use the latest
- e2fsprogs version available on
- <link xl:href="http://downloads.whamcloud.com/e2fsprogs/">
+ <para>Lustre requires a version of e2fsprogs that supports quota
+ to be installed on the server nodes when using the ldiskfs backend
+ (e2fsprogs is not needed with ZFS backend). In general, we recommend
+ to use the latest e2fsprogs version available on
+ <link xl:href="http://downloads.whamcloud.com/e2fsprogs/">
http://downloads.whamcloud.com/public/e2fsprogs/</link>.</para>
<para>The ldiskfs OSD relies on the standard Linux quota to maintain
accounting information on disk. As a consequence, the Linux kernel
<literal>CONFIG_QUOTACTL</literal> and
<literal>CONFIG_QFMT_V2</literal> enabled.</para>
</caution>
- <para>As of Lustre software release 2.4.0, quota enforcement is thus
- turned on/off independently of space accounting which is always enabled.
- <literal>lfs quota
- <replaceable>on|off</replaceable></literal> as well as the per-target
- <literal>quota_type</literal> parameter are deprecated in favor of a
- single per-file system quota parameter controlling inode/block quota
- enforcement. Like all permanent parameters, this quota parameter can be
- set via
- <literal>lctl conf_param</literal> on the MGS via the following
- syntax:</para>
+ <para>Quota enforcement is turned on/off independently of space
+ accounting which is always enabled. There is a single per-file
+ system quota parameter controlling inode/block quota enforcement.
+ Like all permanent parameters, this quota parameter can be set via
+ <literal>lctl conf_param</literal> on the MGS via the command:</para>
<screen>
lctl conf_param <replaceable>fsname</replaceable>.quota.<replaceable>ost|mdt</replaceable>=<replaceable>u|g|p|ugp|none</replaceable>
</screen>
<para>To turn on user, group, and project quotas for block only on
file system
<literal>testfs1</literal>, <emphasis>on the MGS</emphasis> run:</para>
- <screen>$ lctl conf_param testfs1.quota.ost=ugp
-</screen>
+ <screen>mgs# lctl conf_param testfs1.quota.ost=ugp </screen>
<para>To turn on group quotas for inodes on file system
<literal>testfs2</literal>, on the MGS run:</para>
- <screen>$ lctl conf_param testfs2.quota.mdt=g
-</screen>
+ <screen>mgs# lctl conf_param testfs2.quota.mdt=g </screen>
<para>To turn off user, group, and project quotas for both inode and block
on file system
<literal>testfs3</literal>, on the MGS run:</para>
- <screen>$ lctl conf_param testfs3.quota.ost=none
-</screen>
- <screen>$ lctl conf_param testfs3.quota.mdt=none
-</screen>
+ <screen>mgs# lctl conf_param testfs3.quota.ost=none</screen>
+ <screen>mgs# lctl conf_param testfs3.quota.mdt=none</screen>
<section xml:id="quota_verification">
- <title>
- <indexterm>
-
<primary>Quotas</primary>
- <secondary>verifying</secondary>
- </indexterm>Quota Verification</title>
-
<para>Once the quota parameters have been configured, all targets
+ <title>
+ <indexterm>
+ <primary>Quotas</primary>
+ <secondary>verifying</secondary>
+ </indexterm>Quota Verification</title>
+ <para>Once the quota parameters have been configured, all targets
which are part of the file system will be automatically notified of the
new quota settings and enable/disable quota enforcement as needed. The
per-target enforcement status can still be verified by running the
group uptodate: glb[1],slv[1],reint[0]
</screen>
</section>
- </section>
</section>
<section xml:id="quota_administration">
<title>
<primary>Quotas</primary>
<secondary>creating</secondary>
</indexterm>Quota Administration</title>
-
<para>Once the file system is up and running, quota limits on blocks
+ <para>Once the file system is up and running, quota limits on blocks
and inodes can be set for user, group, and project. This is <emphasis>
controlled entirely from a client</emphasis> via three quota
parameters:</para>
<screen>
lctl get_param osd-*.*.quota_slave.limit*
</screen></para>
- <note>
- <para>Prior to 2.4, global quota limits used to be stored in
- administrative quota files using the on-disk format of the linux quota
- file. When upgrading MDT0000 to 2.4, those administrative quota files are
- converted into IAM indexes automatically, conserving existing quota
- limits previously set by the administrator.</para>
- </note>
</section>
<section condition='l2C' xml:id="default_quota">
<title>
<primary>Quotas</primary>
<secondary>Interoperability</secondary>
</indexterm>Quotas and Version Interoperability</title>
- <para>The new quota protocol introduced in Lustre software release 2.4.0
- <emphasis role="bold">is not compatible</emphasis> with previous
- versions. As a consequence,
- <emphasis role="bold">all Lustre servers must be upgraded to release 2.4.0
- for quota to be functional</emphasis>. Quota limits set on the Lustre file
- system prior to the upgrade will be automatically migrated to the new quota
- index format. As for accounting information with ldiskfs backend, they will
- be regenerated by running
- <literal>tunefs.lustre --quota</literal> against all targets. It is worth
- noting that running
- <literal>tunefs.lustre --quota</literal> is
- <emphasis role="bold">mandatory</emphasis> for all targets formatted with a
- Lustre software release older than release 2.4.0, otherwise quota
- enforcement as well as accounting won't be functional.</para>
- <para>Besides, the quota protocol in release 2.4 takes for granted that the
- Lustre client supports the
- <literal>OBD_CONNECT_EINPROGRESS</literal> connect flag. Clients supporting
- this flag will retry indefinitely when the server returns
- <literal>EINPROGRESS</literal> in a reply. Here is the list of Lustre client
- version which are compatible with release 2.4:</para>
- <itemizedlist>
- <listitem>
- <para>Release 2.3-based clients and later</para>
- </listitem>
- <listitem>
- <para>Release 1.8 clients newer or equal to release 1.8.9-wc1</para>
- </listitem>
- <listitem>
- <para>Release 2.1 clients newer or equal to release 2.1.4</para>
- </listitem>
- </itemizedlist>
<para condition="l2A">To use the project quota functionality introduced in
Lustre 2.10, <emphasis role="bold">all Lustre servers and clients must be
upgraded to Lustre release 2.10 or later for project quota to work
clients and not be accounted for on OSTs. Furthermore, the
<emphasis role="bold">servers may be required to use a patched kernel,
</emphasis> for more information see
- <xref linkend="enabling_disk_quota_after24"/>.</para>
+ <xref linkend="enabling_disk_quotas"/>.</para>
</section>
<section xml:id="granted_cache_and_quota_limits">
<title>
</section>
<section remap="h3">
<title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>external journal</tertiary></indexterm>Choosing Parameters for an External Journal</title>
- <para>If you have configured a RAID array and use it directly as an OST, it contains both data and metadata. For better performance, we recommend putting the OST journal on a separate device, by creating a small RAID 1 array and using it as an external journal for the OST.</para>
- <para>In a Lustre file system, the default journal size is 400 MB. A journal size of up to 1
- GB has shown increased performance but diminishing returns are seen for larger journals.
- Additionally, a copy of the journal is kept in RAM. Therefore, make sure you have enough
- memory available to hold copies of all the journals.</para>
+ <para>If you have configured a RAID array and use it directly as an OST,
+ it contains both data and metadata. For better performance, we
+ recommend putting the OST journal on a separate device, by creating a
+ small RAID 1 array and using it as an external journal for the OST.
+ </para>
+ <para>In a typical Lustre file system, the default OST journal size is
+ up to 1GB, and the default MDT journal size is up to 4GB, in order to
+ handle a high transaction rate without blocking on journal flushes.
+ Additionally, a copy of the journal is kept in RAM. Therefore, make
+ sure you have enough RAM on the servers to hold copies of all journals.
+ </para>
<para>The file system journal options are specified to <literal>mkfs.lustre</literal> using
the <literal>--mkfsoptions</literal> parameter. For example:</para>
<screen>--mkfsoptions "<replaceable>other_options</replaceable> -j -J device=/dev/mdJ" </screen>
</glossdiv>
<glossdiv>
<title>D</title>
- <glossentry xml:id="defaultstrippattern">
+ <glossentry xml:id="defaultstripepattern">
<glossterm>Default stripe pattern</glossterm>
<glossdef>
<para>Information in the LOV descriptor that describes the default
new subdirectories at the time they are created.</para>
</glossdef>
</glossentry>
- <glossentry xml:id="DNE" condition='l24'>
+ <glossentry xml:id="DNE">
<glossterm>Distributed namespace (DNE)</glossterm>
<glossdef>
<para>A collection of metadata targets serving a single file
- system namespace. Prior to DNE, Lustre file systems were limited to a
- single metadata target for the entire name space. Without the ability
- to distribute metadata load over multiple targets, Lustre file system
- performance is limited. Lustre was enhanced with DNE functionality in
- two development phases. After completing the first phase of development
- in Lustre software version 2.4, <emphasis>Remote Directories</emphasis>
- allows the metadata for sub-directories to be serviced by an
- independent MDT(s). After completing the second phase of development in
- Lustre software version 2.8, <emphasis>Striped Directories</emphasis>
- allows files in a single directory to be serviced by multiple MDTs.
+ system namespace. Without DNE, Lustre file systems are limited to a
+ single metadata target for the entire name space. Without the ability
+ to distribute metadata load over multiple targets, Lustre file system
+ performance may be limited. The DNE functionality has two types of
+ scalability. <emphasis>Remote Directories</emphasis> (DNE1) allows
+ sub-directories to be serviced by an independent MDT(s), increasing
+ aggregate metadata capacity and performance for independent sub-trees
+ of the filesystem. This also allows performance isolation of workloads
+ running in a specific sub-directory on one MDT from workloads on other
+ MDTs. In Lustre 2.8 and later <emphasis>Striped Directories</emphasis>
+ (DNE2) allows a single directory to be serviced by multiple MDTs.
</para>
</glossdef>
</glossentry>
the file data.</para>
</glossdef>
</glossentry>
- <glossentry xml:id="mdt0" condition='l24'>
- <glossterm>MDT0</glossterm>
+ <glossentry xml:id="mdt0">
+ <glossterm>MDT0000</glossterm>
<glossdef>
- <para>The metadata target for the file system root. Since Lustre
- software release 2.4, multiple metadata targets are possible in the
- same file system. MDT0 is the root of the file system, which must be
- available for the file system to be accessible.</para>
+ <para>The metadata target storing the file system root directory, as
+ well as some core services such as quota tables. Multiple metadata
+ targets are possible in the same file system. MDT0000 must be
+ available for the file system to be accessible.</para>
</glossdef>
</glossentry>
<glossentry xml:id="mgs">
Examples include MDC, OSC, LOV, MDT, and OST.</para>
</glossdef>
</glossentry>
- <glossentry xml:id="odbapi">
- <glossterm>OBD API</glossterm>
- <glossdef>
- <para>The programming interface for configuring OBD devices. This was
- formerly also the API for accessing object IO and attribute methods on
- both the client and server, but has been replaced by the OSD API in
- most parts of the code.</para>
- </glossdef>
- </glossentry>
<glossentry xml:id="odbtype">
<glossterm>OBD type</glossterm>
<glossdef>
Examples of OBD types include the LOV, OSC and OSD.</para>
</glossdef>
</glossentry>
- <glossentry xml:id="odbfilter">
- <glossterm>Obdfilter</glossterm>
- <glossdef>
- <para>An older name for the OBD API data object operation device driver
- that sits between the OST and the OSD. In Lustre software release 2.4
- this device has been renamed OFD."</para>
- </glossdef>
- </glossentry>
<glossentry xml:id="objectstorage">
<glossterm>Object storage</glossterm>
<glossdef>
server restarts.</para>
</glossdef>
</glossentry>
- <glossentry xml:id="remotedirectories" condition='l24'>
+ <glossentry xml:id="remotedirectories">
<glossterm>Remote directory</glossterm>
<glossdef>
- <para>A remote directory describes a feature of
- Lustre where metadata for files in a given directory may be
- stored on a different MDT than the metadata for the parent
- directory. Remote directories only became possible with the
- advent of DNE Phase 1, which arrived in Lustre version
- 2.4. Remote directories are available to system
- administrators who wish to provide individual metadata
- targets for individual workloads.</para>
+ <para>A remote directory describes a feature of Lustre where metadata
+ for files in a given directory may be stored on a different MDT than
+ the metadata for the parent directory. This is sometimes referred
+ to as DNE1.</para>
</glossdef>
</glossentry>
<glossentry xml:id="replay">
<glossentry xml:id="stripeddirectory" condition='l28'>
<glossterm>Striped Directory</glossterm>
<glossdef>
- <para>A striped directory is a feature of Lustre
- software where metadata for files in a given directory are
- distributed evenly over multiple MDTs. Striped directories
+ <para>A striped directory is when metadata for files in a given
+ directory are distributed evenly over multiple MDTs. Striped directories
are only available in Lustre software version 2.8 or later.
- An administrator can use a striped directory to increase
- metadata performance by distributing the metadata requests
- in a single directory over two or more MDTs.</para>
+ A user can create a striped directory to increase metadata
+ performance of large directories by distributing the metadata
+ requests in a single directory over two or more MDTs.</para>
</glossdef>
</glossentry>
<glossentry xml:id="stridesize">
<entry>
<code>lustre-tests-<replaceable>ver</replaceable>_lustre.<replaceable>arch</replaceable></code>
</entry>
- <entry>Lustre I/O Kit benchmarking tools
- <emphasis role="italic">(Included in Lustre software as of
- release 2.2)</emphasis></entry>
+ <entry>Scripts and programs used for running regression
+ tests for Lustre, but likely only of interest to
+ Lustre developers or testers.
+ </entry>
</row>
</tbody>
</tgroup>
<section xml:id="lustremaint.changingservernid">
<title><indexterm><primary>maintenance</primary><secondary>changing a NID</secondary></indexterm>
Changing a Server NID</title>
- <para>In Lustre software release 2.3 or earlier, the <literal>tunefs.lustre
- --writeconf</literal> command is used to rewrite all of the configuration files.</para>
- <para condition="l24">If you need to change the NID on the MDT or OST, a new
- <literal>replace_nids</literal> command was added in Lustre software release 2.4 to simplify
- this process. The <literal>replace_nids</literal> command differs from <literal>tunefs.lustre
- --writeconf</literal> in that it does not erase the entire configuration log, precluding the
- need the need to execute the <literal>writeconf</literal> command on all servers and
- re-specify all permanent parameter settings. However, the <literal>writeconf</literal> command
- can still be used if desired.</para>
+ <para>In order to totally rewrite the Lustre configuration, the
+ <literal>tunefs.lustre --writeconf</literal> command is used to
+ rewrite all of the configuration files.</para>
+ <para>If you need to change only the NID of the MDT or OST, the
+ <literal>replace_nids</literal> command can simplify this process.
+ The <literal>replace_nids</literal> command differs from
+ <literal>tunefs.lustre --writeconf</literal> in that it does not
+ erase the entire configuration log, precluding the need the need to
+ execute the <literal>writeconf</literal> command on all servers and
+ re-specify all permanent parameter settings. However, the
+ <literal>writeconf</literal> command can still be used if desired.
+ </para>
<para>Change a server NID in these situations:</para>
<itemizedlist>
<listitem>
</listitem>
</orderedlist>
</section>
- <section xml:id="lustremaint.adding_new_mdt" condition='l24'>
+ <section xml:id="lustremaint.adding_new_mdt">
<title><indexterm>
<primary>maintenance</primary>
<secondary>adding an MDT</secondary>
user or application workloads from other users of the filesystem. It
is possible to have multiple remote sub-directories reference the
same MDT. However, the root directory will always be located on
- MDT0. To add a new MDT into the file system:</para>
+ MDT0000. To add a new MDT into the file system:</para>
<orderedlist>
<listitem>
<para>Discover the maximum MDT index. Each MDT must have unique index.</para>
desire to continue using the filesystem before it is repaired.</para>
</listitem>
</itemizedlist>
- <section condition="l24" xml:id="lustremaint.rmremotedir">
+ <section xml:id="lustremaint.rmremotedir">
<title><indexterm><primary>maintenance</primary><secondary>removing an MDT</secondary></indexterm>Removing an MDT from the File System</title>
<para>If the MDT is permanently inaccessible,
<literal>lfs rm_entry {directory}</literal> can be used to delete the
<para>The <literal>lfs getstripe --mdt-index</literal> command
returns the index of the MDT that is serving the given directory.</para>
</section>
- <section xml:id="lustremaint.inactivemdt" condition='l24'>
+ <section xml:id="lustremaint.inactivemdt">
<title>
<indexterm><primary>maintenance</primary></indexterm>
<indexterm><primary>maintenance</primary><secondary>inactive MDTs</secondary></indexterm>Working with Inactive MDTs</title>
<indexterm><primary>monitoring</primary><secondary>jobstats</secondary></indexterm>
Lustre Jobstats</title>
- <para>The Lustre jobstats feature is available starting in Lustre software
- release 2.3. It collects file system operation statistics for user processes
- running on Lustre clients, and exposes them via procfs on the server using
- the unique Job Identifier (JobID) provided by the job scheduler for each
- job. Job schedulers known to be able to work with jobstats include:
+ <para>The Lustre jobstats feature collects file system operation statistics
+ for user processes running on Lustre clients, and exposes on the server
+ using the unique Job Identifier (JobID) provided by the job scheduler for
+ each job. Job schedulers known to be able to work with jobstats include:
SLURM, SGE, LSF, Loadleveler, PBS and Maui/MOAB.</para>
<para>Since jobstats is implemented in a scheduler-agnostic manner, it is
likely that it will be able to work with other schedulers also, and also
<listitem>
<para>Mount the MDT.</para>
<note>
- <para condition='l24'>Mount all MDTs if multiple MDTs are
- present.</para>
+ <para>Mount all MDTs if multiple MDTs are present.</para>
</note>
</listitem>
<listitem>
client# mount -t lustre mgsnode@tcp0:/bar /mnt/bar
</screen>
</section>
- <section xml:id="dbdoclet.lfsmkdir" condition='l24'>
+ <section xml:id="dbdoclet.lfsmkdir">
<title>
<indexterm>
<primary>operations</primary>
<secondary>remote directory</secondary>
- </indexterm>Creating a sub-directory on a given MDT</title>
- <para>Lustre 2.4 enables individual sub-directories to be serviced by
- unique MDTs. An administrator can allocate a sub-directory to a given MDT
- using the command:</para>
+ </indexterm>Creating a sub-directory on a specific MDT</title>
+ <para>It is possible to create individual directories, along with its
+ files and sub-directories, to be stored on specific MDTs. To create
+ a sub-directory on a given MDT use the command:
+ </para>
<screen>
client# lfs mkdir –i
<replaceable>mdt_index</replaceable>
<warning>
<para>An administrator can allocate remote sub-directories to separate
MDTs. Creating remote sub-directories in parent directories not hosted on
- MDT0 is not recommended. This is because the failure of the parent MDT
+ MDT0000 is not recommended. This is because the failure of the parent MDT
will leave the namespace below it inaccessible. For this reason, by
- default it is only possible to create remote sub-directories off MDT0. To
- relax this restriction and enable remote sub-directories off any MDT, an
- administrator must issue the following command on the MGS:
+ default it is only possible to create remote sub-directories off MDT0000.
+ To relax this restriction and enable remote sub-directories off any MDT,
+ an administrator must issue the following command on the MGS:
<screen>mgs# lctl conf_param <replaceable>fsname</replaceable>.mdt.enable_remote_dir=1</screen>
For Lustre filesystem 'scratch', the command executed is:
<screen>mgs# lctl conf_param scratch.mdt.enable_remote_dir=1</screen>
<literal>enable_remote_dir_gid</literal>. For example, setting this
parameter to the 'wheel' or 'admin' group ID allows users with that GID
to create and delete remote and striped directories. Setting this
- parameter to <literal>-1</literal> on MDT0 to permanently allow any
+ parameter to <literal>-1</literal> on MDT0000 to permanently allow any
non-root users create and delete remote and striped directories.
On the MGS execute the following command:
<screen>mgs# lctl conf_param <replaceable>fsname</replaceable>.mdt.enable_remote_dir_gid=-1</screen>
software tries the first one, and if that fails, it tries the second
one.)</para>
<para>Two options to
- <literal>mkfs.lustre</literal> can be used to specify failover nodes.
- Introduced in Lustre software release 2.0, the
+ <literal>mkfs.lustre</literal> can be used to specify failover nodes. The
<literal>--servicenode</literal> option is used to specify all service NIDs,
including those for primary nodes and failover nodes. When the
<literal>--servicenode</literal> option is used, the first service node to
load the target device becomes the primary service node, while nodes
corresponding to the other specified NIDs become failover locations for the
- target device. An older option,
- <literal>--failnode</literal>, specifies just the NIDS of failover nodes.
- For more information about the
+ target device. An older option, <literal>--failnode</literal>, specifies
+ just the NIDs of failover nodes. For more information about the
<literal>--servicenode</literal> and
<literal>--failnode</literal> options, see
<xref xmlns:xlink="http://www.w3.org/1999/xlink"
</listitem>
<listitem>
<para><literal>llite.<replaceable>fsname_instance</replaceable>.max_cache_mb</literal>
- - Maximum amount of inactive data cached by the client. The
+ - Maximum amount of read+write data cached by the client. The
default value is 3/4 of the client RAM.
</para>
</listitem>
<section>
<title>Interpreting OST Statistics</title>
<note>
- <para>See also <xref linkend="dbdoclet.50438219_84890"/> (<literal>llobdstat</literal>) and
+ <para>See also
<xref linkend="dbdoclet.50438273_80593"/> (<literal>collectl</literal>).</para>
</note>
<para>OST <literal>stats</literal> files can be used to provide statistics showing activity
<section>
<title>Interpreting MDT Statistics</title>
<note>
- <para>See also <xref linkend="dbdoclet.50438219_84890"/> (<literal>llobdstat</literal>) and
+ <para>See also
<xref linkend="dbdoclet.50438273_80593"/> (<literal>collectl</literal>).</para>
</note>
<para>MDT <literal>stats</literal> files can be used to track MDT
<para> Transient network partition</para>
</listitem>
</itemizedlist>
- <para>For Lustre software release 2.1.x and all earlier releases, all Lustre file system failure
- and recovery operations are based on the concept of connection failure; all imports or exports
- associated with a given connection are considered to fail if any of them fail. Lustre software
- release 2.2.x adds the <xref linkend="imperativerecovery"/> feature which enables the MGS to
- actively inform clients when a target restarts after a failure, failover or other
- interruption.</para>
- <para>For information on Lustre file system recovery, see <xref linkend="metadatereplay"/>. For
- information on recovering from a corrupt file system, see <xref linkend="commitonshare"/>. For
- information on resolving orphaned objects, a common issue after recovery, see <xref
- linkend="dbdoclet.50438225_13916"/>. For information on imperative recovery see <xref
- linkend="imperativerecovery"/>
+ <para>For Lustre, all Lustre file system failure and recovery operations
+ are based on the concept of connection failure; all imports or exports
+ associated with a given connection are considered to fail if any of
+ them fail. The <xref linkend="imperativerecovery"/> feature allows
+ the MGS to actively inform clients when a target restarts after a
+ failure, failover, or other interruption to speed up recovery.</para>
+ <para>For information on Lustre file system recovery, see
+ <xref linkend="metadatereplay"/>. For information on recovering from a
+ corrupt file system, see <xref linkend="commitonshare"/>. For
+ information on resolving orphaned objects, a common issue after recovery,
+ see <xref linkend="dbdoclet.50438225_13916"/>. For information on
+ imperative recovery see <xref linkend="imperativerecovery"/>
</para>
<section remap="h3">
<title><indexterm><primary>recovery</primary><secondary>client failure</secondary></indexterm>Client Failure</title>
at the time of MDS failure are permitted to reconnect during the recovery
window, to avoid the introduction of state changes that might conflict
with what is being replayed by previously-connected clients.</para>
- <para condition="l24">Lustre software release 2.4 introduces multiple
- metadata targets. If multiple MDTs are in use, active-active failover
+ <para>If multiple MDTs are in use, active-active failover
is possible (e.g. two MDS nodes, each actively serving one or more
different MDTs for the same filesystem). See
<xref linkend="dbdoclet.mdtactiveactive"/> for more information.</para>
</section>
<section xml:id="imperativerecovery">
<title><indexterm><primary>imperative recovery</primary></indexterm>Imperative Recovery</title>
- <para>Imperative Recovery (IR) was first introduced in Lustre software release 2.2.0.</para>
<para>Large-scale Lustre file system implementations have historically experienced problems
recovering in a timely manner after a server failure. This is due to the way that clients
detect the server failure and how the servers perform their recovery. Many of the processes
$ lctl get_param osc.testfs-OST0001-osc-*.import |grep instance
instance: 5
</screen>
- </section>
- </section>
- <section remap="h3" xml:id="imperativerecoveryrecomendations">
- <title><indexterm><primary>imperative recovery</primary><secondary>Configuration Suggestions</secondary></indexterm>Configuration Suggestions for Imperative Recovery</title>
-<para>We used to build the MGS and MDT0 on the same target to save a server node. However, to make
- IR work efficiently, we strongly recommend running the MGS node on a separate node for any
- significant Lustre file system installation. There are three main advantages of doing this: </para>
-<orderedlist>
-<listitem><para>Be able to notify clients if the MDT0 is dead</para></listitem>
-<listitem><para>Load balance. The load on the MDS may be very high which may make the MGS unable to notify the clients in time</para></listitem>
-<listitem><para>Safety. The MGS code is simpler and much smaller compared to the code of MDT. This means the chance of MGS down time due to a software bug is very low.</para></listitem>
-</orderedlist>
- </section>
+ </section>
+ </section>
+ <section remap="h3" xml:id="imperativerecoveryrecomendations">
+ <title><indexterm><primary>imperative recovery</primary><secondary>Configuration Suggestions</secondary></indexterm>Configuration Suggestions for Imperative Recovery</title>
+ <para>We used to build the MGS and MDT0000 on the same target to save
+ a server node. However, to make IR work efficiently, we strongly
+ recommend running the MGS node on a separate node for any
+ significant Lustre file system installation. There are three main
+ advantages of doing this: </para>
+ <orderedlist>
+ <listitem><para>Be able to notify clients when MDT0000 recovered.
+ </para></listitem>
+ <listitem><para>Improved load balance. The load on the MDS may be
+ very high which may make the MGS unable to notify the clients in
+ time.</para></listitem>
+ <listitem><para>Robustness. The MGS code is simpler and much smaller
+ compared to the MDS code. This means the chance of an MGS downtime
+ due to a software bug is very low.
+ </para></listitem>
+ </orderedlist>
+ </section>
</section>
<section xml:id="suppressingpings">
<title><indexterm><primary>suppressing pings</primary></indexterm>Suppressing Pings</title>
- <para>On clusters with large numbers of clients and OSTs, OBD_PING messages may impose
- significant performance overheads. As an intermediate solution before a more self-contained
- one is built, Lustre software release 2.4 introduces an option to suppress pings, allowing
- ping overheads to be considerably reduced. Before turning on this option, administrators
- should consider the following requirements and understand the trade-offs involved:</para>
+ <para>On clusters with large numbers of clients and OSTs,
+ <literal>OBD_PING</literal> messages may impose significant performance
+ overheads. There is an option to suppress pings, allowing ping overheads
+ to be considerably reduced. Before turning on this option, administrators
+ should consider the following requirements and understand the trade-offs
+ involved:</para>
<itemizedlist>
<listitem>
- <para>When suppressing pings, a target can not detect client deaths, since clients do not
- send pings that are only to keep their connections alive. Therefore, a mechanism external
- to the Lustre file system shall be set up to notify Lustre targets of client deaths in a
- timely manner, so that stale connections do not exist for too long and lock callbacks to
+ <para>When suppressing pings, a server cannot detect client deaths,
+ since clients do not send pings that are only to keep their
+ connections alive. Therefore, a mechanism external to the Lustre
+ file system shall be set up to notify Lustre targets of client
+ deaths in a timely manner, so that stale connections do not exist
+ for too long and lock callbacks to
dead clients do not always have to wait for timeouts.</para>
</listitem>
<listitem>
<literal>ksocklnd</literal> using the
<literal>nscheds</literal> parameter. This adjusts the number of threads for
each partition, not the overall number of threads on the LND.</para>
- <note>
- <para>Lustre software release 2.3 has greatly decreased the default
- number of threads for
- <literal>ko2iblnd</literal> and
- <literal>ksocklnd</literal> on high-core count machines. The current
- default values are automatically set and are chosen to work well across a
- number of typical scenarios.</para>
- </note>
<section>
<title>ko2iblnd Tuning</title>
<para>The following table outlines the ko2iblnd module parameters to be used
</informaltable>
</section>
</section>
- <section xml:id="dbdoclet.nrstuning" condition='l24'>
+ <section xml:id="dbdoclet.nrstuning">
<title>
<indexterm>
<primary>tuning</primary>
</section>
<section remap="h3">
<title>Ptlrpc Thread Pool</title>
- <para>Releases prior to Lustre software release 2.2 used two portal RPC
- daemons for each client/server pair. One daemon handled all synchronous
- IO requests, and the second daemon handled all asynchronous (non-IO)
- RPCs. The increasing use of large SMP nodes for Lustre servers exposed
- some scaling issues. The lack of threads for large SMP nodes resulted in
- cases where a single CPU would be 100% utilized and other CPUs would be
- relativity idle. This is especially noticeable when a single client
- traverses a large directory.</para>
- <para>Lustre software release 2.2.x implements a ptlrpc thread pool, so
+ <para>The increasing use of large SMP nodes for Lustre servers requires
+ good scaling of many application threads generating large amounts of IO.
+ Lustre implements a ptlrpc thread pool, so
that multiple threads can be created to serve asynchronous RPC requests.
The number of threads spawned is controlled at module load time using
module options. By default one thread is spawned per CPU, with a minimum
"even" servers with <literal>o2ib2</literal> on <literal>rail1</literal>.</para>
</section>
</section>
- <section xml:id="managinglnet.configuringroutes" condition='l24'>
+ <section xml:id="managinglnet.configuringroutes">
<title><indexterm><primary>LNet</primary></indexterm>Dynamically Configuring
LNet Routes</title>
<para>Two scripts are provided:
<primary>striping</primary>
<secondary>remote directories</secondary>
</indexterm>Locating the MDT for a remote directory</title>
- <para condition="l24">Lustre software release 2.4 can be configured with
- multiple MDTs in the same file system. Each sub-directory can have a
- different MDT. To identify on which MDT a given subdirectory is
- located, pass the <literal>getstripe [--mdt-index|-M]</literal>
- parameters to <literal>lfs</literal>. An example of this command is
- provided in the section <xref linkend="lustremaint.rmremotedir"/>.</para>
+ <para>Lustre can be configured with multiple MDTs in the same file
+ system. Each directory and file could be located on a different MDT.
+ To identify which MDT a given subdirectory is located, pass the
+ <literal>getstripe [--mdt-index|-M]</literal> parameter to
+ <literal>lfs</literal>. An example of this command is provided in
+ the section <xref linkend="lustremaint.rmremotedir"/>.</para>
</section>
</section>
<section xml:id="pfl" condition='l2A'>
<link
xl:href="http://wiki.lustre.org/Lustre_Manual_Changes">http://wiki.lustre.org/Lustre_Manual_Changes</link>.</para>
- <para>This manual currently covers all the 2.x Lustre software releases.
-Features that are specific to individual releases are identified within the
-table of contents using a short hand notation (i.e. 'L24' is a Lustre software
-release 2.4 specific feature), and within the text using a distinct box. For
-example:</para>
-
- <para condition='l24'>Lustre software release version 2.4 includes support
-for multiple metadata servers.</para>
+ <para condition="l28">This manual covers a range of Lustre 2.x software
+ releases. Features that are specific to individual releases are
+ identified within the table of contents using a short hand notation
+ (i.e. this paragraph is tagged for a Lustre 2.8 specific feature),
+ and within the text using a distinct box. For example:</para>
<note xml:id="whichversion"><title>which version?</title>
<indexterm><primary>version</primary>
<secondary>which version of Lustre am I running?</secondary></indexterm>
- <para>The current version of Lustre
- that is in use on the client can be found using the command
- <literal>lctl get_param version</literal>, for example:
+ <para>The current version of Lustre that is in use on the node can be found
+ using the command <literal>lctl get_param version</literal>, for example:
<screen>$ lctl get_param version
-version=
-lustre: 2.7.59
-kernel: patchless_client
-build: v2_7_59_0-g703195a-CHANGED-3.10.0.lustreopa</screen>
- </para></note>
+version=2.10.5
+</screen>
+ </para></note>
-
<para>Only the latest revision of this document is made readily available
-because changes are continually arriving. The current and latest revision of
-this manual is available from links maintained at: <link
-xl:href="http://lustre.opensfs.org/documentation/">http://lustre.opensfs.org/documentation/</link>.
+ <para>Only the latest revision of this document is made readily available
+ because changes are continually arriving. The current and latest revision
+ of this manual is available from links maintained at:
+ <link xl:href="http://lustre.opensfs.org/documentation/">http://lustre.opensfs.org/documentation/</link>.
<revhistory>
<revision>
<tbody>
<row>
<entry>
- <para> <literal>LUSTRE_Q_QUOTAON</literal></para>
- </entry>
- <entry>
- <para>Turns on quotas for a Lustre file system. Deprecated as of 2.4.0.
- <emphasis>qc_type</emphasis> is <literal>USRQUOTA</literal>,
- <literal>GRPQUOTA</literal> or <literal>UGQUOTA</literal> (both user and group
- quota). The quota files must exist. They are normally created with the
- <literal>llapi_quotacheck</literal> call. This call is restricted to the super
- user privilege. As of 2.4.0, quota is now enabled on a per file system basis via
- <literal>lctl conf_param</literal> (see <xref linkend="enabling_disk_quotas"/>)
- on the MGS node and quotacheck isn't needed any more.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal>LUSTRE_Q_QUOTAOFF</literal></para>
- </entry>
- <entry>
- <para>Turns off quotas for a Lustre file system. Deprecated as of 2.4.0. <emphasis>qc_type</emphasis> is <literal>USRQUOTA</literal>, <literal>GRPQUOTA</literal> or <literal>UGQUOTA</literal> (both user and group quota). This call is restricted to the super user privilege. As of 2.4.0, quota is disabled via <literal>lctl conf_param</literal> (see <xref linkend="enabling_disk_quotas"/>).</para>
- </entry>
- </row>
- <row>
- <entry>
<para> <literal>LUSTRE_Q_GETQUOTA</literal></para>
</entry>
<entry>
chance that even two disk failures can cause the loss of the whole MDT
device. The first failure disables an entire half of the mirror and the
second failure has a 50% chance of disabling the remaining mirror.</para>
- <para condition='l24'>If multiple MDTs are going to be present in the
+ <para>If multiple MDTs are going to be present in the
system, each MDT should be specified for the anticipated usage and load.
For details on how to add additional MDTs to the filesystem, see
<xref linkend="lustremaint.adding_new_mdt"/>.</para>
- <warning condition='l24'><para>MDT0 contains the root of the Lustre file
- system. If MDT0 is unavailable for any reason, the file system cannot be
- used.</para></warning>
- <note condition='l24'><para>Using the DNE feature it is possible to
- dedicate additional MDTs to sub-directories off the file system root
- directory stored on MDT0, or arbitrarily for lower-level subdirectories.
- using the <literal>lfs mkdir -i <replaceable>mdt_index</replaceable></literal> command.
- If an MDT serving a subdirectory becomes unavailable, any subdirectories
- on that MDT and all directories beneath it will also become inaccessible.
- Configuring multiple levels of MDTs is an experimental feature for the
- 2.4 release, and is fully functional in the 2.8 release. This is
- typically useful for top-level directories to assign different users
- or projects to separate MDTs, or to distribute other large working sets
- of files to multiple MDTs.</para></note>
+ <warning><para>MDT0000 contains the root of the Lustre file system. If
+ MDT0000 is unavailable for any reason, the file system cannot be used.
+ </para></warning>
+ <note><para>Using the DNE feature it is possible to dedicate additional
+ MDTs to sub-directories off the file system root directory stored on
+ MDT0000, or arbitrarily for lower-level subdirectories, using the
+ <literal>lfs mkdir -i <replaceable>mdt_index</replaceable></literal>
+ command. If an MDT serving a subdirectory becomes unavailable, any
+ subdirectories on that MDT and all directories beneath it will also
+ become inaccessible. This is typically useful for top-level directories
+ to assign different users or projects to separate MDTs, or to distribute
+ other large working sets of files to multiple MDTs.</para></note>
<note condition='l28'><para>Starting in the 2.8 release it is possible
to spread a single large directory across multiple MDTs using the DNE
striped directory feature by specifying multiple stripes (or shards)
data. This reserved space is unusable for general storage. Thus, at least
this much space will be used per OST before any file object data is saved.
</para>
- <para condition="l24">With a ZFS backing filesystem for the MDT or OST,
+ <para>With a ZFS backing filesystem for the MDT or OST,
the space allocation for inodes and file data is dynamic, and inodes are
allocated as needed. A minimum of 4kB of usable space (before mirroring)
is needed for each inode, exclusive of other overhead such as directories,
Inodes will be added approximately in proportion to space added.
</para>
</note>
- <note condition='l24'>
+ <note>
<para>Note that the number of total and free inodes reported by
<literal>lfs df -i</literal> for ZFS MDTs and OSTs is estimated based
on the current average space used per inode. When a ZFS filesystem is
better reflect actual site usage.
</para>
</note>
- <note condition='l24'>
- <para>Starting in release 2.4, using the DNE remote directory feature
+ <note>
+ <para>Using the DNE remote directory feature
it is possible to increase the total number of inodes of a Lustre
filesystem, as well as increasing the aggregate metadata performance,
by configuring additional MDTs into the filesystem, see
<para>Maximum number of MDTs</para>
</entry>
<entry>
- <para condition='l24'>256</para>
+ <para>256</para>
</entry>
<entry>
- <para>The Lustre software release 2.3 and earlier allows a
- maximum of 1 MDT per file system, but a single MDS can host
- multiple MDTs, each one for a separate file system.</para>
- <para condition="l24">The Lustre software release 2.4 and later
- requires one MDT for the filesystem root. Up to 255 more
- MDTs can be added to the filesystem and attached into
+ <para>A single MDS can host
+ multiple MDTs, either for separate file systems, or up to 255
+ additional MDTs can be added to the filesystem and attached into
the namespace with DNE remote or striped directories.</para>
</entry>
</row>
<para>Maximum number of files in the file system</para>
</entry>
<entry>
- <para>4 billion (ldiskfs), 256 trillion (ZFS)</para>
- <para condition='l24'>this is a per-MDT limit</para>
+ <para>4 billion (ldiskfs), 256 trillion (ZFS) per MDT</para>
</entry>
<entry>
<para>The ldiskfs filesystem imposes an upper limit of
increased initially at the time of MDT filesystem creation.
For more information, see
<xref linkend="settinguplustresystem"/>.</para>
- <para condition="l24">The ZFS filesystem dynamically allocates
+ <para>The ZFS filesystem dynamically allocates
inodes and does not have a fixed ratio of inodes per unit of MDT
space, but consumes approximately 4KiB of mirrored space per
inode, depending on the configuration.</para>
- <para condition="l24">Each additional MDT can hold up to the
+ <para>Each additional MDT can hold up to the
above maximum number of additional files, depending on
available space and the distribution directories and files
in the filesystem.</para>
<para><xref linkend="dbdoclet.50438219_44971"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438219_84890"/></para>
- </listitem>
- <listitem>
<para><xref linkend="dbdoclet.50438219_90386"/></para>
</listitem>
<listitem>
<section xml:id="dbdoclet.50438219_58217">
<title><indexterm><primary>ll_decode_filter_fid</primary></indexterm>
ll_decode_filter_fid</title>
- <para>The ll_decode_filter_fid utility displays the Lustre object ID and MDT parent FID.</para>
+ <para>The <literal>ll_decode_filter_fid</literal> utility displays the
+ Lustre object ID and MDT parent FID.</para>
<section remap="h5">
<title>Synopsis</title>
<screen>ll_decode_filter_fid object_file [object_file ...]</screen>
</section>
<section remap="h5">
<title>Description</title>
- <para>The ll_decode_filter_fid utility decodes and prints the Lustre OST object ID, MDT FID,
- stripe index for the specified OST object(s), which is stored in the "trusted.fid"
- attribute on each OST object. This is accessible to <literal>ll_decode_filter_fid</literal>
- when the OST file system is mounted locally as type ldiskfs for maintenance.</para>
- <para>The "trusted.fid" extended attribute is stored on each OST object when it is first modified (data written or attributes set), and is not accessed or modified by Lustre after that time.</para>
- <para>The OST object ID (objid) is useful in case of OST directory corruption, though normally the ll_recover_lost_found_objs(8) utility is able to reconstruct the entire OST object directory hierarchy. The MDS FID can be useful to determine which MDS inode an OST object is (or was) used by. The stripe index can be used in conjunction with other OST objects to reconstruct the layout of a file even if the MDT inode was lost.</para>
+ <para>The ll_decode_filter_fid utility decodes and prints the Lustre OST
+ object ID, MDT FID, stripe index for the specified OST object(s), which
+ is stored in the "trusted.fid" attribute on each OST object.
+ This is accessible to <literal>ll_decode_filter_fid</literal> when the
+ OST file system is mounted locally as type ldiskfs for maintenance.
+ </para>
+ <para>The "trusted.fid" extended attribute is stored on each
+ OST object when it is first modified (data written or attributes set),
+ and is not accessed or modified by Lustre after that time.</para>
+ <para>The OST object ID (objid) may be useful in case of OST directory
+ corruption, though LFSCK can normally reconstruct the entire OST object
+ directory tree, see <xref linkend="dbdoclet.lfsckadmin" /> for details.
+ The MDS FID can be useful to determine which MDS inode an OST object
+ is (or was) used by. The stripe index can be used in conjunction with
+ other OST objects to reconstruct the layout of a file even if the MDT
+ inode was lost.
+ </para>
</section>
<section remap="h5">
<title>Examples</title>
online scanning will automatically move objects from
<literal>lost+found</literal> to the proper place in the OST.</para>
</note>
- <note condition='l25'>
- <para>The <literal>ll_recover_lost_found_objs</literal> tool is not
- strictly necessary to bring an OST back online, it just avoids losing
- access to objects that were moved to the lost+found directory due to
- directory corruption on the OST.</para>
- </note>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>$ ll_recover_lost_found_objs [-hv] -d directory</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The first time Lustre modifies an object, it saves the MDS inode number and the objid as an extended attribute on the object, so in case of directory corruption of the OST, it is possible to recover the objects. Running e2fsck fixes the corrupted OST directory, but it puts all of the objects into a lost and found directory, where they are inaccessible to Lustre. Use the ll_recover_lost_found_objs utility to recover all (or at least most) objects from a lost and found directory and return them to the O/0/d* directories.</para>
- <para>To use ll_recover_lost_found_objs, mount the file system locally (using the <literal>-t ldiskfs</literal>, or <literal>-t zfs</literal> command), run the utility and then unmount it again. The OST must not be mounted by Lustre when ll_recover_lost_found_objs is run.</para>
- </section>
- <section remap="h5">
- <title>Options</title>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry>
- <para><emphasis role="bold">Option</emphasis></para>
- </entry>
- <entry>
- <para><emphasis role="bold">Description</emphasis></para>
- </entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry>
- <para> <literal>-h</literal></para>
- </entry>
- <entry>
- <para> Prints a help message</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal>-v</literal></para>
- </entry>
- <entry>
- <para> Increases verbosity</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal>-d <replaceable>directory</replaceable></literal></para>
- </entry>
- <entry>
- <para> Sets the lost and found directory path</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Example</title>
- <screen>ll_recover_lost_found_objs -d /mnt/ost/lost+found </screen>
- </section>
- </section>
- <section xml:id="dbdoclet.50438219_84890">
- <title><indexterm><primary>llodbstat</primary></indexterm>
-llobdstat</title>
- <para>The llobdstat utility displays OST statistics.</para>
<section remap="h5">
<title>Synopsis</title>
<screen>llobdstat ost_name [interval]</screen>
restoring from a file-level MDT backup (
<xref linkend="dbdoclet.backup_device" />), or in case the OI Table is
otherwise corrupted. Later phases of LFSCK will add further checks to the
- Lustre distributed file system state.</para>
- <para condition='l24'>In Lustre software release 2.4, LFSCK namespace
- scanning can verify and repair the directory FID-in-dirent and LinkEA
- consistency.</para>
+ Lustre distributed file system state. LFSCK namespace scanning can verify
+ and repair the directory FID-in-dirent and LinkEA consistency.</para>
<para condition='l26'>In Lustre software release 2.6, LFSCK layout scanning
can verify and repair MDT-OST file layout inconsistencies. File layout
inconsistencies between MDT-objects and OST-objects that are checked and
started. Anytime the LFSCK is triggered, the OI scrub will
run automatically, so there is no need to specify
OI_scrub in that case.</para>
- <para condition='l24'>
- <literal>namespace</literal>: check and repair
+ <para><literal>namespace</literal>: check and repair
FID-in-dirent and LinkEA consistency.</para>
<para condition='l27'> Lustre-2.7 enhances
namespace consistency verification under DNE mode.</para>
</informaltable>
</section>
</section>
- <section condition='l24'>
+ <section>
<title>LFSCK status of namespace via
<literal>procfs</literal></title>
<section>
from the failed node.</para>
</listitem>
</itemizedlist>
- <para>In Lustre software releases previous to Lustre software release
- 2.4, MDSs can be configured as an active/passive pair, while OSSs can be
- deployed in an active/active configuration that provides redundancy
+ <para>If there is a single MDT in a filesystem, two MDSes can be
+ configured as an active/passive pair, while pairs of OSSes can be
+ deployed in an active/active configuration that improves OST availability
without extra overhead. Often the standby MDS is the active MDS for
another Lustre file system or the MGS, so no nodes are idle in the
- cluster.</para>
- <para condition="l24">Lustre software release 2.4 introduces metadata
- targets for individual sub-directories. Active-active failover
- configurations are available for MDSs that serve MDTs on shared
+ cluster. If there are multiple MDTs in a filesystem, active-active
+ failover configurations are available for MDSs that serve MDTs on shared
storage.</para>
</section>
</section>
<itemizedlist>
<listitem>
<para>For MDT failover, two MDSs can be configured to serve the same
- MDT. Only one MDS node can serve an MDT at a time.</para>
- <para condition="l24">Lustre software release 2.4 allows multiple MDTs.
- By placing two or more MDT partitions on storage shared by two MDSs,
- one MDS can fail and the remaining MDS can begin serving the unserved
- MDT. This is described as an active/active failover pair.</para>
+ MDT. Only one MDS node can serve any MDT at one time.
+ By placing two or more MDT devices on storage shared by two MDSs,
+ one MDS can fail and the remaining MDS can begin serving the unserved
+ MDT. This is described as an active/active failover pair.</para>
</listitem>
<listitem>
<para>For OST failover, multiple OSS nodes can be configured to be able
</mediaobject>
</figure>
</section>
- <section xml:id='dbdoclet.mdtactiveactive' condition='l24'>
+ <section xml:id='dbdoclet.mdtactiveactive'>
<title>
<indexterm>
<primary>failover</primary>
<secondary>MDT</secondary>
</indexterm>MDT Failover Configuration (Active/Active)</title>
- <para>Multiple MDTs became available with the advent of Lustre software
- release 2.4. MDTs can be setup as an active/active failover
+ <para>MDTs can be configured as an active/active failover
configuration. A failover cluster is built from two MDSs as shown in
<xref linkend="understandingfailover.fig.configmdts" />.</para>
<figure xml:id="understandingfailover.fig.configmdts">
<para>
<emphasis>MDS count:</emphasis>
</para>
- <para>1 primary + 1 standby</para>
- <para condition="l24">256 MDSs, with up to 256 MDTs</para>
+ <para>256 MDSs, with up to 256 MDTs</para>
</entry>
<entry>
<para>
additional functionality needed by the Lustre file system.</para>
</listitem>
<listitem>
- <para condition="l24">With the Lustre software release 2.4 and later,
- it is also possible to use ZFS as the backing filesystem for Lustre
- for the MDT, OST, and MGS storage. This allows Lustre to leverage the
- scalability and data integrity features of ZFS for individual storage
- targets.</para>
+ <para>It is also possible to use ZFS as the backing filesystem for
+ Lustre for the MDT, OST, and MGS storage. This allows Lustre to
+ leverage the scalability and data integrity features of ZFS for
+ individual storage targets.</para>
</listitem>
<listitem>
<para>
<para>
<emphasis role="bold">High-availability:</emphasis>The Lustre file
system supports active/active failover using shared storage
- partitions for OSS targets (OSTs). Lustre software release 2.3 and
- earlier releases offer active/passive failover using a shared storage
- partition for the MDS target (MDT). The Lustre file system can work
+ partitions for OSS targets (OSTs), and for MDS targets (MDTs).
+ The Lustre file system can work
with a variety of high availability (HA) managers to allow automated
failover and has no single point of failure (NSPF). This allows
application transparent recovery. Multiple mount protection (MMP)
systems that would otherwise cause file system corruption.</para>
</listitem>
<listitem>
- <para condition="l24">With Lustre software release 2.4 or later
- servers and clients it is possible to configure active/active
- failover of multiple MDTs. This allows scaling the metadata
- performance of Lustre filesystems with the addition of MDT storage
- devices and MDS nodes.</para>
+ <para>It is possible to configure active/active failover of multiple
+ MDTs. This allows scaling the metadata performance of Lustre
+ filesystems with the addition of MDT storage devices and MDS nodes.
+ </para>
</listitem>
<listitem>
<para>
</listitem>
<listitem>
<para>
- <emphasis role="bold">Metadata Targets (MDT</emphasis>) - For Lustre
- software release 2.3 and earlier, each file system has one MDT. The
+ <emphasis role="bold">Metadata Targets (MDT</emphasis>) - Each
+ filesystem has at least one MDT. The
MDT stores metadata (such as filenames, directories, permissions and
file layout) on storage attached to an MDS. Each file system has one
MDT. An MDT on a shared storage target can be available to multiple
MDSs, although only one can access it at a time. If an active MDS
fails, a standby MDS can serve the MDT and make it available to
clients. This is referred to as MDS failover.</para>
- <para condition="l24">Since Lustre software release 2.4, multiple
- MDTs are supported in the Distributed Namespace Environment (DNE).
+ <para>Multiple MDTs are supported in the Distributed Namespace
+ Environment (DNE).
In addition to the primary MDT that holds the filesystem root, it
is possible to add additional MDS nodes, each with their own MDTs,
to hold sub-directory trees of the filesystem.</para>
<primary>Lustre</primary>
<secondary>I/O</secondary>
</indexterm>Lustre File System Storage and I/O</title>
- <para>In Lustre software release 2.0, Lustre file identifiers (FIDs) were
- introduced to replace UNIX inode numbers for identifying files or objects.
- A FID is a 128-bit identifier that contains a unique 64-bit sequence
- number, a 32-bit object ID (OID), and a 32-bit version number. The sequence
- number is unique across all Lustre targets in a file system (OSTs and
- MDTs). This change enabled future support for multiple MDTs (introduced in
- Lustre software release 2.4) and ZFS (introduced in Lustre software release
- 2.4).</para>
- <para>Also introduced in release 2.0 is an ldiskfs feature named
- <emphasis role="italic">FID-in-dirent</emphasis>(also known as
- <emphasis role="italic">dirdata</emphasis>) in which the FID is stored as
- part of the name of the file in the parent directory. This feature
- significantly improves performance for
- <literal>ls</literal> command executions by reducing disk I/O. The
- FID-in-dirent is generated at the time the file is created.</para>
- <note>
- <para>The FID-in-dirent feature is not backward compatible with the
- release 1.8 ldiskfs disk format. Therefore, when an upgrade from
- release 1.8 to release 2.x is performed, the FID-in-dirent feature is
- not automatically enabled. For upgrades from release 1.8 to releases
- 2.0 through 2.3, FID-in-dirent can be enabled manually but only takes
- effect for new files.</para>
- <para>For more information about upgrading from Lustre software release
- 1.8 and enabling FID-in-dirent for existing files, see
- <xref xmlns:xlink="http://www.w3.org/1999/xlink"
- linkend="upgradinglustre" />Chapter 16 “Upgrading a Lustre File
- System”.</para>
- </note>
- <para condition="l24">The LFSCK file system consistency checking tool
- released with Lustre software release 2.4 provides functionality that
- enables FID-in-dirent for existing files. It includes the following
- functionality:
+ <para>Lustre uses file identifiers (FIDs) to replace UNIX inode numbers
+ for identifying files or objects. A FID is a 128-bit identifier that
+ contains a unique 64-bit sequence number, a 32-bit object ID (OID), and
+ a 32-bit version number. The sequence number is unique across all Lustre
+ targets in a file system (OSTs and MDTs). This allows Lustre to identify
+ files on multiple MDTs independent of the underlying filesystem type.
+ </para>
+ <para>The LFSCK file system consistency checking tool verifies the
+ consistency of file objects between MDTs and OSTs. It includes the
+ following functionality:
<itemizedlist>
<listitem>
- <para>Generates IGIF mode FIDs for existing files from a 1.8 version
- file system files.</para>
- </listitem>
- <listitem>
<para>Verifies the FID-in-dirent for each file and regenerates the
FID-in-dirent if it is invalid or missing.</para>
</listitem>
31.25 PiB for ldiskfs or 8EiB with ZFS. Note that a Lustre file system can
support files up to 2^63 bytes (8EiB), limited only by the space available
on the OSTs.</para>
- <note>
- <para>Versions of the Lustre software prior to Release 2.2 limited the
- maximum stripe count for a single file to 160 OSTs.</para>
- </note>
<para>Although a single file can only be striped over 2000 objects,
Lustre file systems can have thousands of OSTs. The I/O bandwidth to
access a single file is the aggregated I/O bandwidth to the objects in a
<secondary>multiple MDSs</secondary>
</indexterm>
<indexterm>
- <primary>large_xattr</primary>
- <secondary>ea_inode</secondary>
+ <primary>ea_inode</primary>
+ <secondary>large_xattr</secondary>
</indexterm>
<indexterm>
<primary>wide striping</primary>
- <secondary>large_xattr</secondary>
- <tertiary>ea_inode</tertiary>
+ <secondary>ea_inode</secondary>
+ <tertiary>large_xattr</tertiary>
</indexterm>Upgrading to Lustre Software Release 2.x (Major
Release)</title>
<para>The procedure for upgrading from a Lustre software release 2.x to a
- more recent 2.x release of the Lustre software is described in this
- section.</para>
- <note>
- <para>This procedure can also be used to upgrade Lustre software release
- 1.8.6-wc1 or later to any Lustre software release 2.x. To upgrade other
- versions of Lustre software release 1.8.x, contact your support
- provider.</para>
- </note>
- <note>
- <para>In Lustre software release 2.2, a feature has been added for
- ldiskfs-based MDTs that allows striping a single file across up to 2000
- OSTs. By default, this "wide striping" feature is disabled. It is
- activated by setting the <literal>ea_inode</literal> option on the MDT
- using either <literal>mkfs.lustre</literal> or <literal>tune2fs</literal>.
- For example after upgrading an existing file system to Lustre software
- release 2.2 or later, wide striping can be enabled by running the
- following command on the MDT device before mounting it:
- <screen>tune2fs -O large_xattr</screen>
- Once the wide striping feature is enabled and in use on the MDT, it is
- not possible to directly downgrade the MDT file system to an earlier
- version of the Lustre software that does not support wide striping. To
- disable wide striping:
- <orderedlist>
- <listitem>
- <para>Delete all wide-striped files, <emphasis>OR</emphasis>
- use <literal>lfs_migrate -c 160</literal> (or fewer stripes)
- to migrate the files to use fewer OSTs. This does not affect the
- total number of OSTs that the whole filesystem can access.</para>
- </listitem>
- <listitem>
- <para>Unmount the MDT.</para>
- </listitem>
- <listitem>
- <para>Run the following command to turn off the
- <literal>large_xattr</literal> option:
- <screen>tune2fs -O ^large_xattr</screen></para>
- </listitem>
- </orderedlist>Using either
- <literal>mkfs.lustre</literal> or
- <literal>tune2fs</literal> with
- <literal>large_xattr</literal> or
- <literal>ea_inode</literal> option reseults in
- <literal>ea_inode</literal> in the file system feature list.</para>
- </note>
- <note>
- <para>To generate a list of all files with more than 160 stripes use
- <literal>lfs find</literal> with the
- <literal>--stripe-count</literal> option:
- <screen>lfs find ${mountpoint} --stripe-count=+160</screen></para>
- </note>
- <note condition="l24">
- <para>In Lustre software release 2.4, a new feature allows using multiple
- MDTs, which can each serve one or more remote sub-directories in the file
- system. The
- <literal>root</literal> directory is always located on MDT0.</para>
- <para>Note that clients running a release prior to the Lustre software
- release 2.4 can only see the namespace hosted by MDT0 and will return an
- IO error if an attempt is made to access a directory on another
- MDT.</para>
- </note>
- <para>To upgrade a Lustre software release 2.x to a more recent major
- release, complete these steps:</para>
+ more recent 2.y major release of the Lustre software is described in this
+ section. To upgrade an existing 2.x installation to a more recent major
+ release, complete the following steps:</para>
<orderedlist>
<listitem>
<para>Create a complete, restorable file system backup.</para>
</orderedlist>
</listitem>
<listitem>
- <para>(Optional) For upgrades to Lustre software release 2.2 or higher,
- to enable wide striping on an existing MDT, run the following command
- on the MDT:
- <screen>tune2fs -O ea_inode /dev/<replaceable>mdtdev</replaceable></screen>
+ <para condition="l2D">Lustre allows striping a single file across up to
+ 2000 OSTs. Before Lustre 2.13, the "wide striping" feature that
+ allowed creating files with more than 160 stripes was not enabled by
+ default. From the 2.13 release onward, the <literal>ea_inode</literal>
+ feature is enabled for newly-formatted MDTs. The feature can also
+ be enabled by the <literal>tune2fs</literal> command on existing MDTs:
+ <screen>mds# tune2fs -O ea_inode /dev/<replaceable>mdtdev</replaceable></screen>
</para>
<para>For more information about wide striping, see
<xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping" />.</para>
</listitem>
<listitem>
- <para>(Optional) For upgrades to Lustre software release 2.4 or higher,
- to format an additional MDT, complete these steps:
+ <para>(Optional) To format an additional MDT, complete these steps:
<orderedlist numeration="loweralpha">
<listitem>
<para>Determine the index used for the first MDT (each MDT must
</orderedlist></para>
</listitem>
<listitem>
- <para>(Optional) If you are upgrading to Lustre software release 2.3 or
- higher from Lustre software release 2.2 or earlier and want to enable
- the quota feature, complete these steps:
- <orderedlist numeration="loweralpha">
- <listitem>
- <para>Before setting up the file system, enter on both the MDS and
- OSTs:
- <screen>tunefs.lustre --quota</screen></para>
- </listitem>
- <listitem>
<para>(Optional) If you are upgrading before Lustre software release
2.10, to enable the project quota feature enter the following on every
ldiskfs backend target:
should only be enabled if the project quota feature is required and/or
after it is known that the upgraded release does not need to be
downgraded.</para></note>
- </listitem>
- <listitem>
- <para>When setting up the file system, enter:
- <screen>conf_param $FSNAME.quota.mdt=$QUOTA_TYPE
+ <para>When setting up the file system, enter:
+ <screen>conf_param $FSNAME.quota.mdt=$QUOTA_TYPE
conf_param $FSNAME.quota.ost=$QUOTA_TYPE</screen></para>
- </listitem>
- </orderedlist></para>
- </listitem>
- <listitem>
- <para>(Optional) If you are upgrading from Lustre software release 1.8,
- you must manually enable the FID-in-dirent feature. On the MDS, enter:
- <screen>tune2fs –O dirdata /dev/<replaceable>mdtdev</replaceable></screen></para>
- <warning>
- <para>This step is not reversible. Do not complete this step until
- you are sure you will not be downgrading the Lustre software.</para>
- </warning>
- <para condition='l24'>This step only enables FID-in-dirent for newly
- created files. If you are upgrading to Lustre software release 2.4,
- you can use namespace LFSCK to enable FID-in-dirent for the existing
- files. For the case of upgrading from Lustre software release 1.8, it is
- important to note that if you do NOT enable <literal>dirdata</literal> via
- the <literal>tune2fs</literal> command above, the namespace LFSCK will NOT
- generate FID-in-dirent for the existing files. For more information about
- FID-in-dirent and related functionalities in LFSCK, see
- <xref xmlns:xlink="http://www.w3.org/1999/xlink"
- linkend="understandinglustre.storageio" />.</para>
</listitem>
<listitem>
<para>Start the Lustre file system by starting the components in the
default values for unspecified fields. If the striping EA is
not set, 0, 0, and -1 will be printed for the stripe count,
size, and offset respectively.</para>
- <para condition="l24">The
- <literal>--mdt-index</literal> prints the index of the MDT for a given
- directory. See
+ <para>The <literal>--mdt-index</literal> prints the index of
+ the MDT for a given directory. See
<xref linkend="lustremaint.rmremotedir" />.</para>
</entry>
</row>
<xsl:template name="condition-decorator">
<xsl:param name='content'/>
<xsl:choose>
- <xsl:when test="@condition = 'l23'">
- <xsl:call-template name='textdecoration-1'>
- <xsl:with-param name='version' select="'Introduced in Lustre 2.3'"/>
- <xsl:with-param name='content' select="$content"/>
- </xsl:call-template>
- </xsl:when>
- <xsl:when test="@condition = 'l24'">
- <xsl:call-template name='textdecoration-1'>
- <xsl:with-param name='version' select="'Introduced in Lustre 2.4'"/>
- <xsl:with-param name='content' select="$content"/>
- </xsl:call-template>
- </xsl:when>
<xsl:when test="@condition = 'l25'">
<xsl:call-template name='textdecoration-1'>
<xsl:with-param name='version' select="'Introduced in Lustre 2.5'"/>
<xsl:param name='condition'/>
<!-- add another span to hold the lustre version annotation -->
<xsl:choose>
- <xsl:when test="$condition = 'l23'">
- <span class='floatright'>L 2.3 </span>
- </xsl:when>
- <xsl:when test="$condition = 'l24'">
- <span class='floatright'>L 2.4 </span>
- </xsl:when>
<xsl:when test="$condition = 'l25'">
<span class='floatright'>L 2.5 </span>
</xsl:when>
</xsl:variable>
<xsl:variable name="versionstr">
<xsl:choose>
- <xsl:when test="@condition = 'l23'">Introduced in Lustre 2.3</xsl:when>
- <xsl:when test="@condition = 'l24'">Introduced in Lustre 2.4</xsl:when>
<xsl:when test="@condition = 'l25'">Introduced in Lustre 2.5</xsl:when>
<xsl:when test="@condition = 'l26'">Introduced in Lustre 2.6</xsl:when>
<xsl:when test="@condition = 'l27'">Introduced in Lustre 2.7</xsl:when>
<xsl:when test="@condition = 'l2B'">Introduced in Lustre 2.11</xsl:when>
<xsl:when test="@condition = 'l2C'">Introduced in Lustre 2.12</xsl:when>
<xsl:when test="@condition = 'l2D'">Introduced in Lustre 2.13</xsl:when>
+ <xsl:when test="@condition = 'l2E'">Introduced in Lustre 2.14</xsl:when>
+ <xsl:when test="@condition = 'l2F'">Introduced in Lustre 2.15</xsl:when>
+ <xsl:when test="@condition = 'l2G'">Introduced in Lustre 2.16</xsl:when>
<xsl:otherwise>Documentation Error: unrecognised condition attribute</xsl:otherwise>
</xsl:choose>
</xsl:variable>
git://git.whamcloud.com / doc / manual.git / commitdiff