described in this section. File-level restore of an MDT is not possible
before Lustre software release 2.3, as the Object Index (OI) file cannot
be rebuilt after restore without the OI Scrub functionality.
- <emphasis role="bold">Since Lustre software release 2.3</emphasis>,
- Object Index files are automatically rebuilt at first mount after a
- restore is detected (see
+ Since Lustre 2.3, Object Index files are automatically rebuilt at first
+ mount after a restore is detected (see
<link xl:href="http://jira.whamcloud.com/browse/LU-957">LU-957</link>),
and file-level backup is supported (see
<xref linkend="backup_fs_level"/>).</para>
<screen>[oss]# lctl get_param -n osd-${fstype}.${fsname}-${target}.oi_scrub</screen>
</listitem>
</orderedlist>
- <para condition='l23'>If the file system was used between the time the
- backup was made and when it was restored, then the online
- <literal>LFSCK</literal> tool (part of Lustre code after version 2.3)
- will automatically be
- run to ensure the file system is coherent. If all of the device file
- systems were backed up at the same time after the entire Lustre file system
- was stopped, this step is unnecessary. In either case, the file system will
- be immediately although there may be I/O errors reading
- from files that are present on the MDT but not the OSTs, and files that
- were created after the MDT backup will not be accessible or visible. See
- <xref linkend="dbdoclet.lfsckadmin" />for details on using LFSCK.</para>
+ <para>If the file system was used between the time the backup was made and
+ when it was restored, then the online <literal>LFSCK</literal> tool will
+ automatically be run to ensure the filesystem is coherent. If all of the
+ device filesystems were backed up at the same time after Lustre was
+ was stopped, this step is unnecessary. In either case, the filesystem
+ will be immediately although there may be I/O errors reading
+ from files that are present on the MDT but not the OSTs, and files that
+ were created after the MDT backup will not be accessible or visible. See
+ <xref linkend="dbdoclet.lfsckadmin" />for details on using LFSCK.</para>
</section>
<section xml:id="dbdoclet.backup_lvm_snapshot">
<title>
<section>
<title><indexterm><primary>configuring</primary><secondary>network</secondary><tertiary>rnet_htable_size</tertiary></indexterm>
<literal>rnet_htable_size</literal></title>
- <para condition='l23'><literal>rnet_htable_size</literal> is an integer that indicates how many remote networks the internal LNet hash table is configured to handle. <literal>rnet_htable_size</literal> is used for optimizing the hash table size and does not put a limit on how many remote networks you can have. The default hash table size when this parameter is not specified is: 128.</para>
+ <para><literal>rnet_htable_size</literal> is an integer that indicates how many remote networks the internal LNet hash table is configured to handle. <literal>rnet_htable_size</literal> is used for optimizing the hash table size and does not put a limit on how many remote networks you can have. The default hash table size when this parameter is not specified is: 128.</para>
</section>
</section>
<section remap="h3" xml:id="section_ngq_qhy_zl">
<literal>crc32</literal>,
<literal>crc32c</literal>
</para>
+ <para condition="l2C">
+ In Lustre release 2.12 additional checksum types were added to
+ allow end-to-end checksum integration with T10-PI capable
+ hardware. The client will compute the appropriate checksum
+ type, based on the checksum type used by the storage, for the
+ RPC checksum, which will be verified by the server and passed
+ on to the storage. The T10-PI checksum types are:
+ <literal>t10ip512</literal>,
+ <literal>t10ip4K</literal>,
+ <literal>t10crc512</literal>,
+ <literal>t10crc4K</literal>
+ </para>
</listitem>
<listitem>
<para><literal>osc.<replaceable>osc_instance</replaceable>.max_dirty_mb</literal>
<section remap="h4">
<title>Tuning File Readahead</title>
<para>File readahead is triggered when two or more sequential reads
- by an application fail to be satisfied by data in the Linux buffer
- cache. The size of the initial readahead is 1 MB. Additional
- readaheads grow linearly and increment until the readahead cache on
- the client is full at 40 MB.</para>
+ by an application fail to be satisfied by data in the Linux buffer
+ cache. The size of the initial readahead is determined by the RPC
+ size and the file stripe size, but will typically be at least 1 MiB.
+ Additional readaheads grow linearly and increment until the per-file
+ or per-system readahead cache limit on the client is reached.</para>
<para>Readahead tunables include:</para>
<itemizedlist>
<listitem>
service immediately and disables automatic thread creation behavior.
</para>
</note>
- <para condition='l23'>Lustre software release 2.3 introduced new
- parameters to provide more control to administrators.</para>
+ <para>Parameters are available to provide administrators control
+ over the number of service threads.</para>
<itemizedlist>
<listitem>
<para>
</itemizedlist>
</section>
</section>
- <section xml:id="dbdoclet.mdsbinding" condition='l23'>
+ <section xml:id="dbdoclet.mdsbinding">
<title>
<indexterm>
<primary>tuning</primary>
<secondary>MDS binding</secondary>
</indexterm>Binding MDS Service Thread to CPU Partitions</title>
- <para>With the introduction of Node Affinity (
- <xref linkend="nodeaffdef" />) in Lustre software release 2.3, MDS threads
- can be bound to particular CPU partitions (CPTs) to improve CPU cache
- usage and memory locality. Default values for CPT counts and CPU core
+ <para>With the Node Affinity (<xref linkend="nodeaffdef" />) feature,
+ MDS threads can be bound to particular CPU partitions (CPTs) to improve CPU
+ cache usage and memory locality. Default values for CPT counts and CPU core
bindings are selected automatically to provide good overall performance for
a given CPU count. However, an administrator can deviate from these setting
if they choose. For details on specifying the mapping of CPU cores to
<para>By default, this parameter is off. As always, you should test the
performance to compare the impact of changing this parameter.</para>
</section>
- <section condition='l23'>
+ <section>
<title>
<indexterm>
<primary>tuning</primary>
<secondary>Network interface binding</secondary>
</indexterm>Binding Network Interface Against CPU Partitions</title>
- <para>Lustre software release 2.3 and beyond provide enhanced network
- interface control. The enhancement means that an administrator can bind
- an interface to one or more CPU partitions. Bindings are specified as
- options to the LNet modules. For more information on specifying module
- options, see
+ <para>Lustre allows enhanced network interface control. This means that
+ an administrator can bind an interface to one or more CPU partitions.
+ Bindings are specified as options to the LNet modules. For more
+ information on specifying module options, see
<xref linkend="dbdoclet.50438293_15350" /></para>
<para>For example,
<literal>o2ib0(ib0)[0,1]</literal> will ensure that all messages for
<screen>
ko2iblnd credits=256
</screen>
- <note condition="l23">
- <para>In Lustre software release 2.3 and beyond, LNet may revalidate
- the NI credits, so the administrator's request may not persist.</para>
+ <note>
+ <para>LNet may revalidate the NI credits, so the administrator's
+ request may not persist.</para>
</note>
</section>
<section>
<screen>
lnet large_router_buffers=8192
</screen>
- <note condition="l23">
- <para>In Lustre software release 2.3 and beyond, LNet may revalidate
- the router buffer setting, so the administrator's request may not
- persist.</para>
+ <note>
+ <para>LNet may revalidate the router buffer setting, so the
+ administrator's request may not persist.</para>
</note>
</section>
<section>
be MAX.</para>
</section>
</section>
- <section xml:id="dbdoclet.libcfstuning" condition='l23'>
+ <section xml:id="dbdoclet.libcfstuning">
<title>
<indexterm>
<primary>tuning</primary>
<secondary>libcfs</secondary>
</indexterm>libcfs Tuning</title>
- <para>Lustre software release 2.3 introduced binding service threads via
- CPU Partition Tables (CPTs). This allows the system administrator to
- fine-tune on which CPU cores the Lustre service threads are run, for both
- OSS and MDS services, as well as on the client.
+ <para>Lustre allows binding service threads via CPU Partition Tables
+ (CPTs). This allows the system administrator to fine-tune on which CPU
+ cores the Lustre service threads are run, for both OSS and MDS services,
+ as well as on the client.
</para>
<para>CPTs are useful to reserve some cores on the OSS or MDS nodes for
system functions such as system monitoring, HA heartbeat, or similar
<para><xref linkend="dbdoclet.50438219_44971"/></para>
</section>
</section>
- <section xml:id="dbdoclet.50438219_44971">
+ <section xml:id="dbdoclet.50438219_44971" condition='l28'>
<title><indexterm><primary>ll_recover_lost_found_objs</primary></indexterm>
ll_recover_lost_found_objs</title>
- <para>The ll_recover_lost_found_objs utility helps recover Lustre OST objects (file data) from a lost and found directory and return them to their correct locations.</para>
- <note>
- <para>Running the ll_recover_lost_found_objs tool is not strictly necessary to bring an OST back online, it just avoids losing access to objects that were moved to the lost and found directory due to directory corruption.</para>
+ <para>The <literal>ll_recover_lost_found_objs</literal> utility was
+ used to help recover Lustre OST objects (file data) from the
+ <literal>lost+found</literal> directory of an OST and return them to
+ their correct locations based on information stored in the
+ <literal>trusted.fid</literal> extended attribute stored on every
+ OST object containing data.</para>
+ <note condition="l26"><para>This utility is not needed with Lustre 2.6
+ and later, and is removed in Lustre 2.8 since <literal>LFSCK</literal>
+ online scanning will automatically move objects from
+ <literal>lost+found</literal> to the proper place in the OST.</para>
+ </note>
+ <note condition='l25'>
+ <para>The <literal>ll_recover_lost_found_objs</literal> tool is not
+ strictly necessary to bring an OST back online, it just avoids losing
+ access to objects that were moved to the lost+found directory due to
+ directory corruption on the OST.</para>
</note>
<section remap="h5">
<title>Synopsis</title>
<title>Description</title>
<para>The first time Lustre modifies an object, it saves the MDS inode number and the objid as an extended attribute on the object, so in case of directory corruption of the OST, it is possible to recover the objects. Running e2fsck fixes the corrupted OST directory, but it puts all of the objects into a lost and found directory, where they are inaccessible to Lustre. Use the ll_recover_lost_found_objs utility to recover all (or at least most) objects from a lost and found directory and return them to the O/0/d* directories.</para>
<para>To use ll_recover_lost_found_objs, mount the file system locally (using the <literal>-t ldiskfs</literal>, or <literal>-t zfs</literal> command), run the utility and then unmount it again. The OST must not be mounted by Lustre when ll_recover_lost_found_objs is run.</para>
- <para condition="l26">This utility is not needed for 2.6 and later,
- since the <literal>LFSCK</literal> online scanning will move objects
- from <literal>lost+found</literal> to the proper place in the OST.</para>
</section>
<section remap="h5">
<title>Options</title>
<para> <literal>user_fid2path</literal></para>
</entry>
<entry>
- <para condition='l23'>Enable FID to path translation by regular
- users. Note: This option allows a potential security hole
- because it allows regular users direct access to a file by its
- File ID, bypassing POSIX path-based permission checks which
- could otherwise prevent the user from accessing a file in a
- directory that they do not have access to. Regular permission
- checks are still performed on the file itself, so the user
- cannot access a file to which they have no access rights.
- </para>
+ <para>Enable FID-to-path translation by regular users.
+ </para>
+ <note><para>This option allows a potential security hole because
+ it allows regular users direct access to a file by its Lustre
+ File ID. This bypasses POSIX path-based permission checks,
+ and could allow the user to access a file in a directory that
+ they do not have access to. Regular POSIX file mode and ACL
+ permission checks are still performed on the file itself, so
+ users cannot access a file to which they have no permission.
+ </para></note>
</entry>
</row>
<row>
<para> <literal>nouser_fid2path</literal></para>
</entry>
<entry>
- <para condition='l23'> Disable FID to path translation by
+ <para> Disable FID to path translation by
regular users. Root and processes with
<literal>CAP_DAC_READ_SEARCH</literal> can still perform FID
to path translation.
<para>In the vast majority of cases, the Lustre software can cope with any
inconsistencies found on the disk and between other devices in the file
system.</para>
- <note>
- <para>The legacy offline-LFSCK tool included with e2fsprogs is rarely
- required for Lustre file system operation. offline-LFSCK is not to be
- confused with LFSCK tool, which is part of Lustre and provides online
- consistency checking.</para>
- </note>
<para>For problem analysis, it is strongly recommended that
- <literal>e2fsck</literal> be run under a logger, like script, to record all
+ <literal>e2fsck</literal> be run under a logger, like
+ <literal>script</literal>, to record all
of the output and changes that are made to the file system in case this
information is needed later.</para>
<para>If time permits, it is also a good idea to first run
<secondary>corruption of Lustre file system</secondary>
</indexterm>Recovering from Corruption in the Lustre File System</title>
<para>In cases where an ldiskfs MDT or OST becomes corrupt, you need to run
- e2fsck to correct the local filesystem consistency, then use
+ <literal>e2fsck</literal> to ensure local filesystem consistency, then use
<literal>LFSCK</literal> to run a distributed check on the file system to
resolve any inconsistencies between the MDTs and OSTs, or among MDTs.</para>
<orderedlist>
<xref linkend="lustrerecovery" />(Version-based Recovery).</para>
</note>
</section>
- <section xml:id="dbdoclet.lfsckadmin" condition='l23'>
+ <section xml:id="dbdoclet.lfsckadmin">
<title>
<indexterm>
<primary>recovery</primary>
<primary>recovery</primary>
<secondary>LFSCK</secondary>
</indexterm>Checking the file system with LFSCK</title>
- <para condition='l23'>LFSCK is an administrative tool introduced in Lustre
- software release 2.3 for checking and repair of the attributes specific to a
- mounted Lustre file system. It is similar in concept to an offline fsck repair
- tool for a local filesystem, but LFSCK is implemented to run as part of the
- Lustre file system while the file system is mounted and in use. This allows
- consistency of checking and repair by the Lustre software without unnecessary
- downtime, and can be run on the largest Lustre file systems with negligible
- disruption to normal operations.</para>
- <para condition='l23'>Since Lustre software release 2.3, LFSCK can verify
- and repair the Object Index (OI) table that is used internally to map
- Lustre File Identifiers (FIDs) to MDT internal ldiskfs inode numbers, in
- an internal table called the OI Table. An OI Scrub traverses the OI table
- and makes corrections where necessary. An OI Scrub is required after
- restoring from a file-level MDT backup (
- <xref linkend="dbdoclet.backup_device" />), or in case the OI Table is
- otherwise corrupted. Later phases of LFSCK will add further checks to the
- Lustre distributed file system state.</para>
+ <para>LFSCK is an administrative tool for checking and repair of the
+ attributes specific to a mounted Lustre file system. It is similar
+ in concept to an offline fsck repair tool for a local filesystem,
+ but LFSCK is implemented to run as part of the Lustre file system
+ while the file system is mounted and in use. This allows consistency
+ checking and repair of Lustre-specific metadata without unnecessary
+ downtime, and can be run on the largest Lustre file systems with
+ minimal impact to normal operations.</para>
+ <para>LFSCK can verify
+ and repair the Object Index (OI) table that is used internally to map
+ Lustre File Identifiers (FIDs) to MDT internal ldiskfs inode numbers, in
+ an internal table called the OI Table. An OI Scrub traverses the OI table
+ and makes corrections where necessary. An OI Scrub is required after
+ restoring from a file-level MDT backup (
+ <xref linkend="dbdoclet.backup_device" />), or in case the OI Table is
+ otherwise corrupted. Later phases of LFSCK will add further checks to the
+ Lustre distributed file system state.</para>
<para condition='l24'>In Lustre software release 2.4, LFSCK namespace
- scanning can verify and repair the directory FID-in-dirent and LinkEA
- consistency.</para>
+ scanning can verify and repair the directory FID-in-dirent and LinkEA
+ consistency.</para>
<para condition='l26'>In Lustre software release 2.6, LFSCK layout scanning
- can verify and repair MDT-OST file layout inconsistencies. File layout
- inconsistencies between MDT-objects and OST-objects that are checked and
- corrected include dangling reference, unreferenced OST-objects, mismatched
- references and multiple references.</para>
+ can verify and repair MDT-OST file layout inconsistencies. File layout
+ inconsistencies between MDT-objects and OST-objects that are checked and
+ corrected include dangling reference, unreferenced OST-objects, mismatched
+ references and multiple references.</para>
<para condition='l27'>In Lustre software release 2.7, LFSCK layout scanning
- is enhanced to support verify and repair inconsistencies between multiple
- MDTs.</para>
+ is enhanced to support verify and repair inconsistencies between multiple
+ MDTs.</para>
<para>Control and monitoring of LFSCK is through LFSCK and the
<literal>/proc</literal> file system interfaces. LFSCK supports three types
of interface: switch interface, status interface, and adjustment interface.
<literal>ea_inode</literal> option reseults in
<literal>ea_inode</literal> in the file system feature list.</para>
</note>
- <note condition="l23">
+ <note>
<para>To generate a list of all files with more than 160 stripes use
<literal>lfs find</literal> with the
<literal>--stripe-count</literal> option:
</para>
</entry>
<entry>
- <para condition="l23">Enables/disables FID to path translation by
+ <para>Enables/disables FID to path translation by
regular users</para>
</entry>
</row>
<xsl:with-param name='content' select="$content"/>
</xsl:call-template>
</xsl:when>
+ <xsl:when test="@condition = 'l2C'">
+ <xsl:call-template name='textdecoration-1'>
+ <xsl:with-param name='version' select="'Introduced in Lustre 2.12'"/>
+ <xsl:with-param name='content' select="$content"/>
+ </xsl:call-template>
+ </xsl:when>
<xsl:when test="@condition != ''">
<xsl:call-template name='textdecoration-1'>
- <xsl:with-param name='version' select="'Introduced in Lustre 2.10'"/>
+ <xsl:with-param name='version' select="'Introduced before Lustre 2.3'"/>
<xsl:with-param name='content' select="$content"/>
</xsl:call-template>
</xsl:when>
<xsl:when test="$condition = 'l2B'">
<span class='floatright'>L 2.11 </span>
</xsl:when>
+ <xsl:when test="$condition = 'l2C'">
+ <span class='floatright'>L 2.12 </span>
+ </xsl:when>
<xsl:when test="$condition != ''">
<span class='floatright'>L ?.? </span>
</xsl:when>
<xsl:when test="@condition = 'l29'">Introduced in Lustre 2.9</xsl:when>
<xsl:when test="@condition = 'l2A'">Introduced in Lustre 2.10</xsl:when>
<xsl:when test="@condition = 'l2B'">Introduced in Lustre 2.11</xsl:when>
+ <xsl:when test="@condition = 'l2C'">Introduced in Lustre 2.11</xsl:when>
<xsl:otherwise>Documentation Error: unrecognised condition attribute</xsl:otherwise>
</xsl:choose>
</xsl:variable>
<xsl:when test="@condition='l29'">L 2.9</xsl:when>
<xsl:when test="@condition='l2A'">L 2.10</xsl:when>
<xsl:when test="@condition='l2B'">L 2.11</xsl:when>
+ <xsl:when test="@condition='l2C'">L 2.11</xsl:when>
<xsl:otherwise></xsl:otherwise>
</xsl:choose>
</xsl:variable>
git://git.whamcloud.com / doc / manual.git / commitdiff