<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="systemconfigurationutilities">
<title xml:id="systemconfigurationutilities.title">System Configuration Utilities</title>
<para>This chapter includes system configuration utilities and includes the following sections:</para>
<title><indexterm><primary>e2scan</primary></indexterm>
e2scan</title>
<para>The e2scan utility is an ext2 file system-modified inode scan program. The e2scan program uses libext2fs to find inodes with ctime or mtime newer than a given time and prints out their pathname. Use e2scan to efficiently generate lists of files that have been modified. The e2scan tool is included in the e2fsprogs package, located at:</para>
- <para><link xl:href="http://downloads.whamcloud.com/public/e2fsprogs/latest/">http://downloads.whamcloud.com/public/e2fsprogs/latest/</link></para>
+ <para><link xl:href="http://downloads.hpdd.intel.com/public/e2fsprogs/latest/">http://downloads.hpdd.intel.com/public/e2fsprogs/latest/</link></para>
<section remap="h5">
<title>Synopsis</title>
<screen>e2scan [options] [-f file] block_device</screen>
<para>The l_getidentity utility handles Lustre user / group cache upcall.</para>
<section remap="h5">
<title>Synopsis</title>
- <screen>l_getidentity {mdtname} {uid}</screen>
+ <screen>l_getidentity ${FSNAME}-MDT{xxxx} {uid}</screen>
</section>
<section remap="h5">
<title>Description</title>
- <para>The group upcall file contains the path to an executable file that is invoked to resolve a numeric UID to a group membership list. This utility opens <literal>/proc/fs/lustre/mdt/{mdtname}/identity_info</literal> and writes the related <literal>identity_downcall_data</literal> structure (see <xref linkend='dbdoclet.50438291_33759'/>.) The data is persisted with <literal>lctl set_param mdt.{mdtname}.identity_info</literal>.</para>
+ <para>The group upcall file contains the path to an executable file that is invoked to resolve
+ a numeric UID to a group membership list. This utility opens
+ <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info</literal> and writes the
+ related <literal>identity_downcall_data</literal> structure (see <xref
+ linkend="dbdoclet.50438291_33759"/>.) The data is persisted with <literal>lctl set_param
+ mdt.${FSNAME}-MDT{xxxx}.identity_info</literal>.</para>
<para>The l_getidentity utility is the reference implementation of the user or group cache upcall.</para>
</section>
<section remap="h5">
<tbody>
<row>
<entry>
- <para> <literal>mdtname</literal></para>
+ <para>
+ <literal>${FSNAME}-MDT{xxxx}</literal></para>
</entry>
<entry>
<para> Metadata server target name</para>
<title>Setting Parameters with lctl</title>
<para>Lustre parameters are not always accessible using the procfs interface, as it is platform-specific. As a solution, lctl {get,set}_param has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to /proc/{fs,sys}/{lustre,lnet}. For future portability, use lctl {get,set}_param .</para>
<para>When the file system is running, use the <literal>lctl set_param</literal> command on the affected node(s) to <emphasis>temporarily</emphasis> set parameters (mapping to items in /proc/{fs,sys}/{lnet,lustre}). The <literal>lctl set_param</literal> command uses this syntax:</para>
- <screen>lctl set_param [-n] <replaceable>obdtype.obdname.property</replaceable>=<replaceable>value</replaceable></screen>
+ <screen>lctl set_param [-n] [-P] [-d] <replaceable>obdtype.obdname.property</replaceable>=<replaceable>value</replaceable></screen>
<para>For example:</para>
<screen>mds# lctl set_param mdt.testfs-MDT0000.identity_upcall=NONE</screen>
+ <para condition='l25'>Use <literal>-P</literal> option to set parameters permanently. Option <literal>-d </literal>deletes permanent parameters. For example:
+ <screen>mgs# lctl set_param -P mdt.testfs-MDT0000.identity_upcall=NONE
+mgs# lctl set_param -P -d mdt.testfs-MDT0000.identity_upcall</screen></para>
<para>Many permanent parameters can be set with <literal>lctl conf_param</literal>. In general, <literal>lctl conf_param</literal> can be used to specify any OBD device parameter settable in a /proc/fs/lustre file. The <literal>lctl conf_param</literal> command must be run on the MGS node, and uses this syntax:</para>
<screen><replaceable>obd|fsname</replaceable>.obdtype.property=<replaceable>value</replaceable>) </screen>
<para>For example:</para>
</tgroup>
</informaltable>
<note>
- <para>Lustre tunables are not always accessible using the procfs interface, as it is platform-specific. As a solution, lctl {get,set,list}_param has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to /proc/{fs,sys}/{lustre,lnet}. For future portability, use lctl {get,set,list}_param instead.</para>
+ <para>Lustre tunables are not always accessible using the procfs interface, as it is platform-specific. As a solution, <literal>lctl {get,set,list}_param</literal> has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to <literal>/proc/{fs,sys}/{lustre,lnet}</literal>. For future portability, use <literal>lctl {get,set,list}_param</literal> instead.</para>
</note>
<para><emphasis role="bold">Virtual Block Device Operations</emphasis></para>
<para>Lustre can emulate a virtual block device upon a regular file. This emulation is needed when you are trying to set up a swap space via the file.</para>
<tbody>
<row>
<entry>
- <para>blockdev_attach <replaceable>filename</replaceable> <replaceable>/dev/lloop_device</replaceable></para>
+ <para><literal>blockdev_attach <replaceable>filename</replaceable> <replaceable>/dev/lloop_device</replaceable></literal></para>
</entry>
<entry>
- <para> Attaches a regular Lustre file to a block device. If the device node does not exist, <literal>lctl</literal> creates it. We recommend that you create the device node by <literal>lctl</literal> since the emulator uses a dynamical major number.</para>
+ <para> Attaches a regular Lustre file to a block device. If the device node does not exist, <literal>lctl</literal> creates it. It is recommend that a device node is created by <literal>lctl</literal> since the emulator uses a dynamical major number.</para>
</entry>
</row>
<row>
</section>
<section remap="h5">
<title>Description</title>
- <para>The ll_decode_filter_fid utility decodes and prints the Lustre OST object ID, MDT FID, stripe index for the specified OST object(s), which is stored in the "trusted.fid" attribute on each OST object. This is accessible to ll_decode_filter_fid when the OST filesystem is mounted locally as type ldiskfs for maintenance.</para>
+ <para>The ll_decode_filter_fid utility decodes and prints the Lustre OST object ID, MDT FID,
+ stripe index for the specified OST object(s), which is stored in the "trusted.fid"
+ attribute on each OST object. This is accessible to <literal>ll_decode_filter_fid</literal>
+ when the OST file system is mounted locally as type ldiskfs for maintenance.</para>
<para>The "trusted.fid" extended attribute is stored on each OST object when it is first modified (data written or attributes set), and is not accessed or modified by Lustre after that time.</para>
<para>The OST object ID (objid) is useful in case of OST directory corruption, though normally the ll_recover_lost_found_objs(8) utility is able to reconstruct the entire OST object directory hierarchy. The MDS FID can be useful to determine which MDS inode an OST object is (or was) used by. The stripe index can be used in conjunction with other OST objects to reconstruct the layout of a file even if the MDT inode was lost.</para>
</section>
</section>
<section remap="h5">
<title>Description</title>
- <para>The first time Lustre writes to an object, it saves the MDS inode number and the objid as an extended attribute on the object, so in case of directory corruption of the OST, it is possible to recover the objects. Running e2fsck fixes the corrupted OST directory, but it puts all of the objects into a lost and found directory, where they are inaccessible to Lustre. Use the ll_recover_lost_found_objs utility to recover all (or at least most) objects from a lost and found directory and return them to the O/0/d* directories.</para>
- <para>To use ll_recover_lost_found_objs, mount the file system locally (using the -t ldiskfs command), run the utility and then unmount it again. The OST must not be mounted by Lustre when ll_recover_lost_found_objs is run.</para>
+ <para>The first time Lustre modifies an object, it saves the MDS inode number and the objid as an extended attribute on the object, so in case of directory corruption of the OST, it is possible to recover the objects. Running e2fsck fixes the corrupted OST directory, but it puts all of the objects into a lost and found directory, where they are inaccessible to Lustre. Use the ll_recover_lost_found_objs utility to recover all (or at least most) objects from a lost and found directory and return them to the O/0/d* directories.</para>
+ <para>To use ll_recover_lost_found_objs, mount the file system locally (using the <literal>-t ldiskfs</literal>, or <literal>-t zfs</literal> command), run the utility and then unmount it again. The OST must not be mounted by Lustre when ll_recover_lost_found_objs is run.</para>
+ <para condition="l26">This utility is not needed for 2.6 and later,
+ since the <literal>LFSCK</literal> online scanning will move objects
+ from <literal>lost+found</literal> to the proper place in the OST.</para>
</section>
<section remap="h5">
<title>Options</title>
<section xml:id="dbdoclet.50438219_90386">
<title><indexterm><primary>llog_reader</primary></indexterm>
llog_reader</title>
- <para>The llog_reader utility parses Lustre's on-disk configuration logs.</para>
+ <para>The llog_reader utility translates a Lustre configuration log into human-readable form.</para>
<section remap="h5">
<title>Synopsis</title>
<screen>llog_reader filename</screen>
<section remap="h5">
<title>Description</title>
<para>The llog_reader utility parses the binary format of Lustre's on-disk configuration logs. Llog_reader can only read logs; use tunefs.lustre to write to them.</para>
- <para>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs, then use llog_reader to dump the log file's contents, for example:</para>
+ <para>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs or zfs, then use llog_reader to dump the log file's contents, for example:</para>
<screen>mount -t ldiskfs /dev/sda /mnt/mgs
llog_reader /mnt/mgs/CONFIGS/tfs-client</screen>
<para>To examine the same log file on a running Lustre server, use the ldiskfs-enabled debugfs utility (called debug.ldiskfs on some distributions) to extract the file, for example:</para>
<para> <literal>-t <replaceable>timestamp</replaceable></literal></para>
</entry>
<entry>
- <para> Sets the test start time as printed at the start of a previously-interrupted test to ensure that validation data is the same across the entire filesystem (default value is the current time()).</para>
+ <para> Sets the test start time as printed at the start of a previously-interrupted
+ test to ensure that validation data is the same across the entire file system
+ (default value is the current time()).</para>
</entry>
</row>
<row>
<section xml:id="dbdoclet.50438219_75432">
<title><indexterm><primary>mkfs.lustre</primary></indexterm>
mkfs.lustre</title>
- <para>The mkfs.lustre utility formats a disk for a Lustre service.</para>
+ <para>The <literal>mkfs.lustre</literal> utility formats a disk for a Lustre service.</para>
<section remap="h5">
<title>Synopsis</title>
<screen>mkfs.lustre <replaceable>target_type</replaceable> [options] <replaceable>device</replaceable></screen>
<para> <literal>--ost</literal></para>
</entry>
<entry>
- <para> Object Storage Target (OST)</para>
+ <para> Object storage target (OST)</para>
</entry>
</row>
<row>
<para> <literal>--mdt</literal></para>
</entry>
<entry>
- <para> Metadata Storage Target (MDT)</para>
+ <para> Metadata storage target (MDT)</para>
</entry>
</row>
<row>
<para> <literal>--mgs</literal></para>
</entry>
<entry>
- <para> Configuration Management Service (MGS), one per site. This service can be combined with one <literal>--mdt</literal> service by specifying both types.</para>
+ <para> Configuration management service (MGS), one per site. This service can be
+ combined with one <literal>--mdt</literal> service by specifying both
+ types.</para>
</entry>
</row>
</tbody>
</section>
<section remap="h5">
<title>Description</title>
- <para>mkfs.lustre is used to format a disk device for use as part of a Lustre file system. After formatting, a disk can be mounted to start the Lustre service defined by this command.</para>
- <para>When the file system is created, parameters can simply be added as a --param option to the mkfs.lustre command. See <xref linkend="dbdoclet.50438194_17237"/>.</para>
+ <para><literal>mkfs.lustre</literal> is used to format a disk device for use as part of a
+ Lustre file system. After formatting, a disk can be mounted to start the Lustre service
+ defined by this command.</para>
+ <para>When the file system is created, parameters can simply be added as a
+ <literal>--param</literal> option to the <literal>mkfs.lustre</literal> command. See <xref
+ linkend="dbdoclet.50438194_17237"/>.</para>
<informaltable frame="all">
<tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
+ <colspec colname="c1" colwidth="1*"/>
+ <colspec colname="c2" colwidth="1*"/>
+ <colspec colname="c3" colwidth="3*"/>
<thead>
<row>
<entry nameend="c2" namest="c1">
<para> <literal>--backfstype=<replaceable>fstype</replaceable></literal></para>
</entry>
<entry>
- <para> Forces a particular format for the backing file system (such as ext3, ldiskfs).</para>
+ <para> Forces a particular format for the backing file system such as ldiskfs (the default) or zfs.</para>
</entry>
</row>
<row>
<para> <literal>--comment=<replaceable>comment</replaceable></literal></para>
</entry>
<entry>
- <para> Sets a user comment about this disk, ignored by Lustre.</para>
+ <para> Sets a user comment about this disk, ignored by the Lustre software.</para>
</entry>
</row>
<row>
<para> <literal>--device-size=<replaceable>#</replaceable>>KB</literal></para>
</entry>
<entry>
- <para> Sets the device size for loop devices.</para>
+ <para>Sets the device size for loop devices.</para>
</entry>
</row>
<row>
<para> <literal>--dryrun</literal></para>
</entry>
<entry>
- <para> Only prints what would be done; it does not affect the disk.</para>
+ <para>Only prints what would be done; it does not affect the disk.</para>
</entry>
</row>
<row>
- <entry nameend="c2" namest="c1">
- <para> <literal>--failnode=<replaceable>nid,...</replaceable></literal></para>
- </entry>
- <entry>
- <para> Sets the NID(s) of a failover partner. This option can be repeated as needed.</para>
- <warning><para>This cannot be used with <literal>--servicenode</literal>.</para></warning>
- </entry>
+ <entry nameend="c2" namest="c1"
+ ><literal>--servicenode=<replaceable>nid,...</replaceable></literal></entry>
+ <entry>Sets the NID(s) of all service nodes, including primary and failover partner
+ service nodes. The <literal>--servicenode</literal> option cannot be used with
+ <literal>--failnode</literal> option. See <xref
+ xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438188_92688"/> for
+ more details.</entry>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <literal>--servicenode=<replaceable>nid,...</replaceable></literal></para>
+ <para> <literal>--failnode=<replaceable>nid,...</replaceable></literal></para>
</entry>
<entry>
- <para> Sets the NID(s) of all service node, including failover partner as well as primary node service nids. This option can be repeated as needed.</para>
- <warning><para>This cannot be used with <literal>--failnode</literal>.</para></warning>
+ <para>Sets the NID(s) of a failover service node for a primary server for a target.
+ The <literal>--failnode</literal> option cannot be used with
+ <literal>--servicenode</literal> option. See <xref
+ xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438188_92688"/>
+ for more details.<note>
+ <para>When the <literal>--failnode</literal> option is used, certain
+ restrictions apply (see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="dbdoclet.50438188_92688"/>).</para>
+ </note></para>
</entry>
</row>
<row>
<para> <literal>--fsname=<replaceable>filesystem_name</replaceable></literal></para>
</entry>
<entry>
- <para> The Lustre file system of which this service/node will be a part. The default file system name is 'lustreâ€.</para>
+ <para> The Lustre file system of which this service/node will be a part. The default
+ file system name is <literal>lustre</literal>.</para>
<para> </para>
<note>
<para>The file system name is limited to 8 characters.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <literal>--index=<replaceable>index</replaceable></literal></para>
+ <para>
+ <literal>--index=<replaceable>index_number</replaceable></literal></para>
</entry>
<entry>
- <para> Specifies the OST or MDT number. This should always be used when formatting OSTs, in order to ensure that there is a simple mapping between the OST index and the OSS node and device it is located on.</para>
+ <para>Specifies the OST or MDT number (0...N). This allows mapping between the OSS
+ and MDS node and the device on which the OST or MDT is located.</para>
</entry>
</row>
<row>
</entry>
<entry>
<para> Sets the mount options used when the backing file system is mounted.</para>
- <warning><para>Unlike earlier versions of mkfs.lustre, this version completely replaces the default mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para></warning>
+ <warning><para>Unlike earlier versions of <literal>mkfs.lustre</literal>, this version completely replaces
+ the default mount options with those specified on the command line, and issues a
+ warning on stderr if any default mount options are omitted.</para></warning>
<para>The defaults for ldiskfs are:</para>
- <para>OST: <literal>errors=remount-ro</literal>;</para>
<para>MGS/MDT: <literal>errors=remount-ro,iopen_nopriv,user_xattr</literal></para>
- <para>Do not alter the default mount options unless you know what you are doing.</para>
+ <para>OST: <literal>errors=remount-ro,extents,mballoc</literal></para>
+ <para condition='l25'>OST: <literal>errors=remount-ro</literal></para>
+ <para>Use care when altering the default mount options.</para>
</entry>
</row>
<row>
<title>Examples</title>
<para>Creates a combined MGS and MDT for file system <literal>testfs</literal> on, e.g., node <literal>cfs21</literal>:</para>
<screen>mkfs.lustre --fsname=testfs --mdt --mgs /dev/sda1</screen>
- <para>Creates an OST for file system <literal>testfs</literal> on any node (using the above MGS):</para>
+ <para>Creates an OST for file system <literal>testfs</literal> on any node (using the above
+ MGS):</para>
<screen>mkfs.lustre --fsname=testfs --mgsnode=cfs21@tcp0 --ost --index=0 /dev/sdb</screen>
<para>Creates a standalone MGS on, e.g., node <literal>cfs22</literal>:</para>
<screen>mkfs.lustre --mgs /dev/sda1</screen>
<para> </para>
</entry>
<entry>
- <para> Mounts the Lustre file system named <literal>fsname</literal> on the client by contacting the Management Service at <literal>mgsspec</literal> on the pathname given by <literal>directory</literal>. The format for <literal>mgsspec</literal> is defined below. A mounted client file system appears in fstab(5) and is usable, like any local file system, and provides a full POSIX-compliant interface.</para>
+ <para> Mounts the Lustre file system named <literal>fsname</literal> on the client
+ by contacting the Management Service at <literal>mgsspec</literal> on the pathname
+ given by <literal>directory</literal>. The format for <literal>mgsspec</literal>
+ is defined below. A mounted client file system appears in fstab(5) and is usable,
+ like any local file system, and provides a full POSIX standard-compliant
+ interface.</para>
</entry>
</row>
<row>
<para> Disables Access Control List support.</para>
</entry>
</row>
- </tbody>
- </tgroup>
- </informaltable>
- <para>In addition to the standard mount options and backing disk type (e.g. ext3) options, Lustre understands the following server-specific options:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
<row>
<entry>
- <para><emphasis role="bold">Option</emphasis></para>
+ <para> <literal>verbose</literal></para>
</entry>
<entry>
- <para><emphasis role="bold">Description</emphasis></para>
+ <para> Enable mount/umount console messages.</para>
</entry>
</row>
- </thead>
- <tbody>
<row>
<entry>
- <para> <literal>nosvc</literal></para>
+ <para> <literal>noverbose</literal></para>
</entry>
<entry>
- <para> Starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para>
+ <para> Disable mount/umount console messages.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>nomsgs</literal></para>
+ <para> <literal>user_fid2path</literal></para>
</entry>
<entry>
- <para> Starts only the MDT (with a co-located MGS), without starting the MGS.</para>
+ <para condition='l23'>Enable FID to path translation by regular users. Note: This option allows a potential security hole because it allows regular users direct access to a file by its FID, bypassing POSIX path-based permission checks which could otherwise prevent the user from accessing a file in a directory that they do not have access to. Regular permission checks are still performed on the file itself, so the user cannot access a file to which they have no access rights.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>exclude=<replaceable>ostlist</replaceable></literal></para>
+ <para> <literal>nouser_fid2path</literal></para>
</entry>
<entry>
- <para> Starts a client or MDT with a colon-separated list of known inactive OSTs.</para>
+ <para condition='l23'> Disable FID to path translation by regular users. Root and processes with CAP_DAC_READ_SEARCH can still perform FID to path translation.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para>In addition to the standard mount options and backing disk type (e.g. ext3) options, Lustre understands the following server-specific options:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
</entry>
</row>
+ </thead>
+ <tbody>
<row>
<entry>
<para> <literal>nosvc</literal></para>
</entry>
<entry>
- <para> Only starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para>
+ <para> Starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>nomsgs</literal></para>
+ <para> <literal>nomgs</literal></para>
</entry>
<entry>
- <para> Starts a MDT with a co-located MGS, without starting the MGS.</para>
+ <para> Starts only the MDT (with a co-located MGS), without starting the MGS.</para>
</entry>
</row>
<row>
<para> <literal>exclude=<replaceable>ostlist</replaceable></literal></para>
</entry>
<entry>
- <para> Starts a client or MDT with a (colon-separated) list of known inactive OSTs.</para>
+ <para> Starts a client or MDT with a colon-separated list of known inactive OSTs.</para>
</entry>
</row>
<row>
<para> <literal>recovery_time_soft=<replaceable>timeout</replaceable></literal></para>
</entry>
<entry>
- <para> Allows <literal>timeout</literal> seconds for clients to reconnect for recovery after a server crash. This timeout is incrementally extended if it is about to expire and the server is still handling new connections from recoverable clients. The default soft recovery timeout is 300 seconds (5 minutes).</para>
+ <para> Allows <literal>timeout</literal> seconds for clients to reconnect for recovery after a server crash. This timeout is incrementally extended if it is about to expire and the server is still handling new connections from recoverable clients.</para>
+ <para> The default soft recovery timeout is 3 times the value of the Lustre timeout parameter (see <xref linkend="section_c24_nt5_dl"/>). The default Lustre timeout is 100 seconds, which would make the soft recovery timeout default to 300 seconds (5 minutes). The soft recovery timeout is set at mount time and will not change if the Lustre timeout is changed after mount time.</para>
</entry>
</row>
<row>
<para> <literal>recovery_time_hard=<replaceable>timeout</replaceable></literal></para>
</entry>
<entry>
- <para> The server is allowed to incrementally extend its timeout up to a hard maximum of <literal>timeout</literal> seconds. The default hard recovery timeout is set to 900 seconds (15 minutes).</para>
+ <para> The server is allowed to incrementally extend its timeout up to a hard maximum of <literal>timeout</literal> seconds.</para>
+ <para> The default hard recovery timeout is 9 times the value of the Lustre timeout parameter (see <xref linkend="section_c24_nt5_dl"/>). The default Lustre timeout is 100 seconds, which would make the hard recovery timeout default to 900 seconds (15 minutes). The hard recovery timeout is set at mount time and will not change if the Lustre timeout is changed after mount time.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>noscrub</literal></para>
+ </entry>
+ <entry>
+ <para>Typically the MDT will detect restoration from a file-level backup during mount. This mount option prevents the OI Scrub from starting automatically when the MDT is mounted. Manually starting LFSCK after mounting provides finer control over the starting conditions. This mount option also prevents OI scrub from occurring automatically when OI inconsistency is detected (see <xref linkend="dbdoclet.lfsck_auto_scrub"/>)</para>
</entry>
</row>
</tbody>
</section>
<section remap="h5">
<title>Description</title>
- <para>The routerstat utility watches LNET router statistics. If no <literal><replaceable>interval</replaceable></literal> is specified, then statistics are sampled and printed only one time. Otherwise, statistics are sampled and printed at the specified <literal><replaceable>interval</replaceable></literal> (in seconds).</para>
+ <para>The routerstat utility displays LNET router statistics. If no <literal><replaceable>interval</replaceable></literal> is specified, then statistics are sampled and printed only one time. Otherwise, statistics are sampled and printed at the specified <literal><replaceable>interval</replaceable></literal> (in seconds).</para>
</section>
<section remap="h5">
- <title>Options</title>
+ <title>Output</title>
<para>The routerstat output includes the following fields:</para>
<informaltable frame="all">
<tgroup cols="2">
<thead>
<row>
<entry>
- <para><emphasis role="bold">Option</emphasis></para>
+ <para><emphasis role="bold">Output</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>M</literal></para>
+ </entry>
+ <entry>
+ <para> Number of messages currently being processed by LNET (The maximum number of messages ever processed by LNET concurrently)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>E</literal></para>
+ </entry>
+ <entry>
+ <para> Number of LNET errors</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>S</literal></para>
+ </entry>
+ <entry>
+ <para> Total size (length) of messages sent in bytes/ Number of messages sent</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>R</literal></para>
+ </entry>
+ <entry>
+ <para> Total size (length) of messages received in bytes/ Number of messages received</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>F</literal></para>
+ </entry>
+ <entry>
+ <para> Total size (length) of messages routed in bytes/ Number of messages routed</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>D</literal></para>
+ </entry>
+ <entry>
+ <para> Total size (length) of messages dropped in bytes/ Number of messages dropped</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para>When an <literal><replaceable>interval</replaceable></literal> is specified, additional lines of statistics are printed including the following fields:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Output</emphasis></para>
</entry>
<entry>
<para><emphasis role="bold">Description</emphasis></para>
<para> <literal>M</literal></para>
</entry>
<entry>
- <para> msgs_alloc(msgs_max)</para>
+ <para> Number of messages currently being processed by LNET (The maximum number of messages ever processed by LNET concurrently)</para>
</entry>
</row>
<row>
<para> <literal>E</literal></para>
</entry>
<entry>
- <para> errors</para>
+ <para> Number of LNET errors per second</para>
</entry>
</row>
<row>
<para> <literal>S</literal></para>
</entry>
<entry>
- <para> send_count/send_length</para>
+ <para> Rate of data sent in Mbytes per second/ Count of messages sent per second</para>
</entry>
</row>
<row>
<para> <literal>R</literal></para>
</entry>
<entry>
- <para> recv_count/recv_length</para>
+ <para> Rate of data received in Mbytes per second/ Count of messages received per second</para>
</entry>
</row>
<row>
<para> <literal>F</literal></para>
</entry>
<entry>
- <para> route_count/route_length</para>
+ <para> Rate of data routed in Mbytes per second/ Count of messages routed per second</para>
</entry>
</row>
<row>
<para> <literal>D</literal></para>
</entry>
<entry>
- <para> drop_count/drop_length</para>
+ <para> Rate of data dropped in Mbytes per second/ Count of messages dropped per second</para>
</entry>
</row>
</tbody>
</informaltable>
</section>
<section remap="h5">
+ <title>Example</title>
+ <screen># routerstat 1
+M 0(13) E 0 S 117379184/4250 R 878480/4356 F 0/0 D 0/0
+M 0( 13) E 0 S 7.00/ 7 R 0.00/ 14 F 0.00/ 0 D 0.00/0
+M 0( 13) E 0 S 7.00/ 7 R 0.00/ 14 F 0.00/ 0 D 0.00/0
+M 0( 13) E 0 S 8.00/ 8 R 0.00/ 16 F 0.00/ 0 D 0.00/0
+M 0( 13) E 0 S 7.00/ 7 R 0.00/ 14 F 0.00/ 0 D 0.00/0
+M 0( 13) E 0 S 7.00/ 7 R 0.00/ 14 F 0.00/ 0 D 0.00/0
+M 0( 13) E 0 S 7.00/ 7 R 0.00/ 14 F 0.00/ 0 D 0.00/0
+M 0( 13) E 0 S 7.00/ 7 R 0.00/ 14 F 0.00/ 0 D 0.00/0
+M 0( 13) E 0 S 8.00/ 8 R 0.00/ 16 F 0.00/ 0 D 0.00/0
+M 0( 13) E 0 S 7.00/ 7 R 0.00/ 14 F 0.00/ 0 D 0.00/0
+...</screen>
+ </section>
+ <section remap="h5">
<title>Files</title>
<para>The routerstat utility extracts statistics data from:</para>
<screen>/proc/sys/lnet/stats</screen>
</row>
<row>
<entry>
- <para> <literal>--failnode=<replaceable>nid,...</replaceable></literal></para>
- </entry>
- <entry>
- <para> Sets the NID(s) of a failover partner. This option can be repeated as needed.</para>
- <warning><para>Cannot be used with <literal>--servicenode</literal>.</para></warning>
- </entry>
+ <literal>--servicenode=<replaceable>nid,...</replaceable></literal></entry>
+ <entry>Sets the NID(s) of all service nodes, including primary and failover partner
+ service nodes. The <literal>--servicenode</literal> option cannot be used with
+ <literal>--failnode</literal> option. See <xref
+ xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438188_92688"/> for
+ more details.</entry>
</row>
<row>
<entry>
- <para> <literal>--servicenode=<replaceable>nid,...</replaceable></literal></para>
+ <para> <literal>--failnode=<replaceable>nid,...</replaceable></literal></para>
</entry>
<entry>
- <para> Sets the NID(s) of all service node, including failover partner as well as local service nids. This option can be repeated as needed.</para>
- <warning><para>: Cannot be used with <literal>--failnode</literal>.</para></warning>
+ <para>Sets the NID(s) of a failover service node for a primary server for a target.
+ The <literal>--failnode</literal> option cannot be used with
+ <literal>--servicenode</literal> option. See <xref
+ xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438188_92688"/>
+ for more details.<note>
+ <para>When the <literal>--failnode</literal> option is used, certain
+ restrictions apply (see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="dbdoclet.50438188_92688"/>).</para>
+ </note></para>
</entry>
</row>
<row>
<para> <literal>--fsname=<replaceable>filesystem_name</replaceable></literal></para>
</entry>
<entry>
- <para> The Lustre file system of which this service will be a part. The default file system name is 'lustreâ€.</para>
+ <para> The Lustre file system of which this service will be a part. The default file
+ system name is <literal>lustre</literal>.</para>
</entry>
</row>
<row>
<para> Sets the mount options used when the backing file system is mounted.</para>
<warning><para> Unlike earlier versions of tunefs.lustre, this version completely replaces the existing mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para></warning>
<para>The defaults for ldiskfs are:</para>
- <para>OST: <literal>errors=remount-ro,mballoc,extents</literal>;</para>
<para>MGS/MDT: <literal>errors=remount-ro,iopen_nopriv,user_xattr</literal></para>
+ <para>OST: <literal>errors=remount-ro,extents,mballoc</literal></para>
+ <para condition='l25'>OST: <literal>errors=remount-ro</literal></para>
<para>Do not alter the default mount options unless you know what you are doing.</para>
</entry>
</row>
<para> <literal>--writeconf</literal></para>
</entry>
<entry>
- <para> Erases all configuration logs for the file system to which this MDT belongs, and regenerates them. This is dangerous operation. All clients must be unmounted and servers for this file system should be stopped. All targets (OSTs/MDTs) must then be restarted to regenerate the logs. No clients should be started until all targets have restarted.</para>
- <para> </para>
+ <para> Erases all configuration logs for the file system to which this MDT belongs,
+ and regenerates them. This is dangerous operation. All clients must be unmounted
+ and servers for this file system should be stopped. All targets (OSTs/MDTs) must
+ then be restarted to regenerate the logs. No clients should be started until all
+ targets have restarted.</para>
<para>The correct order of operations is:</para>
- <para>* Unmount all clients on the file system</para>
- <para>* Unmount the MDT and all OSTs on the file system</para>
- <para>* Run <literal>tunefs.lustre --writeconf <replaceable>device</replaceable></literal> on every server</para>
- <para>* Mount the MDT and OSTs</para>
- <para>* Mount the clients</para>
+ <orderedlist>
+ <listitem>
+ <para>Unmount all clients on the file system</para>
+ </listitem>
+ <listitem>
+ <para>Unmount the MDT and all OSTs on the file system</para>
+ </listitem>
+ <listitem>
+ <para>Run <literal>tunefs.lustre --writeconf
+ <replaceable>device</replaceable></literal> on every server</para>
+ </listitem>
+ <listitem>
+ <para>Mount the MDT and OSTs</para>
+ </listitem>
+ <listitem>
+ <para>Mount the clients</para>
+ </listitem>
+ </orderedlist>
</entry>
</row>
</tbody>
<para>The following utilities are located in /usr/bin.</para>
<para><literal>lustre_req_history.sh</literal></para>
<para>The lustre_req_history.sh utility (run from a client), assembles as much Lustre RPC request history as possible from the local node and from the servers that were contacted, providing a better picture of the coordinated network activity.</para>
- <para><literal>llstat.sh</literal></para>
- <para>The llstat.sh utility handles a wider range of statistics files, and has command line switches to produce more graphable output.</para>
- <para><literal>plot-llstat.sh</literal></para>
- <para>The plot-llstat.sh utility plots the output from llstat.sh using gnuplot.</para>
</section>
<section remap="h3">
<title>More /proc Statistics for Application Profiling</title>
quit
EOF
</screen>
- <para>Feature Requests</para>
- <para>The loadgen utility is intended to grow into a more comprehensive test tool; feature requests are encouraged. The current feature requests include:</para>
- <itemizedlist>
- <listitem>
- <para> Locking simulation</para>
- </listitem>
- <listitem>
- <itemizedlist>
- <listitem>
- <para> Many (echo) clients cache locks for the specified resource at the same time.</para>
- </listitem>
- <listitem>
- <para> Many (echo) clients enqueue locks for the specified resource simultaneously.</para>
- </listitem>
- </itemizedlist>
- </listitem>
- <listitem>
- <para> obdsurvey functionality</para>
- </listitem>
- <listitem>
- <itemizedlist>
- <listitem>
- <para> Fold the Lustre I/O kit's obdsurvey script functionality into loadgen</para>
- </listitem>
- </itemizedlist>
- </listitem>
- </itemizedlist>
- </section>
- <section remap="h5">
- <title><indexterm><primary>llog_reader</primary></indexterm>
-llog_reader</title>
- <para>The llog_reader utility translates a Lustre configuration log into human-readable form.</para>
- </section>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>llog_reader filename
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>llog_reader parses the binary format of Lustre's on-disk configuration logs. It can only read the logs. Use tunefs.lustre to write to them.</para>
- <para>To examine a log file on a stopped Lustre server, mount its backing file system as ldiskfs, then use llog_reader to dump the log file's contents. For example:</para>
- <screen>mount -t ldiskfs /dev/sda /mnt/mgs
-llog_reader /mnt/mgs/CONFIGS/tfs-client
-</screen>
- <para>To examine the same log file on a running Lustre server, use the ldiskfs-enabled debugfs utility (called debug.ldiskfs on some distributions) to extract the file. For example:</para>
- <screen>debugfs -c -R 'dump CONFIGS/tfs-client /tmp/tfs-client' /dev/sda
-llog_reader /tmp/tfs-client
-</screen>
- <caution>
- <para>Although they are stored in the CONFIGS directory, mountdata files do not use the config log format and will confuse llog_reader.</para>
- </caution>
- <para>See Also <xref linkend="dbdoclet.50438219_39574"/></para>
</section>
<section remap="h5">
<title><indexterm><primary>lr_reader</primary></indexterm>
<para>The following utilities are part of the Lustre I/O kit. For more information, see <xref linkend="benchmarkingtests"/>.</para>
</section>
<section remap="h5">
- <title><indexterm><primary>sgpdd_survey</primary></indexterm>
-sgpdd_survey</title>
- <para>The sgpdd_survey utility tests 'bare metal' performance, bypassing as much of the kernel as possible. The sgpdd_survey tool does not require Lustre, but it does require the sgp_dd package.</para>
+ <title><indexterm>
+ <primary>sgpdd-survey</primary>
+ </indexterm> sgpdd-survey</title>
+ <para>The <literal>sgpdd-survey</literal> utility tests 'bare metal' performance,
+ bypassing as much of the kernel as possible. The <literal>sgpdd-survey</literal> tool does
+ not require Lustre, but it does require the sgp_dd package.</para>
<caution>
- <para>The sgpdd_survey utility erases all data on the device.</para>
+ <para>The <literal>sgpdd-survey</literal> utility erases all data on the device.</para>
</caution>
</section>
<section remap="h5">
- <title><indexterm><primary>obdfilter_survey</primary></indexterm>obdfilter_survey</title>
- <para>The obdfilter_survey utility is a shell script that tests performance of isolated OSTS, the network via echo clients, and an end-to-end test.</para>
+ <title><indexterm>
+ <primary>obdfilter-survey</primary>
+ </indexterm>obdfilter-survey</title>
+ <para>The <literal>obdfilter-survey</literal> utility is a shell script that tests
+ performance of isolated OSTS, the network via echo clients, and an end-to-end test.</para>
</section>
<section remap="h5">
<title><indexterm><primary>ior-survey</primary></indexterm>ior-survey</title>
<para>The ior-survey utility is a script used to run the IOR benchmark. Lustre includes IOR version 2.8.6.</para>
</section>
<section remap="h5">
- <title><indexterm><primary>ost_survey</primary></indexterm>ost_survey</title>
- <para>The ost_survey utility is an OST performance survey that tests client-to-disk performance of the individual OSTs in a Lustre file system.</para>
+ <title><indexterm>
+ <primary>ost-survey</primary>
+ </indexterm>ost-survey</title>
+ <para>The <literal>ost-survey</literal> utility is an OST performance survey that tests
+ client-to-disk performance of the individual OSTs in a Lustre file system.</para>
</section>
<section remap="h5">
<title><indexterm><primary>stats-collect</primary></indexterm>stats-collect</title>
git://git.whamcloud.com / doc / manual.git / blobdiff