<title><indexterm><primary>e2scan</primary></indexterm>
e2scan</title>
<para>The e2scan utility is an ext2 file system-modified inode scan program. The e2scan program uses libext2fs to find inodes with ctime or mtime newer than a given time and prints out their pathname. Use e2scan to efficiently generate lists of files that have been modified. The e2scan tool is included in the e2fsprogs package, located at:</para>
- <para><link xl:href="http://downloads.hpdd.intel.com/public/e2fsprogs/latest/">http://downloads.hpdd.intel.com/public/e2fsprogs/latest/</link></para>
+ <para><link xl:href="http://downloads.whamcloud.com/public/e2fsprogs/latest/">http://downloads.whamcloud.com/public/e2fsprogs/latest/</link></para>
<section remap="h5">
<title>Synopsis</title>
<screen>e2scan [options] [-f file] block_device</screen>
<tbody>
<row>
<entry>
- <para> <literal>network up|down|tcp|elan|myrinet</literal></para>
+ <para> <literal>network up|down|tcp|elan</literal></para>
</entry>
<entry>
- <para> Starts or stops LNET, or selects a network type for other <literal>lctl</literal> LNET commands.</para>
+ <para> Starts or stops LNet, or selects a network type for other <literal>lctl</literal> LNet commands.</para>
</entry>
</row>
<row>
<para> <literal>list_nids</literal></para>
</entry>
<entry>
- <para> Prints all NIDs on the local node. LNET must be running.</para>
+ <para> Prints all NIDs on the local node. LNet must be running.</para>
</entry>
</row>
<row>
<para> <literal>ping <replaceable>nid</replaceable></literal></para>
</entry>
<entry>
- <para> Checks LNET connectivity via an LNET ping. This uses the fabric appropriate to the specified NID.</para>
+ <para> Checks LNet connectivity via an LNet ping. This uses the fabric appropriate to the specified NID.</para>
</entry>
</row>
<row>
<para> <literal>list_param [-F|-R] <replaceable>parameter</replaceable> <replaceable>[parameter ...]</replaceable></literal></para>
</entry>
<entry>
- <para> Lists the Lustre or LNET parameter name.</para>
+ <para> Lists the Lustre or LNet parameter name.</para>
<para> </para>
</entry>
</row>
<para> <literal>get_param [-n|-N|-F] <replaceable>parameter</replaceable> <replaceable>[parameter ...]</replaceable></literal></para>
</entry>
<entry>
- <para> Gets the value of a Lustre or LNET parameter from the specified path.</para>
+ <para> Gets the value of a Lustre or LNet parameter from the specified path.</para>
</entry>
</row>
<row>
<para> <literal>set_param [-n] <replaceable>parameter</replaceable>=<replaceable>value</replaceable></literal></para>
</entry>
<entry>
- <para> Sets the value of a Lustre or LNET parameter from the specified path.</para>
+ <para> Sets the value of a Lustre or LNet parameter from the specified path.</para>
</entry>
</row>
<row>
<para> <literal>changelog_register</literal></para>
</entry>
<entry>
- <para> Registers a new changelog user for a particular device. Changelog entries are not purged beyond a registered user's set point (see <literal>lfs changelog_clear</literal>).</para>
+ <para> Registers a new changelog user for a particular device.
+ Changelog entries are saved persistently on the MDT with each
+ filesystem operation, and are only purged beyond all registered
+ user's minimum set point (see
+ <literal>lfs changelog_clear</literal>). This may cause the
+ Changelog to consume a large amount of space, eventually
+ filling the MDT, if a changelog user is registered but never
+ consumes those records.</para>
</entry>
</row>
<row>
<para>changelog_deregister <replaceable>id</replaceable></para>
</entry>
<entry>
- <para> Unregisters an existing changelog user. If the user's "clear" record number is the minimum for the device, changelog records are purged until the next minimum.</para>
+ <para> Unregisters an existing changelog user. If the
+ user's "clear" record number is the minimum for
+ the device, changelog records are purged until the next minimum.
+ </para>
</entry>
</row>
</tbody>
<para><xref linkend="dbdoclet.50438219_44971"/></para>
</section>
</section>
- <section xml:id="dbdoclet.50438219_44971">
+ <section xml:id="dbdoclet.50438219_44971" condition='l28'>
<title><indexterm><primary>ll_recover_lost_found_objs</primary></indexterm>
ll_recover_lost_found_objs</title>
- <para>The ll_recover_lost_found_objs utility helps recover Lustre OST objects (file data) from a lost and found directory and return them to their correct locations.</para>
- <note>
- <para>Running the ll_recover_lost_found_objs tool is not strictly necessary to bring an OST back online, it just avoids losing access to objects that were moved to the lost and found directory due to directory corruption.</para>
+ <para>The <literal>ll_recover_lost_found_objs</literal> utility was
+ used to help recover Lustre OST objects (file data) from the
+ <literal>lost+found</literal> directory of an OST and return them to
+ their correct locations based on information stored in the
+ <literal>trusted.fid</literal> extended attribute stored on every
+ OST object containing data.</para>
+ <note condition="l26"><para>This utility is not needed with Lustre 2.6
+ and later, and is removed in Lustre 2.8 since <literal>LFSCK</literal>
+ online scanning will automatically move objects from
+ <literal>lost+found</literal> to the proper place in the OST.</para>
+ </note>
+ <note condition='l25'>
+ <para>The <literal>ll_recover_lost_found_objs</literal> tool is not
+ strictly necessary to bring an OST back online, it just avoids losing
+ access to objects that were moved to the lost+found directory due to
+ directory corruption on the OST.</para>
</note>
<section remap="h5">
<title>Synopsis</title>
<title>Description</title>
<para>The first time Lustre modifies an object, it saves the MDS inode number and the objid as an extended attribute on the object, so in case of directory corruption of the OST, it is possible to recover the objects. Running e2fsck fixes the corrupted OST directory, but it puts all of the objects into a lost and found directory, where they are inaccessible to Lustre. Use the ll_recover_lost_found_objs utility to recover all (or at least most) objects from a lost and found directory and return them to the O/0/d* directories.</para>
<para>To use ll_recover_lost_found_objs, mount the file system locally (using the <literal>-t ldiskfs</literal>, or <literal>-t zfs</literal> command), run the utility and then unmount it again. The OST must not be mounted by Lustre when ll_recover_lost_found_objs is run.</para>
- <para condition="l26">This utility is not needed for 2.6 and later,
- since the <literal>LFSCK</literal> online scanning will move objects
- from <literal>lost+found</literal> to the proper place in the OST.</para>
</section>
<section remap="h5">
<title>Options</title>
<title>Files</title>
<para>The llstat files are located at:</para>
<screen>/proc/fs/lustre/mdt/MDS/*/stats
-/proc/fs/lustre/mds/*/exports/*/stats
+/proc/fs/lustre/mdt/*/exports/*/stats
/proc/fs/lustre/mdc/*/stats
/proc/fs/lustre/ldlm/services/*/stats
/proc/fs/lustre/ldlm/namespaces/*/pool/stats
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <literal>>-f|--force</literal>></para>
+ <para> <literal>-f|--force</literal></para>
</entry>
<entry>
<para> Forces the test to run without a confirmation that the device will be overwritten and all data will be permanently destroyed.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <literal>-o <replaceable>>offset</replaceable></literal></para>
+ <para> <literal>-o <replaceable>offset</replaceable></literal></para>
</entry>
<entry>
<para> Offset (in kilobytes) of the start of the test (default value is 0).</para>
<section xml:id="dbdoclet.50438219_90218">
<title><indexterm><primary>lst</primary></indexterm>
lst</title>
- <para>The lst utility starts LNET self-test.</para>
+ <para>The lst utility starts LNet self-test.</para>
<section remap="h5">
<title>Synopsis</title>
<screen>lst</screen>
</section>
<section remap="h5">
<title>Description</title>
- <para>LNET self-test helps site administrators confirm that Lustre Networking (LNET) has been properly installed and configured. The self-test also confirms that LNET and the network software and hardware underlying it are performing as expected.</para>
- <para>Each LNET self-test runs in the context of a session. A node can be associated with only one session at a time, to ensure that the session has exclusive use of the nodes on which it is running. A session is create, controlled and monitored from a single node; this is referred to as the self-test console.</para>
+ <para>LNet self-test helps site administrators confirm that Lustre Networking (LNet) has been properly installed and configured. The self-test also confirms that LNet and the network software and hardware underlying it are performing as expected.</para>
+ <para>Each LNet self-test runs in the context of a session. A node can be associated with only one session at a time, to ensure that the session has exclusive use of the nodes on which it is running. A session is create, controlled and monitored from a single node; this is referred to as the self-test console.</para>
<para>Any node may act as the self-test console. Nodes are named and allocated to a self-test session in groups. This allows all nodes in a group to be referenced by a single name.</para>
<para>Test configurations are built by describing and running test batches. A test batch is a named collection of tests, with each test composed of a number of individual point-to-point tests running in parallel. These individual point-to-point tests are instantiated according to the test type, source group, target group and distribution specified when the test is added to the test batch.</para>
</section>
<section remap="h5">
<title>Modules</title>
- <para>To run LNET self-test, load these modules: libcfs, lnet, lnet_selftest and any one of the klnds (ksocklnd, ko2iblnd...). To load all necessary modules, run modprobe lnet_selftest, which recursively loads the modules on which lnet_selftest depends.</para>
- <para>There are two types of nodes for LNET self-test: the console node and test nodes. Both node types require all previously-specified modules to be loaded. (The userspace test node does not require these modules).</para>
+ <para>To run LNet self-test, load these modules: libcfs, lnet, lnet_selftest and any one of the klnds (ksocklnd, ko2iblnd...). To load all necessary modules, run modprobe lnet_selftest, which recursively loads the modules on which lnet_selftest depends.</para>
+ <para>There are two types of nodes for LNet self-test: the console node and test nodes. Both node types require all previously-specified modules to be loaded. (The userspace test node does not require these modules).</para>
<para>Test nodes can be in either kernel or in userspace. A console user can invite a kernel test node to join the test session by running lst add_group NID, but the user cannot actively add a userspace test node to the test session. However, the console user can passively accept a test node to the test session while the test node runs lst client to connect to the console.</para>
</section>
<section remap="h5">
<title>Utilities</title>
- <para>LNET self-test includes two user utilities, lst and lstclient.</para>
+ <para>LNet self-test includes two user utilities, lst and lstclient.</para>
<para>lst is the user interface for the self-test console (run on the console node). It provides a list of commands to control the entire test system, such as create session, create test groups, etc.</para>
- <para>lstclient is the userspace self-test program which is linked with userspace LNDs and LNET. A user can invoke lstclient to join a self-test session:</para>
+ <para>lstclient is the userspace self-test program which is linked with userspace LNDs and LNet. A user can invoke lstclient to join a self-test session:</para>
<screen>lstclient -sesid CONSOLE_NID group NAME</screen>
</section>
<section remap="h5">
<title>Example Script</title>
- <para>This is a sample LNET self-test script which simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an IB network (connected via LNET routers), with half the clients reading and half the clients writing.</para>
+ <para>This is a sample LNet self-test script which simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an IB network (connected via LNet routers), with half the clients reading and half the clients writing.</para>
<screen>#!/bin/bash
export LST_SESSION=$$
lst new_session read/write
<section xml:id="dbdoclet.50438219_54734">
<title><indexterm><primary>lustre_rmmod.sh</primary></indexterm>
lustre_rmmod.sh</title>
- <para>The lustre_rmmod.sh utility removes all Lustre and LNET modules (assuming no Lustre services are running). It is located in /usr/bin.</para>
+ <para>The lustre_rmmod.sh utility removes all Lustre and LNet modules (assuming no Lustre services are running). It is located in /usr/bin.</para>
<note>
<para>The lustre_rmmod.sh utility does not work if Lustre modules are being used or if you have manually run the lctl network up command.</para>
</note>
<para>The mount.lustre utility starts a Lustre client or target service.</para>
<section remap="h5">
<title>Synopsis</title>
- <screen>mount -t lustre [-o options] directory
+ <screen>mount -t lustre [-o options] device mountpoint
</screen>
</section>
<section remap="h5">
<tbody>
<row>
<entry>
- <para> <literal><replaceable>mgs_nid</replaceable>:/<replaceable>fsname</replaceable></literal></para>
- <para> </para>
+ <para> <literal><replaceable>mgsname</replaceable>:/<replaceable>fsname</replaceable><replaceable>[/subdir]</replaceable></literal></para>
</entry>
<entry>
- <para> Mounts the Lustre file system named <literal>fsname</literal> on the client
- by contacting the Management Service at <literal>mgsspec</literal> on the pathname
- given by <literal>directory</literal>. The format for <literal>mgsspec</literal>
- is defined below. A mounted client file system appears in fstab(5) and is usable,
- like any local file system, and provides a full POSIX standard-compliant
- interface.</para>
+ <para> Mounts the Lustre file system named
+ <replaceable>fsname</replaceable> (optionally starting at
+ subdirectory <replaceable>subdir</replaceable> within the
+ filesystem, if specified) on the client at the directory
+ <replaceable>mountpoint</replaceable>, by contacting the Lustre
+ Management Service at <replaceable>mgsname</replaceable>. The
+ format for <replaceable>mgsname</replaceable> is defined below. A
+ client file system can be listed in <literal>fstab(5)</literal>
+ for automatic mount at boot time, is usable like any local file
+ system, and provides a full POSIX standard-compliant interface.
+ </para>
</entry>
</row>
<row>
<para> <replaceable>block_device</replaceable></para>
</entry>
<entry>
- <para> Starts the target service defined by the mkfs.lustre command on the physical disk <replaceable>block_device</replaceable>. A mounted target service file system is only useful for df(1) operations and appears in fstab(5) to show the device is in use.</para>
+ <para> Starts the target service defined by the
+ <literal>mkfs.lustre(8)</literal> command on the physical disk
+ <replaceable>block_device</replaceable>. The
+ <replaceable>block_device</replaceable> may be specified using
+ <literal>-L <replaceable>label</replaceable></literal> to find
+ the first block device with that label (e.g.
+ <literal>testfs-MDT0000</literal>), or by UUID using the
+ <literal>-U <replaceable>uuid</replaceable></literal> option.
+ Care should be taken if there is a device-level backup of the
+ target filesystem on the same node, which would have a
+ duplicate label and UUID if it has not been changed with
+ <literal>tune2fs(8)</literal> or similar. The mounted target
+ service filesystem mounted at
+ <replaceable>mountpoint</replaceable> is only useful for
+ <literal>df(1)</literal> operations and appears in
+ <literal>/proc/mounts</literal> to show the device is in use.
+ </para>
</entry>
</row>
</tbody>
<tbody>
<row>
<entry>
- <para> <literal>mgsspec:=<replaceable>mgsnode</replaceable>[:<replaceable>mgsnode</replaceable>]</literal></para>
- <para> </para>
+ <para> <literal>mgsname=<replaceable>mgsnode</replaceable>[:<replaceable>mgsnode</replaceable>]</literal></para>
+ </entry>
+ <entry>
+ <para><replaceable>mgsname</replaceable> is a colon-separated
+ list of <replaceable>mgsnode</replaceable> names where the MGS
+ service may run. Multiple <replaceable>mgsnode</replaceable>
+ values can be specified if the MGS service is configured for
+ HA failover and may be running on any one of the nodes.
+ </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>mgsnode=<replaceable>mgsnid</replaceable>[,<replaceable>mgsnid</replaceable>]</literal></para>
</entry>
<entry>
- <para> The MGS specification may be a colon-separated list of nodes.</para>
+ <para> Each <replaceable>mgsnode</replaceable> may specify a
+ comma-separated list of NIDs, if there are different LNet
+ interfaces for that <literal>mgsnode</literal>.
+ </para>
</entry>
</row>
<row>
<entry>
- <para> <literal>mgsnode:=<replaceable>mgsnid</replaceable>[,<replaceable>mgsnid</replaceable>]</literal></para>
+ <para> <literal>mgssec=<replaceable>flavor</replaceable></literal></para>
</entry>
<entry>
- <para> Each node may be specified by a comma-separated list of NIDs.</para>
+ <para>Specifies the encryption flavor for the initial network
+ RPC connection to the MGS. Non-security flavors are:
+ <literal>null</literal>, <literal>plain</literal>, and
+ <literal>gssnull</literal>, which respectively disable, or
+ have no encryption or integrity features for testing purposes.
+ Kerberos flavors are: <literal>krb5n</literal>,
+ <literal>krb5a</literal>, <literal>krb5i</literal>, and
+ <literal>krb5p</literal>. Shared-secret key flavors are:
+ <literal>skn</literal>, <literal>ska</literal>,
+ <literal>ski</literal>, and <literal>skpi</literal>, see the
+ <xref linkend="lustressk"/> for more details. The security
+ flavor for client-to-server connections is specified in the
+ filesystem configuration that the client fetches from the MGS.
+ </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>skpath=<replaceable>file|directory</replaceable></literal></para>
+ </entry>
+ <entry>
+ <para condition='l29'>
+ Path to a file or directory with the keyfile(s) to load for
+ this mount command. Keys are inserted into the
+ <literal>KEY_SPEC_SESSION_KEYRING</literal> keyring in the
+ kernel with a description containing
+ <literal>lustre:</literal> and a suffix which depends on
+ whether the context of the mount command is for an MGS,
+ MDT/OST, or client.
+ </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>exclude=<replaceable>ostlist</replaceable></literal></para>
+ </entry>
+ <entry>
+ <para>Starts a client or MDT with a colon-separated list of
+ known inactive OSTs that it will not try to connect to.</para>
</entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para>In addition to the standard mount options, Lustre understands the following client-specific options:</para>
+ <para>In addition to the standard mount(8) options, Lustre understands
+ the following client-specific options:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<tbody>
<row>
<entry>
+ <para><literal>always_ping</literal></para>
+ </entry>
+ <entry>
+ <para condition='l29'>The client will periodically ping the server when it is
+ idle, even if the server <literal>ptlrpc</literal> module
+ is configured with the <literal>suppress_pings</literal>
+ option. This allows clients to reliably use the filesystem
+ even if they are not part of an external client health
+ monitoring mechanism.
+ </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
<para> <literal>flock</literal></para>
</entry>
<entry>
- <para> Enables full flock support, coherent across all client nodes.</para>
+ <para>Enables advisory file locking support between
+ participating applications using the <literal>flock(2)</literal>
+ system call. This causes file locking to be coherent across all
+ client nodes also using this mount option. This is useful if
+ applications need coherent userspace file locking across
+ multiple client nodes, but also imposes communications overhead
+ in order to maintain locking consistency between client nodes.
+ </para>
</entry>
</row>
<row>
<para> <literal>localflock</literal></para>
</entry>
<entry>
- <para> Enables local flock support, using only client-local flock (faster, for applications that require flock, but do not run on multiple nodes).</para>
+ <para>Enables client-local <literal>flock(2)</literal> support,
+ using only client-local advisory file locking. This is faster
+ than using the global <literal>flock</literal> option, and can
+ be used for applications that depend on functioning
+ <literal>flock(2)</literal> but run only on a single node.
+ It has minimal overhead using only the Linux kernel's locks.
+ </para>
</entry>
</row>
<row>
<para> <literal>noflock</literal></para>
</entry>
<entry>
- <para> Disables flock support entirely. Applications calling flock get an error. It is up to the administrator to choose either <literal>localflock</literal> (fastest, low impact, not coherent between nodes) or <literal>flock</literal> (slower, performance impact for use, coherent between nodes).</para>
+ <para>Disables <literal>flock(2)</literal> support entirely,
+ and is the default option. Applications calling
+ <literal>flock(2)</literal> get an
+ <literal>ENOSYS</literal> error. It is up to the administrator
+ to choose either the <literal>localflock</literal> or
+ <literal>flock</literal> mount option based on their
+ requirements. It is possible to mount clients with different
+ options, and only those mounted with <literal>flock</literal>
+ will be coherent amongst each other.
+ </para>
</entry>
</row>
<row>
<entry>
- <para> <literal>user_xattr</literal></para>
+ <para> <literal>lazystatfs</literal></para>
</entry>
<entry>
- <para> Enables get/set of extended attributes by regular users. See the attr(5) manual page.</para>
+ <para>Allows <literal>statfs(2)</literal> (as used by
+ <literal>df(1)</literal> and <literal>lfs-df(1)</literal>) to
+ return even if some OST or MDT is unresponsive or has been
+ temporarily or permanently disabled in the configuration.
+ This avoids blocking until all of the targets are available.
+ This is the default behavior since Lustre 2.9.0.
+ </para>
</entry>
</row>
<row>
<entry>
- <para> <literal>nouser_xattr</literal></para>
+ <para> <literal>nolazystatfs</literal></para>
</entry>
<entry>
- <para> Disables use of extended attributes by regular users. Root and system processes can still use extended attributes.</para>
+ <para>Requires that <literal>statfs(2)</literal> block until all
+ OSTs and MDTs are available and have returned space usage.
+ </para>
</entry>
</row>
<row>
<entry>
- <para> <literal>acl</literal></para>
+ <para> <literal>user_xattr</literal></para>
</entry>
<entry>
- <para> Enables POSIX Access Control List support. See the acl(5) manual page.</para>
+ <para>Enables get/set of extended attributes by regular users
+ in the <literal>user.*</literal> namespace. See the
+ <literal>attr(5)</literal> manual page for more details.
+ </para>
</entry>
</row>
<row>
<entry>
- <para> <literal>noacl</literal></para>
+ <para> <literal>nouser_xattr</literal></para>
</entry>
<entry>
- <para> Disables Access Control List support.</para>
+ <para>Disables use of extended attributes in the
+ <literal>user.*</literal> namespace by regular users. Root
+ and system processes can still use extended attributes.</para>
</entry>
</row>
<row>
<para> <literal>verbose</literal></para>
</entry>
<entry>
- <para> Enable mount/umount console messages.</para>
+ <para> Enable extra mount/umount console messages.</para>
</entry>
</row>
<row>
<para> <literal>user_fid2path</literal></para>
</entry>
<entry>
- <para condition='l23'>Enable FID to path translation by regular users. Note: This option allows a potential security hole because it allows regular users direct access to a file by its FID, bypassing POSIX path-based permission checks which could otherwise prevent the user from accessing a file in a directory that they do not have access to. Regular permission checks are still performed on the file itself, so the user cannot access a file to which they have no access rights.</para>
+ <para>Enable FID-to-path translation by regular users.
+ </para>
+ <note><para>This option allows a potential security hole because
+ it allows regular users direct access to a file by its Lustre
+ File ID. This bypasses POSIX path-based permission checks,
+ and could allow the user to access a file in a directory that
+ they do not have access to. Regular POSIX file mode and ACL
+ permission checks are still performed on the file itself, so
+ users cannot access a file to which they have no permission.
+ </para></note>
</entry>
</row>
<row>
<para> <literal>nouser_fid2path</literal></para>
</entry>
<entry>
- <para condition='l23'> Disable FID to path translation by regular users. Root and processes with CAP_DAC_READ_SEARCH can still perform FID to path translation.</para>
+ <para> Disable FID to path translation by
+ regular users. Root and processes with
+ <literal>CAP_DAC_READ_SEARCH</literal> can still perform FID
+ to path translation.
+ </para>
</entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para>In addition to the standard mount options and backing disk type (e.g. ext3) options, Lustre understands the following server-specific options:</para>
+ <para>In addition to the standard mount options and backing disk type
+ (e.g. ldiskfs) options, Lustre understands the following server-specific
+ mount options:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
</row>
<row>
<entry>
- <para> <literal>exclude=<replaceable>ostlist</replaceable></literal></para>
+ <para> <literal>abort_recov</literal></para>
</entry>
<entry>
- <para> Starts a client or MDT with a colon-separated list of known inactive OSTs.</para>
+ <para> Aborts client recovery on that server and starts the target service immediately.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>abort_recov</literal></para>
+ <para> <literal>max_sectors_kb=<replaceable>KB</replaceable></literal></para>
</entry>
<entry>
- <para> Aborts client recovery and starts the target service immediately.</para>
+ <para condition='l210'>Sets the block device parameter
+ <literal>max_sectors_kb</literal> limit for the MDT or OST
+ target being mounted to specified maximum number of kilobytes.
+ When <literal>max_sectors_kb</literal> isn't specified as a
+ mount option, it will automatically be set to the
+ <literal>max_hw_sectors_kb</literal> (up to a maximum of 16MiB)
+ for that block device. This default behavior is suited for
+ most users. When <literal>max_sectors_kb=0</literal> is used,
+ the current value for this tunable will be kept.
+ </para>
</entry>
</row>
<row>
<para> <literal>recovery_time_soft=<replaceable>timeout</replaceable></literal></para>
</entry>
<entry>
- <para> Allows <literal>timeout</literal> seconds for clients to reconnect for recovery after a server crash. This timeout is incrementally extended if it is about to expire and the server is still handling new connections from recoverable clients.</para>
- <para> The default soft recovery timeout is 3 times the value of the Lustre timeout parameter (see <xref linkend="section_c24_nt5_dl"/>). The default Lustre timeout is 100 seconds, which would make the soft recovery timeout default to 300 seconds (5 minutes). The soft recovery timeout is set at mount time and will not change if the Lustre timeout is changed after mount time.</para>
+ <para>Allows <literal>timeout</literal> seconds for clients to
+ reconnect for recovery after a server crash. This timeout is
+ incrementally extended if it is about to expire and the server
+ is still handling new connections from recoverable clients.
+ </para>
+ <para>The default soft recovery timeout is 3 times the value
+ of the Lustre timeout parameter (see
+ <xref linkend="section_c24_nt5_dl"/>). The default Lustre
+ timeout is 100 seconds, which would make the soft recovery
+ timeout default to 300 seconds (5 minutes). The soft recovery
+ timeout is set at mount time and will not change if the Lustre
+ timeout is changed after mount time.
+ </para>
</entry>
</row>
<row>
<para> <literal>recovery_time_hard=<replaceable>timeout</replaceable></literal></para>
</entry>
<entry>
- <para> The server is allowed to incrementally extend its timeout up to a hard maximum of <literal>timeout</literal> seconds.</para>
- <para> The default hard recovery timeout is 9 times the value of the Lustre timeout parameter (see <xref linkend="section_c24_nt5_dl"/>). The default Lustre timeout is 100 seconds, which would make the hard recovery timeout default to 900 seconds (15 minutes). The hard recovery timeout is set at mount time and will not change if the Lustre timeout is changed after mount time.</para>
+ <para>The server is allowed to incrementally extend its timeout
+ up to a hard maximum of <replaceable>timeout</replaceable>
+ seconds.
+ </para>
+ <para>The default hard recovery timeout is 9 times the value
+ of the Lustre timeout parameter (see
+ <xref linkend="section_c24_nt5_dl"/>). The default Lustre
+ timeout is 100 seconds, which would make the hard recovery
+ timeout default to 900 seconds (15 minutes). The hard recovery
+ timeout is set at mount time and will not change if the Lustre
+ timeout is changed after mount time.
+ </para>
</entry>
</row>
<row>
<para> <literal>noscrub</literal></para>
</entry>
<entry>
- <para>Typically the MDT will detect restoration from a file-level backup during mount. This mount option prevents the OI Scrub from starting automatically when the MDT is mounted. Manually starting LFSCK after mounting provides finer control over the starting conditions. This mount option also prevents OI scrub from occurring automatically when OI inconsistency is detected (see <xref linkend="dbdoclet.lfsck_auto_scrub"/>)</para>
+ <para>Typically the MDT will detect restoration from a
+ file-level backup during mount. This mount option prevents
+ the OI Scrub from starting automatically when the MDT is
+ mounted. Manually starting LFSCK after mounting provides finer
+ control over the starting conditions. This mount option also
+ prevents OI scrub from occurring automatically when OI
+ inconsistency is detected (see
+ <xref linkend="dbdoclet.lfsck_auto_scrub"/>).
+ </para>
</entry>
</row>
</tbody>
</section>
<section remap="h5">
<title>Examples</title>
- <para>Starts a client for the Lustre file system testfs at mount point /mnt/myfilesystem. The Management Service is running on a node reachable from this client via the cfs21@tcp0 NID.</para>
- <screen>mount -t lustre cfs21@tcp0:/testfs /mnt/myfilesystem</screen>
+ <para>Starts a client for the Lustre file system
+ <replaceable>chipfs</replaceable> at mount point
+ <replaceable>/mnt/chip</replaceable>. The Management Service is running on
+ a node reachable from this client via the cfs21@tcp0 NID.</para>
+ <screen>mount -t lustre cfs21@tcp0:/chipfs /mnt/chip</screen>
+ <para condition='l29'>Similar to the above example, but mounting a
+ subdirectory under <replaceable>chipfs</replaceable> as a fileset.
+ <screen>mount -t lustre cfs21@tcp0:/chipfs/v1_0 /mnt/chipv1_0</screen>
+ </para>
<para>Starts the Lustre metadata target service from /dev/sda1 on mount point /mnt/test/mdt.</para>
<screen>mount -t lustre /dev/sda1 /mnt/test/mdt</screen>
<para>Starts the testfs-MDT0000 service (using the disk label), but aborts the recovery process.</para>
</section>
<section remap="h5">
<title>Description</title>
- <para>The routerstat utility displays LNET router statistics. If no <literal><replaceable>interval</replaceable></literal> is specified, then statistics are sampled and printed only one time. Otherwise, statistics are sampled and printed at the specified <literal><replaceable>interval</replaceable></literal> (in seconds).</para>
+ <para>The routerstat utility displays LNet router statistics. If no <literal><replaceable>interval</replaceable></literal> is specified, then statistics are sampled and printed only one time. Otherwise, statistics are sampled and printed at the specified <literal><replaceable>interval</replaceable></literal> (in seconds).</para>
</section>
<section remap="h5">
<title>Output</title>
<para> <literal>M</literal></para>
</entry>
<entry>
- <para> Number of messages currently being processed by LNET (The maximum number of messages ever processed by LNET concurrently)</para>
+ <para> Number of messages currently being processed by LNet (The maximum number of messages ever processed by LNet concurrently)</para>
</entry>
</row>
<row>
<para> <literal>E</literal></para>
</entry>
<entry>
- <para> Number of LNET errors</para>
+ <para> Number of LNet errors</para>
</entry>
</row>
<row>
<para> <literal>M</literal></para>
</entry>
<entry>
- <para> Number of messages currently being processed by LNET (The maximum number of messages ever processed by LNET concurrently)</para>
+ <para> Number of messages currently being processed by LNet (The maximum number of messages ever processed by LNet concurrently)</para>
</entry>
</row>
<row>
<para> <literal>E</literal></para>
</entry>
<entry>
- <para> Number of LNET errors per second</para>
+ <para> Number of LNet errors per second</para>
</entry>
</row>
<row>
<para> Per-client statistics tracked on the servers</para>
</listitem>
</itemizedlist>
- <para>Each MDT and OST now tracks LDLM and operations statistics for every connected client, for comparisons and simpler collection of distributed job statistics.</para>
+ <para>Each MDS and OSS now tracks LDLM and operations statistics for
+ every connected client, for comparisons and simpler collection of
+ distributed job statistics.</para>
<screen>/proc/fs/lustre/mds|obdfilter/*/exports/
</screen>
<itemizedlist>
<para> Improved MDT statistics</para>
</listitem>
</itemizedlist>
- <para>More detailed MDT operations statistics are collected for better profiling.</para>
- <screen>/proc/fs/lustre/mds/*/stats
+ <para>More detailed MDT operations statistics are collected for better
+ profiling.</para>
+ <screen>/proc/fs/lustre/mdt/*/md_stats
</screen>
</section>
<section remap="h3">
Testing / Debugging Utilities</title>
<para>Lustre offers the following test and debugging utilities.</para>
<section remap="h5">
- <title><indexterm><primary>loadgen</primary></indexterm>
-loadgen</title>
- <para>The Load Generator (loadgen) is a test program designed to simulate large numbers of Lustre clients connecting and writing to an OST. The loadgen utility is located at lustre/utils/loadgen (in a build directory) or at /usr/sbin/loadgen (from an RPM).</para>
- <para>Loadgen offers the ability to run this test:</para>
- <orderedlist>
- <listitem>
- <para>Start an arbitrary number of (echo) clients.</para>
- </listitem>
- <listitem>
- <para>Start and connect to an echo server, instead of a real OST.</para>
- </listitem>
- <listitem>
- <para>Create/bulk_write/delete objects on any number of echo clients simultaneously.</para>
- </listitem>
- </orderedlist>
- <para>Currently, the maximum number of clients is limited by MAX_OBD_DEVICES and the amount of memory available.</para>
- </section>
- <section remap="h5">
- <title>Usage</title>
- <para>The loadgen utility can be run locally on the OST server machine or remotely from any LNET host. The device command can take an optional NID as a parameter; if unspecified, the first local NID found is used.</para>
- <para>The obdecho module must be loaded by hand before running loadgen.</para>
- <screen># cd lustre/utils/
-# insmod ../obdecho/obdecho.ko
-# ./loadgen
-loadgen> h
-This is a test program used to simulate large numbers of clients. The echo \
-obds are used, so the obdecho module must be loaded.
-
-Typical usage would be:
-loadgen> dev lustre-OST0000 set the target device
-loadgen> start 20 start 20 echo clients
-loadgen> wr 10 5 have 10 clients do simultaneous brw_write \
-tests 5 times each
-
-Available commands are:
- device
- dl
- echosrv
- start
- verbose
- wait
- write
- help
- exit
- quit
-
-For more help type: help command-name
-loadgen>
-loadgen> device lustre-OST0000 192.168.0.21@tcp
-Added uuid OSS_UUID: 192.168.0.21@tcp
-Target OST name is 'lustre-OST0000'
-loadgen>
-loadgen> st 4
-start 0 to 4
-./loadgen: running thread #1
-./loadgen: running thread #2
-./loadgen: running thread #3
-./loadgen: running thread #4
-loadgen> wr 4 5
-Estimate 76 clients before we run out of grant space (155872K / 2097152)
-1: i0
-2: i0
-4: i0
-3: i0
-1: done (0)
-2: done (0)
-4: done (0)
-3: done (0)
-wrote 25MB in 1.419s (17.623 MB/s)
-loadgen>
-</screen>
- <para>The loadgen utility prints periodic status messages; message output can be controlled with the verbose command.</para>
- <para>To insure that a file can be written to (a requirement of write cache), OSTs reserve ("grants"), chunks of space for each newly-created file. A grant may cause an OST to report that it is out of space, even though there is plenty of space on the disk, because the space is "reserved" by other files. The loadgen utility estimates the number of simultaneous open files as the disk size divided by the grant size and reports that number when the write tests are first started.</para>
- <para>Echo Server</para>
- <para>The loadgen utility can start an echo server. On another node, loadgen can specify the echo server as the device, thus creating a network-only test environment.</para>
- <screen>loadgen> echosrv
-loadgen> dl
- 0 UP obdecho echosrv echosrv 3
- 1 UP ost OSS OSS 3
-</screen>
- <para>On another node:</para>
- <screen>loadgen> device echosrv cfs21@tcp
-Added uuid OSS_UUID: 192.168.0.21@tcp
-Target OST name is 'echosrv'
-loadgen> st 1
-start 0 to 1
-./loadgen: running thread #1
-loadgen> wr 1
-start a test_brw write test on X clients for Y iterations
-usage: write <num_clients> <num_iter> [<delay>]
-loadgen> wr 1 1
-loadgen>
-1: i0
-1: done (0)
-wrote 1MB in 0.029s (34.023 MB/s)
-</screen>
- <para>Scripting</para>
- <para>The threads all perform their actions in non-blocking mode; use the wait command to block for the idle state. For example:</para>
- <screen>#!/bin/bash
-./loadgen << EOF
-device lustre-OST0000
-st 1
-wr 1 10
-wait
-quit
-EOF
-</screen>
- </section>
- <section remap="h5">
<title><indexterm><primary>lr_reader</primary></indexterm>
lr_reader</title>
- <para>The lr_reader utility translates a last received (last_rcvd) file into human-readable form.</para>
+ <para>The lr_reader utility translates the content of the <literal>last_rcvd</literal> and <literal>reply_data</literal> files into human-readable form.</para>
<para>The following utilities are part of the Lustre I/O kit. For more information, see <xref linkend="benchmarkingtests"/>.</para>
</section>
<section remap="h5">
<para>The stats-collect utility contains scripts used to collect application profiling information from Lustre clients and servers.</para>
</section>
</section>
- <section remap="h3">
- <title><indexterm><primary>flock</primary></indexterm>Flock Feature</title>
- <para>Lustre now includes the flock feature, which provides file locking support. Flock describes classes of file locks known as 'flocks'. Flock can apply or remove a lock on an open file as specified by the user. However, a single file may not, simultaneously, have both shared and exclusive locks.</para>
- <para>By default, the flock utility is disabled on Lustre. Two modes are available.</para>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <tbody>
- <row>
- <entry>
- <para> <literal>local mode</literal></para>
- </entry>
- <entry>
- <para> In this mode, locks are coherent on one node (a single-node flock), but not across all clients. To enable it, use -o localflock. This is a client-mount option.</para>
- <note>
- <para>This mode does not impact performance and is appropriate for single-node databases.</para>
- </note>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal>consistent mode</literal></para>
- </entry>
- <entry>
- <para> In this mode, locks are coherent across all clients.</para>
- <para> To enable it, use the -o flock. This is a client-mount option.</para>
- <warning><para>This mode affects the performance of the file being flocked and may affect stability, depending on the Lustre version used. Consider using a newer Lustre version which is more stable. If the consistent mode is enabled and no applications are using flock, then it has no effect.</para></warning>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para>A call to use flock may be blocked if another process is holding an incompatible lock. Locks created using flock are applicable for an open file table entry. Therefore, a single process may hold only one type of lock (shared or exclusive) on a single file. Subsequent flock calls on a file that is already locked converts the existing lock to the new lock mode.</para>
+ <section remap="h3" condition='l29'>
+ <title><indexterm><primary>fileset</primary></indexterm>Fileset Feature</title>
+ <para> With the fileset feature, Lustre now provides subdirectory mount
+ support. Subdirectory mounts, also referred to as filesets, allow a
+ client to mount a child directory of a parent filesystem, thereby limiting
+ the filesystem namespace visibility on a specific client. A common use
+ case is for a client to use a subdirectory mount when there is a desire to
+ limit the visibility of the entire filesystem namesapce to aid in the
+ prevention of accidental file deletions outside of the subdirectory
+ mount.</para>
+ <para>It is important to note that invocation of the subdirectory mount is
+ voluntary by the client and not does prevent access to files that are
+ visible in multiple subdirectory mounts via hard links. Furthermore, it
+ does not prevent the client from subsequently mounting the whole file
+ system without a subdirectory being specified.</para>
+ <figure xml:id="understandinglustre.fig.fileset">
+ <title>
+ <indexterm>
+ <primary>Lustre</primary>
+ <secondary>fileset</secondary>
+ </indexterm>Lustre fileset</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata scalefit="1" width="100%"
+ fileref="./figures/fileset.png" />
+ </imageobject>
+ <textobject>
+ <phrase>Lustre file system fileset feature</phrase>
+ </textobject>
+ </mediaobject>
+ </figure>
<section remap="h4">
- <title>Example</title>
- <screen>$ mount -t lustre -o flock mds@tcp0:/lustre /mnt/client</screen>
- <para>You can check it in /etc/mtab. It should look like,</para>
- <screen>mds@tcp0:/lustre /mnt/client lustre rw,flock 0 0</screen>
+ <title>Examples</title>
+ <para>The following example will mount the
+ <literal>chipfs</literal> filesystem on client1 and create a
+ subdirectory <literal>v1_1</literal> within that filesystem. Client2
+ will then mount only the <literal>v1_1</literal> subdirectory as a
+ fileset, thereby limiting access to anything else in the
+ <literal>chipfs</literal> filesystem from client2.</para>
+ <screen>client1# mount -t lustre mgs@tcp:/chipfs /mnt/chip
+client1# mkdir /mnt/chip/v1_1</screen>
+ <screen>client2# mount -t lustre mgs@tcp:/chipfs/v1_1 /mnt/chipv1_1</screen>
+ <para>You can check the created mounts in /etc/mtab. It should look like
+ the following:</para>
+ <screen><replaceable>client1</replaceable>
+mds@tcp0:/chipfs/ /mnt/chip lustre rw 0 0
+</screen><screen>
+<replaceable>client2</replaceable>
+mds@tcp0:/chipfs/v1_1 /mnt/chipv1_1 lustre rw 0 0</screen>
+ <para>Create a directory under the /mnt/chip mount, and get its FID</para>
+ <screen>client1# mkdir /mnt/chip/v1_2
+client1# lfs path2fid /mnt/chip/v1_2
+[0x200000400:0x2:0x0]
+</screen>
+ <para>If you try resolve the FID of the <literal>/mnt/chip/v1_2</literal>
+ path (as created in the example above) on client2, an error will be returned
+ as the FID can not be resolved on client2 since it is not part of the
+ mounted fileset on that client. Recall that the fileset on client2 mounted
+ the <literal>v1_1</literal> subdirectory beneath the top level
+ <literal>chipfs</literal> filesystem.
+ </para>
+ <screen>client2# lfs fid2path /mnt/chip/v1_2 [0x200000400:0x2:0x0]
+fid2path: error on FID [0x200000400:0x2:0x0]: No such file or directory</screen>
+ <para>Subdirectory mounts do not have the <literal>.lustre</literal>
+ pseudo directory, which prevents clients from opening or accessing files
+ only by FID.</para>
+ <screen>client1# ls /mnt/chipfs/.lustre
+ fid lost+found</screen>
+ <screen>client2# ls /mnt/chipv1_1/.lustre
+ ls: cannot access /mnt/chipv1_1/.lustre: No such file or directory
+ </screen>
</section>
</section>
</section>
git://git.whamcloud.com / doc / manual.git / blobdiff