<tbody>
<row>
<entry>
- <para> <literal>--source=<src></literal></para>
+ <para> <literal>--source=<replaceable>src</replaceable></literal></para>
</entry>
<entry>
<para>The path to the root of the Lustre file system (source) which will be synchronized. This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statuslog</literal>) is not specified.</para>
</row>
<row>
<entry>
- <para> <literal>--target=<tgt></literal></para>
+ <para> <literal>--target=<replaceable>tgt</replaceable></literal></para>
</entry>
<entry>
<para>The path to the root where the source file system will be synchronized (target). This is a mandatory option if the status log created during a previous synchronization operation (<literal>--statuslog</literal>) is not specified. This option can be repeated if multiple synchronization targets are desired.</para>
</row>
<row>
<entry>
- <para> <literal>--mdt=<mdt></literal></para>
+ <para> <literal>--mdt=<replaceable>mdt</replaceable></literal></para>
</entry>
<entry>
<para>The metadata device to be synchronized. A changelog user must be registered for this device. This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statuslog</literal>) is not specified.</para>
</row>
<row>
<entry>
- <para> <literal>--user=<user id></literal></para>
+ <para> <literal>--user=<replaceable>userid</replaceable></literal></para>
</entry>
<entry>
<para>The changelog user ID for the specified MDT. To use <literal>lustre_rsync</literal>, the changelog user must be registered. For details, see the <literal>changelog_register</literal> parameter in <xref linkend="systemconfigurationutilities"/> (<literal>lctl</literal>). This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statuslog</literal>) is not specified.</para>
</row>
<row>
<entry>
- <para> <literal>--statuslog=<log></literal></para>
+ <para> <literal>--statuslog=<replaceable>log</replaceable></literal></para>
</entry>
<entry>
<para>A log file to which synchronization status is saved. When the <literal>lustre_rsync</literal> utility starts, if the status log from a previous synchronization operation is specified, then the state is read from the log and otherwise mandatory <literal>--source</literal>, <literal>--target</literal> and <literal>--mdt</literal> options can be skipped. Specifying the <literal>--source</literal>, <literal>--target</literal> and/or <literal>--mdt</literal> options, in addition to the <literal>--statuslog</literal> option, causes the specified parameters in the status log to be overridden. Command line options take precedence over options in the status log.</para>
</row>
<row>
<entry>
- <literal> --xattr <yes|no> </literal>
+ <literal> --xattr <replaceable>yes|no</replaceable> </literal>
</entry>
<entry>
<para>Specifies whether extended attributes (<literal>xattrs</literal>) are synchronized or not. The default is to synchronize extended attributes.</para>
<itemizedlist>
<listitem>
<para>File containing standard output data (same as <literal>stdout</literal>)</para>
- <screen>${rslt}_<emphasis><date/time></emphasis>.summary</screen>
+ <screen><replaceable>rslt_date_time</replaceable>.summary</screen>
</listitem>
<listitem>
<para>Temporary (tmp) files</para>
- <screen>${rslt}_<emphasis><date/time></emphasis>_*
+ <screen><replaceable>rslt_date_time</replaceable>_*
</screen>
</listitem>
<listitem>
<para>Collected tmp files for post-mortem</para>
- <screen>${rslt}_<emphasis><date/time></emphasis>.detail
+ <screen><replaceable>rslt_date_time</replaceable>.detail
</screen>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para><emphasis role="bold">Network</emphasis> - In this mode, the Lustre client generates I/O requests over the network but these requests are not sent to the OST file system. The OSS node runs the obdecho server to receive the requests but discards them before they are sent to the disk.</para>
- <para>Pass the parameters <literal>case=network</literal> and <literal>target=<replaceable><hostname</replaceable>|<replaceable>IP_of_server></replaceable></literal> to the script. For each network case, the script does the required setup.</para>
+ <para>Pass the parameters <literal>case=network</literal> and <literal>target=<replaceable>hostname|IP_of_server</replaceable></literal> to the script. For each network case, the script does the required setup.</para>
<para>For more details, see <xref linkend="dbdoclet.50438212_36037"/></para>
</listitem>
<listitem>
</listitem>
<listitem>
<para>List all OSTs you want to test.</para>
- <para>Use the <literal>target=parameter</literal> to list the OSTs separated by spaces. List the individual OSTs by name using the format <emphasis>
- <literal><fsname>-<OSTnumber></literal>
- </emphasis> (for example, lustre-OST0001). You do not have to specify an MDS or LOV.</para>
+ <para>Use the <literal>target=parameter</literal> to list the OSTs separated by spaces. List the individual OSTs by name using the format
+ <literal><replaceable>fsname</replaceable>-<replaceable>OSTnumber</replaceable></literal>
+ (for example, <literal>lustre-OST0001</literal>). You do not have to specify an MDS or LOV.</para>
</listitem>
<listitem>
<para>Run the <literal>obdfilter_survey</literal> script with the <literal>target=parameter</literal>.</para>
<screen>lctl dl</screen>
</listitem>
<listitem>
- <para>Run the <literal>obdfilter_survey</literal> script with the parameters <literal>case=network</literal> and <literal>targets=<replaceable><hostname</replaceable>|<replaceable>ip_of_server></replaceable></literal>. For example:</para>
+ <para>Run the <literal>obdfilter_survey</literal> script with the parameters <literal>case=network</literal> and <literal>targets=<replaceable>hostname|ip_of_server</replaceable></literal>. For example:</para>
<screen>$ nobjhi=2 thrhi=2 size=1024 targets="oss0 oss1" \
case=network sh odbfilter-survey</screen>
</listitem>
<listitem>
<para>On the server side, view the statistics at:</para>
- <screen>/proc/fs/lustre/obdecho/<emphasis><echo_srv></emphasis>/stats</screen>
- <para>where <emphasis>
- <literal><echo_srv></literal>
- </emphasis> is the <literal>obdecho</literal> server created by the script.</para>
+ <screen>/proc/fs/lustre/obdecho/<replaceable>echo_srv</replaceable>/stats</screen>
+ <para>where <literal><replaceable>echo_srv</replaceable></literal>
+ is the <literal>obdecho</literal> server created by the script.</para>
</listitem>
</orderedlist>
</section>
</listitem>
<listitem>
<para>List all OSCs you want to test.</para>
- <para>Use the <literal>target=parameter</literal> to list the OSCs separated by spaces. List the individual OSCs by name separated by spaces using the format <literal><replaceable><fsname>-<OST_name></replaceable>-osc-<replaceable><OSC_number></replaceable></literal> (for example, <literal>lustre-OST0000-osc-ffff88007754bc00</literal>). You <replaceable role="bold">do not have to specify an MDS or LOV.</replaceable></para>
+ <para>Use the <literal>target=parameter</literal> to list the OSCs separated by spaces. List the individual OSCs by name separated by spaces using the format <literal><replaceable>fsname</replaceable>-<replaceable>OST_name</replaceable>-osc-<replaceable>instance</replaceable></literal> (for example, <literal>lustre-OST0000-osc-ffff88007754bc00</literal>). You <emphasis>do not have to specify an MDS or LOV.</emphasis></para>
</listitem>
<listitem>
- <para>Run the <literal><replaceable role="bold">o</replaceable>bdfilter_survey</literal> script with the <literal>target=parameter</literal> and <literal>case=netdisk</literal>.</para>
+ <para>Run the <literal>obdfilter_survey</literal> script with the <literal>target=<replaceable>osc</replaceable></literal> and <literal>case=netdisk</literal>.</para>
<para>An example of a local test run with up to two objects (<literal>nobjhi</literal>), up to two threads (<literal>thrhi</literal>), and 1024 Mb (size) transfer size is shown below:</para>
<screen>$ nobjhi=2 thrhi=2 size=1024 \
targets="lustre-OST0000-osc-ffff88007754bc00 \
<section remap="h3">
<title>Using <literal>stats-collect</literal></title>
<para>The stats-collect utility is configured by including profiling configuration variables in the config.sh script. Each configuration variable takes the following form, where 0 indicates statistics are to be collected only when the script starts and stops and <emphasis>n</emphasis> indicates the interval in seconds at which statistics are to be collected:</para>
- <screen><emphasis><statistic></emphasis>_INTERVAL=<emphasis>[</emphasis>0<emphasis>|n]</emphasis></screen>
+ <screen><replaceable>statistic</replaceable>_INTERVAL=<replaceable>0|n</replaceable></screen>
<para>Statistics that can be collected include:</para>
<itemizedlist>
<listitem>
<listitem>
<para>Stop collecting statistics on each node, clean up the temporary file, and create a profiling tarball.</para>
<para>Enter:</para>
- <screen>sh gather_stats_everywhere.sh config.sh stop <emphasis><log_name.tgz></emphasis></screen>
- <para>When <emphasis>
- <literal><log_name.tgz></literal>
- </emphasis> is specified, a profile tarball <literal>/tmp/<replaceable><log_name.tgz></replaceable></literal> is created.</para>
+ <screen>sh gather_stats_everywhere.sh config.sh stop <replaceable>log_name</replaceable>.tgz</screen>
+ <para>When <literal><replaceable>log_name</replaceable>.tgz</literal>
+ is specified, a profile tarball <literal><replaceable>/tmp/log_name</replaceable>.tgz</literal> is created.</para>
</listitem>
<listitem>
<para>Analyze the collected statistics and create a csv tarball for the specified profiling data.</para>
<para>The acceptor is a TCP/IP service that some LNDs use to establish communications. If a local network requires it and it has not been disabled, the acceptor listens on a single port for connection requests that it redirects to the appropriate local network. The acceptor is part of the LNET module and configured by the following options:</para>
<itemizedlist>
<listitem>
- <para><literal>secure</literal> - Accept connections only from reserved TCP ports (< 1023).</para>
+ <para><literal>secure</literal> - Accept connections only from reserved TCP ports (below 1023).</para>
</listitem>
<listitem>
<para><literal>all</literal> - Accept connections from any TCP port. </para>
<para><emphasis role="bold">TX Descriptors</emphasis></para>
<para>The <literal>ptllnd</literal> has a pool of so-called "tx descriptors", which it uses not only for outgoing messages, but also to hold state for bulk transfers requested by incoming messages. This pool should scale with the total number of peers.</para>
<para>To enable the building of the Portals LND (<literal>ptllnd.ko</literal>) configure with this option:</para>
- <screen>./configure --with-portals=<path-to-portals-headers></screen>
+ <screen>./configure --with-portals=<replaceable>/path/to/portals/headers</replaceable></screen>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<para> <literal>cksum</literal></para>
</entry>
<entry>
- <para>Enables small message (< 4 KB) checksums if set to a non-zero value.</para>
+ <para>Enables small message (below 4 KB) checksums if set to a non-zero value.</para>
</entry>
</row>
<row>
<para> <literal>polling</literal></para>
</entry>
<entry>
- <para>Use zero (0) to block (wait). A value > 0 will poll that many times before blocking.</para>
+ <para>Use zero (0) to block (wait). A value greater than 0 will poll that many times before blocking.</para>
</entry>
</row>
<row>
Overview of LNET Module Parameters</title>
<para>LNET kernel module (lnet) parameters specify how LNET is to be configured to work with Lustre, including which NICs will be configured to work with Lustre and the routing to be used with Lustre.</para>
<para>Parameters for LNET can be specified in the <literal>/etc/modprobe.d/lustre.conf</literal> file. In some cases the parameters may have been stored in <literal>/etc/modprobe.conf</literal>, but this has been deprecated since before RHEL5 and SLES10, and having a separate <literal>/etc/modprobe.d/lustre.conf</literal> file simplifies administration and distribution of the Lustre networking configuration. This file contains one or more entries with the syntax:</para>
- <screen>options lnet <<emphasis>parameter</emphasis>>=<<emphasis>parameter value</emphasis>></screen>
+ <screen>options lnet <replaceable>parameter</replaceable>=<replaceable>value</replaceable></screen>
<para>To specify the network interfaces that are to be used for Lustre, set either the networks parameter or the <literal>ip2nets</literal> parameter (only one of these parameters can be used at a time):</para>
<itemizedlist>
<listitem>
<section remap="h3">
<title><indexterm><primary>LNET</primary><secondary>using NID</secondary></indexterm>Using a Lustre Network Identifier (NID) to Identify a Node</title>
<para>A Lustre network identifier (NID) is used to uniquely identify a Lustre network endpoint by node ID and network type. The format of the NID is:</para>
- <screen><<emphasis>network</emphasis><emphasis>id</emphasis>>@<<emphasis>network</emphasis><emphasis>type</emphasis>></screen>
+ <screen><replaceable>network_id</replaceable>@<replaceable>network_type</replaceable></screen>
<para>Examples are:</para>
<screen>10.67.73.200@tcp0
10.67.75.100@o2ib</screen>
<para>To determine the appropriate NID to specify in the mount command, use the <literal>lctl</literal> command. To display MDS NIDs, run on the MDS :</para>
<screen>lctl list_nids</screen>
<para>To determine if a client can reach the MDS using a particular NID, run on the client:</para>
- <screen>lctl which_nid <<emphasis>MDS NID</emphasis>></screen>
+ <screen>lctl which_nid <replaceable>MDS_NID</replaceable></screen>
</section>
</section>
<section xml:id="dbdoclet.50438216_46279">
<title><indexterm><primary>LNET</primary><secondary>module parameters</secondary></indexterm>Setting the LNET Module networks Parameter</title>
<para>If a node has more than one network interface, you'll typically want to dedicate a specific interface to Lustre. You can do this by including an entry in the <literal>lustre.conf</literal> file on the node that sets the LNET module <literal>networks</literal> parameter:</para>
- <screen>options lnet networks=<<emphasis>comma-separated list of networks</emphasis>></screen>
+ <screen>options lnet networks=<replaceable>comma-separated list of networks</replaceable></screen>
<para>This example specifies that a Lustre node will use a TCP/IP interface and an InfiniBand interface:</para>
<screen>options lnet networks=tcp0(eth0),o2ib(ib0)</screen>
<para>This example specifies that the Lustre node will use the TCP/IP interface <literal>eth1</literal>:</para>
<title><indexterm><primary>LNET</primary><secondary>routes</secondary></indexterm>Setting the LNET Module routes Parameter</title>
<para>The LNET module routes parameter is used to identify routers in a Lustre configuration. These parameters are set in <literal>modprobe.conf</literal> on each Lustre node.</para>
<para>The LNET routes parameter specifies a colon-separated list of router definitions. Each route is defined as a network number, followed by a list of routers:</para>
- <screen>routes=<<emphasis>net type</emphasis>> <<emphasis>router NID(s)</emphasis>></screen>
+ <screen>routes=<replaceable>net_type router_NID(s)</replaceable></screen>
<para>This example specifies bi-directional routing in which TCP clients can reach Lustre resources on the IB networks and IB servers can access the TCP networks:</para>
<screen>options lnet 'ip2nets="tcp0 192.168.0.*; \
o2ib0(ib0) 132.6.1.[1-128]"' 'routes="tcp0 132.6.1.[1-8]@o2ib0; \
<title><indexterm><primary>LNET</primary><secondary>route checker</secondary></indexterm>Configuring the Router Checker</title>
<para>In a Lustre configuration in which different types of networks, such as a TCP/IP network and an Infiniband network, are connected by routers, a router checker can be run on the clients and servers in the routed configuration to monitor the status of the routers. In a multi-hop routing configuration, router checkers can be configured on routers to monitor the health of their next-hop routers.</para>
<para>A router checker is configured by setting lnet parameters in <literal>lustre.conf</literal> by including an entry in this form:</para>
- <screen>options lnet <<emphasis>router checker parameter</emphasis>>=<<emphasis>parameter value</emphasis>></screen>
+ <screen>options lnet <replaceable>router_checker_parameter</replaceable>=<replaceable>value</replaceable></screen>
<para>The router checker parameters are:</para>
<itemizedlist>
<listitem>
</emphasis>. For more information about hardware requirements, see <xref linkend="settinguplustresystem"/>.</para>
</listitem>
<listitem>
- <para><emphasis><emphasis role="bold">Downloaded and installed the Lustre software</emphasis>.</emphasis> For more information about preparing for and installing the Lustre software, see <xref linkend="installinglustre"/>.</para>
+ <para><emphasis role="bold">Downloaded and installed the Lustre software.</emphasis> For more information about preparing for and installing the Lustre software, see <xref linkend="installinglustre"/>.</para>
</listitem>
</itemizedlist>
<para>The following optional steps should also be completed, if needed, before the Lustre software is configured:</para>
<orderedlist>
<listitem>
<para>Create a combined MGS/MDT file system on a block device. On the MDS node, run:</para>
- <screen>mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgs --mdt --index=0 <<emphasis>block device name</emphasis>></screen>
+ <screen>mkfs.lustre --fsname=<replaceable>fsname</replaceable> --mgs --mdt --index=0 <replaceable>/dev/block_device</replaceable></screen>
<para>The default file system name (<literal>fsname</literal>) is <literal>lustre</literal>.</para>
<note>
<para>If you plan to create multiple file systems, the MGS should be created separately on its own dedicated block device, by running:</para>
- <screen>mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgs <<emphasis>block device name</emphasis>></screen>
+ <screen>mkfs.lustre --fsname=<replaceable>fsname</replaceable> --mgs <replaceable>/dev/block_device</replaceable></screen>
<para>See <xref linkend="dbdoclet.50438194_88063"/> for more details.</para>
</note>
</listitem>
<listitem xml:id="dbdoclet.addmdtindex" condition='l24'>
<para>Optional for Lustre 2.4 and later. Add in additional MDTs.</para>
- <screen>mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgsnode=<nid> --mdt --index=1 <<emphasis>block device name</emphasis>></screen>
+ <screen>mkfs.lustre --fsname=<replaceable>fsname</replaceable> --mgsnode=<replaceable>nid</replaceable> --mdt --index=1 <replaceable>/dev/block_device</replaceable></screen>
<note><para>Up to 4095 additional MDTs can be added.</para></note>
</listitem>
<listitem>
<para>Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</para>
- <screen>mount -t lustre <<emphasis>block device name</emphasis>> <<emphasis>mount point</emphasis>></screen>
+ <screen>mount -t lustre <replaceable>/dev/block_device</replaceable> <replaceable>/mount_point</replaceable></screen>
<note>
<para>If you have created and MGS and an MDT on separate block devices, mount them both.</para>
</note>
</listitem>
<listitem xml:id="dbdoclet.50438267_pgfId-1290915">
<para>Create the OST. On the OSS node, run:</para>
- <screen>mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgsnode=<<emphasis>NID</emphasis>> --ost --index=<<emphasis>OST index</emphasis>> <<emphasis>block device name</emphasis>></screen>
+ <screen>mkfs.lustre --fsname=<replaceable>fsname</replaceable> --mgsnode=<replaceable>MGS_NID</replaceable> --ost --index=<replaceable>OST_index</replaceable> <replaceable>/dev/block_device</replaceable></screen>
<para>When you create an OST, you are formatting a <literal>ldiskfs</literal> file system on a block storage device like you would with any local file system.</para>
<para>You can have as many OSTs per OSS as the hardware or drivers allow. For more information about storage and memory requirements for a Lustre file system, see <xref linkend="settinguplustresystem"/>.</para>
<para>You can only configure one OST per block device. You should create an OST that uses the raw block device and does not use partitioning.</para>
</listitem>
<listitem xml:id="dbdoclet.50438267_pgfId-1293955">
<para>Mount the OST. On the OSS node where the OST was created, run:</para>
- <screen>mount -t lustre <emphasis><block device name> <mount point></emphasis></screen>
+ <screen>mount -t lustre <replaceable>/dev/block_device</replaceable> <replaceable>/mount_point</replaceable></screen>
<note>
<para>
To create additional OSTs, repeat Step <xref linkend="dbdoclet.50438267_pgfId-1290915"/> and Step <xref linkend="dbdoclet.50438267_pgfId-1293955"/>, specifying the next higher OST index number.</para>
</listitem>
<listitem xml:id="dbdoclet.50438267_pgfId-1290934">
<para>Mount the Lustre file system on the client. On the client node, run:</para>
- <screen>mount -t lustre <<emphasis>MGS node</emphasis>>:/<<emphasis>fsname</emphasis>> <<emphasis>mount point</emphasis>>
+ <screen>mount -t lustre <replaceable>MGS_node</replaceable>:/<replaceable>fsname</replaceable> <replaceable>/mount_point</replaceable>
</screen>
<note>
<para>To create additional clients, repeat Step <xref linkend="dbdoclet.50438267_pgfId-1290934"/>.</para>
<para>The ldiskfs OSD relies on the standard Linux quota to maintain accounting information on disk. As a consequence, the Linux kernel running on the Lustre servers using ldiskfs backend must have <literal>CONFIG_QUOTA</literal>, <literal>CONFIG_QUOTACTL</literal> and <literal>CONFIG_QFMT_V2</literal> enabled.</para>
</caution>
- <para>As of Lustre 2.4.0, quota enforcement is thus turned on/off independently of space accounting which is always enabled. <literal>lfs quota<on|off></literal> as well as the per-target <literal>quota_type</literal> parameter are deprecated in favor of a single per-filesystem quota parameter controlling inode/block quota enforcement. Like all permanent parameters, this quota parameter can be set via <literal>lctl conf_param</literal> on the MGS via the following syntax:</para>
- <screen>lctl conf_param ${FSNAME}.quota.<ost|mdt>=<u|g|ug|none></screen>
+ <para>As of Lustre 2.4.0, quota enforcement is thus turned on/off independently of space accounting which is always enabled. <literal>lfs quota<replaceable>on|off</replaceable></literal> as well as the per-target <literal>quota_type</literal> parameter are deprecated in favor of a single per-filesystem quota parameter controlling inode/block quota enforcement. Like all permanent parameters, this quota parameter can be set via <literal>lctl conf_param</literal> on the MGS via the following syntax:</para>
+ <screen>lctl conf_param <replaceable>fsname</replaceable>.quota.<replaceable>ost|mdt</replaceable>=<replaceable>u|g|ug|none</replaceable></screen>
<itemizedlist>
<listitem>
<para><literal>ost</literal> -- to configure block quota managed by OSTs</para>
</listitem>
</itemizedlist>
<para> Usage:</para>
- <screen>lfs quota [-q] [-v] [-o obd_uuid] [-u|-g <uname>|uid|gname|gid>] <filesystem>
-lfs quota -t <-u|-g> <filesystem>
-lfs setquota <-u|--user|-g|--group> <username|groupname> [-b <block-softlimit>] \
- [-B <block-hardlimit>] [-i <inode-softlimit>] \
- [-I <inode-hardlimit>] <filesystem></screen>
+ <screen>lfs quota [-q] [-v] [-o obd_uuid] [-u|-g <replaceable>uname|uid|gname|gid</replaceable>] <replaceable>/mount_point</replaceable>
+lfs quota -t <replaceable>-u|-g</replaceable> <replaceable>/mount_point</replaceable>
+lfs setquota <replaceable>-u|--user|-g|--group</replaceable> <replaceable>username|groupname</replaceable> [-b <replaceable>block-softlimit</replaceable>] \
+ [-B <replaceable>block_hardlimit</replaceable>] [-i <replaceable>inode_softlimit</replaceable>] \
+ [-I <replaceable>inode_hardlimit</replaceable>] <replaceable>/mount_point</replaceable></screen>
<para>To display general quota information (disk usage and limits) for the user running the command and his primary group, run:</para>
<screen>$ lfs quota /mnt/testfs </screen>
<para>To display general quota information for a specific user ("<literal>bob</literal>" in this example), run:</para>
<title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>RAID options</tertiary></indexterm>Formatting Options for RAID Devices</title>
<para>When formatting a file system on a RAID device, it is beneficial to ensure that I/O requests are aligned with the underlying RAID geometry. This ensures that the Lustre RPCs do not generate unnecessary disk operations which may reduce performance dramatically. Use the <literal>--mkfsoptions</literal> parameter to specify additional parameters when formatting the OST or MDT.</para>
<para>For RAID 5, RAID 6, or RAID 1+0 storage, specifying the following option to the <literal>--mkfsoptions</literal> parameter option improves the layout of the file system metadata, ensuring that no single disk contains all of the allocation bitmaps:</para>
- <screen>-E stride = <chunk_blocks> </screen>
- <para>The <literal><chunk_blocks></literal> variable is in units of 4096-byte blocks and represents the amount of contiguous data written to a single disk before moving to the next disk. This is alternately referred to as the RAID stripe size. This is applicable to both MDT and OST file systems.</para>
+ <screen>-E stride = <replaceable>chunk_blocks</replaceable> </screen>
+ <para>The <literal><replaceable>chunk_blocks</replaceable></literal> variable is in units of 4096-byte blocks and represents the amount of contiguous data written to a single disk before moving to the next disk. This is alternately referred to as the RAID stripe size. This is applicable to both MDT and OST file systems.</para>
<para>For more information on how to override the defaults while formatting MDT or OST file systems, see <xref linkend="dbdoclet.50438256_84701"/>.</para>
<section remap="h3">
<title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>for mkfs</tertiary></indexterm>Computing file system parameters for mkfs</title>
- <para>For best results, use RAID 5 with 5 or 9 disks or RAID 6 with 6 or 10 disks, each on a different controller. The stripe width is the optimal minimum I/O size. Ideally, the RAID configuration should allow 1 MB Lustre RPCs to fit evenly on a single RAID stripe without an expensive read-modify-write cycle. Use this formula to determine the <emphasis>
- <literal><stripe_width></literal>
- </emphasis>, where <emphasis>
- <literal><number_of_data_disks></literal>
- </emphasis> does <emphasis>not</emphasis> include the RAID parity disks (1 for RAID 5 and 2 for RAID 6):</para>
- <screen><emphasis><stripe_width_blocks></emphasis> = <emphasis><chunk_blocks></emphasis> * <emphasis><number_of_data_disks></emphasis> = 1 MB </screen>
- <para>If the RAID configuration does not allow <emphasis>
- <literal><chunk_blocks></literal>
- </emphasis> to fit evenly into 1 MB, select <emphasis>
- <literal><chunkblocks></literal>
- </emphasis><emphasis>
- <literal><stripe_width_blocks></literal>
- </emphasis>, such that is close to 1 MB, but not larger.</para>
- <para>The <emphasis>
- <literal><stripe_width_blocks></literal>
- </emphasis> value must equal <emphasis>
- <literal><chunk_blocks>*<number_of_data_disks></literal>
- </emphasis>. Specifying the <emphasis>
- <literal><stripe_width_blocks></literal>
- </emphasis> parameter is only relevant for RAID 5 or RAID 6, and is not needed for RAID 1 plus 0.</para>
+ <para>For best results, use RAID 5 with 5 or 9 disks or RAID 6 with 6 or 10 disks, each on a different controller. The stripe width is the optimal minimum I/O size. Ideally, the RAID configuration should allow 1 MB Lustre RPCs to fit evenly on a single RAID stripe without an expensive read-modify-write cycle. Use this formula to determine the
+ <literal><replaceable>stripe_width</replaceable></literal>, where
+ <literal><replaceable>number_of_data_disks</replaceable></literal>
+ does <emphasis>not</emphasis> include the RAID parity disks (1 for RAID 5 and 2 for RAID 6):</para>
+ <screen><replaceable>stripe_width_blocks = chunk_blocks * number_of_data_disks</replaceable> = 1 MB </screen>
+ <para>If the RAID configuration does not allow
+ <literal><replaceable>chunk_blocks</replaceable></literal>
+ to fit evenly into 1 MB, select
+ <literal><replaceable>stripe_width_blocks</replaceable></literal>,
+ such that is close to 1 MB, but not larger.</para>
+ <para>The
+ <literal><replaceable>stripe_width_blocks</replaceable></literal>
+ value must equal
+ <literal><replaceable>chunk_blocks</replaceable> * <replaceable>number_of_data_disks</replaceable></literal>.
+ Specifying the
+ <literal><replaceable>stripe_width_blocks</replaceable></literal>
+ parameter is only relevant for RAID 5 or RAID 6, and is not needed for RAID 1 plus 0.</para>
<para>Run <literal>--reformat</literal> on the file system device (<literal>/dev/sdc</literal>), specifying the RAID geometry to the underlying ldiskfs file system, where:</para>
- <screen>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, stripe_width=<emphasis><stripe_width_blocks></emphasis>"</screen>
+ <screen>--mkfsoptions "<replaceable>other_options</replaceable> -E stride=<replaceable>chunk_blocks</replaceable>, stripe_width=<replaceable>stripe_width_blocks</replaceable>"</screen>
<informalexample>
- <para>A RAID 6 configuration with 6 disks has 4 data and 2 parity disks. The <emphasis>
- <literal><chunk_blocks></literal>
- </emphasis> <= 1024KB/4 = 256KB.</para>
+ <para>A RAID 6 configuration with 6 disks has 4 data and 2 parity disks. The
+ <literal><replaceable>chunk_blocks</replaceable></literal>
+ <= 1024KB/4 = 256KB.</para>
</informalexample>
<para>Because the number of data disks is equal to the power of 2, the stripe width is equal to 1 MB.</para>
- <screen>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, stripe_width=<emphasis><stripe_width_blocks></emphasis>"...</screen>
+ <screen>--mkfsoptions "<replaceable>other_options</replaceable> -E stride=<replaceable>chunk_blocks</replaceable>, stripe_width=<replaceable>stripe_width_blocks</replaceable>"...</screen>
</section>
<section remap="h3">
<title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>external journal</tertiary></indexterm>Choosing Parameters for an External Journal</title>
<para>If you have configured a RAID array and use it directly as an OST, it contains both data and metadata. For better performance, we recommend putting the OST journal on a separate device, by creating a small RAID 1 array and using it as an external journal for the OST.</para>
<para>Lustre's default journal size is 400 MB. A journal size of up to 1 GB has shown increased performance but diminishing returns are seen for larger journals. Additionally, a copy of the journal is kept in RAM. Therefore, make sure you have enough memory available to hold copies of all the journals.</para>
<para>The file system journal options are specified to mkfs.luster using the <literal>--mkfsoptions</literal> parameter. For example:</para>
- <screen>--mkfsoptions "<other options> -j -J device=/dev/mdJ" </screen>
+ <screen>--mkfsoptions "<replaceable>other_options</replaceable> -j -J device=/dev/mdJ" </screen>
<para>To create an external journal, perform these steps for each OST on the OSS:</para>
<orderedlist>
<listitem>
</listitem>
<listitem>
<para>Create a journal device on the partition. Run:</para>
- <screen>[oss#] mke2fs -b 4096 -O journal_dev /dev/sdb <emphasis><journal_size></emphasis></screen>
- <para>The value of <emphasis>
- <literal><journal_size></literal>
- </emphasis> is specified in units of 4096-byte blocks. For example, 262144 for a 1 GB journal size.</para>
+ <screen>oss# mke2fs -b 4096 -O journal_dev /dev/sdb <replaceable>journal_size</replaceable></screen>
+ <para>The value of
+ <literal><replaceable>journal_size</replaceable></literal>
+ is specified in units of 4096-byte blocks. For example, 262144 for a 1 GB journal size.</para>
</listitem>
<listitem>
<para>Create the OST.</para>
<para>  </para>
</entry>
<entry>
- <para>kernel-<ver>_lustre.<arch></para>
+ <para>kernel-<replaceable>ver</replaceable>_lustre.<replaceable>arch</replaceable></para>
</entry>
<entry>
<para>Lustre patched server kernel.</para>
<para>  </para>
</entry>
<entry>
- <para> lustre-modules-<ver></para>
+ <para> lustre-modules-<replaceable>ver</replaceable></para>
</entry>
<entry>
<para> For Lustre-patched kernel.</para>
<para>  </para>
</entry>
<entry>
- <para> lustre-client-modules-<ver></para>
+ <para> lustre-client-modules-<replaceable>ver</replaceable></para>
</entry>
<entry>
<para> For clients.</para>
<para>  </para>
</entry>
<entry>
- <para> lustre-<ver></para>
+ <para> lustre-<replaceable>ver</replaceable></para>
</entry>
<entry>
<para> Lustre utilities package. This includes userspace utilities to configure and run Lustre.</para>
<para>  </para>
</entry>
<entry>
- <para> lustre-client-<ver></para>
+ <para> lustre-client-<replaceable>ver</replaceable></para>
</entry>
<entry>
<para> Lustre utilities for clients.</para>
<para>  </para>
</entry>
<entry>
- <para> lustre-ldiskfs-<ver></para>
+ <para> lustre-ldiskfs-<replaceable>ver</replaceable></para>
</entry>
<entry>
<para> Lustre-patched backing file system kernel module package for the ldiskfs file system.</para>
<para>  </para>
</entry>
<entry>
- <para> e2fsprogs-<ver></para>
+ <para> e2fsprogs-<replaceable>ver</replaceable></para>
</entry>
<entry>
<para> Utilities package used to maintain the ldiskfs backing file system.</para>
<para>It is not recommended that you use the <literal>rpm -Uvh</literal> command to install a kernel, because this may leave you with an unbootable system if the new kernel doesn't work for some reason.</para>
</note>
<para>For example, the command in the following example would install required packages on a server with Infiniband networking</para>
- <screen>$ rpm -ivh kernel-<ver>_lustre-<ver> kernel-ib-<ver> \
-lustre-modules-<ver> lustre-ldiskfs-<ver>
+ <screen>$ rpm -ivh kernel-<replaceable>ver</replaceable>_lustre-<replaceable>ver</replaceable> kernel-ib-<replaceable>ver</replaceable> \
+lustre-modules-<replaceable>ver</replaceable> lustre-ldiskfs-<replaceable>ver</replaceable>
</screen>
</listitem>
<listitem>
<listitem>
<para>Install the utilities/userspace packages.</para>
<para>Use the <literal>rpm -ivh</literal> command to install the utilities packages. For example:</para>
- <screen>$ rpm -ivh lustre-<ver>
+ <screen>$ rpm -ivh lustre-<replaceable>ver</replaceable>
</screen>
</listitem>
<listitem>
<para>Install the <literal>e2fsprogs</literal> package.</para>
<para>Use the <literal>rpm -ivh</literal> command to install the <literal>e2fsprogs</literal> package. For example:</para>
- <screen>$ rpm -ivh e2fsprogs-<ver>
+ <screen>$ rpm -ivh e2fsprogs-<replaceable>ver</replaceable>
</screen>
<para>If <literal>e2fsprogs</literal> is already installed on your Linux system, install the Lustre-specific <literal>e2fsprogs</literal> version by using <literal>rpm -Uvh</literal> to upgrade the existing <literal>e2fsprogs</literal> package. For example:</para>
- <screen>$ rpm -Uvh e2fsprogs-<ver> </screen>
+ <screen>$ rpm -Uvh e2fsprogs-<replaceable>ver</replaceable> </screen>
<note>
<para>The <literal>rpm</literal> command options <literal>--force</literal> or <literal>--nodeps</literal> should not be used to install or update the Lustre-specific <literal>e2fsprogs</literal> package. If errors are reported, file a bug (for instructions see the topic
<link xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</link>
</listitem>
<listitem>
<para>Install the module packages for clients.</para>
- <para>Use the <literal>rpm -ivh</literal> command to install the lustre-client and <literal>lustre-client-modules-<ver></literal> packages. For example:</para>
- <screen>$ rpm -ivh lustre-client-modules-<ver> kernel-ib-<ver></screen>
+ <para>Use the <literal>rpm -ivh</literal> command to install the lustre-client and <literal>lustre-client-modules-<replaceable>ver</replaceable></literal> packages. For example:</para>
+ <screen>$ rpm -ivh lustre-client-modules-<replaceable>ver</replaceable> kernel-ib-<replaceable>ver</replaceable></screen>
</listitem>
<listitem>
<para>Install the utilities/userspace packages for clients.</para>
<orderedlist>
<listitem>
<para>Configure the patched kernel to run with Lustre. Run:</para>
- <screen>$ cd <path to kernel tree>
+ <screen>$ cd <replaceable>/path/to/kernel/tree</replaceable>
$ cp /boot/config-`uname -r` .config
$ make oldconfig || make menuconfig
$ make include/asm
</listitem>
<listitem>
<para>Run the Lustre configure script against the patched kernel and create the Lustre packages.</para>
- <screen>$ cd <path to lustre source tree>
-$ ./configure --with-linux=<path to kernel tree>
+ <screen>$ cd <replaceable>/path/to/lustre/source/tree</replaceable>
+$ ./configure --with-linux=<replaceable>/path/to/kernel/tree</replaceable>
$ make rpms</screen>
- <para>This creates a set of <literal>.rpms</literal> in <literal>/usr/src/redhat/RPMS/<arch></literal> with an appended date-stamp. The SuSE path is <literal>/usr/src/packages</literal>.</para>
+ <para>This creates a set of <literal>.rpms</literal> in <literal>/usr/src/redhat/RPMS/<replaceable>arch</replaceable></literal> with an appended date-stamp. The SuSE path is <literal>/usr/src/packages</literal>.</para>
<note>
<para>You do not need to run the Lustre configure script against an unpatched kernel.</para>
</note>
<listitem>
<para>Install the kernel, modules and <literal>ldiskfs</literal> packages.</para>
<para>Navigate to the directory where the RPMs are stored, and use the <literal>rpm -ivh</literal> command to install the kernel, module and ldiskfs packages.</para>
- <screen>$ rpm -ivh kernel-lustre-smp-<ver> \
-kernel-ib-<ver> \
-lustre-modules-<ver> \
-lustre-ldiskfs-<ver></screen>
+ <screen>$ rpm -ivh kernel-lustre-smp-<replaceable>ver</replaceable> \
+kernel-ib-<replaceable>ver</replaceable> \
+lustre-modules-<replaceable>ver</replaceable> \
+lustre-ldiskfs-<replaceable>ver</replaceable></screen>
</listitem>
<listitem>
<para>Install the utilities/userspace packages.</para>
<para>Use the <literal>rpm -ivh</literal> command to install the utilities packages. For example:</para>
- <screen>$ rpm -ivh lustre-<ver></screen>
+ <screen>$ rpm -ivh lustre-<replaceable>ver</replaceable></screen>
</listitem>
<listitem>
<para>Install the <literal>e2fsprogs</literal> package.</para>
<para>Make sure the <literal>e2fsprogs</literal> package is unpacked, and use the <literal>rpm -i</literal> command to install it. For example:</para>
- <screen>$ rpm -i e2fsprogs-<ver></screen>
+ <screen>$ rpm -i e2fsprogs-<replaceable>ver</replaceable></screen>
</listitem>
<listitem>
<para>(Optional) If you want to add optional packages to your Lustre system, install them now.</para>
</section>
<section xml:id="dbdoclet.50438223_42848">
<title>Defining and Running the Tests</title>
- <para>A <emphasis>test</emphasis> generates a network load between two groups of nodes, a source group identified using the <literal>--from</literal> parameter and a target group identified using the <literal>--to</literal> parameter. When a test is running, each node in the <literal>--from<replaceable><group></replaceable></literal> simulates a client by sending requests to nodes in the <literal>--to<replaceable><group></replaceable></literal>, which are simulating a set of servers, and then receives responses in return. This activity is designed to mimic Lustre RPC traffic.</para>
+ <para>A <emphasis>test</emphasis> generates a network load between two groups of nodes, a source group identified using the <literal>--from</literal> parameter and a target group identified using the <literal>--to</literal> parameter. When a test is running, each node in the <literal>--from <replaceable>group</replaceable></literal> simulates a client by sending requests to nodes in the <literal>--to <replaceable>group</replaceable></literal>, which are simulating a set of servers, and then receives responses in return. This activity is designed to mimic Lustre RPC traffic.</para>
<para>A <emphasis>batch</emphasis> is a collection of tests that are started and stopped together and run in parallel. A test must always be run as part of a batch, even if it is just a single test. Users can only run or stop a test batch, not individual tests.</para>
<para>Tests in a batch are non-destructive to the file system, and can be run in a normal Lustre environment (provided the performance impact is acceptable).</para>
- <para>A simple batch might contain a single test, for example, to determine whether the network bandwidth presents an I/O bottleneck. In this example, the <literal>--to<replaceable><group></replaceable></literal> could be comprised of Lustre OSSs and <literal>--from<replaceable><group></replaceable></literal> the compute nodes. A second test could be added to perform pings from a login node to the MDS to see how checkpointing affects the <literal>ls -l</literal> process.</para>
+ <para>A simple batch might contain a single test, for example, to determine whether the network bandwidth presents an I/O bottleneck. In this example, the <literal>--to <replaceable>group</replaceable></literal> could be comprised of Lustre OSSs and <literal>--from <replaceable>group</replaceable></literal> the compute nodes. A second test could be added to perform pings from a login node to the MDS to see how checkpointing affects the <literal>ls -l</literal> process.</para>
<para>Two types of tests are available:</para>
<itemizedlist>
<listitem>
<tbody>
<row>
<entry>
- <para><literal>--timeout<replaceable><seconds></replaceable></literal></para>
+ <para><literal>--timeout <replaceable>seconds</replaceable></literal></para>
</entry>
<entry>
<para>Console timeout value of the session. The session ends automatically if it remains idle (i.e., no commands are issued) for this period.</para>
</row>
<row>
<entry>
- <para> <emphasis>
- <literal><name></literal>
- </emphasis></para>
+ <para><literal><replaceable>name</replaceable></literal></para>
</entry>
<entry>
<para>A human-readable string to print when listing sessions or reporting session conflicts.</para>
<para><emphasis role="bold">Example:</emphasis></para>
<screen>$ lst new_session --force read_write</screen>
<para><literal>
- <replaceable role="bold">end_session</replaceable>
+ end_session
</literal></para>
<para>Stops all operations and tests in the current session and clears the session's status.</para>
<screen>$ lst end_session</screen>
<para><literal>
- <replaceable role="bold">show_session</replaceable>
+ show_session
</literal></para>
<para>Shows the session information. This command prints information about the current session. It does not require LST_SESSION to be defined in the process environment.</para>
<screen>$ lst show_session</screen>
<title>Group Commands</title>
<para>This section describes <literal>lst</literal> group commands.</para>
<para><literal>
- <replaceable role="bold">add_group</replaceable>
- <replaceable> <name> <NIDS> [<NIDs>...]</replaceable>
+ add_group
+ <replaceable>name</replaceable> <replaceable>NIDs [NIDs ...]</replaceable>
</literal></para>
<para>Creates the group and adds a list of test nodes to the group.</para>
<informaltable frame="all">
<row>
<entry>
<para> <literal>
- <replaceable><name></replaceable>
+ <replaceable>name</replaceable>
</literal></para>
</entry>
<entry>
<row>
<entry>
<para> <literal>
- <replaceable><NIDs></replaceable>
+ <replaceable>NIDs</replaceable>
</literal></para>
</entry>
<entry>
$ lst add_group clients 192.168.1.[10-100]@tcp 192.168.[2,4].\
[10-20]@tcp</screen>
<para><literal>
- <replaceable role="bold">update_group</replaceable>
- <replaceable><name></replaceable>
- <replaceable role="bold">[--refresh] [--clean</replaceable>
- <replaceable><status></replaceable>
- <replaceable role="bold">] [--remove</replaceable>
- <replaceable><NIDs></replaceable>
- <replaceable role="bold">]</replaceable>
+ update_group
+ <replaceable>name</replaceable>
+ [--refresh] [--clean <replaceable>status</replaceable>]
+ [--remove <replaceable>NIDs</replaceable>]
</literal></para>
<para>Updates the state of nodes in a group or adjusts a group's membership. This command is useful if some nodes have crashed and should be excluded from the group.</para>
<informaltable frame="all">
</row>
<row>
<entry>
- <para> <literal>--clean<replaceable><status></replaceable></literal></para>
+ <para> <literal>--clean <replaceable>status</replaceable></literal></para>
</entry>
<entry nameend="c3" namest="c2">
<para> Removes nodes with a specified status from the group. Status may be:</para>
</row>
<row>
<entry>
- <para> <literal>--remove<replaceable><NIDs></replaceable></literal></para>
+ <para> <literal>--remove <replaceable>NIDs</replaceable></literal></para>
</entry>
<entry nameend="c3" namest="c2">
<para> Removes specified nodes from the group.</para>
invalid == busy || down || unknown
$ lst update_group clients --remove \192.168.1.[10-20]@tcp</screen>
<para><literal>
- <replaceable role="bold">list_group [</replaceable>
- <replaceable><name></replaceable>
- <replaceable role="bold">] [--active] [--busy] [--down] [--unknown] [--all]</replaceable>
+ list_group [
+ name
+ ] [--active] [--busy] [--down] [--unknown] [--all]
</literal></para>
<para>Prints information about a group or lists all groups in the current session if no group is specified.</para>
<informaltable frame="all">
<row>
<entry>
<para><literal>
- <replaceable><name></replaceable>
+ <replaceable>name</replaceable>
</literal></para>
</entry>
<entry>
192.168.1.12@tcp Busy
Total 1 node</screen>
<para><literal>
- <replaceable role="bold">del_group</replaceable>
- <replaceable> <name></replaceable>
+ del_group
+ <replaceable>name</replaceable>
</literal></para>
<para>Removes a group from the session. If the group is referred to by any test, then the operation fails. If nodes in the group are referred to only by this group, then they are kicked out from the current session; otherwise, they are still in the current session.</para>
<screen>$ lst del_group clients</screen>
- <para><literal><replaceable role="bold">lstclient --sesid</replaceable><replaceable> <NID> </replaceable><replaceable role="bold">--group</replaceable><replaceable> <name></replaceable> [--server_mode]</literal></para>
+ <para><literal>lstclient --sesid <replaceable>NID</replaceable> --group <replaceable>name</replaceable> [--server_mode]</literal></para>
<para>Use <literal>lstclient</literal> to run the userland self-test client. The <literal>lstclient</literal> command should be executed after creating a session on the console. There are only two mandatory options for <literal>lstclient</literal>:</para>
<informaltable frame="all">
<tgroup cols="2">
<tbody>
<row>
<entry>
- <literal> --sesid<replaceable><NID></replaceable></literal>
+ <literal> --sesid <replaceable>NID</replaceable></literal>
</entry>
<entry>
<para> The first console's NID.</para>
</row>
<row>
<entry>
- <literal> --group<replaceable><name></replaceable></literal>
+ <literal> --group <replaceable>name</replaceable></literal>
</entry>
<entry>
<para> The test group to join.</para>
<title>Batch and Test Commands</title>
<para>This section describes <literal>lst</literal> batch and test commands.</para>
<para><literal>
- <replaceable role="bold">add_batch NAME</replaceable>
+ add_batch <replaceable>name</replaceable>
</literal></para>
<para>A default batch test set named batch is created when the session is started. You can specify a batch name by using add_batch:</para>
<screen>$ lst add_batch bulkperf</screen>
<para>Creates a batch test called <literal>bulkperf</literal>.</para>
<screen>
- <replaceable role="bold">add_test --batch <replaceable><<replaceable role="italic">batchname</replaceable>></replaceable> [--loop<#>] <replaceable role="bold">[--concurrency</replaceable><replaceable><#></replaceable><replaceable role="bold">] [--distribute</replaceable><replaceable><#:#></replaceable><replaceable role="bold">]</replaceable> \ --from<replaceable> <<replaceable role="italic">group</replaceable>></replaceable> --to <replaceable><<replaceable role="italic">group</replaceable>></replaceable> {brw|ping} <test options></replaceable>
+add_test --batch <replaceable>batchname</replaceable> [--loop <replaceable>loop_count</replaceable>] [--concurrency <replaceable>active_count</replaceable>] [--distribute <replaceable>source_count</replaceable>:<replaceable>sink_count</replaceable>] \
+ --from <replaceable>group</replaceable> --to <replaceable>group</replaceable> brw|ping <replaceable>test_options</replaceable>
</screen>
<para>Adds a test to a batch. The parameters are described below.</para>
<informaltable frame="all">
<tbody>
<row>
<entry>
- <para> <literal>--batch<replaceable><batchname></replaceable></literal></para>
+ <para> <literal>--batch <replaceable>batchname</replaceable></literal></para>
</entry>
<entry nameend="c3" namest="c2">
<para>Names a group of tests for later execution.</para>
</row>
<row>
<entry>
- <literal> --loop<replaceable><#></replaceable></literal>
+ <literal> --loop <replaceable>loop_count</replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
<para>Number of times to run the test.</para>
</row>
<row>
<entry>
- <literal> --concurrency<replaceable><#></replaceable></literal>
+ <literal> --concurrency <replaceable>active_count</replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
<para>The number of requests that are active at one time.</para>
</row>
<row>
<entry>
- <literal> --distribute<replaceable><#:#></replaceable></literal>
+ <literal> --distribute <replaceable>source_count</replaceable>:<replaceable>sink_count</replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
<para>Determines the ratio of client nodes to server nodes for the specified test. This allows you to specify a wide range of topologies, including one-to-one and all-to-all. Distribution divides the source group into subsets, which are paired with equivalent subsets from the target group so only nodes in matching subsets communicate.</para>
</row>
<row>
<entry>
- <literal> --from<replaceable><group></replaceable></literal>
+ <literal> --from <replaceable>group</replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
<para>The source group (test client).</para>
</row>
<row>
<entry>
- <literal> --to<replaceable><group></replaceable></literal>
+ <literal> --to <replaceable>group</replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
<para>The target group (test server).</para>
<row>
<entry/>
<entry>
- <literal> size=<#>| <#>K | <#>M </literal>
+ <literal> size=<replaceable>bytes[KM]</replaceable></literal>
</entry>
<entry>
- <para>I/O size in bytes, KB or MB (i.e., <literal>size=1024</literal>, <literal>size=4</literal>K, <literal>size=1M</literal>). The default is 4K bytes.</para>
+ <para>I/O size in bytes, kilobytes, or Megabytes (i.e., <literal>size=1024</literal>, <literal>size=4K</literal>, <literal>size=1M</literal>). The default is 4 kilobytes.</para>
</entry>
</row>
<row>
--distribute 4:2 (C1,C2,C3,C4->S1,S2), (C5, C6->S3, S1)
--distribute 6:3 (C1,C2,C3,C4,C5,C6->S1,S2,S3)</screen>
<para>The setting <literal>--distribute 1:1</literal> is the default setting where each source node communicates with one target node.</para>
- <para>When the setting <literal>--distribute 1:<replaceable><n></replaceable></literal> (where <literal>
- <replaceable><n></replaceable>
+ <para>When the setting <literal>--distribute 1:<replaceable>n</replaceable></literal> (where <literal>
+ <replaceable>n</replaceable>
</literal> is the size of the target group) is used, each source node communicates with every node in the target group.</para>
<para>Note that if there are more source nodes than target nodes, some source nodes may share the same target nodes. Also, if there are more target nodes than source nodes, some higher-ranked target nodes will be idle.</para>
<para><emphasis role="bold">Example showing a <literal>brw</literal> test:</emphasis></para>
</listitem>
</itemizedlist>
<para><emphasis role="bold">
- <literal><replaceable role="bold">list_batch [</replaceable><replaceable><replaceable role="italic"><name</replaceable>></replaceable><replaceable role="bold">] [--test</replaceable><replaceable> <<replaceable role="italic">index</replaceable>></replaceable><replaceable role="bold">] [--active] [--invalid]</replaceable> [--server | client<replaceable role="bold">]</replaceable></literal>
+ <literal>list_batch [<replaceable>name</replaceable>] [--test <replaceable>index</replaceable>] [--active] [--invalid] [--server|client]</literal>
</emphasis></para>
<para>Lists batches in the current session or lists client and server nodes in a batch or a test.</para>
<informaltable frame="all">
<tbody>
<row>
<entry>
- <literal> --test<replaceable><index></replaceable></literal>
+ <literal> --test <replaceable>index</replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
<para>Lists tests in a batch. If no option is used, all tests in the batch are listed. If one of these options are used, only specified tests in the batch are listed:</para>
192.168.10.102@tcp Active
192.168.10.103@tcp Active</screen>
<para><literal>
- <replaceable role="bold">run</replaceable>
- <replaceable role="bold">
- <replaceable> <<replaceable role="italic">name</replaceable>></replaceable>
- </replaceable>
+ run
+ <replaceable>name</replaceable>
</literal></para>
<para>Runs the batch.</para>
<screen>$ lst run bulkperf</screen>
<para><literal>
- <replaceable role="bold">
- <replaceable role="bold">stop </replaceable>
- <replaceable><<replaceable role="italic">name</replaceable>></replaceable>
- </replaceable>
+ stop
+ <replaceable>name</replaceable>
</literal></para>
<para>Stops the batch.</para>
<screen>$ lst stop bulkperf</screen>
<para><emphasis role="bold">
<literal>
- <replaceable role="bold">query </replaceable>
- <replaceable><<replaceable role="italic">name</replaceable>> </replaceable>
- <replaceable role="bold">[--test</replaceable>
- <replaceable> <<replaceable role="italic">index</replaceable>></replaceable>
- <replaceable role="bold">] [--timeout</replaceable>
- <replaceable> <<replaceable role="italic">seconds</replaceable>></replaceable>
- <replaceable role="bold">] [--loop</replaceable>
- <replaceable><#></replaceable>
- <replaceable role="bold">] [--delay</replaceable>
- <replaceable><<replaceable role="italic">seconds</replaceable>></replaceable>
- <replaceable role="bold">] [--all]</replaceable>
+ query
+ <replaceable>name</replaceable>
+ [--test <replaceable>index</replaceable>]
+ [--timeout <replaceable>seconds</replaceable>]
+ [--loop <replaceable>#</replaceable>]
+ [--delay <replaceable>seconds</replaceable>]
+ [--all]
</literal>
</emphasis></para>
<para>Queries the batch status.</para>
<tbody>
<row>
<entry>
- <literal> --test<replaceable><index></replaceable></literal>
+ <literal> --test <replaceable>index</replaceable></literal>
</entry>
<entry>
<para> Only queries the specified test. The test index starts from 1.</para>
</row>
<row>
<entry>
- <literal> --timeout<replaceable><seconds></replaceable></literal>
+ <literal> --timeout <replaceable>seconds</replaceable></literal>
</entry>
<entry>
<para> The timeout value to wait for RPC. The default is 5 seconds.</para>
</row>
<row>
<entry>
- <literal> --loop<replaceable><#></replaceable></literal>
+ <literal> --loop <replaceable>loop_count</replaceable></literal>
</entry>
<entry>
<para> The loop count of the query.</para>
</row>
<row>
<entry>
- <literal> --delay<replaceable><seconds></replaceable></literal>
+ <literal> --delay <replaceable>seconds</replaceable></literal>
</entry>
<entry>
<para> The interval of each query. The default is 5 seconds.</para>
<title>Other Commands</title>
<para>This section describes other <literal>lst</literal> commands.</para>
<para><literal>
- <replaceable role="bold">
- <replaceable role="bold">ping [-session] [--group</replaceable>
- <replaceable><<replaceable role="italic">name</replaceable>></replaceable>
- <replaceable role="bold">] [--nodes</replaceable>
- <replaceable> <NIDs></replaceable>
- <replaceable role="bold">] [--batch</replaceable>
- <replaceable> <<replaceable role="italic">name</replaceable>></replaceable>
- <replaceable role="bold">] [--server] [--timeout</replaceable>
- <replaceable> <<replaceable role="italic">seconds</replaceable>></replaceable>
- <replaceable role="bold">]</replaceable>
- </replaceable>
+ ping [-session] [--group <replaceable>name</replaceable>]
+ [--nodes <replaceable>NIDs</replaceable>]
+ [--batch <replaceable>name</replaceable>]
+ [--server]
+ [--timeout <replaceable>seconds</replaceable>]
</literal></para>
<para>Sends a 'hello' query to the nodes.</para>
<informaltable frame="all">
</row>
<row>
<entry>
- <literal> --group<replaceable><name></replaceable></literal>
+ <literal> --group <replaceable>name</replaceable></literal>
</entry>
<entry>
<para>Pings all nodes in a specified group.</para>
</row>
<row>
<entry>
- <literal> --nodes<replaceable><NIDs></replaceable></literal>
+ <literal> --nodes <replaceable>NIDs</replaceable></literal>
</entry>
<entry>
<para>Pings all specified nodes.</para>
</row>
<row>
<entry>
- <literal> --batch<replaceable><name></replaceable></literal>
+ <literal> --batch<replaceable>name</replaceable></literal>
</entry>
<entry>
<para>Pings all client nodes in a batch.</para>
<literal> --server </literal>
</entry>
<entry>
- <para>Sends RPC to all server nodes instead of client nodes. This option is only used with <literal>--batch<replaceable><name></replaceable></literal>.</para>
+ <para>Sends RPC to all server nodes instead of client nodes. This option is only used with <literal>--batch <replaceable>name</replaceable></literal>.</para>
</entry>
</row>
<row>
<entry>
- <literal> --timeout<replaceable><seconds></replaceable></literal>
+ <literal> --timeout <replaceable>seconds</replaceable></literal>
</entry>
<entry>
<para>The RPC timeout value.</para>
</tgroup>
</informaltable>
<para><emphasis role="bold">Example:</emphasis></para>
- <screen>$ lst ping 192.168.10.[15-20]@tcp
+ <screen># lst ping 192.168.10.[15-20]@tcp
192.168.1.15@tcp Active [session: liang id: 192.168.1.3@tcp]
192.168.1.16@tcp Active [session: liang id: 192.168.1.3@tcp]
192.168.1.17@tcp Active [session: liang id: 192.168.1.3@tcp]
192.168.1.19@tcp Down [session: <NULL> id: LNET_NID_ANY]
192.168.1.20@tcp Down [session: <NULL> id: LNET_NID_ANY]</screen>
<para><literal>
- <replaceable role="bold">
- <replaceable role="bold">stat [--bw] [--rate] [--read] [--write] [--max] [--min] [--avg] " " [--timeout</replaceable>
- <replaceable> <<replaceable role="italic">seconds</replaceable>></replaceable>
- <replaceable role="bold">] [--delay </replaceable>
- <replaceable><<replaceable role="italic">seconds</replaceable>></replaceable>
- <replaceable role="bold">] </replaceable>
- <replaceable><<replaceable role="italic">group</replaceable>></replaceable>
- <replaceable role="bold">|<</replaceable>
- <replaceable><replaceable role="italic">NIDs</replaceable>> </replaceable>
- <replaceable role="bold">[</replaceable>
- <replaceable><<replaceable role="italic">group</replaceable>></replaceable>
- <replaceable role="bold">|</replaceable>
- <replaceable><<replaceable role="italic">NIDs</replaceable>></replaceable>
- <replaceable role="bold">]</replaceable>
- </replaceable>
+ stat [--bw] [--rate] [--read] [--write] [--max] [--min] [--avg] " "
+ [--timeout <replaceable>seconds</replaceable>]
+ [--delay <replaceable>seconds</replaceable>]
+ <replaceable>group|NIDs</replaceable>
+ [<replaceable>group|NIDs</replaceable>]
</literal></para>
<para>The collection performance and RPC statistics of one or more nodes.</para>
<informaltable frame="all">
</row>
<row>
<entry>
- <literal> --timeout<replaceable><seconds></replaceable></literal>
+ <literal> --timeout <replaceable>seconds</replaceable></literal>
</entry>
<entry>
<para>The timeout of the statistics RPC. The default is 5 seconds.</para>
</row>
<row>
<entry>
- <literal> --delay<replaceable><seconds></replaceable></literal>
+ <literal> --delay <replaceable>seconds</replaceable></literal>
</entry>
<entry>
<para>The interval of the statistics (in seconds).</para>
<para>Only LNET performance statistics are available. By default, all statistics
information is displayed. Users can specify additional information with these options.</para>
<para><literal>
- <replaceable role="bold">show_error [--session] [<group>|<NIDs>]...</replaceable>
+ show_error [--session] <replaceable>[group|NIDs]...</replaceable>
</literal></para>
<para>Lists the number of failed RPCs on test nodes.</para>
<informaltable frame="all">
</emphasis> - This tool is used with the debug_kernel option to manually dump the Lustre debugging log or post-process debugging logs that are dumped automatically. For more information about the lctl tool, see <xref linkend="dbdoclet.50438274_62472"/> and <xref linkend="dbdoclet.50438219_38274"/>.</para>
</listitem>
<listitem>
- <para><replaceable role="bold">Lustre subsystem asserts</replaceable> - A panic-style assertion (LBUG) in the kernel causes Lustre to dump the debug log to the file <literal>/tmp/lustre-log.<replaceable><timestamp></replaceable></literal> where it can be retrieved after a reboot. For more information, see <xref linkend="dbdoclet.50438198_40669"/>.</para>
+ <para>Lustre subsystem asserts - A panic-style assertion (LBUG) in the kernel causes Lustre to dump the debug log to the file <literal>/tmp/lustre-log.<replaceable>timestamp</replaceable></literal> where it can be retrieved after a reboot. For more information, see <xref linkend="dbdoclet.50438198_40669"/>.</para>
</listitem>
<listitem>
<para>
<itemizedlist>
<listitem>
<para><literal>
- <replaceable role="bold">leak_finder.pl</replaceable>
+ <replaceable>leak_finder.pl</replaceable>
</literal> . This program provided with Lustre is useful for finding memory leaks in the code.</para>
</listitem>
</itemizedlist>
<itemizedlist>
<listitem>
<para>Obtain a list of all the types and subsystems:</para>
- <screen>lctl > debug_list <emphasis>{subs | types}</emphasis></screen>
+ <screen>lctl > debug_list <replaceable>subsystems|types</replaceable></screen>
</listitem>
</itemizedlist>
<itemizedlist>
<listitem>
<para>Filter the debug log:</para>
- <screen>lctl > filter <emphasis>{subsystem name | debug type}</emphasis></screen>
+ <screen>lctl > filter <replaceable>subsystem_name|debug_type</replaceable></screen>
</listitem>
</itemizedlist>
<note>
<itemizedlist>
<listitem>
<para>Show debug messages belonging to certain subsystem or type:</para>
- <screen>lctl > show <emphasis>{subsystem name | debug type}</emphasis></screen>
+ <screen>lctl > show <replaceable>subsystem_name|debug_type</replaceable></screen>
<para><literal>debug_kernel</literal> pulls the data from the kernel logs, filters it appropriately, and displays or saves it as per the specified options</para>
- <screen>lctl > debug_kernel [<emphasis>output filename</emphasis>]</screen>
+ <screen>lctl > debug_kernel [<replaceable>output filename</replaceable>]</screen>
<para>If the debugging is being done on User Mode Linux (UML), it might be useful to save the logs on the host machine so that they can be used at a later time.</para>
</listitem>
</itemizedlist>
<itemizedlist>
<listitem>
<para>Filter a log on disk, if you already have a debug log saved to disk (likely from a crash):</para>
- <screen>lctl > debug_file <emphasis>{input filename}</emphasis> [<emphasis>output filename</emphasis>] </screen>
+ <screen>lctl > debug_file <replaceable>input_file</replaceable> [<replaceable>output_file</replaceable>] </screen>
<para>During the debug session, you can add markers or breaks to the log for any reason:</para>
<screen>lctl > mark [marker text] </screen>
<para>The marker text defaults to the current date and time in the debug log (similar to the example shown below):</para>
<section remap="h4">
<title><literal>lctl debug_daemon</literal> Commands</title>
<para>To initiate the <literal>debug_daemon</literal> to start dumping the <literal>debug_buffer</literal> into a file, run as the root user:</para>
- <screen>lctl debug_daemon start {filename} [{megabytes}]</screen>
+ <screen>lctl debug_daemon start <replaceable>filename</replaceable> [<replaceable>megabytes</replaceable>]</screen>
<para>The debug log will be written to the specified filename from the kernel. The file will be limited to the optionally specified number of megabytes.</para>
<para>The daemon wraps around and dumps data to the beginning of the file when the output file size is over the limit of the user-specified file size. To decode the dumped file to ASCII and sort the log entries by time, run:</para>
- <screen>lctl debug_file {filename} > {newfile}</screen>
+ <screen>lctl debug_file <replaceable>filename</replaceable> > <replaceable>newfile</replaceable></screen>
<para>The output is internally sorted by the <literal>lctl</literal> command.</para>
<para>To stop the <literal>debug_daemon</literal> operation and flush the file output, run:</para>
<screen>lctl debug_daemon stop</screen>
<para>This is an example using <literal>debug_daemon</literal> with the interactive mode of <literal>lctl</literal> to dump debug logs to a 40 MB file.</para>
<screen>lctl</screen>
<screen>lctl > debug_daemon start /var/log/lustre.40.bin 40 </screen>
- <screen>{run filesystem operations to debug}</screen>
+ <screen><replaceable>run filesystem operations to debug</replaceable></screen>
<screen>lctl > debug_daemon stop </screen>
<screen>lctl > debug_file /var/log/lustre.bin /var/log/lustre.log</screen>
<para>To start another daemon with an unlimited file size, run:</para>
</section>
<section remap="h3">
<title><indexterm><primary>debugging</primary><secondary>kernel debug log</secondary></indexterm>Controlling Information Written to the Kernel Debug Log</title>
- <para>The <literal>lctl set_param subsystem_debug={subsystem_mask}</literal> and <literal>lctl set_param debug={debug_mask}</literal> are used to determine which information is written to the debug log. The subsystem_debug mask determines the information written to the log based on the functional area of the code (such as lnet, osc, or ldlm). The debug mask controls information based on the message type (such as info, error, trace, or malloc).</para>
+ <para>The <literal>lctl set_param subsystem_debug=<replaceable>subsystem_mask</replaceable></literal> and <literal>lctl set_param debug=<replaceable>debug_mask</replaceable></literal> are used to determine which information is written to the debug log. The subsystem_debug mask determines the information written to the log based on the functional area of the code (such as lnet, osc, or ldlm). The debug mask controls information based on the message type (such as info, error, trace, or malloc).</para>
<para>To turn off Lustre debugging completely:</para>
<screen>lctl set_param debug=0 </screen>
<para>To turn on full Lustre debugging:</para>
<title><indexterm><primary>debugging</primary><secondary>using strace</secondary></indexterm>Troubleshooting with <literal>strace</literal></title>
<para>The <literal>strace</literal> utility provided with the Linux distribution enables system calls to be traced by intercepting all the system calls made by a process and recording the system call name, arguments, and return values.</para>
<para>To invoke <literal>strace</literal> on a program, enter:</para>
- <screen>$ strace <emphasis>{program} {args}</emphasis> </screen>
+ <screen>$ strace <replaceable>program</replaceable> <replaceable>[arguments]</replaceable> </screen>
<para>Sometimes, a system call may fork child processes. In this situation, use the <literal>-f</literal> option of <literal>strace</literal> to trace the child processes:</para>
- <screen>$ strace -f <emphasis>{program} {args}</emphasis> </screen>
+ <screen>$ strace -f <replaceable>program</replaceable> <replaceable>[arguments]</replaceable> </screen>
<para>To redirect the <literal>strace</literal> output to a file, enter:</para>
- <screen>$ strace -o <emphasis>{filename} {program} {args}</emphasis> </screen>
+ <screen>$ strace -o <replaceable>filename</replaceable> <replaceable>program</replaceable> <replaceable>[arguments]</replaceable> </screen>
<para>Use the <literal>-ff</literal> option, along with <literal>-o</literal>, to save the trace output in <literal>filename.pid</literal>, where <literal>pid</literal> is the process ID of the process being traced. Use the <literal>-ttt</literal> option to timestamp all lines in the strace output, so they can be correlated to operations in the lustre kernel debug log.</para>
</section>
<section remap="h3">
<para>Lustre has a specific debug type category for tracing lock traffic. Use:</para>
<screen>lctl> filter all_types
lctl> show dlmtrace
-lctl> debug_kernel [filename] </screen>
+lctl> debug_kernel [<replaceable>filename</replaceable>] </screen>
</section>
</section>
<section xml:id="dbdoclet.50438274_80443">
</emphasis></para>
</entry>
<entry>
- <para>Allows insertion of failure points into the Lustre code. This is useful to generate regression tests that can hit a very specific sequence of events. This works in conjunction with "<literal>lctl set_param fail_loc={fail_loc}</literal>" to set a specific failure point for which a given <literal>OBD_FAIL_CHECK()</literal> will test.</para>
+ <para>Allows insertion of failure points into the Lustre code. This is useful to generate regression tests that can hit a very specific sequence of events. This works in conjunction with "<literal>lctl set_param fail_loc=<replaceable>fail_loc</replaceable></literal>" to set a specific failure point for which a given <literal>OBD_FAIL_CHECK()</literal> will test.</para>
</entry>
</row>
<row>
</listitem>
</itemizedlist>
<para>Requests in the history include "live" requests that are currently being handled. Each line in <literal>req_history</literal> looks like:</para>
- <screen><seq>:<target NID>:<client ID>:<xid>:<length>:<phase> <svc specific> </screen>
+ <screen><replaceable>sequence</replaceable>:<replaceable>target_NID</replaceable>:<replaceable>client_NID</replaceable>:<replaceable>cliet_xid</replaceable>:<replaceable>request_length</replaceable>:<replaceable>rpc_phase</replaceable> <replaceable>service_specific_data</replaceable> </screen>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<row>
<entry>
<para> <literal>
- <replaceable role="bold">target NID</replaceable>
+ <replaceable>target NID</replaceable>
</literal></para>
</entry>
<entry>
<row>
<entry>
<para> <literal>
- <replaceable role="bold">client ID</replaceable>
+ <replaceable>client ID</replaceable>
</literal></para>
</entry>
<entry>
</listitem>
<listitem>
<para>Run the leak finder on the newly-created log dump:</para>
- <screen>perl leak_finder.pl {ascii-logname}</screen>
+ <screen>perl leak_finder.pl <replaceable>ascii-logname</replaceable></screen>
</listitem>
</orderedlist>
<para>The output is:</para>
<indexterm><primary>maintenance</primary><secondary>inactive OSTs</secondary></indexterm>
Working with Inactive OSTs</title>
<para>To mount a client or an MDT with one or more inactive OSTs, run commands similar to this:</para>
- <screen>client> mount -o exclude=testfs-OST0000 -t lustre \
+ <screen>client# mount -o exclude=testfs-OST0000 -t lustre \
uml1:/testfs /mnt/testfs
- client> cat /proc/fs/lustre/lov/testfs-clilov-*/target_obd</screen>
+ client# cat /proc/fs/lustre/lov/testfs-clilov-*/target_obd</screen>
<para>To activate an inactive OST on a live client or MDT, use the <literal>lctl activate</literal> command on the OSC device. For example:</para>
<screen>lctl --device 7 activate</screen>
<note>
lustre-OST0000
lustre-OST0001 </screen>
<para>To get the names of all OSTs, run this command on the MDS:</para>
- <screen># cat /proc/fs/lustre/lov/<fsname>-mdtlov/target_obd </screen>
+ <screen># cat /proc/fs/lustre/lov/<replaceable>fsname</replaceable>-mdtlov/target_obd </screen>
<note>
<para>This command must be run on the MDS.
</para>
<title><indexterm><primary>maintenance</primary><secondary>mounting a server</secondary></indexterm>
Mounting a Server Without Lustre Service</title>
<para>If you are using a combined MGS/MDT, but you only want to start the MGS and not the MDT, run this command:</para>
- <screen>mount -t lustre <MDT partition> -o nosvc <mount point></screen>
- <para>The <literal><MDT partition></literal> variable is the combined MGS/MDT.</para>
+ <screen>mount -t lustre <replaceable>/dev/mdt_partition</replaceable> -o nosvc <replaceable>/mount_point</replaceable></screen>
+ <para>The <literal><replaceable>mdt_partition</replaceable></literal> variable is the combined MGS/MDT block device.</para>
<para>In this example, the combined MGS/MDT is <literal>testfs-MDT0000</literal> and the mount point is <literal>/mnt/test/mdt</literal>.</para>
<screen>$ mount -t lustre -L testfs-MDT0000 -o nosvc /mnt/test/mdt</screen>
</section>
<orderedlist>
<listitem>
<para>On the MDT, run:</para>
- <screen><mdt node>$ tunefs.lustre --writeconf <device></screen>
+ <screen>mdt# tunefs.lustre --writeconf <replaceable>/dev/mdt_device</replaceable></screen>
</listitem>
<listitem>
<para>
On each OST, run:
- <screen><ost node>$ tunefs.lustre --writeconf <device></screen>
+ <screen>ost# tunefs.lustre --writeconf <replaceable>/dev/ost_device</replaceable></screen>
</para>
</listitem>
</orderedlist>
<orderedlist>
<listitem>
<para>On the MDT, run:</para>
- <screen><mdt node>$ tunefs.lustre --writeconf <device> </screen>
+ <screen>mdt# tunefs.lustre --writeconf <replaceable>/dev/mdt_device</replaceable> </screen>
</listitem>
<listitem>
<para>On each OST, run:</para>
- <screen><ost node>$ tunefs.lustre --writeconf <device></screen>
+ <screen>ost# tunefs.lustre --writeconf <replaceable>/dev/ost_device</replaceable></screen>
</listitem>
<listitem>
<para>If the NID on the MGS was changed, communicate the new MGS location to each server. Run:</para>
- <screen>tunefs.lustre --erase-param --mgsnode=<new_nid(s)> --writeconf /dev/.. </screen>
+ <screen>tunefs.lustre --erase-param --mgsnode=<replaceable>new_nid(s)</replaceable> --writeconf <replaceable>/dev/device</replaceable></screen>
</listitem>
</orderedlist>
</listitem>
<listitem>
<para>Add the new block device as a new MDT at the next available index. In this example, the next available index is 4.</para>
<screen>
-mkfs.lustre --reformat --fsname=<filesystemname> --mdt --mgsnode=<mgsnode> --index 4 <blockdevice>
+mds# mkfs.lustre --reformat --fsname=<replaceable>filesystem_name</replaceable> --mdt --mgsnode=<replaceable>mgsnode</replaceable> --index 4 <replaceable>/dev/mdt4_device</replaceable>
</screen>
</listitem>
<listitem>
<para>Mount the MDTs.</para>
<screen>
-mount –t lustre <blockdevice> /mnt/mdt4
+mds# mount –t lustre <replaceable>/dev/mdt4_blockdevice</replaceable> /mnt/mdt4
</screen>
</listitem>
</orderedlist>
<orderedlist>
<listitem>
<para> Add a new OST by passing on the following commands, run:</para>
- <screen>$ mkfs.lustre --fsname=spfs --mgsnode=mds16@tcp0 --ost --index=12 /dev/sda
-$ mkdir -p /mnt/test/ost12
-$ mount -t lustre /dev/sda /mnt/test/ost12</screen>
+ <screen>oss# mkfs.lustre --fsname=spfs --mgsnode=mds16@tcp0 --ost --index=12 /dev/sda
+oss# mkdir -p /mnt/test/ost12
+oss# mount -t lustre /dev/sda /mnt/test/ost12</screen>
</listitem>
<listitem>
<para> Migrate the data (possibly).</para>
<para>New files being created will preferentially be placed on the empty OST. As old files are deleted, they will release space on the old OST.</para>
<para>Files existing prior to the expansion can optionally be rebalanced with an in-place copy, which can be done with a simple script. The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
<para>For example, to rebalance all files within <literal>/mnt/lustre/dir</literal>, enter:</para>
- <screen>lfs_migrate /mnt/lustre/file</screen>
+ <screen>client# lfs_migrate /mnt/lustre/file</screen>
<para>To migrate files within the <literal>/test</literal> filesystem on <literal>OST0004</literal> that are larger than 4GB in size, enter:</para>
- <screen>lfs find /test -obd test-OST0004 -size +4G | lfs_migrate -y</screen>
+ <screen>client# lfs find /test -obd test-OST0004 -size +4G | lfs_migrate -y</screen>
<para>See <xref linkend="dbdoclet.50438206_42260"/> for more details.</para>
</listitem>
</orderedlist>
<orderedlist>
<listitem>
<para>List all OSCs on the node, along with their device numbers. Run:</para>
- <screen>lctl dl|grep " osc "</screen>
+ <screen>lctl dl|grep osc</screen>
<para>This is sample <literal>lctl dl | grep</literal></para>
<screen>11 UP osc lustre-OST-0000-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
12 UP osc lustre-OST-0001-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
</listitem>
<listitem>
<para>Temporarily deactivate the OSC on the MDT. On the MDT, run: </para>
- <screen>$ mdt> lctl --device <devno> deactivate</screen>
+ <screen>mds# lctl --device <replaceable>lustre_devno</replaceable> deactivate</screen>
<para>For example, based on the command output in Step 1, to deactivate device 13 (the MDT’s OSC for <literal>OST-0000</literal>), the command would be:
</para>
- <screen>$ mdt> lctl --device 13 deactivate</screen>
+ <screen>mds# lctl --device 13 deactivate</screen>
<para>This marks the OST as inactive on the MDS, so no new objects are assigned to the OST. This does not prevent use of existing objects for reads or writes.
</para>
<note>
<listitem>
<para>If the OST is still online and available, find all files with objects on the deactivated OST, and copy them to other OSTs in the file system to:
</para>
- <screen>[client]# lfs find --obd <OST UUID> <mount_point> | lfs_migrate -y</screen>
+ <screen>client# lfs find --obd <replaceable>ost_name</replaceable> <replaceable>/mount/point</replaceable> | lfs_migrate -y</screen>
</listitem>
<listitem>
<para>If the OST is no longer available, delete the files on that OST and restore them from backup:
- <screen>[client]# lfs find --obd <OST UUID> -print0 <mount_point> | \
+ <screen>client# lfs find --obd <replaceable>ost_uuid</replaceable> -print0 <replaceable>/mount/point</replaceable> | \
tee /tmp/files_to_restore | xargs -0 -n 1 unlink</screen>
The list of files that need to be restored from backup is stored in <literal>/tmp/files_to_restore</literal>. Restoring these files is beyond the scope of this document.</para>
</listitem>
<para>Deactivate the OST.</para>
<orderedlist>
<listitem>
- <para>To temporarily disable the deactivated OST, enter: <screen>[client]# lctl set_param osc.<fsname>-<OST name>-*.active=0</screen>If there is expected to be a replacement OST in some short time (a few days), the OST can temporarily be deactivated on the clients: <note>
+ <para>To temporarily disable the deactivated OST, enter: <screen>[client]# lctl set_param osc.<replaceable>fsname</replaceable>-<replaceable>OSTnumber</replaceable>-*.active=0</screen>If there is expected to be a replacement OST in some short time (a few days), the OST can temporarily be deactivated on the clients: <note>
<para>This setting is only temporary and will be reset if the clients or MDS are rebooted. It needs to be run on all clients.</para>
</note></para>
</listitem>
</orderedlist>
- <para>If there is not expected to be a replacement for this OST in the near future, permanently deactivate the OST on all clients and the MDS: <screen>[mgs]# lctl conf_param {OST name}.osc.active=0</screen></para>
+ <para>If there is not expected to be a replacement for this OST in the near future, permanently deactivate the OST on all clients and the MDS: <screen>[mgs]# lctl conf_param <replaceable>ost_name</replaceable>.osc.active=0</screen></para>
<note>
<para>A removed OST still appears in the file system; do not create a new OST with the same name.</para>
</note>
<orderedlist>
<listitem>
<para>Mount the OST filesystem.
- <screen>[oss]# mkdir -p /mnt/ost
-[oss]# mount -t ldiskfs {ostdev} /mnt/ost</screen>
+ <screen>oss# mkdir -p /mnt/ost
+[oss]# mount -t ldiskfs <replaceable>/dev/ost_device</replaceable> /mnt/ost</screen>
</para>
</listitem>
<listitem>
<para>Back up the OST configuration files.
- <screen>[oss]# tar cvf {ostname}.tar -C /mnt/ost last_rcvd \
+ <screen>oss# tar cvf <replaceable>ost_name</replaceable>.tar -C /mnt/ost last_rcvd \
CONFIGS/ O/0/LAST_ID</screen>
</para>
</listitem>
<listitem>
<para>
Unmount the OST filesystem.
- <screen>[oss]# umount /mnt/ost</screen>
+ <screen>oss# umount /mnt/ost</screen>
</para>
</listitem>
</orderedlist>
<listitem>
<para>
Format the OST file system.
- <screen>[oss]# mkfs.lustre --ost --index={old OST index} {other options} \
- {newdev}</screen>
+ <screen>oss# mkfs.lustre --ost --index=<replaceable>old_ost_index</replaceable> <replaceable>other_options</replaceable> \
+ <replaceable>/dev/new_ost_dev</replaceable></screen>
</para>
</listitem>
<listitem>
<para>
Mount the OST filesystem.
- <screen>[oss]# mkdir /mnt/ost
-[oss]# mount -t ldiskfs {newdev} /mnt/ost</screen>
+ <screen>oss# mkdir /mnt/ost
+oss# mount -t ldiskfs <replaceable>/dev/new_ost_dev</replaceable> <replaceable>/mnt/ost</replaceable></screen>
</para>
</listitem>
<listitem>
<para>Restore the OST configuration files, if available.
- <screen>[oss]# tar xvf {ostname}.tar -C /mnt/ost</screen></para>
+ <screen>oss# tar xvf <replaceable>ost_name</replaceable>.tar -C /mnt/ost</screen></para>
</listitem>
<listitem>
<para>Recreate the OST configuration files, if unavailable.
The <literal>CONFIGS/mountdata</literal> file is created by <literal>mkfs.lustre</literal> at format time, but has flags set that request it to register itself with the MGS. It is possible to copy these flags from another working OST (which should be the same):
-<screen>[oss1]# debugfs -c -R "dump CONFIGS/mountdata /tmp/ldd" {other_osdev}
-[oss1]# scp /tmp/ldd oss:/tmp/ldd
-[oss0]# dd if=/tmp/ldd of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 skip=5</screen></para>
+<screen>oss1# debugfs -c -R "dump CONFIGS/mountdata /tmp/ldd" <replaceable>/dev/other_osdev</replaceable>
+oss1# scp /tmp/ldd oss0:/tmp/ldd
+oss0# dd if=/tmp/ldd of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 skip=5</screen></para>
</listitem>
<listitem>
<para>
Unmount the OST filesystem.
- <screen>[oss]# umount /mnt/ost</screen>
+ <screen>oss# umount /mnt/ost</screen>
</para>
</listitem>
</orderedlist>
<section>
<title><indexterm><primary>maintenance</primary><secondary>reintroducing an OSTs</secondary></indexterm>
Returning a Deactivated OST to Service</title>
- <para> If the OST was permanently deactivated, it needs to be reactivated in the MGS configuration. <screen>[mgs]# lctl conf_param {OST name}.osc.active=1</screen> If the OST was temporarily deactivated, it needs to be reactivated on the MDS and clients. <screen>[mds]# lctl --device <devno> activate
- [client]# lctl set_param osc.<fsname>-<OST name>-*.active=0</screen></para>
+ <para> If the OST was permanently deactivated, it needs to be reactivated in the MGS configuration. <screen>mgs# lctl conf_param <replaceable>ost_name</replaceable>.osc.active=1</screen> If the OST was temporarily deactivated, it needs to be reactivated on the MDS and clients. <screen>mds# lctl --device <replaceable>lustre_devno</replaceable> activate
+ client# lctl set_param osc.<replaceable>fsname</replaceable>-<replaceable>OSTnumber</replaceable>-*.active=0</screen></para>
</section>
</section>
<section xml:id="dbdoclet.50438199_77819">
<title><indexterm><primary>maintenance</primary><secondary>aborting recovery</secondary></indexterm>
<indexterm><primary>backup</primary><secondary>aborting recovery</secondary></indexterm>
Aborting Recovery</title>
- <para>You can abort recovery with either the <literal>lctl</literal> utility or by mounting the target with the <literal>abort_recov</literal> option (<literal>mount -o abort_recov</literal>). When starting a target, run: <screen>$ mount -t lustre -L <MDT name> -o abort_recov <mount_point></screen></para>
+ <para>You can abort recovery with either the <literal>lctl</literal> utility or by mounting the target with the <literal>abort_recov</literal> option (<literal>mount -o abort_recov</literal>). When starting a target, run: <screen>mds# mount -t lustre -L <replaceable>mdt_name</replaceable> -o abort_recov <replaceable>/mount_point</replaceable></screen></para>
<note>
<para>The recovery process is blocked until all OSTs are available. </para>
</note>
<section xml:id="dbdoclet.50438199_12607">
<title><indexterm><primary>maintenance</primary><secondary>identifying OST host</secondary></indexterm>
Determining Which Machine is Serving an OST </title>
- <para>In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the NID that is serving a specific OST, run one of the following commands on a client (you do not need to be a root user): <screen>client$ lctl get_param osc.${fsname}-${OSTname}*.ost_conn_uuid</screen>For example: <screen>client$ lctl get_param osc.*-OST0000*.ost_conn_uuid
+ <para>In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the NID that is serving a specific OST, run one of the following commands on a client (you do not need to be a root user): <screen>client$ lctl get_param osc.<replaceable>fsname</replaceable>-<replaceable>OSTnumber</replaceable>*.ost_conn_uuid</screen>For example: <screen>client$ lctl get_param osc.*-OST0000*.ost_conn_uuid
osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen>- OR - <screen>client$ lctl get_param osc.*.ost_conn_uuid
osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
osc.lustre-OST0001-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
<title><indexterm><primary>maintenance</primary><secondary>changing failover node address</secondary></indexterm>
Changing the Address of a Failover Node</title>
<para>To change the address of a failover node (e.g, to use node X instead of node Y), run this command on the OSS/OST partition:
- <screen>tunefs.lustre --erase-params --failnode=<NID> <device> </screen>
+ <screen>oss# tunefs.lustre --erase-params --failnode=<replaceable>NID</replaceable> <replaceable>/dev/ost_device</replaceable></screen>
or
- <screen>tunefs.lustre --erase-params --servicenode=<NID> <device> </screen>
+ <screen>oss# tunefs.lustre --erase-params --servicenode=<replaceable>NID</replaceable> <replaceable>/dev/ost_device</replaceable></screen>
</para>
</section>
<section xml:id="dbdoclet.50438199_62545">
<listitem>
<para>Stop the MDS.</para>
<para>Unmount the MDT</para>
- <screen>umount -f <device> </screen>
+ <screen>umount -f <replaceable>/dev/mdt_device</replaceable> </screen>
</listitem>
<listitem>
<para>Create the MGS.</para>
- <screen>mkfs.lustre --mgs --device-size=<size> <mgs-device> </screen>
+ <screen>mds# mkfs.lustre --mgs --device-size=<replaceable>size</replaceable> <replaceable>/dev/mgs_device</replaceable></screen>
</listitem>
<listitem>
<para>Copy the configuration data from MDT disk to the new MGS disk.</para>
- <screen>mount -t ldiskfs -o ro <mdt-device> <mdt-mount-point> </screen>
- <screen>mount -t ldiskfs -o rw <mgs-device> <mgs-mount-point> </screen>
- <screen>cp -r <mdt-mount-point>/CONFIGS/<filesystem-name>-* <mgs-mount-point>/CONFIGS/. </screen>
- <screen>umount <mgs-mount-point> </screen>
- <screen>umount <mdt-mount-point> </screen>
+ <screen>mds# mount -t ldiskfs -o ro <replaceable>/dev/mdt_device</replaceable> <replaceable>/mdt_mount_point</replaceable></screen>
+ <screen>mds# mount -t ldiskfs -o rw <replaceable>/dev/mgs_device</replaceable> <replaceable>/mgs_mount_point</replaceable> </screen>
+ <screen>mds# cp -r <replaceable>/mdt_mount_point</replaceable>/CONFIGS/<replaceable>filesystem_name</replaceable>-* <replaceable>/mgs_mount_point</replaceable>/CONFIGS/. </screen>
+ <screen>mds# umount <replaceable>/mgs_mount_point</replaceable></screen>
+ <screen>mds# umount <replaceable>/mdt_mount_point</replaceable></screen>
<para>See <xref linkend="dbdoclet.50438199_54623"/> for alternative method.</para>
</listitem>
<listitem>
<para>Start the MGS.</para>
- <screen>mount -t lustre <mgs-device> <mgs-mount-point></screen>
+ <screen>mgs# mount -t lustre <replaceable>/dev/mgs_device</replaceable> <replaceable>/mgs_mount_point</replaceable></screen>
<para>Check to make sure it knows about all your filesystem</para>
<screen>cat /proc/fs/lustre/mgs/MGS/filesystems</screen>
</listitem>
<listitem>
<para>Remove the MGS option from the MDT, and set the new MGS nid.</para>
- <screen>tunefs.lustre --nomgs --mgsnode=<new-MGS-nid> <mdt-device></screen>
+ <screen>mds# tunefs.lustre --nomgs --mgsnode=<replaceable>new_mgs_nid</replaceable> <replaceable>/dev/mdt-device</replaceable></screen>
</listitem>
<listitem>
<para>Start the MDT.</para>
- <screen>mount -t lustre <mdt-device> <mdt-mount-point></screen>
+ <screen>mds# mount -t lustre <replaceable>/dev/mdt_device /mdt_mount_point</replaceable></screen>
<para>Check to make sure the MGS configuration look right</para>
- <screen>cat /proc/fs/lustre/mgs/MGS/live/<filesystem-name></screen>
+ <screen>mds# cat /proc/fs/lustre/mgs/MGS/live/<replaceable>filesystem_name</replaceable></screen>
</listitem>
</orderedlist>
</section>
</title>
<para>Because changelog records take up space on the MDT, the system administration must register changelog users. The registrants specify which records they are "done with", and the system purges up to the greatest common record.</para>
<para>To register a new changelog user, run:</para>
- <screen>lctl --device <mdt_device> changelog_register
+ <screen>lctl --device /dev/<replaceable>mdt_device</replaceable> changelog_register
</screen>
<para>Changelog entries are not purged beyond a registered user's set point (see <literal>lfs changelog_clear</literal>).</para>
</section>
<literal>lfs changelog</literal>
</title>
<para>To display the metadata changes on an MDT (the changelog records), run:</para>
- <screen>lfs changelog <MDT name> [startrec [endrec]] </screen>
+ <screen>lfs changelog <replaceable>fsname</replaceable>-<replaceable>MDTnumber</replaceable> [startrec [endrec]] </screen>
<para>It is optional whether to specify the start and end records.</para>
<para>These are sample changelog records:</para>
<screen>2 02MKDIR 4298396676 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0]\
<literal>lfs changelog_clear</literal>
</title>
<para>To clear old changelog records for a specific user (records that the user no longer needs), run:</para>
- <screen>lfs changelog_clear <MDT name> <user ID> <endrec></screen>
- <para>The <literal>changelog_clear</literal> command indicates that changelog records previous to <endrec> are no longer of interest to a particular user <user ID>, potentially allowing the MDT to free up disk space. An <literal><endrec></literal> value of 0 indicates the current last record. To run <literal>changelog_clear</literal>, the changelog user must be registered on the MDT node using <literal>lctl</literal>.</para>
+ <screen>lfs changelog_clear <replaceable>mdt_name</replaceable> <replaceable>userid</replaceable> <replaceable>endrec</replaceable></screen>
+ <para>The <literal>changelog_clear</literal> command indicates that changelog records previous to <replaceable>endrec</replaceable> are no longer of interest to a particular user <replaceable>userid</replaceable>, potentially allowing the MDT to free up disk space. An <literal><replaceable>endrec</replaceable></literal> value of 0 indicates the current last record. To run <literal>changelog_clear</literal>, the changelog user must be registered on the MDT node using <literal>lctl</literal>.</para>
<para>When all changelog users are done with records < X, the records are deleted.</para>
</section>
<section remap="h5">
<literal>lctl changelog_deregister</literal>
</title>
<para>To deregister (unregister) a changelog user, run:</para>
- <screen>lctl --device <mdt_device> changelog_deregister <user ID> </screen>
+ <screen>lctl --device <replaceable>mdt_device</replaceable> changelog_deregister <replaceable>userid</replaceable> </screen>
<para> <literal>changelog_deregister cl1</literal> effectively does a <literal>changelog_clear cl1 0</literal> as it deregisters.</para>
</section>
</section>
<indexterm><primary>operations</primary><secondary>mounting by label</secondary></indexterm>
Mounting by Label</title>
<para>The file system name is limited to 8 characters. We have encoded the file system and target information in the disk label, so you can mount by label. This allows system administrators to move disks around without worrying about issues such as SCSI disk reordering or getting the <literal>/dev/device</literal> wrong for a shared target. Soon, file system naming will be made as fail-safe as possible. Currently, Linux disk labels are limited to 16 characters. To identify the target within the file system, 8 characters are reserved, leaving 8 characters for the file system name:</para>
- <screen><fsname>-MDT0000 or <fsname>-OST0a19</screen>
+ <screen><replaceable>fsname</replaceable>-MDT0000 or <replaceable>fsname</replaceable>-OST0a19</screen>
<para>To mount by label, use this command:</para>
- <screen>$ mount -t lustre -L <file system label> <mount point></screen>
+ <screen>mount -t lustre -L <replaceable>file_system_label</replaceable> <replaceable>/mount_point</replaceable></screen>
<para>This is an example of mount-by-label:</para>
- <screen>$ mount -t lustre -L testfs-MDT0000 /mnt/mdt</screen>
+ <screen>mds# mount -t lustre -L testfs-MDT0000 /mnt/mdt</screen>
<caution>
- <para>Mount-by-label should NOT be used in a multi-path environment.</para>
+ <para>Mount-by-label should NOT be used in a multi-path environment or when snapshots are being created of the device, since multiple block devices will have the same label.</para>
</caution>
<para>Although the file system name is internally limited to 8 characters, you can mount the clients at any mount point, so file system users are not subjected to short names. Here is an example:</para>
- <screen>mount -t lustre mds0@tcp0:/shortfs /mnt/<long-file_system-name></screen>
+ <screen>client# mount -t lustre mds0@tcp0:/short <replaceable>/dev/long_mountpoint_name</replaceable></screen>
</section>
<section xml:id="dbdoclet.50438194_24122">
<title><indexterm><primary>operations</primary><secondary>starting</secondary></indexterm>Starting Lustre</title>
</section>
<section xml:id="dbdoclet.50438194_69255">
<title><indexterm><primary>operations</primary><secondary>unmounting</secondary></indexterm>Unmounting a Server</title>
- <para>To stop a Lustre server, use the <literal>umount <mount point></literal> command.</para>
+ <para>To stop a Lustre server, use the <literal>umount <replaceable>/mount</replaceable> <replaceable>point</replaceable></literal> command.</para>
<para>For example, to stop <literal>ost0</literal> on mount point <literal>/mnt/test</literal>, run:</para>
<screen>$ umount /mnt/test</screen>
<para>Gracefully stopping a server with the <literal>umount</literal> command preserves the state of the connected clients. The next time the server is started, it waits for clients to reconnect, and then goes through the recovery procedure.</para>
</listitem>
</itemizedlist>
<para>By default, the Lustre file system uses failover mode for OSTs. To specify failout mode instead, use the <literal>--param="failover.mode=failout"</literal> option:</para>
- <screen>$ mkfs.lustre --fsname=<fsname> --mgsnode=<MGS node NID> --param="failover.mode=failout" --ost --index="OST index" <block device name></screen>
+ <screen>oss# mkfs.lustre --fsname=<replaceable>fsname</replaceable> --mgsnode=<replaceable>mgs_NID</replaceable> --param=failover.mode=failout --ost --index=<replaceable>ost_index</replaceable> <replaceable>/dev/ost_block_device</replaceable></screen>
<para>In this example, failout mode is specified for the OSTs on MGS <literal>mds0</literal>, file system <literal>testfs</literal>.</para>
- <screen>$ mkfs.lustre --fsname=testfs --mgsnode=mds0 --param="failover.mode=failout" --ost --index=3 /dev/sdb </screen>
+ <screen>oss# mkfs.lustre --fsname=testfs --mgsnode=mds0 --param=failover.mode=failout --ost --index=3 /dev/sdb </screen>
<caution>
<para>Before running this command, unmount all OSTs that will be affected by the change in the failover/failout mode.</para>
</caution>
<note>
<para>After initial file system configuration, use the tunefs.lustre utility to change the failover/failout mode. For example, to set the failout mode, run:</para>
- <para><screen>$ tunefs.lustre --param failover.mode=failout <OST partition></screen></para>
+ <para><screen>$ tunefs.lustre --param failover.mode=failout <replaceable>/dev/ost_device</replaceable></screen></para>
</note>
</section>
<section xml:id="dbdoclet.50438194_54138">
<para>Lustre supports multiple file systems provided the combination of <literal>NID:fsname</literal> is unique. Each file system must be allocated a unique name during creation with the <literal>--fsname</literal> parameter. Unique names for file systems are enforced if a single MGS is present. If multiple MGSs are present (for example if you have an MGS on every MDS) the administrator is responsible for ensuring file system names are unique. A single MGS and unique file system names provides a single point of administration and allows commands to be issued against the file system even if it is not mounted.</para>
<para>Lustre supports multiple file systems on a single MGS. With a single MGS fsnames are guaranteed to be unique. Lustre also allows multiple MGSs to co-exist. For example, multiple MGSs will be necessary if multiple file systems on different Lustre software versions are to be concurrently available. With multiple MGSs additional care must be taken to ensure file system names are unique. Each file system should have a unique fsname among all systems that may interoperate in the future.</para>
<para>By default, the <literal>mkfs.lustre</literal> command creates a file system named <literal>lustre</literal>. To specify a different file system name (limited to 8 characters) at format time, use the <literal>--fsname</literal> option:</para>
- <para><screen>mkfs.lustre --fsname=<file system name></screen></para>
+ <para><screen>mkfs.lustre --fsname=<replaceable>file_system_name</replaceable></screen></para>
<note>
<para>The MDT, OSTs and clients in the new file system must use the same filesystem name (prepended to the device name). For example, for a new file system named <literal>foo</literal>, the MDT and two OSTs would be named <literal>foo-MDT0000</literal>, <literal>foo-OST0000</literal>, and <literal>foo-OST0001</literal>.</para>
</note>
<para>To mount a client on the file system, run:</para>
- <screen>mount -t lustre mgsnode:/<new fsname> <mountpoint></screen>
+ <screen>client# mount -t lustre <replaceable>mgsnode</replaceable>:<replaceable>/new_fsname</replaceable> <replaceable>/mount_point</replaceable></screen>
<para>For example, to mount a client on file system foo at mount point /mnt/foo, run:</para>
- <screen>mount -t lustre mgsnode:/foo /mnt/foo</screen>
+ <screen>client# mount -t lustre mgsnode:/foo /mnt/foo</screen>
<note>
<para>If a client(s) will be mounted on several file systems, add the following line to <literal>/etc/xattr.conf</literal> file to avoid problems when files are moved between the file systems: <literal>lustre.* skip</literal></para>
</note>
<note>
- <para>To ensure that a new MDT is added to an existing MGS create the MDT by specifying: <literal>--mdt --mgsnode=<MGS node NID></literal>.</para>
+ <para>To ensure that a new MDT is added to an existing MGS create the MDT by specifying: <literal>--mdt --mgsnode=<replaceable>mgs_NID</replaceable></literal>.</para>
</note>
<para>A Lustre installation with two file systems (<literal>foo</literal> and <literal>bar</literal>) could look like this, where the MGS node is <literal>mgsnode@tcp0</literal> and the mount points are <literal>/mnt/foo</literal> and <literal>/mnt/bar</literal>.</para>
<screen>mgsnode# mkfs.lustre --mgs /dev/sda
ossbarnode# mkfs.lustre --fsname=bar --mgsnode=mgsnode@tcp0 --ost --index=0 /dev/sdc
ossbarnode# mkfs.lustre --fsname=bar --mgsnode=mgsnode@tcp0 --ost --index=1 /dev/sdd</screen>
<para>To mount a client on file system foo at mount point <literal>/mnt/foo</literal>, run:</para>
- <screen>mount -t lustre mgsnode@tcp0:/foo /mnt/foo</screen>
+ <screen>client# mount -t lustre mgsnode@tcp0:/foo /mnt/foo</screen>
<para>To mount a client on file system bar at mount point <literal>/mnt/bar</literal>, run:</para>
- <screen>mount -t lustre mgsnode@tcp0:/bar /mnt/bar</screen>
+ <screen>client# mount -t lustre mgsnode@tcp0:/bar /mnt/bar</screen>
</section>
<section xml:id="dbdoclet.lfsmkdir" condition='l24'>
<title><indexterm><primary>operations</primary><secondary>remote directory</secondary></indexterm>Creating a sub-directory on a given MDT</title>
<para>Lustre 2.4 enables individual sub-directories to be serviced by unique MDTs. An administrator can allocate a sub-directory to a given MDT using the command:</para>
- <screen>lfs mkdir –i <mdtindex> <remote_dir>
+ <screen>client# lfs mkdir –i <replaceable>mdt_index</replaceable> <replaceable>/mount_point/remote_dir</replaceable>
</screen>
<para>This command will allocate the sub-directory <literal>remote_dir</literal> onto the MDT of index <literal>mdtindex</literal>. For more information on adding additional MDTs and <literal>mdtindex</literal> see <xref linkend='dbdoclet.addmdtindex'/>.</para>
<warning><para>An administrator can allocate remote sub-directories to separate MDTs. Creating remote sub-directories in parent directories not hosted on MDT0 is not recommended. This is because the failure of the parent MDT will leave the namespace below it inaccessible. For this reason, by default it is only possible to create remote sub-directories off MDT0. To relax this restriction and enable remote sub-directories off any MDT, an administrator must issue the command <literal>lctl set_param mdd.*.enable_remote_dir=1</literal>.</para></warning>
<section xml:id="dbdoclet.50438194_17237">
<title>Setting Tunable Parameters with <literal>mkfs.lustre</literal></title>
<para>When the file system is first formatted, parameters can simply be added as a <literal>--param</literal> option to the <literal>mkfs.lustre</literal> command. For example:</para>
- <screen>$ mkfs.lustre --mdt --param="sys.timeout=50" /dev/sda</screen>
+ <screen>mds# mkfs.lustre --mdt --param="sys.timeout=50" /dev/sda</screen>
<para>For more details about creating a file system,see <xref linkend="configuringlustre"/>. For more details about <literal>mkfs.lustre</literal>, see <xref linkend="systemconfigurationutilities"/>.</para>
</section>
<section xml:id="dbdoclet.50438194_55253">
<title>Setting Parameters with <literal>tunefs.lustre</literal></title>
<para>If a server (OSS or MDS) is stopped, parameters can be added to an existing filesystem using the <literal>--param</literal> option to the <literal>tunefs.lustre</literal> command. For example:</para>
- <screen>$ tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda</screen>
- <para>With <literal>tunefs.lustre</literal>, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old <literal>tunefs.lustre</literal> parameters and just use newly-specified parameters, run:</para>
- <screen>$ tunefs.lustre --erase-params --param=<new parameters> </screen>
- <para>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <literal><obd|fsname>.<obdtype>.<proc_file_name>=<value></literal>. For example:</para>
- <screen>$ tunefs.lustre --param mdt.identity_upcall=NONE /dev/sda1</screen>
+ <screen>oss# tunefs.lustre --param=failover.node=192.168.0.13@tcp0 /dev/sda</screen>
+ <para>With <literal>tunefs.lustre</literal>, parameters are <emphasis>additive</emphasis> -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old <literal>tunefs.lustre</literal> parameters and just use newly-specified parameters, run:</para>
+ <screen>mds# tunefs.lustre --erase-params --param=<replaceable>new_parameters</replaceable> </screen>
+ <para>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <literal><replaceable>obdname|fsname</replaceable>.<replaceable>obdtype</replaceable>.<replaceable>proc_file_name</replaceable>=<replaceable>value</replaceable></literal>. For example:</para>
+ <screen>mds# tunefs.lustre --param mdt.identity_upcall=NONE /dev/sda1</screen>
<para>For more details about <literal>tunefs.lustre</literal>, see <xref linkend="systemconfigurationutilities"/>.</para>
</section>
<section xml:id="dbdoclet.50438194_51490">
<section remap="h4">
<title>Setting Temporary Parameters</title>
<para>Use <literal>lctl set_param</literal> to set temporary parameters on the node where it is run. These parameters map to items in <literal>/proc/{fs,sys}/{lnet,lustre}</literal>. The <literal>lctl set_param</literal> command uses this syntax:</para>
- <screen>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value></screen>
+ <screen>lctl set_param [-n] <replaceable>obdtype</replaceable>.<replaceable>obdname</replaceable>.<replaceable>proc_file_name</replaceable>=<replaceable>value</replaceable></screen>
<para>For example:</para>
<screen># lctl set_param osc.*.max_dirty_mb=1024
osc.myth-OST0000-osc.max_dirty_mb=32
<section xml:id="dbdoclet.50438194_64195">
<title>Setting Permanent Parameters</title>
<para>Use the <literal>lctl conf_param</literal> command to set permanent parameters. In general, the <literal>lctl conf_param</literal> command can be used to specify any parameter settable in a <literal>/proc/fs/lustre</literal> file, with its own OBD device. The <literal>lctl conf_param</literal> command uses this syntax (same as the <literal>mkfs.lustre</literal> and <literal>tunefs.lustre</literal> commands):</para>
- <screen><obd|fsname>.<obdtype>.<proc_file_name>=<value>) </screen>
+ <screen><replaceable>obdname|fsname</replaceable>.<replaceable>obdtype</replaceable>.<replaceable>proc_file_name</replaceable>=<replaceable>value</replaceable>) </screen>
<para>Here are a few examples of <literal>lctl conf_param</literal> commands:</para>
- <screen>$ mgs> lctl conf_param testfs-MDT0000.sys.timeout=40
+ <screen>mgs# lctl conf_param testfs-MDT0000.sys.timeout=40
$ lctl conf_param testfs-MDT0000.mdt.identity_upcall=NONE
$ lctl conf_param testfs.llite.max_read_ahead_mb=16
$ lctl conf_param testfs-MDT0000.lov.stripesize=2M
<section xml:id="dbdoclet.50438194_88217">
<title>Listing Parameters</title>
<para>To list Lustre or LNET parameters that are available to set, use the <literal>lctl list_param</literal> command. For example:</para>
- <screen>lctl list_param [-FR] <obdtype>.<obdname></screen>
+ <screen>lctl list_param [-FR] <replaceable>obdtype</replaceable>.<replaceable>obdname</replaceable></screen>
<para>The following arguments are available for the <literal>lctl list_param</literal> command.</para>
<para><literal>-F</literal> Add '<literal>/</literal>', '<literal>@</literal>' or '<literal>=</literal>' for directories, symlinks and writeable files, respectively</para>
<para><literal>-R</literal> Recursively lists all parameters under the specified path</para>
<para>For example:</para>
- <screen>$ lctl list_param obdfilter.lustre-OST0000 </screen>
+ <screen>oss# lctl list_param obdfilter.lustre-OST0000 </screen>
</section>
<section xml:id="dbdoclet.50438194_63247">
<title>Reporting Current Parameter Values</title>
<para>To report current Lustre parameter values, use the <literal>lctl get_param</literal> command with this syntax:</para>
- <screen>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name></screen>
+ <screen>lctl get_param [-n] <replaceable>obdtype</replaceable>.<replaceable>obdname</replaceable>.<replaceable>proc_file_name</replaceable></screen>
<para>This example reports data on RPC service times.</para>
- <screen>$ lctl get_param -n ost.*.ost_io.timeouts
+ <screen>oss# lctl get_param -n ost.*.ost_io.timeouts
service : cur 1 worst 30 (at 1257150393, 85d23h58m54s ago) 1 1 1 1 </screen>
<para>This example reports the amount of space this client has reserved for writeback cache with each OST:</para>
- <screen># lctl get_param osc.*.cur_grant_bytes
+ <screen>client# lctl get_param osc.*.cur_grant_bytes
osc.myth-OST0000-osc-ffff8800376bdc00.cur_grant_bytes=2097152
osc.myth-OST0001-osc-ffff8800376bdc00.cur_grant_bytes=33890304
osc.myth-OST0002-osc-ffff8800376bdc00.cur_grant_bytes=35418112
<screen>lctl list_nids</screen>
<para>This displays the server's NIDs (networks configured to work with Lustre).</para>
<para>This example has a combined MGS/MDT failover pair on mds0 and mds1, and a OST failover pair on oss0 and oss1. There are corresponding Elan addresses on mds0 and mds1.</para>
- <screen>mds0> mkfs.lustre --fsname=testfs --mdt --mgs --failnode=mds1,2@elan /dev/sda1
-mds0> mount -t lustre /dev/sda1 /mnt/test/mdt
-oss0> mkfs.lustre --fsname=testfs --failnode=oss1 --ost --index=0 \
+ <screen>mds0# mkfs.lustre --fsname=testfs --mdt --mgs --failnode=mds1,2@elan /dev/sda1
+mds0# mount -t lustre /dev/sda1 /mnt/test/mdt
+oss0# mkfs.lustre --fsname=testfs --failnode=oss1 --ost --index=0 \
--mgsnode=mds0,1@elan --mgsnode=mds1,2@elan /dev/sdb
-oss0> mount -t lustre /dev/sdb /mnt/test/ost0
-client> mount -t lustre mds0,1@elan:mds1,2@elan:/testfs /mnt/testfs
-mds0> umount /mnt/mdt
-mds1> mount -t lustre /dev/sda1 /mnt/test/mdt
-mds1> cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status</screen>
- <para>Where multiple NIDs are specified, comma-separation (for example, <literal>mds1,2@elan</literal>) means that the two NIDs refer to the same host, and that Lustre needs to choose the "best" one for communication. Colon-separation (for example, <literal>mds0:mds1</literal>) means that the two NIDs refer to two different hosts, and should be treated as failover locations (Lustre tries the first one, and if that fails, it tries the second one.)</para>
+oss0# mount -t lustre /dev/sdb /mnt/test/ost0
+client# mount -t lustre mds0,1@elan:mds1,2@elan:/testfs /mnt/testfs
+mds0# umount /mnt/mdt
+mds1# mount -t lustre /dev/sda1 /mnt/test/mdt
+mds1# cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status</screen>
+ <para>Where multiple NIDs are specified, comma-separation (for example, <literal>mds1,2@elan</literal>) means that the two NIDs refer to the same host, and that Lustre needs to choose the <emphasis>best</emphasis> one for communication. Colon-separation (for example, <literal>mds0:mds1</literal>) means that the two NIDs refer to two different hosts, and should be treated as failover locations (Lustre tries the first one, and if that fails, it tries the second one.)</para>
<para>Two options exist to specify failover nodes. <literal>--failnode</literal> and <literal>--servicenode</literal>. <literal>--failnode</literal> specifies the NIDs of failover nodes. <literal>--servicenode</literal> specifies all service NIDs, including those of the primary node and of failover nodes. Option <literal>--servicenode</literal> makes the MDT or OST treat all its service nodes equally. The first service node to load the target device becomes the primary service node. Other node NIDs will become failover locations for the target device.</para>
<note>
<para>If you have an MGS or MDT configured for failover, perform these steps:</para>
</listitem>
<listitem>
<para>If you have a separate MGS (that you do not want to reformat), then add the "--writeconf" flag to <literal>mkfs.lustre</literal> on the MDT, run:</para>
- <screen>$ mkfs.lustre --reformat --writeconf --fsname spfs --mgsnode=<MGS node NID> --mdt --index=0 /dev/<emphasis>{mdsdev}</emphasis></screen>
+ <screen>$ mkfs.lustre --reformat --writeconf --fsname spfs --mgsnode=<replaceable>mgs_nid</replaceable> --mdt --index=0 <replaceable>/dev/mds_device</replaceable></screen>
</listitem>
</orderedlist>
<note>
<itemizedlist>
<listitem>
<para> All known file systems</para>
- <screen># cat /proc/fs/lustre/mgs/MGS/filesystems
+ <screen>mgs# cat /proc/fs/lustre/mgs/MGS/filesystems
spfs
lustre</screen>
</listitem>
<itemizedlist>
<listitem>
<para> The server names participating in a file system (for each file system that has at least one server running)</para>
- <screen># cat /proc/fs/lustre/mgs/MGS/live/spfs
+ <screen>mgs# cat /proc/fs/lustre/mgs/MGS/live/spfs
fsname: spfs
flags: 0x0 gen: 7
spfs-MDT0000
spfs-OST0000</screen>
</listitem>
</itemizedlist>
- <para>All servers are named according to this convention: <literal><fsname>-<MDT|OST><XXXX></literal>. This can be shown for live servers under <literal>/proc/fs/lustre/devices</literal>:</para>
- <screen># cat /proc/fs/lustre/devices
+ <para>All servers are named according to this convention: <literal><replaceable>fsname</replaceable>-<replaceable>MDT|OSTnumber</replaceable></literal>. This can be shown for live servers under <literal>/proc/fs/lustre/devices</literal>:</para>
+ <screen>mds# cat /proc/fs/lustre/devices
0 UP mgs MGS MGS 11
1 UP mgc MGC192.168.10.34@tcp 1f45bb57-d9be-2ddb-c0b0-5431a49226705
2 UP mdt MDS MDS_uuid 3
9 UP osc lustre-OST0000-osc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05
10 UP osc lustre-OST0001-osc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05</screen>
<para>Or from the device label at any time:</para>
- <screen># e2label /dev/sda
+ <screen>mds# e2label /dev/sda
lustre-MDT0000</screen>
</section>
<section remap="h3">
<para>Lustre uses two types of timeouts.</para>
<itemizedlist>
<listitem>
- <para>LND timeouts that ensure point-to-point communications complete in finite time in the presence of failures. These timeouts are logged with the <literal>S_LND</literal> flag set. They may <emphasis>not</emphasis> be printed as console messages, so you should check the Lustre log for <literal>D_NETERROR</literal> messages, or enable printing of <literal>D_NETERROR</literal> messages to the console (<literal>echo + neterror > /proc/sys/lnet/printk</literal>).</para>
+ <para>LND timeouts that ensure point-to-point communications complete in finite time in the presence of failures. These timeouts are logged with the <literal>S_LND</literal> flag set. They may <emphasis>not</emphasis> be printed as console messages, so you should check the Lustre log for <literal>D_NETERROR</literal> messages, or enable printing of <literal>D_NETERROR</literal> messages to the console (<literal>lctl set_param printk=+neterror</literal>).</para>
</listitem>
</itemizedlist>
<para>Congested routers can be a source of spurious LND timeouts. To avoid this, increase the number of LNET router buffers to reduce back-pressure and/or increase LND timeouts on all nodes on all connected networks. You should also consider increasing the total number of LNET router nodes in the system so that the aggregate router bandwidth matches the aggregate server bandwidth.</para>
</tgroup>
</informaltable>
<para>Adaptive timeouts are enabled, by default. To disable adaptive timeouts, at run time, set <literal>at_max</literal> to 0. On the MGS, run:</para>
- <screen>$ lctl conf_param <fsname>.sys.at_max=0</screen>
+ <screen>$ lctl conf_param <replaceable>fsname</replaceable>.sys.at_max=0</screen>
<note>
<para>Changing adaptive timeouts status at runtime may cause transient timeout, reconnect, recovery, etc.</para>
</note>
<section remap="h3">
<title><indexterm><primary>proc</primary><secondary>free space</secondary></indexterm>Free Space Distribution</title>
<para>Free-space stripe weighting, as set, gives a priority of "0" to free space (versus trying to place the stripes "widely" -- nicely distributed across OSSs and OSTs to maximize network balancing). To adjust this priority (as a percentage), use the <literal>qos_prio_free</literal> proc tunable:</para>
- <screen>$ cat /proc/fs/lustre/lov/<fsname>-mdtlov/qos_prio_free</screen>
+ <screen>$ cat /proc/fs/lustre/lov/<replaceable>fsname</replaceable>-mdtlov/qos_prio_free</screen>
<para>Currently, the default is 90%. You can permanently set this value by running this command on the MGS:</para>
- <screen>$ lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90</screen>
+ <screen>$ lctl conf_param <replaceable>fsname</replaceable>-MDT0000.lov.qos_prio_free=90</screen>
<para>Setting the priority to 100% means that OSS distribution does not count in the weighting, but the stripe assignment is still done via weighting. If OST 2 has twice as much free space as OST 1, it is twice as likely to be used, but it is NOT guaranteed to be used.</para>
<para>Also note that free-space stripe weighting does not activate until two OSTs are imbalanced by more than 20%. Until then, a faster round-robin stripe allocator is used. (The new round-robin order also maximizes network balancing.)</para>
<section remap="h4">
<section xml:id="dbdoclet.50438271_78950">
<title><indexterm><primary>proc</primary><secondary>I/O tunables</secondary></indexterm>Lustre I/O Tunables</title>
<para>The section describes I/O tunables.</para>
- <para><literal> /proc/fs/lustre/llite/<fsname>-<uid>/max_cache_mb </literal></para>
- <screen># cat /proc/fs/lustre/llite/lustre-ce63ca00/max_cached_mb 128</screen>
+ <para><literal> llite.<replaceable>fsname-instance</replaceable>/max_cache_mb</literal></para>
+ <screen>client# lctl get_param llite.lustre-ce63ca00.max_cached_mb
+128</screen>
<para>This tunable is the maximum amount of inactive data cached by the client (default is 3/4 of RAM).</para>
<section remap="h3">
<title><indexterm><primary>proc</primary><secondary>RPC tunables</secondary></indexterm>Client I/O RPC Stream Tunables</title>
blocksizefilesfree max_dirty_mb ost_server_uuid stats</screen>
<para>... and so on.</para>
<para>RPC stream tunables are described below.</para>
- <para><literal> /proc/fs/lustre/osc/<object name>/max_dirty_mb </literal></para>
+ <para><literal> osc.<replaceable>osc_instance</replaceable>.max_dirty_mb </literal></para>
<para xml:id='lustreproc.maxdirtymb'>This tunable controls how many MBs of dirty data can be written and queued up in the OSC. POSIX file writes that are cached contribute to this count. When the limit is reached, additional writes stall until previously-cached writes are written to the server. This may be changed by writing a single ASCII integer to the file. Only values between 0 and 512 are allowable. If 0 is given, no writes are cached. Performance suffers noticeably unless you use large writes (1 MB or more).</para>
- <para><literal> /proc/fs/lustre/osc/<object name>/cur_dirty_bytes </literal></para>
+ <para><literal> osc.<replaceable>osc_instance</replaceable>.cur_dirty_bytes </literal></para>
<para>This tunable is a read-only value that returns the current amount of bytes written and cached on this OSC.</para>
- <para><literal> /proc/fs/lustre/osc/<object name>/max_pages_per_rpc </literal></para>
+ <para><literal> osc.<replaceable>osc_instance</replaceable>.max_pages_per_rpc </literal></para>
<para>This tunable is the maximum number of pages that will undergo I/O in a single RPC to the OST. The minimum is a single page and the maximum for this setting is platform dependent (256 for i386/x86_64, possibly less for ia64/PPC with larger <literal>PAGE_SIZE</literal>), though generally amounts to a total of 1 MB in the RPC.</para>
- <para><literal> /proc/fs/lustre/osc/<object name>/max_rpcs_in_flight </literal></para>
+ <para><literal> osc.<replaceable>osc_instance</replaceable>.max_rpcs_in_flight </literal></para>
<para>This tunable is the maximum number of concurrent RPCs in flight from an OSC to its OST. If the OSC tries to initiate an RPC but finds that it already has the same number of RPCs outstanding, it will wait to issue further RPCs until some complete. The minimum setting is 1 and maximum setting is 256. If you are looking to improve small file I/O performance, increase the <literal>max_rpcs_in_flight</literal> value.</para>
<para>To maximize performance, the value for <literal>max_dirty_mb</literal> is recommended to be 4 * <literal>max_pages_per_rpc</literal> * <literal>max_rpcs_in_flight</literal>.</para>
<note>
<para>The
<literal>
- <replaceable><object name></replaceable>
+ <replaceable>osc_instance</replaceable>
</literal>
- varies depending on the specific Lustre configuration. For <literal><object name></literal> examples, refer to the sample command output.</para>
+ is typically <literal><replaceable>fsname</replaceable>-OST<replaceable>ost_index</replaceable>-osc-<replaceable>mountpoint_instance</replaceable></literal>. The <literal><replaceable>mountpoint_instance</replaceable></literal> is a unique value per mountpoint to allow associating osc, mdc, lov, lmv, and llite parameters for the same mountpoint. For <literal><replaceable>osc_instance</replaceable></literal> examples, refer to the sample command output.</para>
</note>
</section>
<section remap="h3">
<para>Difference from the previous range end to the current range start.</para>
<para>For example, Smallest-Extent indicates that the writes in the range 100 to 1110 were sequential, with a minimum write of 10 and a maximum write of 500. This range was started with an offset of -150. That means this is the difference between the last entry's range-end and this entry's range-start for the same file.</para>
<para>The <literal>rw_offset_stats</literal> file can be cleared by writing to it:</para>
- <screen>echo > /proc/fs/lustre/llite/lustre-f57dee00/rw_offset_stats</screen>
+ <screen>lctl set_param llite.*.rw_offset_stats=0</screen>
</entry>
</row>
</tbody>
<title><indexterm><primary>proc</primary><secondary>client stats</secondary></indexterm>Client stats</title>
<para>The stats parameter maintains statistics of activity across the VFS interface of the Lustre file system. Only non-zero parameters are displayed in the file. This section of the manual covers the statistics that will accumulate during typical operation of a client.</para>
<para>Client statistics are enabled by default. The statistics can be cleared by echoing an empty string into the <literal>stats</literal> file or with the command: <literal>lctl set_param llite.*.stats=0</literal>. Statistics for an individual file system can be displayed, for example:</para>
- <screen># cat /proc/fs/lustre/llite/lustre-ffff88000449a800/stats
+ <screen>client# lctl get_param llite.*.stats
snapshot_time 1308343279.169704 secs.usecs
dirty_pages_hits 14819716 samples [regs]
dirty_pages_misses 81473472 samples [regs]
<para><emphasis role="bold">Client-Based I/O Extent Size Survey</emphasis></para>
<para>The <literal>rw_extent_stats</literal> histogram in the <literal>llite</literal> directory shows you the statistics for the sizes of the read-write I/O extents. This file does not maintain the per-process statistics.</para>
<para>Example:</para>
- <screen>$ cat /proc/fs/lustre/llite/lustre-ee5af200/extents_stats
+ <screen>client# lctl get_param llite.testfs-*.extents_stats
snapshot_time: 1213828728.348516 (secs.usecs)
read | write
extents calls % cum% | calls % cum%
512K - 1024K : 0 0 0 | 0 0 86
1M - 2M : 0 0 0 | 11 13 100</screen>
<para>The file can be cleared by issuing the following command:</para>
- <screen>$ echo > cat /proc/fs/lustre/llite/lustre-ee5af200/extents_stats</screen>
+ <screen>client# lctl set_param llite.testfs-*.extents_stats=0</screen>
<para><emphasis role="bold">Per-Process Client I/O Statistics</emphasis></para>
<para>The <literal>extents_stats_per_process</literal> file maintains the I/O extent size statistics on a per-process basis. So you can track the per-process statistics for the last <literal>MAX_PER_PROCESS_HIST</literal> processes.</para>
<para>Example:</para>
- <screen>$ cat /proc/fs/lustre/llite/lustre-ee5af200/extents_stats_per_process
+ <screen>lctl get_param llite.testfs-*.extents_stats_per_process
snapshot_time: 1213828762.204440 (secs.usecs)
read | write
extents calls % cum% | calls % cum%
<section xml:id="dbdoclet.50438271_55057">
<title><indexterm><primary>proc</primary><secondary>block I/O</secondary></indexterm>Watching the OST Block I/O Stream</title>
<para>Similarly, there is a <literal>brw_stats</literal> histogram in the obdfilter directory which shows you the statistics for number of I/O requests sent to the disk, their size and whether they are contiguous on the disk or not.</para>
- <screen>cat /proc/fs/lustre/obdfilter/lustre-OST0000/brw_stats
+ <screen>oss# lctl get_param obdfilter.testfs-OST0000.brw_stats
snapshot_time: 1174875636.764630 (secs:usecs)
read write
pages per brw brws % cum % | rpcs % cum %
<section remap="h4">
<title>Tuning File Readahead</title>
<para>File readahead is triggered when two or more sequential reads by an application fail to be satisfied by the Linux buffer cache. The size of the initial readahead is 1 MB. Additional readaheads grow linearly, and increment until the readahead cache on the client is full at 40 MB.</para>
- <para><literal> /proc/fs/lustre/llite/<fsname>-<uid>/max_read_ahead_mb </literal></para>
+ <para><literal> llite.<replaceable>fsname-instance</replaceable>.max_read_ahead_mb </literal></para>
<para>This tunable controls the maximum amount of data readahead on a file. Files are read ahead in RPC-sized chunks (1 MB or the size of read() call, if larger) after the second sequential read on a file descriptor. Random reads are done at the size of the read() call only (no readahead). Reads to non-contiguous regions of the file reset the readahead algorithm, and readahead is not triggered again until there are sequential reads again. To disable readahead, set this tunable to 0. The default value is 40 MB.</para>
- <para><literal> /proc/fs/lustre/llite/<fsname>-<uid>/max_read_ahead_whole_mb </literal></para>
+ <para><literal> llite.<replaceable>fsname-instance</replaceable>.max_read_ahead_whole_mb </literal></para>
<para>This tunable controls the maximum size of a file that is read in its entirety, regardless of the size of the <literal>read()</literal>.</para>
</section>
<section remap="h4">
<para><literal> /proc/fs/lustre/llite/*/statahead_max </literal></para>
<para>This proc interface controls whether directory statahead is enabled and the maximum statahead windows size (which means how many files can be pre-fetched by the statahead thread). By default, statahead is enabled and the value of <literal>statahead_max</literal> is 32.</para>
<para>To disable statahead, run:</para>
- <screen>echo 0 > /proc/fs/lustre/llite/*/statahead_max</screen>
- <para>or</para>
<screen>lctl set_param llite.*.statahead_max=0</screen>
<para>To set the maximum statahead windows size (n), run:</para>
- <screen>echo n > /proc/fs/lustre/llite/*/statahead_max</screen>
- <para>or</para>
<screen>lctl set_param llite.*.statahead_max=n</screen>
<para>The maximum value of n is 8192.</para>
<para>The AGL can be controlled as follows:</para>
- <screen>echo n > /proc/fs/lustre/llite/*/statahead_agl</screen>
- <para>or</para>
<screen>lctl set_param llite.*.statahead_agl=n</screen>
<para>If "n" is 0, then the AGL is disabled, else the AGL is enabled.</para>
<para><literal> /proc/fs/lustre/llite/*/statahead_stats </literal></para>
<note>
<para>Asynchronous journal commit cannot work with O_DIRECT writes, a journal flush is still forced.</para>
</note>
- <para>When asynchronous journal commit is enabled, client nodes keep data in the page cache (a page reference). Lustre clients monitor the last committed transaction number (transno) in messages sent from the OSS to the clients. When a client sees that the last committed transno reported by the OSS is >=bulk write transno, it releases the reference on the corresponding pages. To avoid page references being held for too long on clients after a bulk write, a 7 second ping request is scheduled (jbd commit time is 5 seconds) after the bulk write reply is received, so the OSS has an opportunity to report the last committed transno.</para>
+ <para>When asynchronous journal commit is enabled, client nodes keep data in the page cache (a page reference). Lustre clients monitor the last committed transaction number (transno) in messages sent from the OSS to the clients. When a client sees that the last committed transno reported by the OSS is at least the bulk write transno, it releases the reference on the corresponding pages. To avoid page references being held for too long on clients after a bulk write, a 7 second ping request is scheduled (jbd commit time is 5 seconds) after the bulk write reply is received, so the OSS has an opportunity to report the last committed transno.</para>
<para>If the OSS crashes before the journal commit occurs, then the intermediate data is lost. However, new OSS recovery functionality (introduced in the asynchronous journal commit feature), causes clients to replay their write requests and compensate for the missing disk updates by restoring the state of the file system.</para>
<para>To enable asynchronous journal commit, set the <literal>sync_journal parameter</literal> to zero (<literal>sync_journal=0</literal>):</para>
<screen>$ lctl set_param obdfilter.*.sync_journal=0
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">
- <literal>stats</literal>
- </emphasis></para>
- </entry>
- <entry>
- <para>Enables/disables the collection of statistics. Collected statistics can be found</para>
- <para>in <literal>/proc/fs/ldiskfs2/<dev>/mb_history</literal>.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal> max_to_scan </literal></para>
- </entry>
- <entry>
- <para>Maximum number of free chunks that <literal>mballoc</literal> finds before a final decision to avoid livelock.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal> min_to_scan </literal></para>
- </entry>
- <entry>
- <para>Minimum number of free chunks that <literal>mballoc</literal> finds before a final decision. This is useful for a very small request, to resist fragmentation of big free chunks.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal> order2_req </literal></para>
- </entry>
- <entry>
- <para>For requests equal to 2^N (where N >= <literal>order2_req</literal>), a very fast search via buddy structures is used.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal> stream_req </literal></para>
- </entry>
- <entry>
- <para>Requests smaller or equal to this value are packed together to form large write I/Os.</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para>The following tunables, providing more control over allocation policy, will be available in the next version:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry>
- <para><emphasis role="bold">Field</emphasis></para>
- </entry>
- <entry>
- <para><emphasis role="bold">Description</emphasis></para>
- </entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry>
- <para> <literal> stats </literal></para>
- </entry>
- <entry>
- <para>Enables/disables the collection of statistics. Collected statistics can be found in <literal>/proc/fs/ldiskfs2/<dev>/mb_history</literal>.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal> max_to_scan </literal></para>
+ <para> <literal>mb_max_to_scan</literal></para>
</entry>
<entry>
<para>Maximum number of free chunks that <literal>mballoc</literal> finds before a final decision to avoid livelock.</para>
</row>
<row>
<entry>
- <para> <literal> min_to_scan </literal></para>
+ <para> <literal>mb_min_to_scan</literal></para>
</entry>
<entry>
- <para>Minimum number of free chunks that <literal>mballoc</literal> finds before a final decision. This is useful for a very small request, to resist fragmentation of big free chunks.</para>
+ <para>Minimum number of free chunks that <literal>mballoc</literal> searches before picking the best chunk for allocation. This is useful for a very small request, to resist fragmentation of big free chunks.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal> order2_req </literal></para>
+ <para> <literal>mb_order2_req</literal></para>
</entry>
<entry>
<para>For requests equal to 2^N (where N >= <literal>order2_req</literal>), a very fast search via buddy structures is used.</para>
</row>
<row>
<entry>
- <para> <literal> small_req </literal></para>
+ <para> <literal>mb_small_req</literal></para>
</entry>
<entry morerows="1">
<para>All requests are divided into 3 categories:</para>
</row>
<row>
<entry>
- <para> <literal> large_req </literal></para>
+ <para> <literal>mb_large_req</literal></para>
</entry>
</row>
<row>
<entry>
- <para> <literal> prealloc_table </literal></para>
+ <para> <literal>mb_prealloc_table</literal></para>
</entry>
<entry>
<para>The amount of space to preallocate depends on the current file size. The idea is that for small files we do not need 1 MB preallocations and for large files, 1 MB preallocations are not large enough; it is better to preallocate 4 MB.</para>
</row>
<row>
<entry>
- <para> <literal> group_prealloc </literal></para>
+ <para> <literal>mb_group_prealloc</literal></para>
</entry>
<entry>
- <para>The amount of space preallocated for small requests to be grouped.</para>
+ <para>The amount of space (in kilobytes) preallocated for groups of small requests.</para>
</entry>
</row>
</tbody>
</section>
<section remap="h3">
<title><indexterm><primary>proc</primary><secondary>locking</secondary></indexterm>Locking</title>
- <para><literal> /proc/fs/lustre/ldlm/ldlm/namespaces/<OSC name|MDC name>/lru_size </literal></para>
+ <para><literal> ldlm.namespaces.<replaceable>osc_name|mdc_name</replaceable>.lru_size </literal></para>
<para>The <literal>lru_size</literal> parameter is used to control the number of client-side locks in an LRU queue. LRU size is dynamic, based on load. This optimizes the number of locks available to nodes that have different workloads (e.g., login/build nodes vs. compute nodes vs. backup nodes).</para>
- <para>The total number of locks available is a function of the server's RAM. The default limit is 50 locks/1 MB of RAM. If there is too much memory pressure, then the LRU size is shrunk. The number of locks on the server is limited to {number of OST/MDT on node} * {number of clients} * {client lru_size}.</para>
+ <para>The total number of locks available is a function of the server's RAM. The default limit is 50 locks/1 MB of RAM. If there is too much memory pressure, then the LRU size is shrunk. The number of locks on the server is limited to <replaceable>targets_on_server</replaceable> * <replaceable>client_count</replaceable> * <replaceable>client_lru_size</replaceable>.</para>
<itemizedlist>
<listitem>
<para>To enable automatic LRU sizing, set the <literal>lru_size</literal> parameter to 0. In this case, the <literal>lru_size</literal> parameter shows the current number of locks being used on the export. LRU sizing is enabled by default starting with Lustre 1.6.5.1.</para>
</listitem>
<listitem>
- <para>To specify a maximum number of locks, set the lru_size parameter to a value > 0 (former numbers are okay, 100 * CPU_NR). We recommend that you only increase the LRU size on a few login nodes where users access the file system interactively.</para>
+ <para>To specify a maximum number of locks, set the lru_size parameter to a value other than 0 (former numbers are okay, 100 * <replaceable>core_count</replaceable>). We recommend that you only increase the LRU size on a few login nodes where users access the file system interactively.</para>
</listitem>
</itemizedlist>
<para>To clear the LRU on a single client, and as a result flush client cache, without changing the <literal>lru_size</literal> value:</para>
- <screen>$ lctl set_param ldlm.namespaces.<osc_name|mdc_name>.lru_size=clear</screen>
+ <screen>$ lctl set_param ldlm.namespaces.<replaceable>osc_name|mdc_name</replaceable>.lru_size=clear</screen>
<para>If you shrink the LRU size below the number of existing unused locks, then the unused locks are canceled immediately. Use echo clear to cancel all locks without changing the value.</para>
<note>
<para>Currently, the lru_size parameter can only be set temporarily with <literal>lctl set_param</literal>; it cannot be set permanently.</para>
# sysctl lnet.debug
lnet.debug = neterror warning</screen>
<para>You can verify and change the debug level using the <literal>/proc</literal> interface in Lustre. To use the flags with <literal>/proc</literal>, run:</para>
- <screen># cat /proc/sys/lnet/debug
+ <screen># lctl get_param debug
+debug=
neterror warning
-# echo "+ha" > /proc/sys/lnet/debug
-# cat /proc/sys/lnet/debug
+# lctl set_param debug=+ha
+# lctl get_param debug
+debug=
neterror warning ha
-# echo "-warning" > /proc/sys/lnet/debug
-# cat /proc/sys/lnet/debug
+# lctl set_param debug=-warning
+# lctl get_param debug
+debug=
neterror ha</screen>
<para><literal> /proc/sys/lnet/subsystem_debug </literal></para>
<para>This controls the debug logs for subsystems (see <literal>S_*</literal> definitions).</para>
<para>To force an OST recovery, unmount the OST and then mount it again. If the OST was connected to clients before it failed, then a recovery process starts after the remount, enabling clients to reconnect to the OST and replay transactions in their queue. When the OST is in recovery mode, all new client connections are refused until the recovery finishes. The recovery is complete when either all previously-connected clients reconnect and their transactions are replayed or a client connection attempt times out. If a connection attempt times out, then all clients waiting to reconnect (and their transactions) are lost.</para>
<note>
<para>If you know an OST will not recover a previously-connected client (if, for example, the client has crashed), you can manually abort the recovery using this command:</para>
- <para><screen>lctl --device <OST device number> abort_recovery</screen></para>
+ <para><screen>oss# lctl --device <replaceable>lustre_device_number</replaceable> abort_recovery</screen></para>
<para>To determine an OST's device number and device name, run the <literal>lctl dl</literal> command. Sample <literal>lctl dl</literal> command output is shown below:</para>
<screen>7 UP obdfilter ddn_data-OST0009 ddn_data-OST0009_UUID 1159 </screen>
<para>In this example, 7 is the OST device number. The device name is <literal>ddn_data-OST0009</literal>. In most instances, the device name can be used in place of the device number.</para>
<para>Lustre logs are dumped to <literal>/proc/sys/lnet/debug_pat</literal>h.</para>
<para>Collect the first group of messages related to a problem, and any messages that precede "LBUG" or "assertion failure" errors. Messages that mention server nodes (OST or MDS) are specific to that server; you must collect similar messages from the relevant server console logs.</para>
<para>Another Lustre debug log holds information for Lustre action for a short period of time which, in turn, depends on the processes on the node to use Lustre. Use the following command to extract debug logs on each of the nodes, run</para>
- <screen>$ lctl dk <filename></screen>
+ <screen>$ lctl dk <replaceable>filename</replaceable></screen>
<note>
<para>LBUG freezes the thread to allow capture of the panic stack. A system reboot is needed to clear the thread.</para>
</note>
<para>You can also post a question to the <link xl:href="http://wiki.whamcloud.com/display/PUB/Community+Resources">lustre-discuss mailing list</link> or search the <link xl:href="http://groups.google.com/group/lustre-discuss-list">lustre-discuss Archives</link> for information about your issue.</para>
<para>A Lustre diagnostics tool is available for downloading at: <link xl:href="http://downloads.whamcloud.com/public/tools/">http://downloads.whamcloud.com/public/tools/</link></para>
<para>You can run this tool to capture diagnostics output to include in the reported bug. To run this tool, enter one of these commands:</para>
- <screen># lustre-diagnostics -t <bugzilla bug #>
+ <screen># lustre-diagnostics -t <replaceable>bug_number</replaceable>
# lustre-diagnostics.
</screen>
<para>Output is sent directly to the terminal. Use normal file redirection to send the output to a file, and then manually attach the file to the bug you are submitting.</para>
</listitem>
<listitem>
<para>Deactivate the OST (on the OSS at the MDS). Run:</para>
- <screen>$ lctl --device <OST device name or number> deactivate</screen>
+ <screen>$ lctl --device <replaceable>lustre_device_number</replaceable> deactivate</screen>
<para>The OST device number or device name is generated by the lctl dl command.</para>
<para>The <literal>deactivate</literal> command prevents clients from creating new objects on the specified OST, although you can still access the OST for reading.</para>
<note>
<para>If the OST later becomes available it needs to be reactivated, run:</para>
- <screen># lctl --device <OST device name or number> activate</screen>
+ <screen># lctl --device <replaceable>lustre_device_number</replaceable> activate</screen>
</note>
</listitem>
<listitem>
<note>
<para>The file system must be stopped on all servers before performing this procedure.</para>
</note>
- <para>For hex < -> decimal translations:</para>
+ <para>For hex-to-decimal translations:</para>
<para>Use GDB:</para>
<screen>(gdb) p /x 15028
$2 = 0x3ab4</screen>
<orderedlist>
<listitem>
<para>Determine a reasonable value for the LAST_ID file. Check on the MDS:</para>
- <screen># mount -t ldiskfs /dev/<mdsdev> /mnt/mds
+ <screen># mount -t ldiskfs <replaceable>/dev/mdt_device</replaceable> /mnt/mds
# od -Ax -td8 /mnt/mds/lov_objid
</screen>
<para>There is one entry for each OST, in OST index order. This is what the MDS thinks is the last in-use object.</para>
<section remap="h3">
<title>Setting SCSI I/O Sizes</title>
<para>Some SCSI drivers default to a maximum I/O size that is too small for good Lustre performance. we have fixed quite a few drivers, but you may still find that some drivers give unsatisfactory performance with Lustre. As the default value is hard-coded, you need to recompile the drivers to change their default. On the other hand, some drivers may have a wrong default set.</para>
- <para>If you suspect bad I/O performance and an analysis of Lustre statistics indicates that I/O is not 1 MB, check <literal>/sys/block/<device>/queue/max_sectors_kb</literal>. If the <literal>max_sectors_kb</literal> value is less than 1024, set it to at least 1024 to improve performance. If changing <literal>max_sectors_kb</literal> does not change the I/O size as reported by Lustre, you may want to examine the SCSI driver code.</para>
+ <para>If you suspect bad I/O performance and an analysis of Lustre statistics indicates that I/O is not 1 MB, check <literal>/sys/block/<replaceable>device</replaceable>/queue/max_sectors_kb</literal>. If the <literal>max_sectors_kb</literal> value is less than 1024, set it to at least 1024 to improve performance. If changing <literal>max_sectors_kb</literal> does not change the I/O size as reported by Lustre, you may want to examine the SCSI driver code.</para>
</section>
</section>
</chapter>
<para>Portal round-robin defines the policy LNet applies to deliver events and messages to the upper layers. The upper layers are ptlrpc service or LNet selftest.</para>
<para>If portal round-robin is disabled, LNet will deliver messages to CPTs based on a hash of the source NID. Hence, all messages from a specific peer will be handled by the same CPT. This can reduce data traffic between CPUs. However, for some workloads, this behavior may result in poorly balancing loads across the CPU.</para>
<para>If portal round-robin is enabled, LNet will round-robin incoming events across all CPTs. This may balance load better across the CPU but can incur a cross CPU overhead.</para>
- <para>The current policy can be changed by an administrator with <literal>echo <VALUE> > /proc/sys/lnet/portal_rotor</literal>. There are four options for <literal><VALUE></literal>:</para>
+ <para>The current policy can be changed by an administrator with <literal>echo <replaceable>value</replaceable> > /proc/sys/lnet/portal_rotor</literal>. There are four options for <literal><replaceable>value</replaceable></literal>:</para>
<itemizedlist>
<listitem>
<para><literal>OFF</literal></para>
</itemizedlist>
<para>With MMP enabled, mounting a clean file system takes at least 10 seconds. If the file system was not cleanly unmounted, then the file system mount may require additional time.</para>
<note>
- <para>The MMP feature is only supported on Linux kernel versions >= 2.6.9.</para>
+ <para>The MMP feature is only supported on Linux kernel versions newer than 2.6.9.</para>
</note>
<section remap="h3">
<title><indexterm><primary>failover</primary><secondary>multiple-mount protection</secondary></indexterm>Working with Multiple-Mount Protection</title>
<para>On a new Lustre file system, MMP is automatically enabled by mkfs.lustre at format time if failover is being used and the kernel and <literal>e2fsprogs</literal> version support it. On an existing file system, a Lustre administrator can manually enable MMP when the file system is unmounted.</para>
<para>Use the following commands to determine whether MMP is running in Lustre and to enable or disable the MMP feature.</para>
<para>To determine if MMP is enabled, run:</para>
- <screen>dumpe2fs -h <device>|grep mmp</screen>
+ <screen>dumpe2fs -h <replaceable>/dev/block_device</replaceable> | grep mmp</screen>
<para>Here is a sample command:</para>
<screen>dumpe2fs -h /dev/sdc | grep mmp
Filesystem features: has_journal ext_attr resize_inode dir_index
filetype extent mmp sparse_super large_file uninit_bg</screen>
<para>To manually disable MMP, run:</para>
- <screen>tune2fs -O ^mmp <device> </screen>
+ <screen>tune2fs -O ^mmp <replaceable>/dev/block_device</replaceable></screen>
<para>To manually enable MMP, run:</para>
- <screen>tune2fs -O mmp <device></screen>
+ <screen>tune2fs -O mmp <replaceable>/dev/block_device</replaceable></screen>
<para>When MMP is enabled, if <literal>ldiskfs</literal> detects multiple mount attempts after the file system is mounted, it blocks these later mount attempts and reports the time when the MMP block was last updated, the node name, and the device name of the node where the file system is currently mounted.</para>
</section>
</section>
<section remap="h3">
<title><indexterm><primary>I/O</primary><secondary>OST space usage</secondary></indexterm>Checking OST Space Usage</title>
<para>The example below shows an unbalanced file system:</para>
- <screen>root@LustreClient01 ~]# lfs df -h
+ <screen>client# lfs df -h
UUID bytes Used Available \
Use% Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G \
filesystem summary: 11.8G 5.4G 5.8G \
45% /mnt/lustre</screen>
<para>In this case, OST:2 is almost full and when an attempt is made to write additional information to the file system (even with uniform striping over all the OSTs), the write command fails as follows:</para>
- <screen>[root@LustreClient01 ~]# lfs setstripe /mnt/lustre 4M 0 -1
-[root@LustreClient01 ~]# dd if=/dev/zero of=/mnt/lustre/test_3 \ bs=10M cou\
-nt=100
-dd: writing `/mnt/lustre/test_3': No space left on device
+ <screen>client# lfs setstripe /mnt/lustre 4M 0 -1
+client# dd if=/dev/zero of=/mnt/lustre/test_3 bs=10M count=100
+dd: writing '/mnt/lustre/test_3': No space left on device
98+0 records in
97+0 records out
1017192448 bytes (1.0 GB) copied, 23.2411 seconds, 43.8 MB/s</screen>
<orderedlist>
<listitem>
<para>Log into the MDS server:</para>
- <screen>[root@LustreClient01 ~]# ssh root@192.168.0.10
+ <screen>client# ssh root@192.168.0.10
root@192.168.0.10's password:
Last login: Wed Nov 26 13:35:12 2008 from 192.168.0.6</screen>
</listitem>
<listitem>
<para>Use the <literal>lctl dl</literal> command to show the status of all file system components:</para>
- <screen>[root@mds ~]# lctl dl
+ <screen>mds# lctl dl
0 UP mgs MGS MGS 9
1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
2 UP mdt MDS MDS_uuid 3
</listitem>
<listitem>
<para>Use <literal>lctl</literal> deactivate to take the full OST offline:</para>
- <screen>[root@mds ~]# lctl --device 7 deactivate</screen>
+ <screen>mds# lctl --device 7 deactivate</screen>
</listitem>
<listitem>
<para>Display the status of the file system components:</para>
- <screen>[root@mds ~]# lctl dl
+ <screen>mds# lctl dl
0 UP mgs MGS MGS 9
1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
2 UP mdt MDS MDS_uuid 3
<listitem>
<para>Identify the file(s) to be moved.</para>
<para>In the example below, output from the <literal>getstripe</literal> command indicates that the file <literal>test_2</literal> is located entirely on OST2:</para>
- <screen>[client]# lfs getstripe /mnt/lustre/test_2
+ <screen>client# lfs getstripe /mnt/lustre/test_2
/mnt/lustre/test_2
obdidx objid objid group
2 8 0x8 0</screen>
</listitem>
<listitem>
<para>To move single object(s), create a new copy and remove the original. Enter:</para>
- <screen>[client]# cp -a /mnt/lustre/test_2 /mnt/lustre/test_2.tmp
-[client]# mv /mnt/lustre/test_2.tmp /mnt/lustre/test_2</screen>
+ <screen>client# cp -a /mnt/lustre/test_2 /mnt/lustre/test_2.tmp
+client# mv /mnt/lustre/test_2.tmp /mnt/lustre/test_2</screen>
</listitem>
<listitem>
<para>To migrate large files from one or more OSTs, enter:</para>
- <screen>[client]# lfs find --ost {OST_UUID} -size +1G | lfs_migrate -y</screen>
+ <screen>client# lfs find --ost <replaceable>ost_name</replaceable> -size +1G | lfs_migrate -y</screen>
</listitem>
<listitem>
<para>Check the file system balance.</para>
<para>The <literal>df</literal> output in the example below shows a more balanced system compared to the <literal>df</literal> output in the example in <xref linkend="dbdoclet.50438211_17536"/>.</para>
- <screen>[client]# lfs df -h
+ <screen>client# lfs df -h
UUID bytes Used Available Use% \
Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% \
<para>Running the <literal>writeconf</literal> command on the MDS erases all pools information (as well as any other parameters set using <literal>lctl conf_param</literal>). We recommend that the pools definitions (and <literal>conf_param</literal> settings) be executed using a script, so they can be reproduced easily after a <literal>writeconf</literal> is performed.</para>
</caution>
<para>To create a new pool, run:</para>
- <screen>lctl pool_new <fsname>.<poolname></screen>
+ <screen>mgs# lctl pool_new <replaceable>fsname</replaceable>.<replaceable>poolname</replaceable></screen>
<note>
<para>The pool name is an ASCII string up to 16 characters.</para>
</note>
<para>To add the named OST to a pool, run:</para>
- <screen>lctl pool_add <fsname>.<poolname> <ost_list></screen>
+ <screen>mgs# lctl pool_add <replaceable>fsname</replaceable>.<replaceable>poolname</replaceable> <replaceable>ost_list</replaceable></screen>
<para>Where:</para>
<itemizedlist>
<listitem>
- <para><literal><ost_list> is <fsname->OST<index_range>[_UUID]</literal></para>
+ <para><literal><replaceable>ost_list</replaceable> is <replaceable>fsname</replaceable>-OST<replaceable>index_range</replaceable></literal></para>
</listitem>
<listitem>
- <para><literal><index_range> is <ost_index_start>-<ost_index_end>[,<index_range>]</literal> or <literal><ost_index_start>-<ost_index_end>/<step></literal></para>
+ <para><literal><replaceable>index_range</replaceable> is <replaceable>ost_index_start</replaceable>-<replaceable>ost_index_end[,index_range]</replaceable></literal> or <literal><replaceable>ost_index_start</replaceable>-<replaceable>ost_index_end/step</replaceable></literal></para>
</listitem>
</itemizedlist>
- <para>If the leading <literal><fsname></literal> and/or ending <literal>_UUID</literal> are missing, they are automatically added.</para>
+ <para>If the leading <literal><replaceable>fsname</replaceable></literal> and/or ending <literal>_UUID</literal> are missing, they are automatically added.</para>
<para> For example, to add even-numbered OSTs to pool1 on file system <literal>lustre</literal>, run a single command (add) to add many OSTs to the pool at one time:</para>
<para><screen>lctl pool_add lustre.pool1 OST[0-10/2]</screen></para>
<note>
<para>Each time an OST is added to a pool, a new <literal>llog</literal> configuration record is created. For convenience, you can run a single command.</para>
</note>
<para>To remove a named OST from a pool, run:</para>
- <screen>lctl pool_remove <fsname>.<poolname> <ost_list></screen>
+ <screen>mgs# lctl pool_remove <replaceable>fsname</replaceable>.<replaceable>poolname</replaceable> <replaceable>ost_list</replaceable></screen>
<para>To destroy a pool, run:</para>
- <screen>lctl pool_destroy <fsname>.<poolname></screen>
+ <screen>mgs# lctl pool_destroy <replaceable>fsname</replaceable>.<replaceable>poolname</replaceable></screen>
<note>
<para>All OSTs must be removed from a pool before it can be destroyed.</para>
</note>
<para>To list pools in the named file system, run:</para>
- <screen>lctl pool_list <fsname> | <pathname></screen>
+ <screen>mgs# lctl pool_list <replaceable>fsname|pathname</replaceable></screen>
<para>To list OSTs in a named pool, run:</para>
- <screen>lctl pool_list <fsname>.<poolname></screen>
+ <screen>lctl pool_list <replaceable>fsname</replaceable>.<replaceable>poolname</replaceable></screen>
<section remap="h4">
<title>Using the lfs Command with OST Pools</title>
<para>Several lfs commands can be run with OST pools. Use the <literal>lfs setstripe</literal> command to associate a directory with an OST pool. This causes all new regular files and directories in the directory to be created in the pool. The lfs command can be used to list pools in a file system and OSTs in a named pool.</para>
<para>To associate a directory with a pool, so all new files and directories will be created in the pool, run:</para>
- <screen>lfs setstripe --pool|-p pool_name <filename|dirname> </screen>
+ <screen>client# lfs setstripe --pool|-p pool_name <replaceable>filename|dirname</replaceable> </screen>
<para>To set striping patterns, run:</para>
- <screen>lfs setstripe [--size|-s stripe_size] [--offset|-o start_ost]
+ <screen>client# lfs setstripe [--size|-s stripe_size] [--offset|-o start_ost]
[--count|-c stripe_count] [--pool|-p pool_name]
- <dir|filename></screen>
+ <replaceable>dir|filename</replaceable></screen>
<note>
<para>If you specify striping with an invalid pool name, because the pool does not exist or the pool name was mistyped, <literal>lfs setstripe</literal> returns an error. Run <literal>lfs pool_list</literal> to make sure the pool exists and the pool name is entered correctly.</para>
</note>
<orderedlist>
<listitem>
<para>Add a new OST by passing on the following commands, run:</para>
- <screen>$ mkfs.lustre --fsname=spfs --mgsnode=mds16@tcp0 --ost --index=12 /dev/sda
-$ mkdir -p /mnt/test/ost12
-$ mount -t lustre /dev/sda /mnt/test/ost12</screen>
+ <screen>oss# mkfs.lustre --fsname=spfs --mgsnode=mds16@tcp0 --ost --index=12 /dev/sda
+oss# mkdir -p /mnt/test/ost12
+oss# mount -t lustre /dev/sda /mnt/test/ost12</screen>
</listitem>
<listitem>
<para>Migrate the data (possibly).</para>
<section remap="h3">
<title>Making File System Objects Immutable</title>
<para>An immutable file or directory is one that cannot be modified, renamed or removed. To do this:</para>
- <screen>chattr +i <file></screen>
+ <screen>chattr +i <replaceable>file</replaceable></screen>
<para>To remove this flag, use <literal>chattr -i</literal></para>
</section>
</section>
<para>To check which checksum algorithm is being used by Lustre, run:</para>
<screen>$ lctl get_param osc.*.checksum_type</screen>
<para>To change the wire checksum algorithm used by Lustre, run:</para>
- <screen>$ lctl set_param osc.*.checksum_type=<algorithm name></screen>
+ <screen>$ lctl set_param osc.*.checksum_type=<replaceable>algorithm</replaceable></screen>
<note>
<para>The in-memory checksum always uses the adler32 algorithm, if available, and only falls back to crc32 if adler32 cannot be used.</para>
</note>
<para>This command tells you the network(s) configured to work with Lustre</para>
<para>If the networks are not correctly setup, see the <literal>modules.conf</literal> "<literal>networks=</literal>" line and make sure the network layer modules are correctly installed and configured.</para>
<para>To get the best remote NID, run:</para>
- <screen>$ lctl which_nid <NID list></screen>
- <para>where <literal><NID list></literal> is the list of available NIDs.</para>
+ <screen>$ lctl which_nid <replaceable>NIDs</replaceable></screen>
+ <para>where <literal><replaceable>NIDs</replaceable></literal> is the list of available NIDs.</para>
<para>This command takes the "best" NID from a list of the NIDs of a remote host. The "best" NID is the one that the local node uses when trying to communicate with the remote node.</para>
<section remap="h4">
<title>Starting Clients</title>
<para>Attempting to remove Lustre modules prior to stopping the network may result in a crash or an LNET hang. if this occurs, the node must be rebooted (in most cases). Make sure that the Lustre network and Lustre are stopped prior to unloading the modules. Be extremely careful using rmmod -f.</para>
</note>
<para>To unconfigure the LNET network, run:</para>
- <screen>modprobe -r <any lnd and the lnet modules></screen>
+ <screen>modprobe -r <replaceable>lnd_and_lnet_modules</replaceable></screen>
<note>
<para>
To remove all Lustre modules, run:</para>
</listitem>
<listitem>
<para>Run the modprobe lnet command and create a combined MGS/MDT file system.</para>
- <para>The following commands create the MGS/MDT file system and mount the servers (MGS/MDT and OSS).</para>
+ <para>The following commands create an MGS/MDT or OST file system and mount the targets on the servers.</para>
<screen>modprobe lnet
-$ mkfs.lustre --fsname lustre --mgs --mdt <block device name>
-$ mkdir -p <mount point>
-$ mount -t lustre <block device> <mount point>
-$ mount -t lustre <block device> <mount point>
-$ mkfs.lustre --fsname lustre --mgs --mdt <block device name>
-$ mkdir -p <mount point>
-$ mount -t lustre <block device> <mount point>
-$ mount -t lustre <block device> <mount point> </screen>
+# mkfs.lustre --fsname lustre --mgs --mdt <replaceable>/dev/mdt_device</replaceable>
+# mkdir -p <replaceable>/mount_point</replaceable>
+# mount -t lustre /dev/<replaceable>mdt_device</replaceable> <replaceable>/mount_point</replaceable></screen>
<para>For example:</para>
<screen>modprobe lnet
-$ mkfs.lustre --fsname lustre --mdt --mgs /dev/sda
-$ mkdir -p /mnt/test/mdt
-$ mount -t lustre /dev/sda /mnt/test/mdt
-$ mount -t lustre mgs@o2ib0:/lustre /mnt/mdt
-$ mkfs.lustre --fsname lustre --mgsnode=mds@o2ib0 --ost --index=0 /dev/sda
-$ mkdir -p /mnt/test/mdt
-$ mount -t lustre /dev/sda /mnt/test/ost
-$ mount -t lustre mgs@o2ib0:/lustre /mnt/ost0</screen>
+mds# mkfs.lustre --fsname lustre --mdt --mgs /dev/sda
+mds# mkdir -p /mnt/test/mdt
+mds# mount -t lustre /dev/sda /mnt/test/mdt
+mds# mount -t lustre mgs@o2ib0:/lustre /mnt/mdt
+oss# mkfs.lustre --fsname lustre --mgsnode=mds@o2ib0 --ost --index=0 /dev/sda
+oss# mkdir -p /mnt/test/mdt
+oss# mount -t lustre /dev/sda /mnt/test/ost
+oss# mount -t lustre mgs@o2ib0:/lustre /mnt/ost0</screen>
</listitem>
<listitem>
<para>Mount the clients.</para>
- <screen>mount -t lustre <MGS node>:/<fsname> <mount point></screen>
+ <screen>client# mount -t lustre <replaceable>mgs_node</replaceable>:/<replaceable>fsname</replaceable> <replaceable>/mount_point</replaceable></screen>
<para>This example shows an IB client being mounted.</para>
- <screen>mount -t lustre
+ <screen>client# mount -t lustre
192.168.10.101@o2ib0,192.168.10.102@o2ib1:/mds/client /mnt/lustre</screen>
</listitem>
</orderedlist>
<para>You could create these configurations:</para>
<itemizedlist>
<listitem>
- <para>A cluster with more clients than servers. The fact that an individual client cannot get two rails of bandwidth is unimportant because the servers are the actual bottleneck.</para>
+ <para>A cluster with more clients than servers. The fact that an individual client cannot get two rails of bandwidth is unimportant because the servers are typically the actual bottleneck.</para>
</listitem>
</itemizedlist>
<screen>ip2nets="o2ib0(ib0), o2ib1(ib1) 192.168.[0-1].* \
</section>
<section xml:id="dbdoclet.50438221_64726">
<title><indexterm><primary>root squash</primary></indexterm>Using Root Squash</title>
- <para>Lustre 1.6 introduced root squash functionality, a security feature which controls super user access rights to an Lustre file system. Before the root squash feature was added, Lustre users could run <literal>rm -rf *</literal> as root, and remove data which should not be deleted. Using the root squash feature prevents this outcome.</para>
+ <para>Root squash is a security feature which restricts super-user access rights to a Lustre file system. Without the root squash feature enabled, Lustre users on untrusted clients could access or modify files owned by root on the filesystem, including deleting them. Using the root squash feature restricts file access/modifications as the root user to only the specified clients. Note, however, that this does <emphasis>not</emphasis> prevent users on insecure clients from accessing files owned by <emphasis>other</emphasis> users.</para>
<para>The root squash feature works by re-mapping the user ID (UID) and group ID (GID) of the root user to a UID and GID specified by the system administrator, via the Lustre configuration management server (MGS). The root squash feature also enables the Lustre administrator to specify a set of client for which UID/GID re-mapping does not apply.</para>
<section remap="h3">
<title><indexterm><primary>root squash</primary><secondary>configuring</secondary></indexterm>Configuring Root Squash</title>
<title><indexterm><primary>root squash</primary><secondary>enabling</secondary></indexterm>Enabling and Tuning Root Squash</title>
<para>The default value for <literal>nosquash_nids</literal> is NULL, which means that root squashing applies to all clients. Setting the root squash UID and GID to 0 turns root squash off.</para>
<para>Root squash parameters can be set when the MDT is created (<literal>mkfs.lustre --mdt</literal>). For example:</para>
- <screen>mkfs.lustre --reformat --fsname=Lustre --mdt --mgs \
+ <screen>mds# mkfs.lustre --reformat --fsname=testfs --mdt --mgs \
--param "mdt.root_squash=500:501" \
--param "mdt.nosquash_nids='0@elan1 192.168.1.[10,11]'" /dev/sda1</screen>
<para>Root squash parameters can also be changed on an unmounted device with <literal>tunefs.lustre</literal>. For example:</para>
--param "mdt.nosquash_nids=192.168.0.13@tcp0" /dev/sda1
</screen>
<para>Root squash parameters can also be changed with the <literal>lctl conf_param</literal> command. For example:</para>
- <screen>lctl conf_param Lustre.mdt.root_squash="1000:101"
-lctl conf_param Lustre.mdt.nosquash_nids="*@tcp"</screen>
+ <screen>mgs# lctl conf_param testfs.mdt.root_squash="1000:101"
+mgs# lctl conf_param testfs.mdt.nosquash_nids="*@tcp"</screen>
<note>
<para>When using the lctl conf_param command, keep in mind:</para>
<itemizedlist>
</itemizedlist>
</note>
<para>The <literal>nosquash_nids</literal> list can be cleared with:</para>
- <screen>lctl conf_param Lustre.mdt.nosquash_nids="NONE"</screen>
+ <screen>mgs# lctl conf_param testfs.mdt.nosquash_nids="NONE"</screen>
<para>- OR -</para>
- <screen>lctl conf_param Lustre.mdt.nosquash_nids="clear"</screen>
+ <screen>mgs# lctl conf_param testfs.mdt.nosquash_nids="clear"</screen>
<para>If the <literal>nosquash_nids</literal> value consists of several NID ranges (e.g. <literal>0@elan</literal>, <literal>1@elan1</literal>), the list of NID ranges must be quoted with single (') or double ('') quotation marks. List elements must be separated with a space. For example:</para>
- <screen>mkfs.lustre ... --param "mdt.nosquash_nids='0@elan1 1@elan2'" /dev/sda1
-lctl conf_param Lustre.mdt.nosquash_nids="24@elan 15@elan1"</screen>
+ <screen>mds# mkfs.lustre ... --param "mdt.nosquash_nids='0@elan1 1@elan2'" /dev/sda1
+lctl conf_param testfs.mdt.nosquash_nids="24@elan 15@elan1"</screen>
<para>These are examples of incorrect syntax:</para>
- <screen>mkfs.lustre ... --param "mdt.nosquash_nids=0@elan1 1@elan2" /dev/sda1
-lctl conf_param Lustre.mdt.nosquash_nids=24@elan 15@elan1</screen>
+ <screen>mds# mkfs.lustre ... --param "mdt.nosquash_nids=0@elan1 1@elan2" /dev/sda1
+lctl conf_param testfs.mdt.nosquash_nids=24@elan 15@elan1</screen>
<para>To check root squash parameters, use the lctl get_param command:</para>
- <screen>lctl get_param mdt.Lustre-MDT0000.root_squash
+ <screen>mds# lctl get_param mdt.testfs-MDT0000.root_squash
lctl get_param mdt.*.nosquash_nids</screen>
<note>
<para>An empty nosquash_nids list is reported as NONE.</para>
<para><literal>mkfs.lustre</literal> and <literal>tunefs.lustre</literal> do not perform parameter syntax checking. If the root squash parameters are incorrect, they are ignored on mount and the default values are used instead.</para>
</listitem>
<listitem>
- <para>Root squash parameters are parsed with rigorous syntax checking. The root_squash parameter should be specified as <literal><decnum>':'<decnum></literal>. The <literal>nosquash_nids</literal> parameter should follow LNET NID range list syntax.</para>
+ <para>Root squash parameters are parsed with rigorous syntax checking. The root_squash parameter should be specified as <literal><decnum>:<decnum></literal>. The <literal>nosquash_nids</literal> parameter should follow LNET NID range list syntax.</para>
</listitem>
</itemizedlist>
<para>LNET NID range syntax:</para>
Setting the File Layout/Striping Configuration (<literal>lfs setstripe</literal>)</title>
<para>Use the <literal>lfs setstripe</literal> command to create new files with a specific file layout (stripe pattern) configuration.</para>
<screen>lfs setstripe [--size|-s stripe_size] [--count|-c stripe_count]
-[--index|-i start_ost] [--pool|-p pool_name] <filename|dirname> </screen>
+[--index|-i start_ost] [--pool|-p pool_name] <replaceable>filename|dirname</replaceable> </screen>
<para><emphasis role="bold">
<literal>stripe_size</literal>
</emphasis>
<para>The start OST is the first OST to which files are written. The default value for <literal>start_ost</literal> is -1, which allows the MDS to choose the starting index. This setting is strongly recommended, as it allows space and load balancing to be done by the MDS as needed. Otherwise, the file starts on the specified OST index. The numbering of the OSTs starts at 0.</para>
<note>
<para>If you pass a <literal>start_ost</literal> value of 0 and a <literal>stripe_count</literal> value of <emphasis>1</emphasis>, all files are written to OST 0, until space is exhausted. This is probably not what you meant to do. If you only want to adjust the stripe count and keep the other parameters at their default settings, do not specify any of the other parameters:</para>
- <para><screen>lfs setstripe -c <stripe_count> <file></screen></para>
+ <para><screen>client# lfs setstripe -c <replaceable>stripe_count</replaceable> <replaceable>filename</replaceable></screen></para>
</note>
<para><emphasis role="bold">
<literal>pool_name</literal>
<section remap="h3">
<title>Inspecting the File Tree</title>
<para>To inspect an entire tree of files, use the <literal>lfs</literal> find command:</para>
- <screen>lfs find [--recursive | -r] <file or directory> ...</screen>
- <para>You can also use <literal>ls -l /proc/<replaceable><pid></replaceable>/fd/</literal> to find open files using Lustre. For example:</para>
+ <screen>lfs find [--recursive | -r] <replaceable>file|directory</replaceable> ...</screen>
+ <para>You can also use <literal>ls -l /proc/<replaceable>pid</replaceable>/fd/</literal> to find open files using Lustre. For example:</para>
<screen>$ lfs getstripe $(readlink /proc/$(pidof cat)/fd/1)</screen>
<para>Typical output is:</para>
<screen>/mnt/lustre/foo
<section remap="h3">
<title><indexterm><primary>space</primary><secondary>location weighting</secondary></indexterm>>Adjusting the Weighting Between Free Space and Location</title>
<para>The weighting priority can be adjusted in the <literal>/proc</literal> file <literal>/proc/fs/lustre/lov/lustre-mdtlov/qos_prio_free proc</literal>. The default value is 90%. Use this command on the MGS to permanently change this weighting:</para>
- <screen>lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90</screen>
+ <screen>lctl conf_param <replaceable>fsname</replaceable>-MDT0000.lov.qos_prio_free=90</screen>
<para>Increasing this value puts more weighting on free space. When the free space priority is set to 100%, then location is no longer used in stripe-ordering calculations and weighting is based entirely on free space.</para>
<note>
<para>Setting the priority to 100% means that OSS distribution does not count in the weighting, but the stripe assignment is still done via a weighting. For example, if OST2 has twice as much free space as OST1, then OST2 is twice as likely to be used, but it is not guaranteed to be used.</para>
<section xml:id="dbdoclet.50438256_pgfId-1293228">
<title><indexterm><primary>inodes</primary><secondary>MDS</secondary></indexterm><indexterm><primary>setup</primary><secondary>inodes</secondary></indexterm>Setting the Number of Inodes for the MDS</title>
<para>The number of inodes on the MDT is determined at format time based on the total size of the file system to be created. The default MDT inode ratio is one inode for every 4096 bytes of file system space. To override the inode ratio, use the following option:</para>
- <screen>-i <emphasis><bytes per inode></emphasis></screen>
+ <screen>-i <replaceable>bytes_per_inode</replaceable></screen>
<para>For example, use the following option to create one inode per 2048 bytes of file system space.</para>
<screen>--mkfsoptions="-i 2048" </screen>
<para>To avoid mke2fs creating an unusable file system, do not specify the -i option with an inode ratio below one inode per 1024 bytes. Instead, specify an absolute number of inodes, using this option:</para>
- <screen>-N<emphasis> <number of inodes></emphasis></screen>
+ <screen>-N <replaceable>number_of_inodes</replaceable></screen>
<para>For example, by default, a 2 TB MDT will have 512M inodes. The largest currently-supported file system size is 16 TB, which would hold 4B inodes, the maximum possible number of inodes in a ldiskfs file system. With an MDS inode ratio of 1024 bytes per inode, a 2 TB MDT would hold 2B inodes, and a 4 TB MDT would hold 4B inodes.</para>
</section>
<section remap="h3">
<title><indexterm><primary>inodes</primary><secondary>MDT</secondary></indexterm>Setting the Inode Size for the MDT</title>
<para>Lustre uses "large" inodes on backing file systems to efficiently store Lustre metadata with each file. On the MDT, each inode is at least 512 bytes in size (by default), while on the OST each inode is 256 bytes in size.</para>
<para>The backing ldiskfs file system also needs sufficient space for other metadata like the journal (up to 400 MB), bitmaps and directories and a few files that Lustre uses to maintain cluster consistency.</para>
- <para>To specify a larger inode size, use the <literal>-I <inodesize></literal> option. We recommend you do NOT specify a smaller-than-default inode size, as this can lead to serious performance problems; and you cannot change this parameter after formatting the file system. The inode ratio must always be larger than the inode size.</para>
+ <para>To specify a larger inode size, use the <literal>-I <replaceable>inode_size</replaceable></literal> option. We recommend you do NOT specify a smaller-than-default inode size, as this can lead to serious performance problems; and you cannot change this parameter after formatting the file system. The inode ratio must always be larger than the inode size.</para>
</section>
<section xml:id="dbdoclet.50438256_53886">
<title><indexterm><primary>inodes</primary><secondary>OST</secondary></indexterm>Setting the Number of Inodes for an OST</title>
<para>When formatting OST file systems, it is normally advantageous to take local file system usage into account. Try to minimize the number of inodes on each OST, while keeping enough margin for potential variance in future usage. This helps reduce the format and file system check time, and makes more space available for data.</para>
<para>The current default is to create one inode per 16 KB of space in the OST file system, but in many environments, this is far too many inodes for the average file size. As a good rule of thumb, the OSTs should have at least:</para>
- <para>num_ost_inodes = 4 * <emphasis><num_mds_inodes></emphasis> * <emphasis><default_stripe_count></emphasis> / <emphasis><number_osts></emphasis></para>
+ <para>num_ost_inodes = 4 * <replaceable>number_of_mds_inodes</replaceable> * <replaceable>default_stripe_count</replaceable> / <replaceable>number_of_osts</replaceable></para>
<table frame="all">
<title xml:id="settinguplustresystem.tab1">Inode Ratio to be considered</title>
<tgroup cols="3">
<tbody>
<row>
<entry>
- <para> < 10GB </para>
+ <para> over 10GB </para>
</entry>
<entry>
<para> 1 inode/16KB </para>
</row>
<row>
<entry>
- <para> > 8TB </para>
+ <para> over 8TB </para>
</entry>
<entry>
<para> 1 inode/1MB </para>
</table>
<para> </para>
<para>You can specify the number of inodes on the OST file systems using the following option to the <literal>--mkfs</literal> option:</para>
- <screen>-N <emphasis><num_inodes></emphasis></screen>
+ <screen>-N <replaceable>num_inodes</replaceable></screen>
<para> Alternately, if you know the average file size, then you can specify the OST inode count for the OST file systems using:</para>
- <screen>-i <emphasis><average_file_size</emphasis> / (<emphasis>number_of_stripes</emphasis> * 4)<emphasis>></emphasis></screen>
+ <screen>-i <replaceable>average_file_size</replaceable> / (<replaceable>number_of_stripes</replaceable> * 4)</screen>
<para>For example, if the average file size is 16 MB and there are, by default 4 stripes per file, then <literal>--mkfsoptions='-i 1048576'</literal> would be appropriate.</para>
<note>
<para>In addition to the number of inodes, file system check time on OSTs is affected by a number of other variables: size of the file system, number of allocated blocks, distribution of allocated blocks on the disk, disk speed, CPU speed, and amount of RAM on the server. Reasonable file system check times (without serious file system problems), are expected to take five and thirty minutes per TB.</para>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">-b inode buffer blocks</emphasis></para>
+ <para> <literal>-b <replaceable>inode buffer blocks</replaceable></literal></para>
</entry>
<entry>
<para> Sets the readahead inode blocks to get excellent performance when scanning the block device.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">-o output file</emphasis></para>
+ <para> <literal>-o <replaceable>output file</replaceable></literal></para>
</entry>
<entry>
<para> If an output file is specified, modified pathnames are written to this file. Otherwise, modified parameters are written to stdout.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">-t inode | pathname</emphasis></para>
+ <para> <literal>-t <replaceable>inode</replaceable>| <replaceable>pathname</replaceable></literal></para>
</entry>
<entry>
<para> Sets the e2scan type if type is inode. The e2scan utility prints modified inode numbers to stdout. By default, the type is set as pathname.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">-u</emphasis></para>
+ <para> <literal>-u</literal></para>
</entry>
<entry>
<para> Rebuilds the parent database from scratch. Otherwise, the current parent database is used.</para>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">mdtname</emphasis></para>
+ <para> <literal>mdtname</literal></para>
</entry>
<entry>
<para> Metadata server target name</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">uid</emphasis></para>
+ <para> <literal>uid</literal></para>
</entry>
<entry>
<para> User identifier</para>
<para>The lctl utility is used for root control and configuration. With lctl you can directly control Lustre via an ioctl interface, allowing various configuration, maintenance and debugging features to be accessed.</para>
<section remap="h5">
<title>Synopsis</title>
- <screen>lctl
-lctl --device <devno> <command [args]></screen>
+ <screen>lctl [--device <replaceable>devno</replaceable>] <replaceable>command [args]</replaceable></screen>
</section>
<section remap="h5">
<title>Description</title>
<screen>dl
dk
device
-network <<emphasis>up/down</emphasis>>
+network <replaceable>up|down</replaceable>
list_nids
-ping <emphasis>nid</emphasis>help
+ping <replaceable>nid</replaceable>help
quit</screen>
- <para>For a complete list of available commands, type help at the lctl prompt. To get basic help on command meaning and syntax, type help<emphasis>command</emphasis>. Command completion is activated with the TAB key, and command history is available via the up- and down-arrow keys.</para>
+ <para>For a complete list of available commands, type <literal>help</literal> at the <literal>lctl</literal> prompt. To get basic help on command meaning and syntax, type <literal>help <replaceable>command</replaceable></literal>. Command completion is activated with the TAB key (depending on compile options), and command history is available via the up- and down-arrow keys.</para>
<para>For non-interactive use, use the second invocation, which runs the command after connecting to the device.</para>
</section>
<section remap="h5">
<title>Setting Parameters with lctl</title>
<para>Lustre parameters are not always accessible using the procfs interface, as it is platform-specific. As a solution, lctl {get,set}_param has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to /proc/{fs,sys}/{lustre,lnet}. For future portability, use lctl {get,set}_param .</para>
- <para>When the file system is running, use the lctl set_param command to set temporary parameters (mapping to items in /proc/{fs,sys}/{lnet,lustre}). The <literal>lctl set_param</literal> command uses this syntax:</para>
- <screen>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value></screen>
+ <para>When the file system is running, use the <literal>lctl set_param</literal> command on the affected node(s) to <emphasis>temporarily</emphasis> set parameters (mapping to items in /proc/{fs,sys}/{lnet,lustre}). The <literal>lctl set_param</literal> command uses this syntax:</para>
+ <screen>lctl set_param [-n] <replaceable>obdtype.obdname.property</replaceable>=<replaceable>value</replaceable></screen>
<para>For example:</para>
- <screen>$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))</screen>
- <para>Many permanent parameters can be set with lctl conf_param. In general, lctl conf_param can be used to specify any parameter settable in a /proc/fs/lustre file, with its own OBD device. The <literal>lctl conf_param</literal> command uses this syntax:</para>
- <screen><obd|fsname>.<obdtype>.<proc_file_name>=<value>) </screen>
+ <screen>mds# lctl set_param mdt.testfs-MDT0000.identity_upcall=NONE</screen>
+ <para>Many permanent parameters can be set with <literal>lctl conf_param</literal>. In general, <literal>lctl conf_param</literal> can be used to specify any OBD device parameter settable in a /proc/fs/lustre file. The <literal>lctl conf_param</literal> command must be run on the MGS node, and uses this syntax:</para>
+ <screen><replaceable>obd|fsname</replaceable>.obdtype.property=<replaceable>value</replaceable>) </screen>
<para>For example:</para>
- <screen>$ lctl conf_param testfs-MDT0000.mdt.identity_upcall=NONE
+ <screen>mgs# lctl conf_param testfs-MDT0000.mdt.identity_upcall=NONE
$ lctl conf_param testfs.llite.max_read_ahead_mb=16 </screen>
<caution>
- <para>The <literal>lctl conf_param</literal> command permanently sets parameters in the file system configuration.</para>
+ <para>The <literal>lctl conf_param</literal> command <emphasis>permanently</emphasis> sets parameters in the file system configuration for all nodes of the specified type.</para>
</caution>
- <para>To get current Lustre parameter settings, use the lctl get_param command with this syntax:</para>
- <screen>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name></screen>
+ <para>To get current Lustre parameter settings, use the <literal>lctl get_param</literal> command on the desired node with the same parameter name as <literal>lctl set_param</literal>:</para>
+ <screen>lctl get_param [-n] <replaceable>obdtype.obdname.parameter</replaceable></screen>
<para>For example:</para>
- <screen>$ lctl get_param -n ost.*.ost_io.timeouts </screen>
- <para>To list Lustre parameters that are available to set, use the lctl list_param command, with this syntax:</para>
- <screen>lctl list_param [-n] <obdtype>.<obdname></screen>
- <para>For example:</para>
- <screen>$ lctl list_param obdfilter.lustre-OST0000</screen>
- <para>For more information on using lctl to set temporary and permanent parameters, see <xref linkend="dbdoclet.50438194_51490"/> (Setting Parameters with lctl).</para>
+ <screen>mds# lctl get_param mdt.testfs-MDT0000.identity_upcall</screen>
+ <para>To list Lustre parameters that are available to set, use the <literal>lctl list_param</literal> command, with this syntax:</para>
+ <screen>lctl list_param [-R] [-F] <replaceable>obdtype.obdname.*</replaceable></screen>
+ <para>For example, to list all of the parameters on the MDT:</para>
+ <screen>oss# lctl list_param -RF mdt</screen>
+ <para>For more information on using lctl to set temporary and permanent parameters, see <xref linkend="dbdoclet.50438194_51490"/>.</para>
<para><emphasis role="bold">Network Configuration</emphasis></para>
<informaltable frame="all">
<tgroup cols="2">
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">network</emphasis> <<emphasis>up/down</emphasis>>|<<emphasis>tcp/elan/myrinet</emphasis>></para>
+ <para> <literal>network up|down|tcp|elan|myrinet</literal></para>
</entry>
<entry>
- <para> Starts or stops LNET, or selects a network type for other <emphasis role="bold">lctl</emphasis> LNET commands.</para>
+ <para> Starts or stops LNET, or selects a network type for other <literal>lctl</literal> LNET commands.</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">list_nids</emphasis></para>
+ <para> <literal>list_nids</literal></para>
</entry>
<entry>
<para> Prints all NIDs on the local node. LNET must be running.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">which_nid</emphasis> <<emphasis>nidlist</emphasis>></para>
+ <para> <literal>which_nid <replaceable>nidlist</replaceable></literal></para>
</entry>
<entry>
<para> From a list of NIDs for a remote node, identifies the NID on which interface communication will occur.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">ping</emphasis> <<emphasis>nid</emphasis>></para>
+ <para> <literal>ping <replaceable>nid</replaceable></literal></para>
</entry>
<entry>
<para> Checks LNET connectivity via an LNET ping. This uses the fabric appropriate to the specified NID.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">interface_list</emphasis></para>
+ <para> <literal>interface_list</literal></para>
</entry>
<entry>
- <para> Prints the network interface information for a given <emphasis role="bold">network</emphasis> type.</para>
+ <para> Prints the network interface information for a given <emphasis>network</emphasis> type.</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">peer_list</emphasis></para>
+ <para> <literal>peer_list</literal></para>
</entry>
<entry>
- <para> Prints the known peers for a given <emphasis role="bold">network</emphasis> type.</para>
+ <para> Prints the known peers for a given <emphasis>network</emphasis> type.</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">conn_list</emphasis></para>
+ <para> <literal>conn_list</literal></para>
</entry>
<entry>
- <para> Prints all the connected remote NIDs for a given <emphasis role="bold">network</emphasis> type.</para>
+ <para> Prints all the connected remote NIDs for a given <emphasis>network</emphasis> type.</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">active_tx</emphasis></para>
+ <para> <literal>active_tx</literal></para>
</entry>
<entry>
- <para> This command prints active transmits. It is only used for the Elan <emphasis role="bold">network</emphasis> type.</para>
+ <para> This command prints active transmits. It is only used for the Elan <emphasis>network</emphasis> type.</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">route_list</emphasis></para>
+ <para> <literal>route_list</literal></para>
</entry>
<entry>
<para> Prints the complete routing table.</para>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">device</emphasis> <<emphasis>devname</emphasis>></para>
+ <para> <literal>device <replaceable>devname</replaceable></literal></para>
</entry>
<entry>
<para>  </para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">device_list</emphasis></para>
+ <para> <literal>device_list</literal></para>
</entry>
<entry>
<para>  </para>
</entry>
<entry>
- <para> Shows the local Lustre OBDs, a/k/a <emphasis role="bold">dl</emphasis>.</para>
+ <para> Shows the local Lustre OBDs, a/k/a <literal>dl</literal>.</para>
</entry>
</row>
</tbody>
<tbody>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">list_param</emphasis><emphasis>[-F|-R]</emphasis> <<emphasis>param_path ...</emphasis>></para>
+ <para> <literal>list_param [-F|-R] <replaceable>parameter</replaceable> <replaceable>[parameter ...]</replaceable></literal></para>
</entry>
<entry>
<para> Lists the Lustre or LNET parameter name.</para>
<para>  </para>
</entry>
<entry>
- <para> -<emphasis role="bold">F</emphasis></para>
+ <para> <literal>-F</literal></para>
</entry>
<entry>
<para> Adds '/', '@' or '=' for directories, symlinks and writeable files, respectively.</para>
<para>  </para>
</entry>
<entry>
- <para> -<emphasis role="bold">R</emphasis></para>
+ <para> <literal>-R</literal></para>
</entry>
<entry>
- <para> Recursively lists all parameters under the specified path. If <emphasis>param_path</emphasis> is unspecified, all parameters are shown.</para>
+ <para> Recursively lists all parameters under the specified path. If <literal>param_path</literal> is unspecified, all parameters are shown.</para>
</entry>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">get_param</emphasis><emphasis>[-n|-N|-F]</emphasis> <<emphasis>param_path ...</emphasis>></para>
+ <para> <literal>get_param [-n|-N|-F] <replaceable>parameter</replaceable> <replaceable>[parameter ...]</replaceable></literal></para>
</entry>
<entry>
<para> Gets the value of a Lustre or LNET parameter from the specified path.</para>
<para>  </para>
</entry>
<entry>
- <para> <emphasis role="bold">-n</emphasis></para>
+ <para> <literal>-n</literal></para>
</entry>
<entry>
<para> Prints only the parameter value and not the parameter name.</para>
<para>  </para>
</entry>
<entry>
- <para> <emphasis role="bold">-N</emphasis></para>
+ <para> <literal>-N</literal></para>
</entry>
<entry>
<para> Prints only matched parameter names and not the values; especially useful when using patterns.</para>
<para>  </para>
</entry>
<entry>
- <para> <emphasis role="bold">-F</emphasis></para>
+ <para> <literal>-F</literal></para>
</entry>
<entry>
- <para> When <emphasis role="bold">-N</emphasis> is specified, adds '/', '@' or '=' for directories, symlinks and writeable files, respectively.</para>
+ <para> When <literal>-N</literal> is specified, adds '/', '@' or '=' for directories, symlinks and writeable files, respectively.</para>
</entry>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">set_param</emphasis><emphasis>[-n]</emphasis><emphasis><param_path=value...></emphasis></para>
+ <para> <literal>set_param [-n] <replaceable>parameter</replaceable>=<replaceable>value</replaceable></literal></para>
</entry>
<entry>
<para> Sets the value of a Lustre or LNET parameter from the specified path.</para>
<para>  </para>
</entry>
<entry>
- <para> <emphasis role="bold">-n</emphasis></para>
+ <para> <literal>-n</literal></para>
</entry>
<entry>
<para> Disables printing of the key name when printing values.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">conf_param</emphasis><emphasis>[-d] <device|fsname>.<parameter>=<value></emphasis></para>
+ <para><literal>conf_param [-d] <replaceable>device|fsname</replaceable> <replaceable>parameter</replaceable>=<replaceable>value</replaceable></literal></para>
</entry>
<entry>
<para> Sets a permanent configuration parameter for any device via the MGS. This command must be run on the MGS node.</para>
- <para>All writeable parameters under <emphasis role="bold">lctl list_param</emphasis> (e.g. <emphasis>lctl list_param -F osc.*.* | grep</emphasis> =) can be permanently set using <emphasis role="bold">lctl conf_param</emphasis>, but the format is slightly different. For <emphasis role="bold">conf_param</emphasis>, the device is specified first, then the obdtype. Wildcards are not supported. Additionally, failover nodes may be added (or removed), and some system-wide parameters may be set as well (sys.at_max, sys.at_min, sys.at_extra, sys.at_early_margin, sys.at_history, sys.timeout, sys.ldlm_timeout). For system-wide parameters, <device> is ignored.</para>
- <para>For more information on setting permanent parameters and <emphasis role="bold">lctl conf_param</emphasis> command examples, see <xref linkend="dbdoclet.50438194_64195"/> (Setting Permanent Parameters).</para>
+ <para>All writeable parameters under <literal>lctl list_param</literal> (e.g. <literal>lctl list_param -F osc.*.* | grep</literal> =) can be permanently set using <literal>lctl conf_param</literal>, but the format is slightly different. For <literal>conf_param</literal>, the device is specified first, then the obdtype. Wildcards are not supported. Additionally, failover nodes may be added (or removed), and some system-wide parameters may be set as well (sys.at_max, sys.at_min, sys.at_extra, sys.at_early_margin, sys.at_history, sys.timeout, sys.ldlm_timeout). For system-wide parameters, <replaceable>device</replaceable> is ignored.</para>
+ <para>For more information on setting permanent parameters and <literal>lctl conf_param</literal> command examples, see <xref linkend="dbdoclet.50438194_64195"/> (Setting Permanent Parameters).</para>
</entry>
</row>
<row>
<para>  </para>
</entry>
<entry>
- <para> <emphasis role="bold">-d <device|fsname>.<parameter></emphasis></para>
+ <para><literal>-d <replaceable>device|fsname</replaceable>.<replaceable>parameter</replaceable></literal></para>
<para> </para>
</entry>
<entry>
- <para> Deletes a parameter setting (use the default value at the next restart). A null value for <emphasis role="bold"><value></emphasis> also deletes the parameter setting.</para>
+ <para> Deletes a parameter setting (use the default value at the next restart). A null value for <replaceable>value</replaceable> also deletes the parameter setting.</para>
</entry>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">activate</emphasis></para>
+ <para> <literal>activate</literal></para>
</entry>
<entry>
- <para> Re-activates an import after the deactivate operation. This setting is only effective until the next restart (see <emphasis role="bold">conf_param</emphasis>).</para>
+ <para> Re-activates an import after the deactivate operation. This setting is only effective until the next restart (see <literal>conf_param</literal>).</para>
</entry>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">deactivate</emphasis></para>
+ <para> <literal>deactivate</literal></para>
</entry>
<entry>
<para> Deactivates an import, in particular meaning do not assign new file stripes to an OSC. Running lctl deactivate on the MDS stops new objects from being allocated on the OST. Running lctl deactivate on Lustre clients causes them to return -EIO when accessing objects on the OST instead of waiting for recovery.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">abort_recovery</emphasis></para>
+ <para> <literal>abort_recovery</literal></para>
</entry>
<entry>
<para> Aborts the recovery process on a re-starting MDT or OST.</para>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">blockdev_attach</emphasis><emphasis><file name> <device node></emphasis></para>
+ <para>blockdev_attach <replaceable>filename</replaceable> <replaceable>/dev/lloop_device</replaceable></para>
</entry>
<entry>
- <para> Attaches a regular Lustre file to a block device. If the device node does not exist, <emphasis role="bold">lctl</emphasis> creates it. We recommend that you create the device node by <emphasis role="bold">lctl</emphasis> since the emulator uses a dynamical major number.</para>
+ <para> Attaches a regular Lustre file to a block device. If the device node does not exist, <literal>lctl</literal> creates it. We recommend that you create the device node by <literal>lctl</literal> since the emulator uses a dynamical major number.</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">blockdev_detach</emphasis><emphasis><device node></emphasis></para>
+ <para><literal>blockdev_detach <replaceable>/dev/lloop_device</replaceable></literal></para>
</entry>
<entry>
<para> Detaches the virtual block device.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">blockdev_info</emphasis><emphasis><device node></emphasis></para>
+ <para><literal>blockdev_info <replaceable>/dev/lloop_device</replaceable></literal></para>
</entry>
<entry>
<para> Provides information about the Lustre file attached to the device node.</para>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">changelog_register</emphasis></para>
+ <para> <literal>changelog_register</literal></para>
</entry>
<entry>
- <para> Registers a new changelog user for a particular device. Changelog entries are not purged beyond a registered user's set point (see <emphasis role="bold">lfs changelog_clear</emphasis>).</para>
+ <para> Registers a new changelog user for a particular device. Changelog entries are not purged beyond a registered user's set point (see <literal>lfs changelog_clear</literal>).</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">changelog_deregister</emphasis><emphasis><id></emphasis></para>
+ <para>changelog_deregister <replaceable>id</replaceable></para>
</entry>
<entry>
<para> Unregisters an existing changelog user. If the user's "clear" record number is the minimum for the device, changelog records are purged until the next minimum.</para>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">debug_daemon</emphasis></para>
+ <para> <literal>debug_daemon</literal></para>
</entry>
<entry>
<para> Starts and stops the debug daemon, and controls the output filename and size.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">debug_kernel</emphasis><emphasis>[file] [raw]</emphasis></para>
+ <para> <literal>debug_kernel <replaceable>[file]</replaceable> [raw]</literal></para>
</entry>
<entry>
<para> Dumps the kernel debug buffer to stdout or a file.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">debug_file</emphasis><emphasis><input> [output]</emphasis></para>
+ <para><literal>debug_file <replaceable>input_file</replaceable> <replaceable>[output_file]</replaceable></literal></para>
</entry>
<entry>
<para> Converts the kernel-dumped debug log from binary to plain text format.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">clear</emphasis></para>
+ <para> <literal>clear</literal></para>
</entry>
<entry>
<para> Clears the kernel debug buffer.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">mark</emphasis><emphasis><text></emphasis></para>
+ <para> <literal>mark <replaceable>text</replaceable></literal></para>
</entry>
<entry>
<para> Inserts marker text in the kernel debug buffer.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">filter</emphasis><emphasis><subsystem id/debug mask></emphasis></para>
+ <para> <literal>filter <replaceable>subsystem_id|debug_mask</replaceable></literal></para>
</entry>
<entry>
<para> Filters kernel debug messages by subsystem or mask.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">show</emphasis><emphasis><subsystem id/debug mask></emphasis></para>
+ <para> <literal>show <replaceable>subsystem_id|debug_mask</replaceable></literal></para>
</entry>
<entry>
<para> Shows specific types of messages.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">debug_list</emphasis><emphasis><subs/types></emphasis></para>
+ <para> <literal>debug_list <replaceable>subsystems|types</replaceable></literal></para>
</entry>
<entry>
<para> Lists all subsystem and debug types.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">modules</emphasis><emphasis><path></emphasis></para>
+ <para> <literal>modules <replaceable>path</replaceable></literal></para>
</entry>
<entry>
<para> Provides GDB-friendly module information.</para>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">--device</emphasis></para>
+ <para> <literal>--device</literal></para>
</entry>
<entry>
<para> Device to be used for the operation (specified by name or number). See device_list.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--ignore_errors | ignore_errors</emphasis></para>
+ <para> <literal>--ignore_errors | ignore_errors</literal></para>
</entry>
<entry>
<para> Ignores errors during script processing.</para>
</section>
<section remap="h5">
<title>Examples</title>
- <para><emphasis role="bold">lctl</emphasis></para>
+ <para><literal>lctl</literal></para>
<screen>$ lctl
lctl > dl
0 UP mgc MGC192.168.0.20@tcp btbb24e3-7deb-2ffa-eab0-44dffe00f692 5
<para><emphasis role="bold">Option</emphasis></para>
</entry>
<entry>
- <para><emphasis role="bold"> Description</emphasis></para>
+ <para><emphasis role="bold">Description</emphasis></para>
</entry>
</row>
</thead>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">-h</emphasis></para>
+ <para> <literal>-h</literal></para>
</entry>
<entry>
<para> Prints a help message</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">-v</emphasis></para>
+ <para> <literal>-v</literal></para>
</entry>
<entry>
<para> Increases verbosity</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">-d</emphasis><emphasis> directory</emphasis></para>
+ <para> <literal>-d <replaceable>directory</replaceable></literal></para>
</entry>
<entry>
<para> Sets the lost and found directory path</para>
</section>
<section remap="h5">
<title>Description</title>
- <para>The llobdstat utility displays a line of OST statistics for the given ost_name every interval seconds. It should be run directly on an OSS node. Type <emphasis>CTRL-C</emphasis> to stop statistics printing.</para>
+ <para>The llobdstat utility displays a line of OST statistics for the given ost_name every interval seconds. It should be run directly on an OSS node. Type <literal>CTRL-C</literal> to stop statistics printing.</para>
</section>
<section remap="h5">
<title>Example</title>
</section>
<section remap="h5">
<title>Files</title>
- <screen>/proc/fs/lustre/obdfilter/<ostname>/stats</screen>
+ <screen>/proc/fs/lustre/obdfilter/<replaceable>ostname</replaceable>/stats</screen>
</section>
</section>
<section xml:id="dbdoclet.50438219_90386">
<para>The llstat utility displays Lustre statistics.</para>
<section remap="h5">
<title>Synopsis</title>
- <screen>llstat [-c] [-g] [-i interval] stats_file
-</screen>
+ <screen>llstat [-c] [-g] [-i <replaceable>interval</replaceable>] <replaceable>stats_file
+</replaceable></screen>
</section>
<section remap="h5">
<title>Description</title>
- <para>The llstat utility displays statistics from any of the Lustre statistics files that share a common format and are updated at <emphasis>interval</emphasis> seconds. To stop statistics printing, use <emphasis>ctrl</emphasis>-<emphasis>c.</emphasis></para>
+ <para>The llstat utility displays statistics from any of the Lustre statistics files that share a common format and are updated at <literal>interval</literal> seconds. To stop statistics printing, use <literal>ctrl</literal>-<literal>c.</literal></para>
</section>
<section remap="h5">
<title>Options</title>
<para><emphasis role="bold">Option</emphasis></para>
</entry>
<entry>
- <para><emphasis role="bold"> Description</emphasis></para>
+ <para><emphasis role="bold">Description</emphasis></para>
</entry>
</row>
</thead>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">-c</emphasis></para>
+ <para> <literal>-c</literal></para>
</entry>
<entry>
<para> Clears the statistics file.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">-i</emphasis></para>
+ <para> <literal>-i</literal></para>
</entry>
<entry>
<para> Specifies the polling period (in seconds).</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">-g</emphasis></para>
+ <para> <literal>-g</literal></para>
</entry>
<entry>
<para> Specifies graphable output format.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">-h</emphasis></para>
+ <para> <literal>-h</literal></para>
</entry>
<entry>
<para> Displays help information.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">stats_file</emphasis></para>
+ <para> <literal>stats_file</literal></para>
</entry>
<entry>
- <para> Specifies either the full path to a statistics file or the shorthand reference, <emphasis role="bold">mds</emphasis> or <emphasis role="bold">ost</emphasis></para>
+ <para> Specifies either the full path to a statistics file or the shorthand reference, <literal>mds</literal> or <literal>ost</literal></para>
</entry>
</row>
</tbody>
<para>The llverdev verifies a block device is functioning properly over its full size.</para>
<section remap="h5">
<title>Synopsis</title>
- <screen>llverdev [-c <emphasis>chunksize</emphasis>] [-f] [-h] [-o <emphasis>offset</emphasis>] [-l] [-p] [-r] [-t <emphasis>timestamp</emphasis>] [-v] [-w] <emphasis>device</emphasis></screen>
+ <screen>llverdev [-c <replaceable>chunksize</replaceable>] [-f] [-h] [-o <replaceable>offset</replaceable>] [-l] [-p] [-r] [-t <replaceable>timestamp</replaceable>] [-v] [-w] <replaceable>device</replaceable></screen>
</section>
<section remap="h5">
<title>Description</title>
<tbody>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">-c|--chunksize</emphasis></para>
+ <para> <literal>-c|--chunksize</literal></para>
</entry>
<entry>
<para> I/O chunk size in bytes (default value is 1048576).</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">-f|--force</emphasis></para>
+ <para> <literal>>-f|--force</literal>></para>
</entry>
<entry>
<para> Forces the test to run without a confirmation that the device will be overwritten and all data will be permanently destroyed.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">-h|--help</emphasis></para>
+ <para> <literal>-h|--help</literal></para>
</entry>
<entry>
<para> Displays a brief help message.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">-o</emphasis><emphasis>offset</emphasis></para>
+ <para> <literal>-o <replaceable>>offset</replaceable></literal></para>
</entry>
<entry>
<para> Offset (in kilobytes) of the start of the test (default value is 0).</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">-l|--long</emphasis></para>
+ <para> <literal>-l|--long</literal></para>
</entry>
<entry>
<para> Runs a full check, writing and then reading and verifying every block on the disk.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">-p|--partial</emphasis></para>
+ <para> <literal>-p|--partial</literal></para>
</entry>
<entry>
<para> Runs a partial check, only doing periodic checks across the device (1 GB steps).</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">-r|--read</emphasis></para>
+ <para> <literal>-r|--read</literal></para>
</entry>
<entry>
- <para> Runs the test in read (verify) mode only, after having previously run the test in <emphasis role="bold">-w</emphasis> mode.</para>
+ <para> Runs the test in read (verify) mode only, after having previously run the test in <literal>-w</literal> mode.</para>
</entry>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">-t</emphasis><emphasis>timestamp</emphasis></para>
+ <para> <literal>-t <replaceable>timestamp</replaceable></literal></para>
</entry>
<entry>
<para> Sets the test start time as printed at the start of a previously-interrupted test to ensure that validation data is the same across the entire filesystem (default value is the current time()).</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">-v|--verbose</emphasis></para>
+ <para> <literal>-v|--verbose</literal></para>
</entry>
<entry>
<para> Runs the test in verbose mode, listing each read and write operation.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">-w|--write</emphasis></para>
+ <para> <literal>-w|--write</literal></para>
</entry>
<entry>
<para> Runs the test in write (test-pattern) mode (default runs both read and write).</para>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">-e|--enumerate</emphasis></para>
+ <para> <literal>-e|--enumerate</literal></para>
</entry>
<entry>
<para> Causes lshowmount to list each client mounted on a separate line instead of trying to compress the list of clients into a hostrange string.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">-h|--help</emphasis></para>
+ <para> <literal>-h|--help</literal></para>
</entry>
<entry>
<para> Causes lshowmount to print out a usage message.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">-l|--lookup</emphasis></para>
+ <para> <literal>-l|--lookup</literal></para>
</entry>
<entry>
<para> Causes lshowmount to try to look up the hostname for NIDs that look like IP addresses.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">-v|--verbose</emphasis></para>
+ <para> <literal>-v|--verbose</literal></para>
</entry>
<entry>
<para> Causes lshowmount to output export information for each service instead of only displaying the aggregate information for all Lustre services on the server.</para>
</section>
<section remap="h5">
<title>Files</title>
- <screen>/proc/fs/lustre/mgs/<server>/exports/<uuid>/nid /proc/fs/lustre/mds/<server>/expo\
-rts/<uuid>/nid /proc/fs/lustre/obdfilter/<server>/exports/<uuid>/nid</screen>
+ <screen>/proc/fs/lustre/mgs/<replaceable>server</replaceable>/exports/<replaceable>uuid</replaceable>/nid
+/proc/fs/lustre/mds/<replaceable>server</replaceable>/exports/<replaceable>uuid</replaceable>/nid
+/proc/fs/lustre/obdfilter/<replaceable>server</replaceable>/exports/<replaceable>uuid</replaceable>/nid</screen>
</section>
</section>
<section xml:id="dbdoclet.50438219_90218">
<para>The lustre_rsync utility synchronizes (replicates) a Lustre file system to a target file system.</para>
<section remap="h5">
<title>Synopsis</title>
- <screen>lustre_rsync --source|-s <src> --target|-t <tgt>
- --mdt|-m <mdt> [--user|-u <user id>]
- [--xattr|-x <yes|no>] [--verbose|-v]
- [--statuslog|-l <log>] [--dry-run] [--abort-on-err]
+ <screen>lustre_rsync --source|-s <replaceable>src</replaceable> --target|-t <replaceable>tgt</replaceable>
+ --mdt|-m <replaceable>mdt</replaceable> [--user|-u <replaceable>userid</replaceable>]
+ [--xattr|-x <replaceable>yes|no</replaceable>] [--verbose|-v]
+ [--statuslog|-l <replaceable>log</replaceable>] [--dry-run] [--abort-on-err]
-lustre_rsync --statuslog|-l <log>
+lustre_rsync --statuslog|-l <replaceable>log</replaceable>
-lustre_rsync --statuslog|-l <log> --source|-s <source>
- --target|-t <tgt> --mdt|-m <mdt></screen>
+lustre_rsync --statuslog|-l <replaceable>log</replaceable> --source|-s <replaceable>source</replaceable>
+ --target|-t <replaceable>tgt</replaceable> --mdt|-m <replaceable>mdt</replaceable></screen>
</section>
<section remap="h5">
<title>Description</title>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">--source=</emphasis><emphasis><src></emphasis></para>
+ <para> <literal>--source=<replaceable>src</replaceable></literal></para>
</entry>
<entry>
<para> The path to the root of the Lustre file system (source) which will be synchronized. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--target=</emphasis><emphasis><tgt></emphasis></para>
+ <para> <literal>--target=<replaceable>tgt</replaceable></literal></para>
</entry>
<entry>
<para> The path to the root where the source file system will be synchronized (target). This is a mandatory option if the status log created during a previous synchronization operation (--statuslog) is not specified. This option can be repeated if multiple synchronization targets are desired.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--mdt=</emphasis><emphasis><mdt></emphasis></para>
+ <para> <literal>--mdt=<replaceable>mdt</replaceable></literal></para>
</entry>
<entry>
<para> The metadata device to be synchronized. A changelog user must be registered for this device. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--user=</emphasis><emphasis><user id></emphasis></para>
+ <para> <literal>--user=<replaceable>userid</replaceable></literal></para>
</entry>
<entry>
<para> The changelog user ID for the specified MDT. To use lustre_rsync, the changelog user must be registered. For details, see the changelog_register parameter in the lctl man page. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--statuslog=</emphasis><emphasis><log></emphasis></para>
+ <para> <literal>--statuslog=<replaceable>log</replaceable></literal></para>
</entry>
<entry>
<para> A log file to which synchronization status is saved. When lustre_rsync starts, the state of a previous replication is read from here. If the status log from a previous synchronization operation is specified, otherwise mandatory options like --source, --target and --mdt options may be skipped. By specifying options like --source, --target and/or --mdt in addition to the --statuslog option, parameters in the status log can be overridden. Command line options take precedence over options in the status log.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--xattr</emphasis><emphasis><yes|no></emphasis></para>
+ <para> <literal>--xattr<replaceable>yes|no</replaceable></literal></para>
</entry>
<entry>
<para> Specifies whether extended attributes (xattrs) are synchronized or not. The default is to synchronize extended attributes.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--verbose</emphasis></para>
+ <para> <literal>--verbose</literal></para>
</entry>
<entry>
<para> Produces a verbose output.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--dry-run</emphasis></para>
+ <para> <literal>--dry-run</literal></para>
</entry>
<entry>
<para> Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--abort-on-err</emphasis></para>
+ <para> <literal>--abort-on-err</literal></para>
</entry>
<entry>
<para> Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para>
<para>The mkfs.lustre utility formats a disk for a Lustre service.</para>
<section remap="h5">
<title>Synopsis</title>
- <screen>mkfs.lustre <target_type> [options] device</screen>
- <para>where <target_type> is one of the following:</para>
+ <screen>mkfs.lustre <replaceable>target_type</replaceable> [options] <replaceable>device</replaceable></screen>
+ <para>where <replaceable>target_type</replaceable> is one of the following:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">--ost</emphasis></para>
+ <para> <literal>--ost</literal></para>
</entry>
<entry>
<para> Object Storage Target (OST)</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--mdt</emphasis></para>
+ <para> <literal>--mdt</literal></para>
</entry>
<entry>
<para> Metadata Storage Target (MDT)</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--network=</emphasis><emphasis>net,...</emphasis></para>
+ <para> <literal>--network=<replaceable>net,...</replaceable></literal></para>
</entry>
<entry>
<para> Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--mgs</emphasis></para>
+ <para> <literal>--mgs</literal></para>
</entry>
<entry>
- <para> Configuration Management Service (MGS), one per site. This service can be combined with one <emphasis role="bold">--mdt</emphasis> service by specifying both types.</para>
+ <para> Configuration Management Service (MGS), one per site. This service can be combined with one <literal>--mdt</literal> service by specifying both types.</para>
</entry>
</row>
</tbody>
<tbody>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--backfstype</emphasis>=<emphasis>fstype</emphasis></para>
+ <para> <literal>--backfstype=<replaceable>fstype</replaceable></literal></para>
</entry>
<entry>
<para> Forces a particular format for the backing file system (such as ext3, ldiskfs).</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--comment</emphasis>=<emphasis>comment</emphasis></para>
+ <para> <literal>--comment=<replaceable>comment</replaceable></literal></para>
</entry>
<entry>
<para> Sets a user comment about this disk, ignored by Lustre.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--device-size</emphasis>=<emphasis>KB</emphasis></para>
+ <para> <literal>--device-size=<replaceable>#</replaceable>>KB</literal></para>
</entry>
<entry>
<para> Sets the device size for loop devices.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--dryrun</emphasis></para>
+ <para> <literal>--dryrun</literal></para>
</entry>
<entry>
<para> Only prints what would be done; it does not affect the disk.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--failnode</emphasis>=<emphasis>nid</emphasis>,...</para>
+ <para> <literal>--failnode=<replaceable>nid,...</replaceable></literal></para>
</entry>
<entry>
<para> Sets the NID(s) of a failover partner. This option can be repeated as needed.</para>
- <para><emphasis role="bold">CAUTION</emphasis>: Cannot be used with <emphasis role="bold">--servicenode</emphasis>.</para>
+ <warning><para>This cannot be used with <literal>--servicenode</literal>.</para></warning>
</entry>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--servicenode</emphasis>=<emphasis>nid</emphasis>,...</para>
+ <para> <literal>--servicenode=<replaceable>nid,...</replaceable></literal></para>
</entry>
<entry>
<para> Sets the NID(s) of all service node, including failover partner as well as primary node service nids. This option can be repeated as needed.</para>
- <para><emphasis role="bold">CAUTION</emphasis>: Cannot be used with <emphasis role="bold">--failnode</emphasis>.</para>
+ <warning><para>This cannot be used with <literal>--failnode</literal>.</para></warning>
</entry>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--fsname</emphasis>=<emphasis>filesystem_name</emphasis></para>
+ <para> <literal>--fsname=<replaceable>filesystem_name</replaceable></literal></para>
</entry>
<entry>
<para> The Lustre file system of which this service/node will be a part. The default file system name is 'lustreâ€.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--index</emphasis>=<emphasis>index</emphasis></para>
+ <para> <literal>--index=<replaceable>index</replaceable></literal></para>
</entry>
<entry>
<para> Specifies the OST or MDT number. This should always be used when formatting OSTs, in order to ensure that there is a simple mapping between the OST index and the OSS node and device it is located on.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--mkfsoptions</emphasis>=<emphasis>opts</emphasis></para>
+ <para> <literal>--mkfsoptions=<replaceable>opts</replaceable></literal></para>
</entry>
<entry>
<para> Formats options for the backing file system. For example, ext3 options could be set here.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--mountfsoptions</emphasis>=<emphasis>opts</emphasis></para>
+ <para> <literal>--mountfsoptions=<replaceable>opts</replaceable></literal></para>
</entry>
<entry>
<para> Sets the mount options used when the backing file system is mounted.</para>
- <para><emphasis role="bold">CAUTION</emphasis>: Unlike earlier versions of mkfs.lustre, this version completely replaces the default mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para>
+ <warning><para>Unlike earlier versions of mkfs.lustre, this version completely replaces the default mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para></warning>
<para>The defaults for ldiskfs are:</para>
- <para>OST: <emphasis>errors=remount-ro</emphasis>;</para>
- <para>MGS/MDT: <emphasis>errors=remount-ro,iopen_nopriv,user_xattr</emphasis></para>
+ <para>OST: <literal>errors=remount-ro</literal>;</para>
+ <para>MGS/MDT: <literal>errors=remount-ro,iopen_nopriv,user_xattr</literal></para>
<para>Do not alter the default mount options unless you know what you are doing.</para>
</entry>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--network</emphasis>=<emphasis>net</emphasis>,...</para>
+ <para> <literal>--network=<replaceable>net,...</replaceable></literal></para>
<para> </para>
</entry>
<entry>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--mgsnode</emphasis>=<emphasis>nid</emphasis>,...</para>
+ <para> <literal>--mgsnode=<replaceable>nid,...</replaceable></literal></para>
</entry>
<entry>
<para> Sets the NIDs of the MGS node, required for all targets other than the MGS.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--param</emphasis><emphasis>key</emphasis>=<emphasis>value</emphasis></para>
+ <para> <literal>--param <replaceable>key</replaceable>=<replaceable>value</replaceable></literal></para>
</entry>
<entry>
- <para> Sets the permanent parameter <emphasis>key</emphasis> to value <emphasis>value</emphasis>. This option can be repeated as necessary. Typical options might include:</para>
+ <para> Sets the permanent parameter <replaceable>key</replaceable> to value <replaceable>value</replaceable>. This option can be repeated as necessary. Typical options might include:</para>
</entry>
</row>
<row>
<para>  </para>
</entry>
<entry>
- <para> --<emphasis>param sys.timeout=40</emphasis></para>
+ <para> <literal>--param sys.timeout=40</literal>></para>
</entry>
<entry>
<para> System obd timeout.</para>
<para>  </para>
</entry>
<entry>
- <para> --<emphasis>param lov.stripesize=2M</emphasis></para>
+ <para> <literal>--param lov.stripesize=2M</literal></para>
</entry>
<entry>
<para> Default stripe size.</para>
<para>  </para>
</entry>
<entry>
- <para> --<emphasis>param lov.stripecount=2</emphasis></para>
+ <para> <literal>param lov.stripecount=2</literal></para>
</entry>
<entry>
<para> Default stripe count.</para>
<para>  </para>
</entry>
<entry>
- <para> --<emphasis>param failover.mode=failout</emphasis></para>
+ <para> <literal>--param failover.mode=failout</literal></para>
</entry>
<entry>
<para> Returns errors instead of waiting for recovery.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--quiet</emphasis></para>
+ <para> <literal>--quiet</literal></para>
</entry>
<entry>
<para> Prints less information.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--reformat</emphasis></para>
+ <para> <literal>--reformat</literal></para>
</entry>
<entry>
<para> Reformats an existing Lustre disk.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--stripe-count-hint=stripes</emphasis></para>
+ <para> <literal>--stripe-count-hint=stripes</literal></para>
</entry>
<entry>
<para> Used to optimize the MDT's inode size.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <emphasis role="bold">--verbose</emphasis></para>
+ <para> <literal>--verbose</literal></para>
</entry>
<entry>
<para> Prints more information.</para>
</section>
<section remap="h5">
<title>Examples</title>
- <para>Creates a combined MGS and MDT for file system <emphasis role="bold">testfs</emphasis> on, e.g., node <emphasis role="bold">cfs21</emphasis>:</para>
+ <para>Creates a combined MGS and MDT for file system <literal>testfs</literal> on, e.g., node <literal>cfs21</literal>:</para>
<screen>mkfs.lustre --fsname=testfs --mdt --mgs /dev/sda1</screen>
- <para>Creates an OST for file system <emphasis role="bold">testfs</emphasis> on any node (using the above MGS):</para>
+ <para>Creates an OST for file system <literal>testfs</literal> on any node (using the above MGS):</para>
<screen>mkfs.lustre --fsname=testfs --mgsnode=cfs21@tcp0 --ost --index=0 /dev/sdb</screen>
- <para>Creates a standalone MGS on, e.g., node <emphasis role="bold">cfs22</emphasis>:</para>
+ <para>Creates a standalone MGS on, e.g., node <literal>cfs22</literal>:</para>
<screen>mkfs.lustre --mgs /dev/sda1</screen>
- <para>Creates an MDT for file system <emphasis role="bold">myfs1</emphasis> on any node (using the above MGS):</para>
+ <para>Creates an MDT for file system <literal>myfs1</literal> on any node (using the above MGS):</para>
<screen>mkfs.lustre --fsname=myfs1 --mdt --mgsnode=cfs22@tcp0 /dev/sda2</screen>
</section>
<section remap="h5">
<tbody>
<row>
<entry>
- <para> <emphasis><mgsspec>:/<fsname></emphasis></para>
+ <para> <literal><replaceable>mgs_nid</replaceable>:/<replaceable>fsname</replaceable></literal></para>
<para> </para>
</entry>
<entry>
- <para> Mounts the Lustre file system named <emphasis>fsname</emphasis> on the client by contacting the Management Service at <emphasis>mgsspec</emphasis> on the pathname given by <emphasis>directory</emphasis>. The format for <emphasis>mgsspec</emphasis> is defined below. A mounted client file system appears in fstab(5) and is usable, like any local file system, and provides a full POSIX-compliant interface.</para>
+ <para> Mounts the Lustre file system named <literal>fsname</literal> on the client by contacting the Management Service at <literal>mgsspec</literal> on the pathname given by <literal>directory</literal>. The format for <literal>mgsspec</literal> is defined below. A mounted client file system appears in fstab(5) and is usable, like any local file system, and provides a full POSIX-compliant interface.</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis><disk_device></emphasis></para>
+ <para> <replaceable>block_device</replaceable></para>
</entry>
<entry>
- <para> Starts the target service defined by the mkfs.lustre command on the physical disk <emphasis>disk_device</emphasis>. A mounted target service file system is only useful for df(1) operations and appears in fstab(5) to show the device is in use.</para>
+ <para> Starts the target service defined by the mkfs.lustre command on the physical disk <replaceable>block_device</replaceable>. A mounted target service file system is only useful for df(1) operations and appears in fstab(5) to show the device is in use.</para>
</entry>
</row>
</tbody>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold"><mgsspec>:</emphasis>=<emphasis><mgsnode>[:<mgsnode>]</emphasis></para>
+ <para> <literal>mgsspec:=<replaceable>mgsnode</replaceable>[:<replaceable>mgsnode</replaceable>]</literal></para>
<para> </para>
</entry>
<entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold"><mgsnode>:</emphasis>=<emphasis><mgsnid>[,<mgsnid>]</emphasis></para>
+ <para> <literal>mgsnode:=<replaceable>mgsnid</replaceable>[,<replaceable>mgsnid</replaceable>]</literal></para>
</entry>
<entry>
<para> Each node may be specified by a comma-separated list of NIDs.</para>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">flock</emphasis></para>
+ <para> <literal>flock</literal></para>
</entry>
<entry>
<para> Enables full flock support, coherent across all client nodes.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">localflock</emphasis></para>
+ <para> <literal>localflock</literal></para>
</entry>
<entry>
<para> Enables local flock support, using only client-local flock (faster, for applications that require flock, but do not run on multiple nodes).</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">noflock</emphasis></para>
+ <para> <literal>noflock</literal></para>
</entry>
<entry>
- <para> Disables flock support entirely. Applications calling flock get an error. It is up to the administrator to choose either <emphasis role="bold">localflock</emphasis> (fastest, low impact, not coherent between nodes) or <emphasis role="bold">flock</emphasis> (slower, performance impact for use, coherent between nodes).</para>
+ <para> Disables flock support entirely. Applications calling flock get an error. It is up to the administrator to choose either <literal>localflock</literal> (fastest, low impact, not coherent between nodes) or <literal>flock</literal> (slower, performance impact for use, coherent between nodes).</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">user_xattr</emphasis></para>
+ <para> <literal>user_xattr</literal></para>
</entry>
<entry>
<para> Enables get/set of extended attributes by regular users. See the attr(5) manual page.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">nouser_xattr</emphasis></para>
+ <para> <literal>nouser_xattr</literal></para>
</entry>
<entry>
<para> Disables use of extended attributes by regular users. Root and system processes can still use extended attributes.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">acl</emphasis></para>
+ <para> <literal>acl</literal></para>
</entry>
<entry>
<para> Enables POSIX Access Control List support. See the acl(5) manual page.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">noacl</emphasis></para>
+ <para> <literal>noacl</literal></para>
</entry>
<entry>
<para> Disables Access Control List support.</para>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">nosvc</emphasis></para>
+ <para> <literal>nosvc</literal></para>
</entry>
<entry>
<para> Starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">nomsgs</emphasis></para>
+ <para> <literal>nomsgs</literal></para>
</entry>
<entry>
<para> Starts only the MDT (with a co-located MGS), without starting the MGS.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">exclude=<ostlist></emphasis></para>
+ <para> <literal>exclude=<replaceable>ostlist</replaceable></literal></para>
</entry>
<entry>
<para> Starts a client or MDT with a colon-separated list of known inactive OSTs.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">nosvc</emphasis></para>
+ <para> <literal>nosvc</literal></para>
</entry>
<entry>
<para> Only starts the MGC (and MGS, if co-located) for a target service, not the actual service.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">nomsgs</emphasis></para>
+ <para> <literal>nomsgs</literal></para>
</entry>
<entry>
<para> Starts a MDT with a co-located MGS, without starting the MGS.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">exclude</emphasis>=<emphasis>ostlist</emphasis></para>
+ <para> <literal>exclude=<replaceable>ostlist</replaceable></literal></para>
</entry>
<entry>
<para> Starts a client or MDT with a (colon-separated) list of known inactive OSTs.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">abort_recov</emphasis></para>
+ <para> <literal>abort_recov</literal></para>
</entry>
<entry>
<para> Aborts client recovery and starts the target service immediately.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">md_stripe_cache_size</emphasis></para>
+ <para> <literal>md_stripe_cache_size</literal></para>
</entry>
<entry>
<para> Sets the stripe cache size for server-side disk with a striped RAID configuration.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">recovery_time_soft</emphasis>=<emphasis>timeout</emphasis></para>
+ <para> <literal>recovery_time_soft=<replaceable>timeout</replaceable></literal></para>
</entry>
<entry>
- <para> Allows <emphasis>timeout</emphasis> seconds for clients to reconnect for recovery after a server crash. This timeout is incrementally extended if it is about to expire and the server is still handling new connections from recoverable clients. The default soft recovery timeout is 300 seconds (5 minutes).</para>
+ <para> Allows <literal>timeout</literal> seconds for clients to reconnect for recovery after a server crash. This timeout is incrementally extended if it is about to expire and the server is still handling new connections from recoverable clients. The default soft recovery timeout is 300 seconds (5 minutes).</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">recovery_time_hard</emphasis>=<emphasis>timeout</emphasis></para>
+ <para> <literal>recovery_time_hard=<replaceable>timeout</replaceable></literal></para>
</entry>
<entry>
- <para> The server is allowed to incrementally extend its timeout up to a hard maximum of <emphasis>timeout</emphasis> seconds. The default hard recovery timeout is set to 900 seconds (15 minutes).</para>
+ <para> The server is allowed to incrementally extend its timeout up to a hard maximum of <literal>timeout</literal> seconds. The default hard recovery timeout is set to 900 seconds (15 minutes).</para>
</entry>
</row>
</tbody>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">results_filename</emphasis></para>
+ <para> <literal>results_filename</literal></para>
</entry>
<entry>
<para> Output generated by plot-llstat</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">parameter_index</emphasis></para>
+ <para> <literal>parameter_index</literal></para>
<para> </para>
</entry>
<entry>
<para>The routerstat utility prints Lustre router statistics.</para>
<section remap="h5">
<title>Synopsis</title>
- <screen>routerstat [interval]</screen>
+ <screen>routerstat [<replaceable>interval</replaceable>]</screen>
</section>
<section remap="h5">
<title>Description</title>
- <para>The routerstat utility watches LNET router statistics. If no <emphasis>interval</emphasis> is specified, then statistics are sampled and printed only one time. Otherwise, statistics are sampled and printed at the specified <emphasis>interval</emphasis> (in seconds).</para>
+ <para>The routerstat utility watches LNET router statistics. If no <literal><replaceable>interval</replaceable></literal> is specified, then statistics are sampled and printed only one time. Otherwise, statistics are sampled and printed at the specified <literal><replaceable>interval</replaceable></literal> (in seconds).</para>
</section>
<section remap="h5">
<title>Options</title>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">M</emphasis></para>
+ <para> <literal>M</literal></para>
</entry>
<entry>
<para> msgs_alloc(msgs_max)</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">E</emphasis></para>
+ <para> <literal>E</literal></para>
</entry>
<entry>
<para> errors</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">S</emphasis></para>
+ <para> <literal>S</literal></para>
</entry>
<entry>
<para> send_count/send_length</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">R</emphasis></para>
+ <para> <literal>R</literal></para>
</entry>
<entry>
<para> recv_count/recv_length</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">F</emphasis></para>
+ <para> <literal>F</literal></para>
</entry>
<entry>
<para> route_count/route_length</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">D</emphasis></para>
+ <para> <literal>D</literal></para>
</entry>
<entry>
<para> drop_count/drop_length</para>
<para>The tunefs.lustre utility modifies configuration information on a Lustre target disk.</para>
<section remap="h5">
<title>Synopsis</title>
- <screen>tunefs.lustre [options] <device></screen>
+ <screen>tunefs.lustre [options] <replaceable>/dev/device</replaceable></screen>
</section>
<section remap="h5">
<title>Description</title>
<para>Changes made here affect a file system only when the target is mounted the next time.</para>
</caution>
<para>With tunefs.lustre, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old tunefs.lustre parameters and just use newly-specified parameters, run:</para>
- <screen>$ tunefs.lustre --erase-params --param=<new parameters> </screen>
- <para>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <obd|fsname>.<obdtype>.<proc_file_name>=<value>. For example:</para>
+ <screen>$ tunefs.lustre --erase-params --param=<replaceable>new_parameters</replaceable> </screen>
+ <para>The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as <replaceable>{obd|fsname}.obdtype.proc_file_name=value</replaceable>. For example:</para>
<screen>$ tunefs.lustre --param mdt.identity_upcall=NONE /dev/sda1</screen>
</section>
<section remap="h5">
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">--comment</emphasis>=<emphasis>comment</emphasis></para>
+ <para> <literal>--comment=<replaceable>comment</replaceable></literal></para>
</entry>
<entry>
<para> Sets a user comment about this disk, ignored by Lustre.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--dryrun</emphasis></para>
+ <para> <literal>--dryrun</literal></para>
</entry>
<entry>
<para> Only prints what would be done; does not affect the disk.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--erase-params</emphasis></para>
+ <para> <literal>--erase-params</literal></para>
</entry>
<entry>
<para> Removes all previous parameter information.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--failnode</emphasis>=<emphasis>nid</emphasis>,...</para>
+ <para> <literal>--failnode=<replaceable>nid,...</replaceable></literal></para>
</entry>
<entry>
<para> Sets the NID(s) of a failover partner. This option can be repeated as needed.</para>
- <para><emphasis role="bold">CAUTION</emphasis>: Cannot be used with <emphasis role="bold">--servicenode</emphasis>.</para>
+ <warning><para>Cannot be used with <literal>--servicenode</literal>.</para></warning>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--servicenode</emphasis>=<emphasis>nid</emphasis>,...</para>
+ <para> <literal>--servicenode=<replaceable>nid,...</replaceable></literal></para>
</entry>
<entry>
<para> Sets the NID(s) of all service node, including failover partner as well as local service nids. This option can be repeated as needed.</para>
- <para><emphasis role="bold">CAUTION</emphasis>: Cannot be used with <emphasis role="bold">--failnode</emphasis>.</para>
+ <warning><para>: Cannot be used with <literal>--failnode</literal>.</para></warning>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--fsname</emphasis>=<emphasis>filesystem_name</emphasis></para>
+ <para> <literal>--fsname=<replaceable>filesystem_name</replaceable></literal></para>
</entry>
<entry>
<para> The Lustre file system of which this service will be a part. The default file system name is 'lustreâ€.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--index</emphasis>=index</para>
+ <para> <literal>--index=<replaceable>index</replaceable></literal></para>
</entry>
<entry>
<para> Forces a particular OST or MDT index.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--mountfsoptions</emphasis>=opts</para>
+ <para> <literal>--mountfsoptions=<replaceable>opts</replaceable></literal></para>
</entry>
<entry>
<para> Sets the mount options used when the backing file system is mounted.</para>
- <para><emphasis role="bold">CAUTION</emphasis>: Unlike earlier versions of tunefs.lustre, this version completely replaces the existing mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para>
+ <warning><para> Unlike earlier versions of tunefs.lustre, this version completely replaces the existing mount options with those specified on the command line, and issues a warning on stderr if any default mount options are omitted.</para></warning>
<para>The defaults for ldiskfs are:</para>
- <para>OST: <emphasis>errors=remount-ro,mballoc,extents</emphasis>;</para>
- <para>MGS/MDT: <emphasis>errors=remount-ro,iopen_nopriv,user_xattr</emphasis></para>
+ <para>OST: <literal>errors=remount-ro,mballoc,extents</literal>;</para>
+ <para>MGS/MDT: <literal>errors=remount-ro,iopen_nopriv,user_xattr</literal></para>
<para>Do not alter the default mount options unless you know what you are doing.</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--network</emphasis>=net,...</para>
+ <para> <literal>--network=<replaceable>net,...</replaceable></literal></para>
</entry>
<entry>
<para> Network(s) to which to restrict this OST/MDT. This option can be repeated as necessary.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--mgs</emphasis></para>
+ <para> <literal>--mgs</literal></para>
</entry>
<entry>
<para> Adds a configuration management service to this target.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--msgnode</emphasis>=<emphasis>nid</emphasis>,...</para>
+ <para> <literal>--msgnode=<replaceable>nid,...</replaceable></literal></para>
</entry>
<entry>
<para> Sets the NID(s) of the MGS node; required for all targets other than the MGS.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--nomgs</emphasis></para>
+ <para> <literal>--nomgs</literal></para>
</entry>
<entry>
<para> Removes a configuration management service to this target.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--quiet</emphasis></para>
+ <para> <literal>--quiet</literal></para>
</entry>
<entry>
<para> Prints less information.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--verbose</emphasis></para>
+ <para> <literal>--verbose</literal></para>
</entry>
<entry>
<para> Prints more information.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">--writeconf</emphasis></para>
+ <para> <literal>--writeconf</literal></para>
</entry>
<entry>
<para> Erases all configuration logs for the file system to which this MDT belongs, and regenerates them. This is dangerous operation. All clients must be unmounted and servers for this file system should be stopped. All targets (OSTs/MDTs) must then be restarted to regenerate the logs. No clients should be started until all targets have restarted.</para>
<para>The correct order of operations is:</para>
<para>* Unmount all clients on the file system</para>
<para>* Unmount the MDT and all OSTs on the file system</para>
- <para>* Run <emphasis role="bold">tunefs.lustre --writeconf <device></emphasis> on every server</para>
+ <para>* Run <literal>tunefs.lustre --writeconf <replaceable>device</replaceable></literal> on every server</para>
<para>* Mount the MDT and OSTs</para>
<para>* Mount the clients</para>
</entry>
<section remap="h5">
<title>Examples</title>
<para>Change the MGS's NID address. (This should be done on each target disk, since they should all contact the same MGS.)</para>
- <screen>tunefs.lustre --erase-param --mgsnode=<new_nid> --writeconf /dev/sda</screen>
+ <screen>tunefs.lustre --erase-param --mgsnode=<replaceable>new_nid</replaceable> --writeconf /dev/sda</screen>
<para>Add a failover NID location for this target.</para>
<screen>tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda </screen>
</section>
<title><indexterm><primary>utilities</primary><secondary>application profiling</secondary></indexterm>
Application Profiling Utilities</title>
<para>The following utilities are located in /usr/bin.</para>
- <para><emphasis role="bold">lustre_req_history.sh</emphasis></para>
+ <para><literal>lustre_req_history.sh</literal></para>
<para>The lustre_req_history.sh utility (run from a client), assembles as much Lustre RPC request history as possible from the local node and from the servers that were contacted, providing a better picture of the coordinated network activity.</para>
- <para><emphasis role="bold">llstat.sh</emphasis></para>
+ <para><literal>llstat.sh</literal></para>
<para>The llstat.sh utility handles a wider range of statistics files, and has command line switches to produce more graphable output.</para>
- <para><emphasis role="bold">plot-llstat.sh</emphasis></para>
+ <para><literal>plot-llstat.sh</literal></para>
<para>The plot-llstat.sh utility plots the output from llstat.sh using gnuplot.</para>
</section>
<section remap="h3">
<title>More /proc Statistics for Application Profiling</title>
<para>The following utilities provide additional statistics.</para>
- <para><emphasis role="bold">vfs_ops_stats</emphasis></para>
+ <para><literal>vfs_ops_stats</literal></para>
<para>The client vfs_ops_stats utility tracks Linux VFS operation calls into Lustre for a single PID, PPID, GID or everything.</para>
<screen>/proc/fs/lustre/llite/*/vfs_ops_stats
/proc/fs/lustre/llite/*/vfs_track_[pid|ppid|gid]
</screen>
- <para><emphasis role="bold">extents_stats</emphasis></para>
+ <para><literal>extents_stats</literal></para>
<para>The client extents_stats utility shows the size distribution of I/O calls from the client (cumulative and by process).</para>
<screen>/proc/fs/lustre/llite/*/extents_stats, extents_stats_per_process
</screen>
- <para><emphasis role="bold">offset_stats</emphasis></para>
+ <para><literal>offset_stats</literal></para>
<para>The client offset_stats utility shows the read/write seek activity of a client by offsets and ranges.</para>
<screen>/proc/fs/lustre/llite/*/offset_stats
</screen>
</screen>
<para>The loadgen utility prints periodic status messages; message output can be controlled with the verbose command.</para>
<para>To insure that a file can be written to (a requirement of write cache), OSTs reserve ("grants"), chunks of space for each newly-created file. A grant may cause an OST to report that it is out of space, even though there is plenty of space on the disk, because the space is "reserved" by other files. The loadgen utility estimates the number of simultaneous open files as the disk size divided by the grant size and reports that number when the write tests are first started.</para>
- <para><emphasis role="bold">Echo Server</emphasis></para>
+ <para>Echo Server</para>
<para>The loadgen utility can start an echo server. On another node, loadgen can specify the echo server as the device, thus creating a network-only test environment.</para>
<screen>loadgen> echosrv
loadgen> dl
quit
EOF
</screen>
- <para><emphasis role="bold">Feature Requests</emphasis></para>
+ <para>Feature Requests</para>
<para>The loadgen utility is intended to grow into a more comprehensive test tool; feature requests are encouraged. The current feature requests include:</para>
<itemizedlist>
<listitem>
<caution>
<para>Although they are stored in the CONFIGS directory, mountdata files do not use the config log format and will confuse llog_reader.</para>
</caution>
- <para><emphasis role="bold">See Also</emphasis></para>
- <para><xref linkend="dbdoclet.50438219_39574"/></para>
+ <para>See Also <xref linkend="dbdoclet.50438219_39574"/></para>
</section>
<section remap="h5">
<title><indexterm><primary>lr_reader</primary></indexterm>
<tbody>
<row>
<entry>
- <para> <emphasis role="bold">local mode</emphasis></para>
+ <para> <literal>local mode</literal></para>
</entry>
<entry>
<para> In this mode, locks are coherent on one node (a single-node flock), but not across all clients. To enable it, use -o localflock. This is a client-mount option.</para>
</row>
<row>
<entry>
- <para> <emphasis role="bold">consistent mode</emphasis></para>
+ <para> <literal>consistent mode</literal></para>
</entry>
<entry>
<para> In this mode, locks are coherent across all clients.</para>
<para> To enable it, use the -o flock. This is a client-mount option.</para>
- <para><emphasis role="bold">CAUTION</emphasis>: This mode affects the performance of the file being flocked and may affect stability, depending on the Lustre version used. Consider using a newer Lustre version which is more stable. If the consistent mode is enabled and no applications are using flock, then it has no effect.</para>
+ <warning><para>This mode affects the performance of the file being flocked and may affect stability, depending on the Lustre version used. Consider using a newer Lustre version which is more stable. If the consistent mode is enabled and no applications are using flock, then it has no effect.</para></warning>
</entry>
</row>
</tbody>
<title>Manually Starting LFSCK</title>
<section>
<title>Synopsis</title>
- <screen>lctl lfsck_start <-M | --device MDT_device> \
- [-e | --error error_handle] \
+ <screen>lctl lfsck_start -M | --device <replaceable>MDT_device</replaceable> \
+ [-e | --error <replaceable>error_handle</replaceable>] \
[-h | --help] \
- [-m | --method iteration_method] \
- [-n | --dryrun switch] \
+ [-m | --method <replaceable>iteration_method</replaceable>] \
+ [-n | --dryrun <replaceable>switch</replaceable>] \
[-r | --reset] \
- [-s | --speed speed_limit] \
- [-t | --type lfsck_type[,lfsck_type...]]
+ [-s | --speed <replaceable>speed_limit</replaceable>] \
+ [-t | --type <replaceable>lfsck_type[,lfsck_type...]</replaceable>]
</screen>
</section>
<section>
<title>Manually Stopping <literal>lfsck</literal></title>
<section>
<title>Synopsis</title>
- <screen>lctl lfsck_stop <-M | --device MDT_device> \
+ <screen>lctl lfsck_stop -M | --device <replaceable>MDT_device</replaceable> \
[-h | --help]
</screen>
</section>
<title>LFSCK status via <literal>procfs</literal></title>
<section>
<title>Synopsis</title>
- <screen>lctl get_param -n osd-ldisk.${FSNAME}-${MDT_device}.oi_scrub
+ <screen>lctl get_param -n osd-ldisk.<replaceable>FSNAME</replaceable>-<replaceable>MDT_device</replaceable>.oi_scrub
</screen>
</section>
<section>
<listitem>
<para>Install the kernel, modules and ldiskfs packages. For example:</para>
<screen>$ rpm -ivh
-kernel-lustre-smp-<ver> \
-kernel-ib-<ver> \
-lustre-modules-<ver> \
-lustre-ldiskfs-<ver></screen>
+kernel-lustre-smp-<replaceable>ver</replaceable> \
+kernel-ib-<replaceable>ver</replaceable> \
+lustre-modules-<replaceable>ver</replaceable> \
+lustre-ldiskfs-<replaceable>ver</replaceable></screen>
</listitem>
<listitem>
<para>Upgrade the utilities/userspace packages. For example:</para>
- <screen>$ rpm -Uvh lustre-<ver></screen>
+ <screen>$ rpm -Uvh lustre-<replaceable>ver</replaceable></screen>
</listitem>
<listitem>
<para>If a new <literal>e2fsprogs</literal> package is available, upgrade it. For example:</para>
- <screen>$ rpm -Uvh e2fsprogs-<ver>
+ <screen>$ rpm -Uvh e2fsprogs-<replaceable>ver</replaceable>
</screen>
<para>Use e2fsprogs-1.41.90-wc3 or later, available at:</para>
<para><link xl:href="http://downloads.whamcloud.com/public/e2fsprogs/latest/">http://downloads.whamcloud.com/public/e2fsprogs/latest/</link></para>
<orderedlist>
<listitem>
<para>Mount the OSTs (be sure to mount all OSTs). On each OSS node, run:</para>
- <screen>mount -a -t lustre</screen>
+ <screen>oss# mount -a -t lustre</screen>
<para>This command assumes that all OSTs are listed in the /etc/fstab file. If the OSTs are not in the /etc/fstab file, they need to be mounted individually by running the mount command:</para>
- <screen>mount -t lustre <block device name> <mount point> </screen>
+ <screen>oss# mount -t lustre <replaceable>/dev/block_device</replaceable> <replaceable>/mount_point</replaceable> </screen>
</listitem>
<listitem>
<para>Mount the MDT. On the MDS node, run:</para>
- <screen>mount -a -t lustre</screen>
+ <screen>mds# mount -a -t lustre</screen>
</listitem>
<listitem>
<para>Mount the file system on the clients. On each client node, run:</para>
- <screen>mount -a -t lustre</screen>
+ <screen>client# mount -a -t lustre</screen>
</listitem>
</orderedlist>
</listitem>
<section remap="h5">
<title>Synopsis</title>
<screen>lfs
-lfs changelog [--follow] <mdtname> [startrec [endrec]]
-lfs changelog_clear <mdtname> <id> <endrec>
-lfs check <mds|osts|servers>
-lfs df [-i] [-h] [--pool]-p <fsname>[.<pool>] [path]
+lfs changelog [--follow] <replaceable>mdt_name</replaceable> [startrec [endrec]]
+lfs changelog_clear <replaceable>mdt_name id endrec</replaceable>
+lfs check <replaceable>mds|osts|servers</replaceable>
+lfs df [-i] [-h] [--pool]-p <replaceable>fsname</replaceable>[.<replaceable>pool</replaceable>] [<replaceable>path</replaceable>]
lfs find [[!] --atime|-A [-+]N] [[!] --mtime|-M [-+]N]
- [[!] --ctime|-C [-+]N] [--maxdepth|-D N] [--name|-n <pattern>]
- [--print|-p] [--print0|-P] [[!] --obd|-O <uuid[s]>]
+ [[!] --ctime|-C [-+]N] [--maxdepth|-D N] [--name|-n <replaceable>pattern</replaceable>]
+ [--print|-p] [--print0|-P] [[!] --obd|-O <replaceable>ost_name</replaceable>[,<replaceable>ost_name...</replaceable>]]
[[!] --size|-S [+-]N[kMGTPE]] --type |-t {bcdflpsD}]
- [[!] --gid|-g|--group|-G <gname>|<gid>]
- [[!] --uid|-u|--user|-U <uname>|<uid>]
- <dirname|filename>
+ [[!] --gid|-g|--group|-G <replaceable>gname|gid</replaceable>]
+ [[!] --uid|-u|--user|-U <replaceable>uname|uid</replaceable>]
+ <replaceable>dirname|filename</replaceable>
lfs getname [-h]|[path...]
-lfs getstripe [--obd|-O <uuid>] [--quiet|-q] [--verbose|-v]
+lfs getstripe [--obd|-O <replaceable>ost_name</replaceable>] [--quiet|-q] [--verbose|-v]
[--count|-c] [--index|-i | --offset|-o]
[--size|-s] [--pool|-p] [--directory|-d]
- [--recursive|-r] [--raw|-R] [-M] <dirname|filename> ...
-lfs setstripe [--size|-s stripe_size] [--count|-c stripe_cnt]
- [--index|-i|--offset|-o start_ost_index]
- [--pool|-p <pool>]
- <dirname|filename>
-lfs setstripe -d <dir>
+ [--recursive|-r] [--raw|-R] [-M] <replaceable>dirname|filename</replaceable> ...
+lfs setstripe [--size|-s stripe_size] [--count|-c <replaceable>stripe_count</replaceable>]
+ [--index|-i|--offset|-o <replaceable>start_ost_index</replaceable>]
+ [--pool|-p <replaceable>pool</replaceable>]
+ <replaceable>dirname|filename</replaceable>
+lfs setstripe -d <replaceable>dir</replaceable>
lfs osts [path]
-lfs poollist <filesystem>[.<pool>]|<pathname>
-lfs quota [-q] [-v] [-o obd_uuid|-I ost_idx|-i mdt_idx]
- [-u <uname>|-u <uid>|-g <gname>|-g <gid>]
- <filesystem>
-lfs quota -t <-u|-g> <filesystem>
-lfs quotacheck [-ug] <filesystem>
-lfs quotachown [-i] <filesystem>
-lfs quotainv [-ug] [-f] <filesystem>
-lfs quotaon [-ugf] <filesystem>
-lfs quotaoff [-ug] <filesystem>
-lfs setquota <-u|--user|-g|--group> <uname|uid|gname|gid>
- [--block-softlimit <block-softlimit>]
- [--block-hardlimit <block-hardlimit>]
- [--inode-softlimit <inode-softlimit>]
- [--inode-hardlimit <inode-hardlimit>]
- <filesystem>
-lfs setquota <-u|--user|-g|--group> <uname|uid|gname|gid>
- [-b <block-softlimit>] [-B <block-hardlimit>]
- [-i <inode-softlimit>] [-I <inode-hardlimit>]
- <filesystem>
-lfs setquota -t <-u|-g>
- [--block-grace <block-grace>]
- [--inode-grace <inode-grace>]
- <filesystem>
-lfs setquota -t <-u|-g>
- [-b <block-grace>] [-i <inode-grace>]
- <filesystem>
+lfs poollist <replaceable>filesystem</replaceable>[.<replaceable>pool</replaceable>]|<replaceable>pathname</replaceable>
+lfs quota [-q] [-v] [-o <replaceable>obd_uuid</replaceable>|-I <replaceable>ost_idx</replaceable>|-i <replaceable>mdt_idx</replaceable>]
+ [-u <replaceable>username|uid|-g</replaceable> <replaceable>group|gid</replaceable>]
+ <replaceable>/mount_point</replaceable>
+lfs quota -t -u|-g <replaceable>/mount_point</replaceable>
+lfs quotacheck [-ug] <replaceable>/mount_point</replaceable>
+lfs quotachown [-i] <replaceable>/mount_point</replaceable>
+lfs quotainv [-ug] [-f] <replaceable>/mount_point</replaceable>
+lfs quotaon [-ugf] <replaceable>/mount_point</replaceable>
+lfs quotaoff [-ug] <replaceable>/mount_point</replaceable>
+lfs setquota {-u|--user|-g|--group} <replaceable>uname|uid|gname|gid</replaceable>
+ [--block-softlimit <replaceable>block_softlimit</replaceable>]
+ [--block-hardlimit <replaceable>block_hardlimit</replaceable>]
+ [--inode-softlimit <replaceable>inode_softlimit</replaceable>]
+ [--inode-hardlimit <replaceable>inode_hardlimit</replaceable>]
+ <replaceable>/mount_point</replaceable>
+lfs setquota -u|--user|-g|--group <replaceable>uname|uid|gname|gid</replaceable>
+ [-b <replaceable>block_softlimit</replaceable>] [-B <replaceable>block_hardlimit</replaceable>]
+ [-i <replaceable>inode-softlimit</replaceable>] [-I <replaceable>inode_hardlimit</replaceable>]
+ <replaceable>/mount_point</replaceable>
+lfs setquota -t -u|-g
+ [--block-grace <replaceable>block_grace</replaceable>]
+ [--inode-grace <replaceable>inode_grace</replaceable>]
+ <replaceable>/mount_point</replaceable>
+lfs setquota -t -u|-g
+ [-b <replaceable>block_grace</replaceable>] [-i <replaceable>inode_grace</replaceable>]
+ <replaceable>/mount_point</replaceable>
lfs help
</screen>
<note>
- <para>In the above example, the <literal><filesystem></literal> parameter refers to the mount point of the Lustre file system. The default mount point is <literal>/mnt/lustre</literal></para>
+ <para>In the above example, the <literal><replaceable>/mount_point</replaceable></literal> parameter refers to the mount point of the Lustre file system.</para>
</note>
<note>
<para>The old lfs quota output was very detailed and contained cluster-wide quota statistics (including cluster-wide limits for a user/group and cluster-wide usage for a user/group), as well as statistics for each MDS/OST. Now, <literal>lfs quota</literal> has been updated to provide only cluster-wide statistics, by default. To obtain the full report of cluster-wide limits, usage and statistics, use the <literal>-v</literal> option with <literal>lfs quota</literal>.</para>
<para> <literal>changelog_clear</literal></para>
</entry>
<entry>
- <para>Indicates that changelog records previous to <literal><endrec></literal> are no longer of interest to a particular consumer <literal><id></literal>, potentially allowing the MDT to free up disk space. An <literal><endrec></literal> of 0 indicates the current last record. Changelog consumers must be registered on the MDT node using <literal>lctl</literal>.</para>
+ <para>Indicates that changelog records previous to <literal><replaceable>endrec</replaceable></literal> are no longer of interest to a particular consumer <literal><replaceable>id</replaceable></literal>, potentially allowing the MDT to free up disk space. An <literal><replaceable>endrec</replaceable></literal> of 0 indicates the current last record. Changelog consumers must be registered on the MDT node using <literal>lctl</literal>.</para>
</entry>
</row>
<row>
</row>
<row>
<entry nameend="c2" namest="c1">
- <literal> df [-i] [-h] [--pool|-p <fsname>[.<pool>] [path] </literal>
+ <literal> df [-i] [-h] [--pool|-p <replaceable>fsname</replaceable>[.<replaceable>pool</replaceable>] [<replaceable>path</replaceable>] </literal>
</entry>
<entry>
<para>Report file system disk space usage or inode usage (with <literal>-i</literal>) of each MDT/OST or a subset of OSTs if a pool is specified with <literal>-p</literal>. By default, prints the usage of all mounted Lustre file systems. Otherwise, if path is specified, prints only the usage of that file system. If <literal>-h</literal> is given, the output is printed in human-readable format, using SI base-2 suffixes for <emphasis role="bold">M</emphasis>ega-, <emphasis role="bold">G</emphasis>iga-, <emphasis role="bold">T</emphasis>era-, <emphasis role="bold">P</emphasis>eta-, or <emphasis role="bold">E</emphasis>xabytes.</para>
<para>  </para>
</entry>
<entry>
- <para> <literal>--obd <uuid></literal></para>
+ <para> <literal>--obd <replaceable>ost_name</replaceable></literal></para>
</entry>
<entry>
<para>Lists files that have an object on a specific OST.</para>
<para>  </para>
</entry>
<entry>
- <para> <literal>--pool <pool></literal></para>
+ <para> <literal>--pool <replaceable>pool</replaceable></literal></para>
</entry>
<entry>
<para>Name of the pre-defined pool of OSTs (see <xref linkend="dbdoclet.50438219_38274"/>) that will be used for striping. The <literal>stripe_cnt</literal>, <literal>stripe_size</literal> and <literal>start_ost</literal> values are used as well. The start-ost value must be part of the pool or an error is returned.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <literal>quota [-q] [-v] [-o obd_uuid|-i mdt_idx|-I ost_idx] [-u|-g <uname>|<uid>|<gname>|<gid>] <filesystem></literal></para>
+ <para> <literal>quota [-q] [-v] [-o <replaceable>obd_uuid</replaceable>|-i <replaceable>mdt_idx</replaceable>|-I <replaceable>ost_idx</replaceable>] [-u|-g <replaceable>uname|uid|gname|gid]</replaceable> <replaceable>/mount_point</replaceable></literal></para>
<para> </para>
</entry>
<entry>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <literal>quota -t <-u|-g> <filesystem></literal></para>
+ <para> <literal>quota -t <replaceable>-u|-g</replaceable> <replaceable>/mount_point</replaceable></literal></para>
</entry>
<entry>
<para>Displays block and inode grace times for user (<literal>-u</literal>) or group (<literal>-g</literal>) quotas.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <literal>quotacheck [-ugf] <filesystem></literal></para>
+ <para> <literal>quotacheck [-ugf] <replaceable>/mount_point</replaceable></literal></para>
</entry>
<entry>
<para>Scans the specified file system for disk usage, and creates or updates quota files. Options specify quota for users (<literal>-u</literal>), groups (<literal>-g</literal>), and force (<literal>-f</literal>).</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <literal>quotaon [-ugf] <filesystem></literal></para>
+ <para> <literal>quotaon [-ugf] <replaceable>/mount_point</replaceable></literal></para>
</entry>
<entry>
<para>Turns on file system quotas. Options specify quota for users (<literal>-u</literal>), groups (<literal>-g</literal>), and force (<literal>-f</literal>).</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <literal>quotaoff [-ugf] <filesystem></literal></para>
+ <para> <literal>quotaoff [-ugf] <replaceable>/mount_point</replaceable></literal></para>
</entry>
<entry>
<para>Turns off file system quotas. Options specify quota for users (<literal>-u</literal>), groups (<literal>-g</literal>), and force (<literal>-f</literal>).</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <literal>quotainv [-ug] [-f] <filesystem></literal></para>
+ <para> <literal>quotainv [-ug] [-f] <replaceable>/mount_point</replaceable></literal></para>
</entry>
<entry>
<para> Clears quota files (administrative quota files if used without <literal>-f</literal>, operational quota files otherwise), all of their quota entries for users (<literal>-u</literal>) or groups (<literal>-g</literal>). After running <literal>quotainv</literal>, you must run <literal>quotacheck</literal> before using quotas.</para>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para> <literal>setquota <-u|-g> <uname>|<uid>|<gname>|<gid> [--block-softlimit <block-softlimit>] [--block-hardlimit <block-hardlimit>] [--inode-softlimit <inode-softlimit>] [--inode-hardlimit <inode-hardlimit>] <filesystem></literal></para>
+ <para> <literal>setquota -u|-g <replaceable>uname|uid|gname|gid}</replaceable> [--block-softlimit <replaceable>block_softlimit</replaceable>] [--block-hardlimit <replaceable>block_hardlimit</replaceable>] [--inode-softlimit <replaceable>inode_softlimit</replaceable>] [--inode-hardlimit <replaceable>inode_hardlimit</replaceable>] <replaceable>/mount_point</replaceable></literal></para>
</entry>
<entry>
<para>Sets file system quotas for users or groups. Limits can be specified with <literal>--{block|inode}-{softlimit|hardlimit}</literal> or their short equivalents <literal>-b</literal>, <literal>-B</literal>, <literal>-i</literal>, <literal>-I</literal>. Users can set 1, 2, 3 or 4 limits.<footnote>
</row>
<row>
<entry nameend="c2" namest="c1">
- <para><literal> setquota -t <-u|-g> [--block-grace <block-grace>] [--inode-grace <inode-grace>] <filesystem></literal></para>
+ <para><literal> setquota -t -u|-g [--block-grace <replaceable>block_grace</replaceable>] [--inode-grace <replaceable>inode_grace</replaceable>] <replaceable>/mount_point</replaceable></literal></para>
</entry>
<entry>
<para>Sets the file system quota grace times for users or groups. Grace time is specified in '<literal>XXwXXdXXhXXmXXs</literal>' format or as an integer seconds value. See <xref linkend="dbdoclet.50438206_11903"/>.</para>
<para>Lists inode usage per OST and MDT.</para>
<screen>$ lfs df -i</screen>
<para>List space or inode usage for a specific OST pool.</para>
- <screen>$ lfs df --pool <filesystem>[.<pool>] | <pathname></screen>
+ <screen>$ lfs df --pool <replaceable>filesystem</replaceable>[.<replaceable>pool</replaceable>] | <replaceable>pathname</replaceable></screen>
<para>List quotas of user 'bob'.</para>
<screen>$ lfs quota -u bob /mnt/lustre</screen>
<para>Show grace times for user quotas on /mnt/lustre.</para>
<para>The <literal>e2fsck</literal> utility is run on each of the local MDS and OST device file systems and verifies that the underlying <literal>ldiskfs</literal> is consistent. After <literal>e2fsck</literal> is run, <literal>lfsck</literal> does distributed coherency checking for the Lustre file system. In most cases, <literal>e2fsck</literal> is sufficient to repair any file system issues and <literal>lfsck</literal> is not required.</para>
<section remap="h5">
<title>Synopsis</title>
- <screen>lfsck [-c|--create] [-d|--delete] [-f|--force] [-h|--help] [-l|--lostfound] [-n|--nofix] [-v|--verbose] --mdsdb mds_database_file --ostdb ost1_database_file [ost2_database_file...] <filesystem>
+ <screen>lfsck [-c|--create] [-d|--delete] [-f|--force] [-h|--help] [-l|--lostfound] [-n|--nofix] [-v|--verbose] --mdsdb <replaceable>mds_database_file</replaceable> --ostdb <replaceable>ost1_database_file</replaceable>[<replaceable>ost2_database_file</replaceable>...] <replaceable>/mount_point</replaceable>
</screen>
<note>
- <para>As shown, the <literal><filesystem></literal> parameter refers to the Lustre file system mount point. The default mount point is <literal>/mnt/lustre</literal>.</para>
+ <para>As shown, the <literal><replaceable>/mount_point</replaceable></literal> parameter refers to the Lustre file system mount point. The default mount point is <literal>/mnt/lustre</literal>.</para>
</note>
<note>
<para>For <literal>lfsck</literal>, database filenames must be provided as absolute pathnames. Relative paths do not work, the databases cannot be properly opened.</para>
<para><literal>mds_database_file</literal></para>
</entry>
<entry>
- <para>MDS database file created by running <literal>e2fsck</literal> <literal>--mdsdb</literal> <literal>mds_database_file</literal> <literal><device></literal> on the MDS backing device. This is required.</para>
+ <para>MDT database file created by running <literal>e2fsck --mdsdb <replaceable>mds_database_file</replaceable> <replaceable>/dev/mdt_device</replaceable></literal> on the MDT backing device. This is required.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>--ostdb ost1_database_file</literal></para>
- <para><literal>[ost2_database_file...]</literal></para>
+ <para> <literal>--ostdb <replaceable>ost1_database_file</replaceable></literal></para>
+ <para><literal>[<replaceable>ost2_database_file</replaceable>...]</literal></para>
</entry>
<entry>
- <para>OST database files created by running <literal>e2fsck</literal> <literal>--ostdb ost_database_file <device></literal> on each of the OST backing devices. These are required unless an OST is unavailable, in which case all objects thereon are considered missing.</para>
+ <para>OST database files created by running <literal>e2fsck --ostdb <replaceable>ost_database_file</replaceable> <replaceable>/dev/ost_device</replaceable></literal> on each of the OST backing devices. These are required unless an OST is unavailable, in which case all objects thereon are considered missing.</para>
</entry>
</row>
</tbody>
</copyright>
<copyright>
<year>2011</year>
- <year>2012</year>
+ <year>2013</year>
<holder>Intel Corporation. (Intel modifications to the original version of this Operations Manual.)</holder>
</copyright>
<legalnotice>
git://git.whamcloud.com / doc / manual.git / commitdiff