<para> <literal>--user=<user id></literal></para>
</entry>
<entry>
- <para>The changelog user ID for the specified MDT. To use <literal>lustre_rsync</literal>, the changelog user must be registered. For details, see the <literal>changelog_register</literal> parameter in <xref linkend="systemconfigurationutilities"/> (<literal>lctl</literal>). This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statusl</literal>) is not specified.</para>
+ <para>The changelog user ID for the specified MDT. To use <literal>lustre_rsync</literal>, the changelog user must be registered. For details, see the <literal>changelog_register</literal> parameter in <xref linkend="systemconfigurationutilities"/> (<literal>lctl</literal>). This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statuslog</literal>) is not specified.</para>
</entry>
</row>
<row>
<para> <literal>--statuslog=<log></literal></para>
</entry>
<entry>
- <para>A log file to which synchronization status is saved. When the <literal>lustre_rsync</literal> utility starts, if the status log from a previous synchronization operation is specified, then the state is read from the log and otherwise mandatory <literal>--source</literal>, <literal>--target</literal> and <literal>--mdt</literal> options can be skipped. Specifying the <literal>--source</literal>, <literal>--target</literal> and/or <literal>--mdt</literal> options, in addition to the <literal>--statuslog</literal> option, causes the specified parameters in the status log to be overriden. Command line options take precedence over options in the status log.</para>
+ <para>A log file to which synchronization status is saved. When the <literal>lustre_rsync</literal> utility starts, if the status log from a previous synchronization operation is specified, then the state is read from the log and otherwise mandatory <literal>--source</literal>, <literal>--target</literal> and <literal>--mdt</literal> options can be skipped. Specifying the <literal>--source</literal>, <literal>--target</literal> and/or <literal>--mdt</literal> options, in addition to the <literal>--statuslog</literal> option, causes the specified parameters in the status log to be overridden. Command line options take precedence over options in the status log.</para>
</entry>
</row>
<row>
fstab passwd</screen>
</section>
<section remap="h3">
- <title><indexterm><primary>backup</primary><secondary>using LVM</secondary><tertiary>createing snapshots</tertiary></indexterm>Creating Snapshot Volumes</title>
+ <title><indexterm><primary>backup</primary><secondary>using LVM</secondary><tertiary>creating snapshots</tertiary></indexterm>Creating Snapshot Volumes</title>
<para>Whenever you want to make a "checkpoint" of the main Lustre file system, create LVM snapshots of all target MDT and OSTs in the LVM-based backup file system. You must decide the maximum size of a snapshot ahead of time, although you can dynamically change this later. The size of a daily snapshot is dependent on the amount of data changed daily in the main Lustre file system. It is likely that a two-day old snapshot will be twice as big as a one-day old snapshot.</para>
<para>You can create as many snapshots as you have room for in the volume group. If necessary, you can dynamically add disks to the volume group.</para>
<para>The snapshots of the target MDT and OSTs should be taken at the same point in time. Make sure that the cronjob updating the backup file system is not running, since that is the only thing writing to the disks. Here is an example:</para>
<note>
<para>Array performance with all LUNs loaded does not always match the performance of a single LUN when tested in isolation.</para>
</note>
- <para><emphasis role="bold">Prequisites:</emphasis></para>
+ <para><emphasis role="bold">Prerequisites:</emphasis></para>
<itemizedlist>
<listitem>
<para><literal>sgp_dd</literal> tool in the <literal>sg3_utils</literal> package</para>
<section remap="h3">
<title>Running sgpdd_survey</title>
<para>The <literal>sgpdd_survey</literal> script must be customized for the particular device being tested and for the location where the script saves its working and result files (by specifying the <literal>${rslt}</literal> variable). Customization variables are described at the beginning of the script.</para>
- <para>When the <literal>sgpdd_survey</literal> script runs, it creates a number of working files and a pair of result files. The names of all the files created start with the prefixdefined in the variable <literal>${rslt}</literal>. (The default value is <literal>/tmp</literal>.) The files include:</para>
+ <para>When the <literal>sgpdd_survey</literal> script runs, it creates a number of working files and a pair of result files. The names of all the files created start with the prefix defined in the variable <literal>${rslt}</literal>. (The default value is <literal>/tmp</literal>.) The files include:</para>
<itemizedlist>
<listitem>
<para>File containing standard output data (same as <literal>stdout</literal>)</para>
<para><literal>thr</literal> - Number of threads generating I/O (1 thread in above example).</para>
</listitem>
<listitem>
- <para><literal>crg</literal> - Current regions, the number of disjount areas on the disk to which I/O is being sent (1 region in above example, indicating that no seeking is done).</para>
+ <para><literal>crg</literal> - Current regions, the number of disjoint areas on the disk to which I/O is being sent (1 region in above example, indicating that no seeking is done).</para>
</listitem>
<listitem>
<para><literal>MB/s</literal> - Aggregate bandwidth measured by dividing the total amount of data by the elapsed time (180.45 MB/s in the above example).</para>
</listitem>
<listitem>
<para>Determine the OST names.</para>
- <para>On the OSS nodes to be tested, run the lctldl command. The OST device names are listed in the fourth column of the output. For example:</para>
+ <para>On the OSS nodes to be tested, run the <literal>lctl dl</literal> command. The OST device names are listed in the fourth column of the output. For example:</para>
<screen>$ lctl dl |grep obdfilter
0 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 1159
2 UP obdfilter lustre-OST0002 lustre-OST0002_UUID 1159
</listitem>
<listitem>
<para>List all OSCs you want to test.</para>
- <para>Use the <literal>target=parameter</literal> to list the OSCs separated by spaces. List the individual OSCs by name seperated by spaces using the format <literal><replaceable><fsname>-<OST_name></replaceable>-osc-<replaceable><OSC_number></replaceable></literal> (for example, <literal>lustre-OST0000-osc-ffff88007754bc00</literal>). You <replaceable role="bold">do not have to specify an MDS or LOV.</replaceable></para>
+ <para>Use the <literal>target=parameter</literal> to list the OSCs separated by spaces. List the individual OSCs by name separated by spaces using the format <literal><replaceable><fsname>-<OST_name></replaceable>-osc-<replaceable><OSC_number></replaceable></literal> (for example, <literal>lustre-OST0000-osc-ffff88007754bc00</literal>). You <replaceable role="bold">do not have to specify an MDS or LOV.</replaceable></para>
</listitem>
<listitem>
<para>Run the <literal><replaceable role="bold">o</replaceable>bdfilter_survey</literal> script with the <literal>target=parameter</literal> and <literal>case=netdisk</literal>.</para>
<para><literal>IO</literal> - Lustre target operations statistics</para>
</listitem>
<listitem>
- <para><literal>JBD</literal> - ldisfs journal statistics</para>
+ <para><literal>JBD</literal> - ldiskfs journal statistics</para>
</listitem>
<listitem>
<para><literal>CLIENT</literal> - Lustre OSC request statistics</para>
<para>Examples are:</para>
<screen>10.67.73.200@tcp0
10.67.75.100@o2ib</screen>
- <para>The first entry above identifes a TCP/IP node, while the second entry identifies an InfiniBand node.</para>
+ <para>The first entry above identifies a TCP/IP node, while the second entry identifies an InfiniBand node.</para>
<para>When a mount command is run on a client, the client uses the NID of the MDS to retrieve configuration information. If an MDS has more than one NID, the client should use the appropriate NID for its local network.</para>
<para>To determine the appropriate NID to specify in the mount command, use the <literal>lctl</literal> command. To display MDS NIDs, run on the MDS :</para>
<screen>lctl list_nids</screen>
<para>LNET lines in <literal>lustre.conf</literal> are only used by the local node to determine what to call its interfaces. They are not used for routing decisions.</para>
</note>
<section remap="h3">
- <title><indexterm><primary>configuing</primary><secondary>multihome</secondary></indexterm>Multihome Server Example</title>
+ <title><indexterm><primary>configuring</primary><secondary>multihome</secondary></indexterm>Multihome Server Example</title>
<para>If a server with multiple IP addresses (multihome server) is connected to a Lustre network, certain configuration setting are required. An example illustrating these setting consists of a network with the following nodes:</para>
<itemizedlist>
<listitem>
</section>
<section xml:id="dbdoclet.50438216_71227">
<title><indexterm><primary>LNET</primary><secondary>routes</secondary></indexterm>Setting the LNET Module routes Parameter</title>
- <para>The LNET module routes parameter is used to identify routers in a Lustre configuration. These parameters are set in <literal>modprob.conf</literal> on each Lustre node.</para>
+ <para>The LNET module routes parameter is used to identify routers in a Lustre configuration. These parameters are set in <literal>modprobe.conf</literal> on each Lustre node.</para>
<para>The LNET routes parameter specifies a colon-separated list of router definitions. Each route is defined as a network number, followed by a list of routers:</para>
<screen>routes=<<emphasis>net type</emphasis>> <<emphasis>router NID(s)</emphasis>></screen>
<para>This example specifies bi-directional routing in which TCP clients can reach Lustre resources on the IB networks and IB servers can access the TCP networks:</para>
<para>On the router nodes, use:</para>
<screen>lnet networks="tcp o2ib" forwarding=enabled </screen>
<para>On the MDS, use the reverse as shown below:</para>
- <screen>lnet networks="o2ib0" rountes="tcp0 132.6.1.[1-8]@o2ib0" </screen>
+ <screen>lnet networks="o2ib0" routes="tcp0 132.6.1.[1-8]@o2ib0" </screen>
<para>To start the routers, run:</para>
<screen>modprobe lnet
lctl network configure</screen>
</listitem>
</itemizedlist>
<para>For examples using these utilities, see the topic <xref linkend="systemconfigurationutilities"/></para>
- <para>The <literal>lfs</literal> utility is usful for configuring and querying a variety of options related to files. For more information, see <xref linkend="userutilities"/>.</para>
+ <para>The <literal>lfs</literal> utility is useful for configuring and querying a variety of options related to files. For more information, see <xref linkend="userutilities"/>.</para>
<note>
<para>Some sample scripts are included in the directory where Lustre is installed. If you have installed the Lustre source code, the scripts are located in the <literal>lustre/tests</literal> sub-directory. These scripts enable quick setup of some simple standard Lustre configurations.</para>
</note>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringstorage">
<title xml:id="configuringstorage.title">Configuring Storage on a Lustre File System</title>
- <para>This chapter describes best practices for storage selection and file system options to optimize perforance on RAID, and includes the following sections:</para>
+ <para>This chapter describes best practices for storage selection and file system options to optimize performance on RAID, and includes the following sections:</para>
<itemizedlist>
<listitem>
<para>
</listitem>
</itemizedlist>
<note>
- <para><emphasis role="bold">It is strongly recommended that hardware RAID be used with Lustre.</emphasis> Lustre currently does not support any redundancy at the file system level and RAID is required to protect agains disk failure.</para>
+ <para><emphasis role="bold">It is strongly recommended that hardware RAID be used with Lustre.</emphasis> Lustre currently does not support any redundancy at the file system level and RAID is required to protect against disk failure.</para>
</note>
<section xml:id="dbdoclet.50438208_60972">
<title>
<glossterm>EA
</glossterm>
<glossdef>
- <para>Extended Attribute. A small amount of data which can be retrieved through a name associated with a particular inode. Lustre uses EAa to store striping information (location of file data on OSTs). Examples of extended attributes are ACLs, striping information, and crypto keys.</para>
+ <para>Extended Attribute. A small amount of data which can be retrieved through a name associated with a particular inode. Lustre uses EAs to store striping information (location of file data on OSTs). Examples of extended attributes are ACLs, striping information, and crypto keys.</para>
</glossdef>
</glossentry>
<glossentry xml:id="eviction">
<para> unknown</para>
</entry>
<entry>
- <para> The node'™s status has yet to be determined.</para>
+ <para> The node's status has yet to be determined.</para>
</entry>
</row>
<row>
<literal> --test<replaceable><index></replaceable></literal>
</entry>
<entry nameend="c3" namest="c2">
- <para>Lists tests in a batch. If no option is used, all tests in the batch are listed. IIf one of these options are used, only specified tests in the batch are listed:</para>
+ <para>Lists tests in a batch. If no option is used, all tests in the batch are listed. If one of these options are used, only specified tests in the batch are listed:</para>
</entry>
</row>
<row>
<literal>
<replaceable>lfs</replaceable>
</literal>
- - This utility provides access to the extended attributes (EAs) of a Lustre file (along with other information). For more inforamtion about lfs, see <xref linkend="dbdoclet.50438206_94597"/>.</para>
+ - This utility provides access to the extended attributes (EAs) of a Lustre file (along with other information). For more information about lfs, see <xref linkend="dbdoclet.50438206_94597"/>.</para>
</listitem>
</itemizedlist>
</section>
<para>The procedures below may be useful to administrators or developers debugging a Lustre files system.</para>
<section remap="h3">
<title><indexterm><primary>debugging</primary><secondary>message format</secondary></indexterm>Understanding the Lustre Debug Messaging Format</title>
- <para>Lustre debug messages are categorized by originating sybsystem, message type, and locaton in the source code. For a list of subsystems and message types, see <xref linkend="dbdoclet.50438274_57603"/>.</para>
+ <para>Lustre debug messages are categorized by originating subsystem, message type, and location in the source code. For a list of subsystems and message types, see <xref linkend="dbdoclet.50438274_57603"/>.</para>
<note>
<para>For a current list of subsystems and debug message types, see <literal>libcfs/include/libcfs/libcfs_debug.h</literal> in the Lustre tree</para>
</note>
</itemizedlist>
<section xml:id="dbdoclet.50438199_42877">
<title>
- <indexterm><primary>maintance</primary></indexterm>
- <indexterm><primary>maintance</primary><secondary>inactive OSTs</secondary></indexterm>
+ <indexterm><primary>maintenance</primary></indexterm>
+ <indexterm><primary>maintenance</primary><secondary>inactive OSTs</secondary></indexterm>
Working with Inactive OSTs</title>
<para>To mount a client or an MDT with one or more inactive OSTs, run commands similar to this:</para>
<screen>client> mount -o exclude=testfs-OST0000 -t lustre \
</note>
</section>
<section xml:id="dbdoclet.50438199_15240">
- <title><indexterm><primary>maintance</primary><secondary>finding nodes</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>finding nodes</secondary></indexterm>
Finding Nodes in the Lustre File System</title>
<para>There may be situations in which you need to find all nodes in your Lustre file system or get the names of all OSTs.</para>
<para>To get a list of all Lustre nodes, run this command on the MGS:</para>
1: lustre-OST0001_UUID ACTIVE </screen>
</section>
<section xml:id="dbdoclet.50438199_26070">
- <title><indexterm><primary>maintance</primary><secondary>mounting a server</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>mounting a server</secondary></indexterm>
Mounting a Server Without Lustre Service</title>
<para>If you are using a combined MGS/MDT, but you only want to start the MGS and not the MDT, run this command:</para>
<screen>mount -t lustre <MDT partition> -o nosvc <mount point></screen>
<screen>$ mount -t lustre -L testfs-MDT0000 -o nosvc /mnt/test/mdt</screen>
</section>
<section xml:id="dbdoclet.50438199_54623">
- <title><indexterm><primary>maintance</primary><secondary>regenerating config logs</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>regenerating config logs</secondary></indexterm>
Regenerating Lustre Configuration Logs</title>
<para>If the Lustre system's configuration logs are in a state where the file system cannot be started, use the <literal>writeconf</literal> command to erase them. After the <literal>writeconf</literal> command is run and the servers restart, the configuration logs are re-generated and stored on the MGS (as in a new file system).</para>
<para>You should only use the <literal>writeconf</literal> command if:</para>
<para>After the <literal>writeconf</literal> command is run, the configuration logs are re-generated as servers restart.</para>
</section>
<section xml:id="dbdoclet.50438199_31353">
- <title><indexterm><primary>maintance</primary><secondary>changing a NID</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>changing a NID</secondary></indexterm>
Changing a Server NID</title>
<para>If you need to change the NID on the MDT or an OST, run the <literal>writeconf</literal> command to erase Lustre configuration information (including server NIDs), and then re-generate the system configuration using updated server NIDs.</para>
<para>Change a server NID in these situations:</para>
<para>After the <literal>writeconf</literal> command is run, the configuration logs are re-generated as servers restart, and server NIDs in the updated <literal>list_nids</literal> file are used.</para>
</section>
<section xml:id="dbdoclet.50438199_22527">
- <title><indexterm><primary>maintance</primary><secondary>adding a OST</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>adding a OST</secondary></indexterm>
Adding a New OST to a Lustre File System</title>
<para>To add an OST to existing Lustre file system:</para>
<orderedlist>
</orderedlist>
</section>
<section xml:id="dbdoclet.50438199_14978">
- <title><indexterm><primary>maintance</primary><secondary>restoring a OST</secondary></indexterm>
- <indexterm><primary>maintance</primary><secondary>removing a OST</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>restoring a OST</secondary></indexterm>
+ <indexterm><primary>maintenance</primary><secondary>removing a OST</secondary></indexterm>
Removing and Restoring OSTs</title>
<para>OSTs can be removed from and restored to a Lustre file system. Currently in Lustre, removing an OST really means that the OST is 'deactivated' in the file system, not permanently removed. A removed OST still appears in the file system; do not create a new OST with the same name.</para>
<para>You may want to remove (deactivate) an OST and prevent new files from being written to it in several situations:</para>
</listitem>
</itemizedlist>
<section remap="h3">
- <title><indexterm><primary>maintance</primary><secondary>removing a OST</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>removing a OST</secondary></indexterm>
Removing an OST from the File System</title>
<para>OSTs can be removed from a Lustre file system. Currently in Lustre, removing an OST actually means that the OST is 'deactivated' from the file system, not permanently removed. A removed OST still appears in the device listing; you should not normally create a new OST with the same name.</para>
<para>You may want to deactivate an OST and prevent new files from being written to it in several situations:</para>
<orderedlist>
<listitem>
<para>List all OSCs on the node, along with their device numbers. Run:</para>
- <screen>lctldl|grep " osc "</screen>
+ <screen>lctl dl|grep " osc "</screen>
<para>This is sample <literal>lctl dl | grep</literal></para>
<screen>11 UP osc lustre-OST-0000-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
12 UP osc lustre-OST-0001-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5
</orderedlist>
</section>
<section remap="h3">
- <title><indexterm><primary>maintance</primary><secondary>backing up OST config</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>backing up OST config</secondary></indexterm>
<indexterm><primary>backup</primary><secondary>OST config</secondary></indexterm>
Backing Up OST Configuration Files</title>
<para>If the OST device is still accessible, then the Lustre configuration files on the OST should be backed up and saved for future use in order to avoid difficulties when a replacement OST is returned to service. These files rarely change, so they can and should be backed up while the OST is functional and accessible. If the deactivated OST is still available to mount (i.e. has not permanently failed or is unmountable due to severe corruption), an effort should be made to preserve these files. </para>
</orderedlist>
</section>
<section>
- <title><indexterm><primary>maintance</primary><secondary>restoring OST config</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>restoring OST config</secondary></indexterm>
<indexterm><primary>backup</primary><secondary>restoring OST config</secondary></indexterm>
Restoring OST Configuration Files</title>
<para>If the original OST is still available, it is best to follow the OST backup and restore procedure given in either <xref linkend="dbdoclet.50438207_71633"/>, or <xref linkend="dbdoclet.50438207_21638"/> and <xref linkend="dbdoclet.50438207_22325"/>.
</orderedlist>
</section>
<section>
- <title><indexterm><primary>maintance</primary><secondary>reintroducing an OSTs</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>reintroducing an OSTs</secondary></indexterm>
Returning a Deactivated OST to Service</title>
<para> If the OST was permanently deactivated, it needs to be reactivated in the MGS configuration. <screen>[mgs]# lctl conf_param {OST name}.osc.active=1</screen> If the OST was temporarily deactivated, it needs to be reactivated on the MDS and clients. <screen>[mds]# lctl --device <devno> activate
[client]# lctl set_param osc.<fsname>-<OST name>-*.active=0</screen></para>
</section>
</section>
<section xml:id="dbdoclet.50438199_77819">
- <title><indexterm><primary>maintance</primary><secondary>aborting recovery</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>aborting recovery</secondary></indexterm>
<indexterm><primary>backup</primary><secondary>aborting recovery</secondary></indexterm>
Aborting Recovery</title>
<para>You can abort recovery with either the <literal>lctl</literal> utility or by mounting the target with the <literal>abort_recov</literal> option (<literal>mount -o abort_recov</literal>). When starting a target, run: <screen>$ mount -t lustre -L <MDT name> -o abort_recov <mount_point></screen></para>
</note>
</section>
<section xml:id="dbdoclet.50438199_12607">
- <title><indexterm><primary>maintance</primary><secondary>identifying OST host</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>identifying OST host</secondary></indexterm>
Determining Which Machine is Serving an OST </title>
<para>In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the NID that is serving a specific OST, run one of the following commands on a client (you do not need to be a root user): <screen>client$ lctl get_param osc.${fsname}-${OSTname}*.ost_conn_uuid</screen>For example: <screen>client$ lctl get_param osc.*-OST0000*.ost_conn_uuid
osc.lustre-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen>- OR - <screen>client$ lctl get_param osc.*.ost_conn_uuid
osc.lustre-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen></para>
</section>
<section xml:id="dbdoclet.50438199_62333">
- <title><indexterm><primary>maintance</primary><secondary>changing failover node address</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>changing failover node address</secondary></indexterm>
Changing the Address of a Failover Node</title>
<para>To change the address of a failover node (e.g, to use node X instead of node Y), run this command on the OSS/OST partition:
<screen>tunefs.lustre --erase-params --failnode=<NID> <device> </screen>
</para>
</section>
<section xml:id="dbdoclet.50438199_62545">
- <title><indexterm><primary>maintance</primary><secondary>separate a combined MGS/MDT</secondary></indexterm>
+ <title><indexterm><primary>maintenance</primary><secondary>separate a combined MGS/MDT</secondary></indexterm>
Separate a combined MGS/MDT</title>
<para>These instructions assume the MGS node will be the same as the MDS node. For instructions on how to move MGS to a different node, see <xref linkend="dbdoclet.50438199_31353"/>.</para>
<para>These instructions are for doing the split without shutting down other servers and clients.</para>
<para>Copy the configuration data from MDT disk to the new MGS disk.</para>
<screen>mount -t ldiskfs -o ro <mdt-device> <mdt-mount-point> </screen>
<screen>mount -t ldiskfs -o rw <mgs-device> <mgs-mount-point> </screen>
- <screen>cp -r <mdt-moint-point>/CONFIGS/<filesystem-name>-* <mgs-mount-point>/CONFIGS/. </screen>
+ <screen>cp -r <mdt-mount-point>/CONFIGS/<filesystem-name>-* <mgs-mount-point>/CONFIGS/. </screen>
<screen>umount <mgs-mount-point> </screen>
<screen>umount <mdt-mount-point> </screen>
<para>See <xref linkend="dbdoclet.50438199_54623"/> for alternative method.</para>
<screen># debugfs -c -R "stat /O/0/d$((34976 % 32))/34976" /dev/lustre/ost_test2 </screen></para>
<para>The command output is:
<screen>debugfs 1.42.3.wc3 (15-Aug-2012)
-/dev/lustre/ost_test2: catastrophic mode - not reading inode or group bitma\
-ps
+/dev/lustre/ost_test2: catastrophic mode - not reading inode or group bitmaps
Inode: 352365 Type: regular Mode: 0666 Flags: 0x80000
Generation: 2393149953 Version: 0x0000002a:00005f81
User: 1000 Group: 1000 Size: 260096
<para>There is no file size/block attributes pre-fetching and the traversing thread has to send synchronous glimpse size RPCs to OST(s).</para>
</listitem>
</orderedlist>
- <para>The first issue is resolved by using statahead local dcache, and the second one is resolved by using asynchronous glimpse lock (AGL) RPCs for per-fetching file size/block attributes from OST(s).</para>
+ <para>The first issue is resolved by using statahead local dcache, and the second one is resolved by using asynchronous glimpse lock (AGL) RPCs for pre-fetching file size/block attributes from OST(s).</para>
<section remap="h4">
<title>Tuning File Readahead</title>
<para>File readahead is triggered when two or more sequential reads by an application fail to be satisfied by the Linux buffer cache. The size of the initial readahead is 1 MB. Additional readaheads grow linearly, and increment until the readahead cache on the client is full at 40 MB.</para>
</section>
<section remap="h3">
<title><indexterm><primary>proc</primary><secondary>read cache</secondary></indexterm>OSS Read Cache</title>
- <para>The OSS read cache feature provides read-only caching of data on an OSS. This functionality uses the regular Linux page cache to store the data. Just like caching from a regular filesytem in Linux, OSS read cache uses as much physical memory as is allocated.</para>
+ <para>The OSS read cache feature provides read-only caching of data on an OSS. This functionality uses the regular Linux page cache to store the data. Just like caching from a regular filesystem in Linux, OSS read cache uses as much physical memory as is allocated.</para>
<para>OSS read cache improves Lustre performance in these situations:</para>
<itemizedlist>
<listitem>
</section>
<section remap="h3">
<title>Description</title>
- <para>The group upcall file contains the path to an executable that is invoked to resolve a numeric UID to a group membership list. This utility opens <literal>/proc/fs/lustre/mdt/{mdtname}/identity_info</literal> and writes the releated <literal>identity_downcall_data</literal> data structure (see <xref linkend="dbdoclet.50438291_33759"/>). The data is persisted with <literal>lctl set_param mdt.{mdtname}.identity_info</literal>.</para>
+ <para>The group upcall file contains the path to an executable that is invoked to resolve a numeric UID to a group membership list. This utility opens <literal>/proc/fs/lustre/mdt/{mdtname}/identity_info</literal> and writes the related <literal>identity_downcall_data</literal> data structure (see <xref linkend="dbdoclet.50438291_33759"/>). The data is persisted with <literal>lctl set_param mdt.{mdtname}.identity_info</literal>.</para>
<para>For a sample upcall program, see <literal>lustre/utils/l_getidentity.c</literal> in the Lustre source distribution.</para>
<section remap="h4">
<title>Primary and Secondary Groups</title>
* the valid values for perms are:
* setuid/setgid/setgrp/rmtacl -- enable corresponding perm
* nosetuid/nosetgid/nosetgrp/normtacl -- disable corresponding perm
-* they can be listed together, seperated by ',',
+* they can be listed together, separated by ',',
* when perm and noperm are in the same line (item), noperm is preferential,
* when they are in different lines (items), the latter is preferential,
* '*' nid is as default perm, and is not preferential.*/</screen>
</section>
<section remap="h5">
<title>Description</title>
- <para>The group upcall file contains the path to an executable that is invoked to resolve a numeric UID to a group membership list. This utility opens <literal>/proc/fs/lustre/mdt/{mdtname}/identity_info</literal> and writes the releated <literal>identity_downcall_data</literal> data structure (see <xref linkend='dbdoclet.50438291_33759'/>). The data is persisted with <literal>lctl set_param mdt.{mdtname}.identity_info</literal>.</para>
+ <para>The group upcall file contains the path to an executable that is invoked to resolve a numeric UID to a group membership list. This utility opens <literal>/proc/fs/lustre/mdt/{mdtname}/identity_info</literal> and writes the related <literal>identity_downcall_data</literal> data structure (see <xref linkend='dbdoclet.50438291_33759'/>). The data is persisted with <literal>lctl set_param mdt.{mdtname}.identity_info</literal>.</para>
<para>l_getidentity is the reference implementation of the user/group cache upcall.</para>
</section>
<section remap="h5">
<title>
<indexterm><primary>recovery</primary></indexterm>
<indexterm><primary>recovery</primary><secondary>VBR</secondary><see>version-based recovery</see></indexterm>
- <indexterm><primary>recovery</primary><secondary>commit on share</secondary><see>ommit on share</see></indexterm>
+ <indexterm><primary>recovery</primary><secondary>commit on share</secondary><see>commit on share</see></indexterm>
<indexterm><primary>lustre</primary><secondary>recovery</secondary><see>recovery</see></indexterm>
Recovery Overview</title>
<para>Lustre's recovery feature is responsible for dealing with node or network failure and returning the cluster to a consistent, performant state. Because Lustre allows servers to perform asynchronous update operations to the on-disk file system (i.e., the server can reply without waiting for the update to synchronously commit to disk), the clients may have state in memory that is newer than what the server can recover from disk after a crash.</para>
</section>
<section remap="h3">
<title>Transaction Numbers</title>
- <para>Each client request processed by the server that involves any state change (metadata update, file open, write, etc., depending on server type) is assigned a transaction number by the server that is a target-unique, monontonically increasing, server-wide 64-bit integer. The transaction number for each file system-modifying request is sent back to the client along with the reply to that client request. The transaction numbers allow the client and server to unambiguously order every modification to the file system in case recovery is needed.</para>
+ <para>Each client request processed by the server that involves any state change (metadata update, file open, write, etc., depending on server type) is assigned a transaction number by the server that is a target-unique, monotonically increasing, server-wide 64-bit integer. The transaction number for each file system-modifying request is sent back to the client along with the reply to that client request. The transaction numbers allow the client and server to unambiguously order every modification to the file system in case recovery is needed.</para>
<para>Each reply sent to a client (regardless of request type) also contains the last committed transaction number that indicates the highest transaction number committed to the file system. The <literal>ldiskfs</literal> backing file system that Lustre uses enforces the requirement that any earlier disk operation will always be committed to disk before a later disk operation, so the last committed transaction number also reports that any requests with a lower transaction number have been committed to disk.</para>
</section>
<section remap="h3">
<footnote>
<para>There are two scenarios under which client RPCs are not replayed: (1) Non-functioning or isolated clients do not reconnect, and they cannot replay their RPCs, causing a gap in the replay sequence. These clients get errors and are evicted. (2) Functioning clients connect, but they cannot replay some or all of their RPCs that occurred after the gap caused by the non-functioning/isolated clients. These clients get errors (caused by the failed clients). With VBR, these requests have a better chance to replay because the "gaps" are only related to specific files that the missing client(s) changed.</para>
</footnote>.</para>
- <para>In pre-VBR versions of Lustre, if the MGS or an OST went down and then recovered, a recovery process was triggered in which clients attempted to replay their requests. Clients were only allowed to replay RPCs in serial order. If a particular client could not replay its requests, then those requests were lost as well as the requests of clients later in the sequence. The ''downstream'' clients never got to replay their requests because of the wait on the earlier client'™s RPCs. Eventually, the recovery period would time out (so the component could accept new requests), leaving some number of clients evicted and their requests and data lost.</para>
+ <para>In pre-VBR versions of Lustre, if the MGS or an OST went down and then recovered, a recovery process was triggered in which clients attempted to replay their requests. Clients were only allowed to replay RPCs in serial order. If a particular client could not replay its requests, then those requests were lost as well as the requests of clients later in the sequence. The ''downstream'' clients never got to replay their requests because of the wait on the earlier client's RPCs. Eventually, the recovery period would time out (so the component could accept new requests), leaving some number of clients evicted and their requests and data lost.</para>
<para>With VBR, the recovery mechanism does not result in the loss of clients or their data, because changes in inode versions are tracked, and more clients are able to reintegrate into the cluster. With VBR, inode tracking looks like this:</para>
<itemizedlist>
<listitem>
<para>Imperative recovery can also be disabled on the client side with the same mount option:</para>
<screen># mount -t lustre -onoir mymgsnid@tcp:/testfs /mnt/testfs</screen>
<note><para>When a single client is deactivated in this manner, the MGS will deactivate imperative recovery for the whole cluster. IR-enabled clients will still get notification of target restart, but targets will not be allowed to shorten the recovery window. </para></note>
- <para>You can also disable imperative recovery globally on the MGS by writing `state=disabled’ to the controling procfs entry</para>
+ <para>You can also disable imperative recovery globally on the MGS by writing `state=disabled’ to the controlling procfs entry</para>
<screen># lctl set_param mgs.MGS.live.testfs="state=disabled"</screen>
<para>The above command will disable imperative recovery for file system named <emphasis>testfs</emphasis></para>
</section>
</section>
</section>
<section xml:id="dbdoclet.50438198_30989">
- <title><indexterm><primary>troubleshooting</primary><secondary>reporting bugs</secondary></indexterm><indexterm><primary>reporting bugs</primary><see>troubleshooing</see></indexterm>
+ <title><indexterm><primary>troubleshooting</primary><secondary>reporting bugs</secondary></indexterm><indexterm><primary>reporting bugs</primary><see>troubleshooting</see></indexterm>
Reporting a Lustre Bug</title>
<para>If, after troubleshooting your Lustre system, you cannot resolve the problem, consider reporting a Lustre bug. The process for reporting a bug is described in the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</link>.</para>
<para>You can also post a question to the <link xl:href="http://wiki.whamcloud.com/display/PUB/Community+Resources">lustre-discuss mailing list</link> or search the <link xl:href="http://groups.google.com/group/lustre-discuss-list">lustre-discuss Archives</link> for information about your issue.</para>
</section>
<section remap="h3">
<title><indexterm><primary>troubleshooting</primary><secondary>timeouts on setup</secondary></indexterm>Handling Timeouts on Initial Lustre Setup</title>
- <para>If you come across timeouts or hangs on the initial setup of your Lustre system, verify that name resolution for servers and clients is working correctly. Some distributions configure <literal>/etc/hosts sts</literal> so the name of the local machine (as reported by the 'hostname' command) is mapped to local host (127.0.0.1) instead of a proper IP address.</para>
+ <para>If you come across timeouts or hangs on the initial setup of your Lustre system, verify that name resolution for servers and clients is working correctly. Some distributions configure <literal>/etc/hosts</literal> so the name of the local machine (as reported by the 'hostname' command) is mapped to local host (127.0.0.1) instead of a proper IP address.</para>
<para>This might produce this error:</para>
<screen>LustreError:(ldlm_handle_cancel()) received cancel for unknown lock cookie
0xe74021a4b41b954e from nid 0x7f000001 (0:127.0.0.1)
</section>
<section xml:id="dbdoclet.50438272_80545">
<title><indexterm><primary>tuning</primary><secondary>for small files</secondary></indexterm>Improving Lustre Performance When Working with Small Files</title>
- <para>A Lustre environment where an application writes small file chunks from many clients to a single file will result in bad I/O performance. To improve Lustre'™s performance with small files:</para>
+ <para>A Lustre environment where an application writes small file chunks from many clients to a single file will result in bad I/O performance. To improve Lustre's performance with small files:</para>
<itemizedlist>
<listitem>
<para>Have the application aggregate writes some amount before submitting them to Lustre. By default, Lustre enforces POSIX coherency semantics, so it results in lock ping-pong between client nodes if they are all writing to the same file at one time.</para>
</listitem>
</itemizedlist>
<note>
- <para>For information about high availability(HA) management software, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Red_Hat_Cluster_Manager_with_Lustre">Using Red Hat Cluster Manager with Lustre</link> or the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre">LuUsing Pacemaker with stre</link>.</para>
+ <para>For information about high availability(HA) management software, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Red_Hat_Cluster_Manager_with_Lustre">Using Red Hat Cluster Manager with Lustre</link> or the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre">Using Pacemaker with Lustre</link>.</para>
</note>
<section xml:id="dbdoclet.50438213_13563">
<title>
<section remap="h3">
<title>
<indexterm><primary>I/O</primary><secondary>migrating data</secondary></indexterm>
- <indexterm><primary>maintance</primary><secondary>full OSTs</secondary></indexterm>
+ <indexterm><primary>maintenance</primary><secondary>full OSTs</secondary></indexterm>
Migrating Data within a File System</title>
<para>As stripes cannot be moved within the file system, data must be migrated manually by copying and renaming the file, removing the original file, and renaming the new file with the original file name. The simplest way to do this is to use the <literal>lfs_migrate</literal> command (see <xref linkend="dbdoclet.50438206_42260"/>). However, the steps for migrating a file by hand are also shown here for reference.</para>
<orderedlist>
<section remap="h3">
<title>
<indexterm><primary>I/O</primary><secondary>bringing OST online</secondary></indexterm>
- <indexterm><primary>maintance</primary><secondary>bringing OST online</secondary></indexterm>
+ <indexterm><primary>maintenance</primary><secondary>bringing OST online</secondary></indexterm>
Returning an Inactive OST Back Online</title>
<para>Once the deactivated OST(s) no longer are severely imbalanced, due to either active or passive data redistribution, they should be reactivated so they will again have new files allocated on them.</para>
<section xml:id="dbdoclet.50438211_75549">
<title>
<indexterm><primary>I/O</primary><secondary>pools</secondary></indexterm>
- <indexterm><primary>maintance</primary><secondary>pools</secondary></indexterm>
+ <indexterm><primary>maintenance</primary><secondary>pools</secondary></indexterm>
<indexterm><primary>pools</primary></indexterm>
Creating and Managing OST Pools</title>
<para>The OST pools feature enables users to group OSTs together to make object placement more flexible. A 'pool' is the name associated with an arbitrary subset of OSTs in a Lustre cluster.</para>
<screen>$ lctl get_param -n mdc.home-MDT0000-mdc-*.connect_flags | grep acl acl</screen>
<para>To mount the client with no ACLs:</para>
<screen>$ mount -t lustre -o noacl ibmds2@o2ib:/home /home</screen>
- <para>ACLs are enabled in Lustre on a system-wide basis; either all clients enable ACLs or none do. Activating ACLs is controlled by MDS mount options <literal>acl</literal> / <literal>noacl</literal> (enable/disableACLs). Client-side mount options acl/noacl are ignored. You do not need to change the client configuration, and the 'acl' string will not appear in the client /etc/mtab. The client acl mount option is no longer needed. If a client is mounted with that option, then this message appears in the MDS syslog:</para>
+ <para>ACLs are enabled in Lustre on a system-wide basis; either all clients enable ACLs or none do. Activating ACLs is controlled by MDS mount options <literal>acl</literal> / <literal>noacl</literal> (enable/disable ACLs). Client-side mount options acl/noacl are ignored. You do not need to change the client configuration, and the 'acl' string will not appear in the client /etc/mtab. The client acl mount option is no longer needed. If a client is mounted with that option, then this message appears in the MDS syslog:</para>
<screen>...MDS requires ACL support but client does not</screen>
<para>The message is harmless but indicates a configuration issue, which should be corrected.</para>
<para>If ACLs are not enabled on the MDS, then any attempts to reference an ACL on a client return an Operation not supported error.</para>
[root@client lustre]# setfacl -m user:chirag:rwx rain
[root@client lustre]# ls -ld rain
drwxrwx---+ 2 root root 4096 Feb 20 06:50 rain
-[root@client lustre]# getfacl --omit-heade rain
+[root@client lustre]# getfacl --omit-header rain
user::rwx
user:chirag:rwx
group::r-x
<indexterm><primary>space</primary><secondary>striping</secondary></indexterm>
How Lustre Striping Works</title>
<para>Lustre uses a round-robin algorithm for selecting the next OST to which a stripe is to be written. Normally the usage of OSTs is well balanced. However, if users create a small number of exceptionally large files or incorrectly specify striping parameters, imbalanced OST usage may result.</para>
- <para>The MDS allocates objects on seqential OSTs. Periodically, it will adjust the striping layout to eliminate some degenerated cases where applications that create very regular file layouts (striping patterns) would preferentially use a particular OST in the sequence.</para>
+ <para>The MDS allocates objects on sequential OSTs. Periodically, it will adjust the striping layout to eliminate some degenerated cases where applications that create very regular file layouts (striping patterns) would preferentially use a particular OST in the sequence.</para>
<para>Stripes are written to sequential OSTs until free space across the OSTs differs by more than 20%. The MDS will then use weighted random allocations with a preference for allocating objects on OSTs with more free space. This can reduce I/O performance until space usage is rebalanced to within 20% again.</para>
<para>For a more detailed description of stripe assignments, see <xref linkend="dbdoclet.50438209_10424"/>.</para>
</section>
</itemizedlist>
</section>
<section remap="h3">
- <title><indexterm><primary>space</primary><secondary>locationg weighting</secondary></indexterm>>Adjusting the Weighting Between Free Space and Location</title>
+ <title><indexterm><primary>space</primary><secondary>location weighting</secondary></indexterm>>Adjusting the Weighting Between Free Space and Location</title>
<para>The weighting priority can be adjusted in the <literal>/proc</literal> file <literal>/proc/fs/lustre/lov/lustre-mdtlov/qos_prio_free proc</literal>. The default value is 90%. Use this command on the MGS to permanently change this weighting:</para>
<screen>lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90</screen>
<para>Increasing this value puts more weighting on free space. When the free space priority is set to 100%, then location is no longer used in stripe-ordering calculations and weighting is based entirely on free space.</para>
<para> <literal>lmm_magic</literal></para>
</entry>
<entry>
- <para>Specifies the format of the returned striping information. <literal>LOV_MAGIC_V1</literal> isused for lov_user_md_v1. LOV_MAGIC_V3 is used for <literal>lov_user_md_v3</literal>.</para>
+ <para>Specifies the format of the returned striping information. <literal>LOV_MAGIC_V1</literal> is used for lov_user_md_v1. LOV_MAGIC_V3 is used for <literal>lov_user_md_v3</literal>.</para>
</entry>
</row>
<row>
<para> <literal>EEXIST</literal></para>
</entry>
<entry>
- <para> triping information has already been set and cannot be altered; <literal>name</literal> already exists.</para>
+ <para>Striping information has already been set and cannot be altered; <literal>name</literal> already exists.</para>
</entry>
</row>
<row>
return rc;
}
-/* Ping all OSTs that belong to this filesysem */
+/* Ping all OSTs that belong to this filesystem */
int ping_osts()
{
}
printf("Getting uuid list\n");
rc = get_my_uuids(file);
- rintf("Write to the file\n");
+ printf("Write to the file\n");
rc = write_file(file);
rc = close_file(file);
printf("Listing LOV data\n");
rc = ping_osts();
/* the results should match lfs getstripe */
- printf("Confirming our results with lfs getsrtipe\n");
+ printf("Confirming our results with lfs getstripe\n");
sprintf(sys_cmd, "/usr/bin/lfs getstripe %s/%s", MY_LUSTRE_DIR, TESTFILE);
system(sys_cmd);
<para>2 KB/inode * 40 million inodes = 80 GB</para>
</informalexample>
<note>
- <para>Inode size depends on stripe count. Without any stripes an inode allocation of 512 bytes might be suffecient. As striping grows to ~80 stripes per file, the inode allocation is around 2KB. At 160 stripes, a rough estimate for inode allocation is 4.5KB.</para>
+ <para>Inode size depends on stripe count. Without any stripes an inode allocation of 512 bytes might be sufficient. As striping grows to ~80 stripes per file, the inode allocation is around 2KB. At 160 stripes, a rough estimate for inode allocation is 4.5KB.</para>
</note>
<para>If the average file size is small, 4 KB for example, Lustre is not very efficient as the MDT uses as much space as the OSTs. However, this is not a common configuration for Lustre.</para>
<note>
<para> 512 PB</para>
</entry>
<entry>
- <para>Each OST or MDT on 64-bit kernel servers can have a file system up to 128 TB. On 32-bit systems, due to page cache limits, 16TB is the maximum block device size, which in turn applies to the size of OSTon 32-bit kernel servers.</para>
+ <para>Each OST or MDT on 64-bit kernel servers can have a file system up to 128 TB. On 32-bit systems, due to page cache limits, 16TB is the maximum block device size, which in turn applies to the size of OST on 32-bit kernel servers.</para>
<para>You can have multiple OST file systems on a single OSS node.</para>
</entry>
</row>
</section>
<section remap="h5">
<title>Description</title>
- <para>The group upcall file contains the path to an executable file that is invoked to resolve a numeric UID to a group membership list. This utility opens <literal>/proc/fs/lustre/mdt/{mdtname}/identity_info</literal> and writes the releated <literal>identity_downcall_data</literal> structure (see <xref linkend='dbdoclet.50438291_33759'/>.) The data is persisted with <literal>lctl set_param mdt.{mdtname}.identity_info</literal>.</para>
+ <para>The group upcall file contains the path to an executable file that is invoked to resolve a numeric UID to a group membership list. This utility opens <literal>/proc/fs/lustre/mdt/{mdtname}/identity_info</literal> and writes the related <literal>identity_downcall_data</literal> structure (see <xref linkend='dbdoclet.50438291_33759'/>.) The data is persisted with <literal>lctl set_param mdt.{mdtname}.identity_info</literal>.</para>
<para>The l_getidentity utility is the reference implementation of the user or group cache upcall.</para>
</section>
<section remap="h5">
<section remap="h5">
<title>Setting Parameters with lctl</title>
<para>Lustre parameters are not always accessible using the procfs interface, as it is platform-specific. As a solution, lctl {get,set}_param has been introduced as a platform-independent interface to the Lustre tunables. Avoid direct references to /proc/{fs,sys}/{lustre,lnet}. For future portability, use lctl {get,set}_param .</para>
- <para>When the file system is running, use the lctl set_param command to set temporary parameters (mapping to items in /proc/{fs,sys}/{lnet,lustre}). The lctl set_param command uses this syntax:</para>
+ <para>When the file system is running, use the lctl set_param command to set temporary parameters (mapping to items in /proc/{fs,sys}/{lnet,lustre}). The <literal>lctl set_param</literal> command uses this syntax:</para>
<screen>lctl set_param [-n] <obdtype>.<obdname>.<proc_file_name>=<value></screen>
<para>For example:</para>
<screen>$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))</screen>
- <para>Many permanent parameters can be set with lctl conf_param. In general, lctl conf_param can be used to specify any parameter settable in a /proc/fs/lustre file, with its own OBD device. The lctl conf_param command uses this syntax:</para>
+ <para>Many permanent parameters can be set with lctl conf_param. In general, lctl conf_param can be used to specify any parameter settable in a /proc/fs/lustre file, with its own OBD device. The <literal>lctl conf_param</literal> command uses this syntax:</para>
<screen><obd|fsname>.<obdtype>.<proc_file_name>=<value>) </screen>
<para>For example:</para>
<screen>$ lctl conf_param testfs-MDT0000.mdt.identity_upcall=NONE
$ lctl conf_param testfs.llite.max_read_ahead_mb=16 </screen>
<caution>
- <para>The lctlconf_param command permanently sets parameters in the file system configuration.</para>
+ <para>The <literal>lctl conf_param</literal> command permanently sets parameters in the file system configuration.</para>
</caution>
<para>To get current Lustre parameter settings, use the lctl get_param command with this syntax:</para>
<screen>lctl get_param [-n] <obdtype>.<obdname>.<proc_file_name></screen>
#123455: objid=614725 seq=0 parent=[0x18d11:0xebba84eb:0x1]
#123458: objid=533088 seq=0 parent=[0x21417:0x19734d61:0x0]</screen>
<para>This shows that the three files in lost+found have decimal object IDs - 690670, 614725, and 533088, respectively. The object sequence number (formerly object group) is 0 for all current OST objects.</para>
- <para>The MDT parent inode FIDs are hexdecimal numbers of the form sequence:oid:idx. Since the sequence number is below 0x100000000 in all these cases, the FIDs are in the legacy Inode and Generation In FID (IGIF) namespace and are mapped directly to the MDT inode = seq and generation = oid values; the MDT inodes are 0x751c5, 0x18d11, and 0x21417 respectively. For objects with MDT parent sequence numbers above 0x200000000, this indicates that the FID needs to be mapped via the MDT Object Index (OI) file on the MDT to determine the internal inode number.</para>
+ <para>The MDT parent inode FIDs are hexadecimal numbers of the form sequence:oid:idx. Since the sequence number is below 0x100000000 in all these cases, the FIDs are in the legacy Inode and Generation In FID (IGIF) namespace and are mapped directly to the MDT inode = seq and generation = oid values; the MDT inodes are 0x751c5, 0x18d11, and 0x21417 respectively. For objects with MDT parent sequence numbers above 0x200000000, this indicates that the FID needs to be mapped via the MDT Object Index (OI) file on the MDT to determine the internal inode number.</para>
<para>The idx field shows the stripe number of this OST object in the Lustre RAID-0 striped file.</para>
</section>
<section remap="h5">
</section>
<section remap="h5">
<title>Description</title>
- <para>The lshowmount utility shows the hosts that have Lustre mounted to a server. Ths utility looks for exports from the MGS, MDS, and obdfilter.</para>
+ <para>The lshowmount utility shows the hosts that have Lustre mounted to a server. This utility looks for exports from the MGS, MDS, and obdfilter.</para>
</section>
<section remap="h5">
<title>Options</title>
<para> <emphasis role="bold">--statuslog=</emphasis><emphasis><log></emphasis></para>
</entry>
<entry>
- <para> A log file to which synchronization status is saved. When lustre_rsync starts, the state of a previous replication is read from here. If the status log from a previous synchronization operation is specified, otherwise mandatory options like --source, --target and --mdt options may be skipped. By specifying options like --source, --target and/or --mdt in addition to the --statuslog option, parameters in the status log can be overriden. Command line options take precedence over options in the status log.</para>
+ <para> A log file to which synchronization status is saved. When lustre_rsync starts, the state of a previous replication is read from here. If the status log from a previous synchronization operation is specified, otherwise mandatory options like --source, --target and --mdt options may be skipped. By specifying options like --source, --target and/or --mdt in addition to the --statuslog option, parameters in the status log can be overridden. Command line options take precedence over options in the status log.</para>
</entry>
</row>
<row>
<title><indexterm><primary>lr_reader</primary></indexterm>
lr_reader</title>
<para>The lr_reader utility translates a last received (last_rcvd) file into human-readable form.</para>
- <para>The following utilites are part of the Lustre I/O kit. For more information, see <xref linkend="benchmarkingtests"/>.</para>
+ <para>The following utilities are part of the Lustre I/O kit. For more information, see <xref linkend="benchmarkingtests"/>.</para>
</section>
<section remap="h5">
<title><indexterm><primary>sgpdd_survey</primary></indexterm>
<para> None</para>
</entry>
<entry>
- <para> Low latency, high bandwith network.</para>
+ <para> Low latency, high bandwidth network.</para>
</entry>
</row>
</tbody>
<note>
<para>Versions of Lustre prior to 2.2 had a maximum stripe count for a single file was limited to 160 OSTs.</para>
</note>
- <para>Athough a single file can only be striped over 2000 objects, Lustre file systems can have thousands of OSTs. The I/O bandwidth to access a single file is the aggregated I/O bandwidth to the objects in a file, which can be as much as a bandwidth of up to 2000 servers. On systems with more than 2000 OSTs, clients can do I/O using multiple files to utilize the full file system bandwidth.</para>
+ <para>Although a single file can only be striped over 2000 objects, Lustre file systems can have thousands of OSTs. The I/O bandwidth to access a single file is the aggregated I/O bandwidth to the objects in a file, which can be as much as a bandwidth of up to 2000 servers. On systems with more than 2000 OSTs, clients can do I/O using multiple files to utilize the full file system bandwidth.</para>
<para>For more information about striping, see <xref linkend="managingstripingfreespace"/>.</para>
</section>
</section>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="understandinglustrenetworking">
<title xml:id="understandinglustrenetworking.title">Understanding Lustre Networking (LNET)</title>
- <para>This chapter introduces Lustre Networking (LNET) and includes the following for intereste sections:</para>
+ <para>This chapter introduces Lustre Networking (LNET) and includes the following sections:</para>
<itemizedlist>
<listitem>
<para>
git://git.whamcloud.com / doc / manual.git / commitdiff