<?xml version='1.0' encoding='UTF-8'?>
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="backupandrestore">
<title xml:id="backupandrestore.title">Backing Up and Restoring a File System</title>
- <para>This chapter describes how to backup and restore on Lustre, at the filesystem-level, device-level and file-level. Each backup approach is described in the the following sections:</para>
+ <para>This chapter describes how to backup and restore on Lustre, at the file system-level,
+ device-level and file-level. Each backup approach is described in the the following
+ sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438207_56395"/></para>
<indexterm><primary>LVM</primary><see>backup</see></indexterm>
<indexterm><primary>rsync</primary><see>backup</see></indexterm>
Backing up a File System</title>
- <para>Backing up a complete file system gives you full control over the files to back up, and allows restoration of individual files as needed. Filesystem-level backups are also the easiest to integrate into existing backup solutions.</para>
+ <para>Backing up a complete file system gives you full control over the files to back up, and
+ allows restoration of individual files as needed. File system-level backups are also the
+ easiest to integrate into existing backup solutions.</para>
<para>File system backups are performed from a Lustre client (or many clients working parallel in different directories) rather than on individual server nodes; this is no different than backing up any other file system.</para>
<para>However, due to the large size of most Lustre file systems, it is not always possible to get a complete backup. We recommend that you back up subsets of a file system. This includes subdirectories of the entire file system, filesets for a single user, files incremented by date, and so on.</para>
<note>
- <para>In order to allow Lustre to scale the filesystem namespace for future applications, Lustre 2.x internally uses a 128-bit file identifier for all files. To interface with user applications, Lustre presents 64-bit inode numbers for the <literal>stat()</literal>, <literal>fstat()</literal>, and <literal>readdir()</literal> system calls on 64-bit applications, and 32-bit inode numbers to 32-bit applications.</para>
+ <para>In order to allow Lustre to scale the file system namespace for future applications,
+ Lustre 2.x internally uses a 128-bit file identifier for all files. To interface with user
+ applications, Lustre presents 64-bit inode numbers for the <literal>stat()</literal>,
+ <literal>fstat()</literal>, and <literal>readdir()</literal> system calls on 64-bit
+ applications, and 32-bit inode numbers to 32-bit applications.</para>
<para>Some 32-bit applications accessing Lustre filesystems (on both 32-bit and 64-bit CPUs) may experience problems with the <literal>stat()</literal>, <literal>fstat()</literal> or<literal> readdir()</literal> system calls under certain circumstances, though the Lustre client should return 32-bit inode numbers to these applications.</para>
- <para>In particular, if the Lustre filesystem is exported from a 64-bit client via NFS to a 32-bit client, the Linux NFS server will export 64-bit inode numbers to applications running on the NFS client. If the 32-bit applications are not compiled with Large File Support (LFS), then they return <literal>EOVERFLOW</literal> errors when accessing the Lustre files. To avoid this problem, Linux NFS clients can use the kernel command-line option "<literal>nfs.enable_ino64=0</literal>" in order to force the NFS client to export 32-bit inode numbers to the client.</para>
+ <para>In particular, if the Lustre file system is exported from a 64-bit client via NFS to a
+ 32-bit client, the Linux NFS server will export 64-bit inode numbers to applications running
+ on the NFS client. If the 32-bit applications are not compiled with Large File Support
+ (LFS), then they return <literal>EOVERFLOW</literal> errors when accessing the Lustre files.
+ To avoid this problem, Linux NFS clients can use the kernel command-line option
+ "<literal>nfs.enable_ino64=0</literal>" in order to force the NFS client to
+ export 32-bit inode numbers to the client.</para>
<para><emphasis role="bold">Workaround</emphasis>: We very strongly recommend that backups using <literal>tar(1)</literal> and other utilities that depend on the inode number to uniquely identify an inode to be run on 64-bit clients. The 128-bit Lustre file identifiers cannot be uniquely mapped to a 32-bit inode number, and as a result these utilities may operate incorrectly on 32-bit clients.</para>
</note>
<section remap="h3">
<para><emphasis role="bold">Back up all file system data.</emphasis></para>
<screen>[oss]# tar czvf {backup file}.tgz [--xattrs] --sparse .</screen>
<note>
- <para>The tar <literal>--sparse</literal> option is vital for
- backing up an MDT. In order to have <literal>--sparse</literal>
- behave correctly, and complete the backup of and MDT in finite
- time, the version of tar must be specified. Correctly functioning
- versions of tar include the Lustre-enhanced tar
- <link xl:href='http://build.whamcloud.com/job/lustre-tar-master/'>http://build.whamcloud.com/job/lustre-tar-master/</link>,
- Red Hat tar from RHEL 6.3 onwards and GNU tar version 1.25 or
- more recent.</para>
+ <para>The tar <literal>--sparse</literal> option is vital for backing up an MDT. In
+ order to have <literal>--sparse</literal> behave correctly, and complete the backup of
+ and MDT in finite time, the version of tar must be specified. Correctly functioning
+ versions of tar include the Lustre software enhanced version of tar at <link
+ xmlns:xlink="http://www.w3.org/1999/xlink"
+ xlink:href="https://wiki.hpdd.intel.com/display/PUB/Lustre+Tools#LustreTools-lustre-tar"
+ />, the tar from a Red Hat Enterprise Linux distribution (version 6.3 or more recent)
+ and the GNU tar version 1.25 or more recent.</para>
</note>
<warning>
<para>The tar <literal>--xattrs</literal> option is only available
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="benchmarkingtests">
<title xml:id="benchmarkingtests.title">Benchmarking Lustre Performance (Lustre I/O Kit)</title>
- <para>This chapter describes the Lustre I/O kit, a collection of I/O benchmarking tools for a Lustre cluster, and PIOS, a parallel I/O simulator for Linux and Solaris. It includes:</para>
+ <para>This chapter describes the Lustre I/O kit, a collection of I/O benchmarking tools for a
+ Lustre cluster, and PIOS, a parallel I/O simulator for Linux and
+ Solaris<superscript>*</superscript> operating systems. It includes:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438212_44437"/></para>
<note>
<para>If you need to create raw devices to use the <literal>sgpdd-survey</literal> tool, note
that raw device 0 cannot be used due to a bug in certain versions of the "raw"
- utility (including that shipped with RHEL4U4.)</para>
+ utility (including the version shipped with Red Hat Enterprise Linux 4U4.)</para>
</note>
<section remap="h3">
<title><indexterm><primary>benchmarking</primary><secondary>tuning storage</secondary></indexterm>Tuning Linux Storage Devices</title>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configurationfilesmoduleparameters">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configurationfilesmoduleparameters">
<title xml:id="configurationfilesmoduleparameters.title">Configuration Files and Module Parameters</title>
<para>This section describes configuration files and module parameters and includes the following sections:</para>
<itemizedlist>
</informaltable>
</section>
<section remap="h3">
- <title><indexterm><primary>configuring</primary><secondary>portals</secondary></indexterm>Portals LND (Linux)</title>
+ <title><indexterm>
+ <primary>configuring</primary>
+ <secondary>portals</secondary>
+ </indexterm>Portals LND Linux (ptllnd)</title>
<para>The Portals LND Linux (<literal>ptllnd</literal>) can be used as a interface layer to communicate with Sandia Portals networking devices. This version is intended to work on Cray XT3 Linux nodes that use Cray Portals as a network transport.</para>
<para><emphasis role="bold">Message Buffers</emphasis></para>
<para>When <literal>ptllnd</literal> starts up, it allocates and posts sufficient message buffers to allow all expected peers (set by concurrent_peers) to send one unsolicited message. The first message that a peer actually sends is (so-called) "<literal>HELLO</literal>" message, used to negotiate how much additional buffering to setup (typically 8 messages). If 10000 peers actually exist, then enough buffers are posted for 80000 messages.</para>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringfailover">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringfailover">
<title xml:id="configuringfailover.title">Configuring Lustre Failover</title>
<para>This chapter describes how to configure Lustre failover using the Heartbeat cluster infrastructure daemon. It includes:</para>
<itemizedlist>
<para>Lustre must be combined with high-availability (HA) software to enable a complete Lustre failover solution. Lustre can be used with several HA packages including:</para>
<itemizedlist>
<listitem>
- <para><emphasis>Red Hat Cluster Manager</emphasis> - For more information about Red Hat
- Cluster Manager, see <link
+ <para><emphasis>Red Hat<superscript>*</superscript>Cluster Manager</emphasis> - For more
+ information about Red Hat Cluster Manager, see <link
xlink:href="http://www.redhat.com/software/rha/cluster/manager/"
xmlns:xlink="http://www.w3.org/1999/xlink"
>http://www.redhat.com/software/rha/cluster/manager/</link>.</para>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlnet">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlnet">
<title xml:id="configuringlnet.title">Configuring Lustre Networking (LNET)</title>
<para>This chapter describes how to configure Lustre Networking (LNET). It includes the following sections:</para>
<itemizedlist>
</itemizedlist>
<note>
<para>Configuring LNET is optional.</para>
- <para>LNET will, by default, use the first TCP/IP interface it discovers on a system (<literal>eth0</literal>). If this network configuration is sufficient, you do not need to configure LNET. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
+ <para>LNET will, by default, use the first TCP/IP interface it discovers on a system
+ (<literal>eth0</literal>). If this network configuration is sufficient, you do not need to
+ configure LNET. LNET configuration is required if you are using InfiniBand or multiple
+ Ethernet interfaces.</para>
</note>
<section xml:id="dbdoclet.50438216_33148">
<title><indexterm><primary>LNET</primary></indexterm>
Overview of LNET Module Parameters</title>
- <para>LNET kernel module (lnet) parameters specify how LNET is to be configured to work with Lustre, including which NICs will be configured to work with Lustre and the routing to be used with Lustre.</para>
- <para>Parameters for LNET can be specified in the <literal>/etc/modprobe.d/lustre.conf</literal> file. In some cases the parameters may have been stored in <literal>/etc/modprobe.conf</literal>, but this has been deprecated since before RHEL5 and SLES10, and having a separate <literal>/etc/modprobe.d/lustre.conf</literal> file simplifies administration and distribution of the Lustre networking configuration. This file contains one or more entries with the syntax:</para>
+ <para>LNET kernel module (lnet) parameters specify how LNET is to be configured to work with the
+ Lustre file system, including which NICs will be configured to work with Lustre file system
+ and the routing to be used with Lustre file system.</para>
+ <para>Parameters for LNET can be specified in the <literal>/etc/modprobe.d/lustre.conf</literal>
+ file. In some cases the parameters may have been stored in
+ <literal>/etc/modprobe.conf</literal>, but this has been deprecated since before Red Hat
+ Enterprise Linux 5 and SUSE Linux Enterprise Server 10 were released. A separate
+ <literal>/etc/modprobe.d/lustre.conf</literal> file simplifies administration and
+ distribution of the Lustre networking configuration. This file contains one or more entries
+ with the syntax:</para>
<screen>options lnet <replaceable>parameter</replaceable>=<replaceable>value</replaceable></screen>
<para>To specify the network interfaces that are to be used for Lustre, set either the <literal>networks</literal> parameter or the <literal>ip2nets</literal> parameter (only one of these parameters can be used at a time):</para>
<itemizedlist>
<para> TCP clients, each with a single TCP interface.</para>
</listitem>
<listitem>
- <para> InfiniBand clients, each with a single Infiniband interface and a TCP/IP interface for administration.</para>
+ <para> InfiniBand clients, each with a single InfiniBand interface and a TCP/IP interface
+ for administration.</para>
</listitem>
</itemizedlist>
<para>To set the <literal>networks</literal> option for this example:</para>
<para>By default, Lustre ignores the loopback (<literal>lo0</literal>) interface. Lustre does not ignore IP addresses aliased to the loopback. If you alias IP addresses to the loopback interface, you must specify all Lustre networks using the LNET networks parameter.</para>
</note>
<note>
- <para>If the server has multiple interfaces on the same subnet, the Linux kernel will send all traffic using the first configured interface. This is a limitation of Linux, not Lustre. In this case, network interface bonding should be used. For more information about network interface bonding, see <xref linkend="settingupbonding"/>.</para>
+ <para>If the server has multiple interfaces on the same subnet, the Linux kernel will send
+ all traffic using the first configured interface. This is a limitation of the Linux
+ kernel, not the Lustre software. In this case, network interface bonding should be used.
+ For more information about network interface bonding, see <xref linkend="settingupbonding"
+ />.</para>
</note>
</section>
</section>
<para>For the example below, the nodes in the network have these IP addresses:</para>
<itemizedlist>
<listitem>
- <para> Server svr1: <literal>eth0</literal> IP address <literal>192.168.0.2</literal>, IP over Infiniband (<literal>o2ib</literal>) address <literal>132.6.1.2</literal>.</para>
+ <para> Server svr1: <literal>eth0</literal> IP address <literal>192.168.0.2</literal>, IP
+ over InfiniBand (<literal>o2ib</literal>) address <literal>132.6.1.2</literal>.</para>
</listitem>
<listitem>
- <para> Server svr2: <literal>eth0</literal> IP address <literal>192.168.0.4</literal>, IP over Infiniband (<literal>o2ib</literal>) address <literal>132.6.1.4</literal>.</para>
+ <para> Server svr2: <literal>eth0</literal> IP address <literal>192.168.0.4</literal>, IP
+ over InfiniBand (<literal>o2ib</literal>) address <literal>132.6.1.4</literal>.</para>
</listitem>
<listitem>
<para> TCP clients have IP addresses <literal>192.168.0.5-255.</literal></para>
</listitem>
<listitem>
- <para> Infiniband clients have IP over Infiniband (<literal>o2ib</literal>) addresses <literal>132.6.[2-3].2, .4, .6, .8</literal>.</para>
+ <para> InfiniBand clients have IP over InfiniBand (<literal>o2ib</literal>) addresses
+ <literal>132.6.[2-3].2, .4, .6, .8</literal>.</para>
</listitem>
</itemizedlist>
<para>The following entry is placed in the <literal>lustre.conf</literal> file on each server and client:</para>
</section>
<section xml:id="dbdoclet.50438216_35668">
<title><indexterm><primary>LNET</primary><secondary>route checker</secondary></indexterm>Configuring the Router Checker</title>
- <para>In a Lustre configuration in which different types of networks, such as a TCP/IP network and an Infiniband network, are connected by routers, a router checker can be run on the clients and servers in the routed configuration to monitor the status of the routers. In a multi-hop routing configuration, router checkers can be configured on routers to monitor the health of their next-hop routers.</para>
+ <para>In a Lustre configuration in which different types of networks, such as a TCP/IP network
+ and an InfiniBand network, are connected by routers, a router checker can be run on the
+ clients and servers in the routed configuration to monitor the status of the routers. In a
+ multi-hop routing configuration, router checkers can be configured on routers to monitor the
+ health of their next-hop routers.</para>
<para>A router checker is configured by setting lnet parameters in <literal>lustre.conf</literal> by including an entry in this form:</para>
<screen>options lnet <replaceable>router_checker_parameter</replaceable>=<replaceable>value</replaceable></screen>
<para>The router checker parameters are:</para>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlustre">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlustre">
<title xml:id="configuringlustre.title">Configuring Lustre</title>
<para>This chapter shows how to configure a simple Lustre system comprised of a combined MGS/MDT, an OST and a client. It includes:</para>
<itemizedlist>
<para><emphasis>Set up network interface bonding on Ethernet interfaces.</emphasis> For information about setting up network interface bonding, see <xref linkend="settingupbonding"/>.</para>
</listitem>
<listitem>
- <para><emphasis>Set</emphasis> lnet <emphasis>module parameters to specify how Lustre Networking (LNET) is to be configured to work with Lustre and test the LNET configuration.</emphasis> LNET will, by default, use the first TCP/IP interface it discovers on a system. If this network configuration is sufficient, you do not need to configure LNET. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
+ <para><emphasis>Set</emphasis> lnet <emphasis>module parameters to specify how Lustre
+ Networking (LNET) is to be configured to work with Lustre and test the LNET
+ configuration.</emphasis> LNET will, by default, use the first TCP/IP interface it
+ discovers on a system. If this network configuration is sufficient, you do not need to
+ configure LNET. LNET configuration is required if you are using InfiniBand or multiple
+ Ethernet interfaces.</para>
</listitem>
</itemizedlist>
<para>For information about configuring LNET, see <xref linkend="configuringlnet"/>. For information about testing LNET, see <xref linkend="lnetselftest"/>.</para>
<para>You should specify the OST index number at format time in order to simplify translating the OST number in error messages or file striping to the OSS node and block device later on.</para>
<para>If you are using block devices that are accessible from multiple OSS nodes, ensure that you mount the OSTs from only one OSS node at at time. It is strongly recommended that multiple-mount protection be enabled for such devices to prevent serious data corruption. For more information about multiple-mount protection, see <xref linkend="managingfailover"/>.</para>
<note>
- <para>Lustre currently supports block devices up to 128 TB on RHEL 5/6 (up to 8 TB on other distributions). If the device size is only slightly larger that 16 TB, it is recommended that you limit the file system size to 16 TB at format time. We recommend that you not place DOS partitions on top of RAID 5/6 block devices due to negative impacts on performance, but instead format the whole disk for the filesystem.</para>
+ <para>Lustre currently supports block devices up to 128 TB on Red Hat Enterprise Linux 5
+ and 6 (up to 8 TB on other distributions). If the device size is only slightly larger
+ that 16 TB, it is recommended that you limit the file system size to 16 TB at format
+ time. We recommend that you not place DOS partitions on top of RAID 5/6 block devices
+ due to negative impacts on performance, but instead format the whole disk for the file
+ system.</para>
</note>
</listitem>
<listitem xml:id="dbdoclet.50438267_pgfId-1293955">
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringquotas">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringquotas">
<title xml:id="configuringquotas.title">Configuring and Managing Quotas</title>
<para>This chapter describes how to configure quotas and includes the following sections:</para>
<itemizedlist>
<para>Quotas are allocated and consumed in a quantized fashion.</para>
</listitem>
<listitem>
- <para>Client does not set the <literal>usrquota</literal> or <literal>grpquota</literal> options to mount. As of Lustre 2.4, space accounting is always enabled by default and quota enforcement can be enabled/disabled on a per-filesystem basis with <literal>lctl conf_param</literal>. It is worth noting that both <literal>lfs quotaon</literal> and <literal>quota_type</literal> are deprecated as of Lustre 2.4.0.</para>
+ <para>Client does not set the <literal>usrquota</literal> or <literal>grpquota</literal>
+ options to mount. As of Lustre 2.4, space accounting is always enabled by default and
+ quota enforcement can be enabled/disabled on a per-file system basis with <literal>lctl
+ conf_param</literal>. It is worth noting that both <literal>lfs quotaon</literal> and
+ <literal>quota_type</literal> are deprecated as of Lustre 2.4.0.</para>
</listitem>
</itemizedlist>
<caution>
flag is present.</para>
</listitem>
<listitem>
- <para>For ZFS backend, accounting ZAPs are created and maintained by the ZFS filesystem
+ <para>For ZFS backend, accounting ZAPs are created and maintained by the ZFS file system
itself. While ZFS tracks per-user and group block usage, it does not handle inode
accounting. The ZFS OSD implements its own support for inode tracking. Two options are
available:</para>
<para>As of Lustre 2.4.0, quota enforcement is thus turned on/off independently of space
accounting which is always enabled. <literal>lfs
quota<replaceable>on|off</replaceable></literal> as well as the per-target
- <literal>quota_type</literal> parameter are deprecated in favor of a single per-filesystem
- quota parameter controlling inode/block quota enforcement. Like all permanent parameters,
- this quota parameter can be set via <literal>lctl conf_param</literal> on the MGS via the
- following syntax:</para>
+ <literal>quota_type</literal> parameter are deprecated in favor of a single per-file
+ system quota parameter controlling inode/block quota enforcement. Like all permanent
+ parameters, this quota parameter can be set via <literal>lctl conf_param</literal> on the
+ MGS via the following syntax:</para>
<screen>lctl conf_param <replaceable>fsname</replaceable>.quota.<replaceable>ost|mdt</replaceable>=<replaceable>u|g|ug|none</replaceable></screen>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<para>Examples:</para>
- <para>To turn on user and group quotas for block only on filesystem testfs1, run:</para>
+ <para>To turn on user and group quotas for block only on file system
+ <literal>testfs1</literal>, run:</para>
<screen>$ lctl conf_param testfs1.quota.ost=ug</screen>
- <para>To turn on group quotas for inodes on filesystem testfs2, run:</para>
- <screen>$ lctl conf_param testfs2.quota.mdt=g</screen>
- <para>To turn off user and group quotas for both inode and block on filesystem testfs3,
+ <para>To turn on group quotas for inodes on file system <literal>testfs2</literal>,
run:</para>
+ <screen>$ lctl conf_param testfs2.quota.mdt=g</screen>
+ <para>To turn off user and group quotas for both inode and block on file system
+ <literal>testfs3</literal>, run:</para>
<screen>$ lctl conf_param testfs3.quota.ost=none</screen>
<screen>$ lctl conf_param testfs3.quota.mdt=none</screen>
- <para>Once the quota parameter set on the MGS, all targets which are part of the filesystem
+ <para>Once the quota parameter set on the MGS, all targets which are part of the file system
will be notified of the new quota settings and enable/disable quota enforcement as needed.
The per-target enforcement status can still be verified by running the following command on
the Lustre servers:</para>
$ lfs setquota -u bob -b 307200 -B 309200 -i 10000 -I 11000 /mnt/testfs</screen></para>
</note>For Lustre versions older than 2.4, <literal>lfs quotacheck</literal> must be first
run from a client node to create quota files on the Lustre targets (i.e. the MDT and OSTs).
- <literal>lfs quotacheck</literal> requires the filesystem to be quiescent (i.e. no
+ <literal>lfs quotacheck</literal> requires the file system to be quiescent (i.e. no
modifying operations like write, truncate, create or delete should run concurrently).
Failure to follow this caution may result in inaccurate user/group disk usage. Operations
that do not change Lustre files (such as read or mount) are okay to run. <literal>lfs
quotacheck</literal> performs a scan on all the Lustre targets to calculates the
- block/inode usage for each user/group. If the Lustre filesystem has many files,
+ block/inode usage for each user/group. If the Lustre file system has many files,
<literal>quotacheck</literal> may take a long time to complete. Several options can be
passed to <literal>lfs quotacheck</literal>:</para>
<screen># lfs quotacheck -ug /mnt/testfs</screen>
</section>
<section xml:id="quota_administration">
<title><indexterm><primary>Quotas</primary><secondary>creating</secondary></indexterm>Quota Administration</title>
- <para>Once the filesystem is up and running, quota limits on blocks and files can be set for both user and group. This is controlled via three quota parameters:</para>
+ <para>Once the file system is up and running, quota limits on blocks and files can be set for
+ both user and group. This is controlled via three quota parameters:</para>
<para><emphasis role="bold">Grace period</emphasis> -- The period of time (in seconds) within which users are allowed to exceed their soft limit. There are four types of grace periods:</para>
<itemizedlist>
<listitem>
</section>
<section xml:id="quota_interoperability">
<title><indexterm><primary>Quotas</primary><secondary>Interoperability</secondary></indexterm>Interoperability</title>
- <para>The new quota protocol introduced in Lustre 2.4.0 <emphasis role="bold">isn't compatible</emphasis> with the old one. As a consequence, <emphasis role="bold">all Lustre servers must be upgraded to 2.4.0 for quota to be functional</emphasis>. Quota limits set on the Lustre filesystem prior to the upgrade will be automatically migrated to the new quota index format. As for accounting information with ldiskfs backend, they will be regenerated by running <literal>tunefs.lustre --quota</literal> against all targets. It is worth noting that running <literal>tunefs.lustre --quota</literal> is <emphasis role="bold">mandatory</emphasis> for all targets formatted with a Lustre version older than 2.4.0, otherwise quota enforcement as well as accounting won't be functional.</para>
+ <para>The new quota protocol introduced in Lustre 2.4.0 <emphasis role="bold">isn't
+ compatible</emphasis> with the old one. As a consequence, <emphasis role="bold">all Lustre
+ servers must be upgraded to 2.4.0 for quota to be functional</emphasis>. Quota limits set on
+ the Lustre file system prior to the upgrade will be automatically migrated to the new quota
+ index format. As for accounting information with ldiskfs backend, they will be regenerated by
+ running <literal>tunefs.lustre --quota</literal> against all targets. It is worth noting that
+ running <literal>tunefs.lustre --quota</literal> is <emphasis role="bold">mandatory</emphasis>
+ for all targets formatted with a Lustre version older than 2.4.0, otherwise quota enforcement
+ as well as accounting won't be functional.</para>
<para>Besides, the 2.4 quota protocol takes for granted that the Lustre client supports the <literal>OBD_CONNECT_EINPROGRESS</literal> connect flag. Clients supporting this flag will retry indefinitely when the server returns <literal>EINPROGRESS</literal> in a reply. Here is the list of Lustre client version which are compatible with 2.4:</para>
<itemizedlist>
<listitem>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringstorage">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringstorage">
<title xml:id="configuringstorage.title">Configuring Storage on a Lustre File System</title>
<para>This chapter describes best practices for storage selection and file system options to optimize performance on RAID, and includes the following sections:</para>
<itemizedlist>
<title><indexterm><primary>storage</primary><secondary>configuring</secondary><tertiary>external journal</tertiary></indexterm>Choosing Parameters for an External Journal</title>
<para>If you have configured a RAID array and use it directly as an OST, it contains both data and metadata. For better performance, we recommend putting the OST journal on a separate device, by creating a small RAID 1 array and using it as an external journal for the OST.</para>
<para>Lustre's default journal size is 400 MB. A journal size of up to 1 GB has shown increased performance but diminishing returns are seen for larger journals. Additionally, a copy of the journal is kept in RAM. Therefore, make sure you have enough memory available to hold copies of all the journals.</para>
- <para>The file system journal options are specified to mkfs.luster using the <literal>--mkfsoptions</literal> parameter. For example:</para>
+ <para>The file system journal options are specified to <literal>mkfs.lustre</literal> using
+ the <literal>--mkfsoptions</literal> parameter. For example:</para>
<screen>--mkfsoptions "<replaceable>other_options</replaceable> -j -J device=/dev/mdJ" </screen>
<para>To create an external journal, perform these steps for each OST on the OSS:</para>
<orderedlist>
<glossterm>configlog </glossterm>
<glossdef>
<para>An llog file used in a node, or retrieved from a management server over the network
- with configuration instructions for the Lustre* file system at startup time.</para>
+ with configuration instructions for the Lustre file system at startup time.</para>
</glossdef>
</glossentry>
<glossentry xml:id="configlock">
</glossterm>
<glossdef>
<para>Lustre network driver. A code module that enables LNET support over particular
- transports, such as TCP and various kinds of InfiniBand* networks.</para>
+ transports, such as TCP and various kinds of InfiniBand networks.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lnet">
<para>The starting point for administering Lustre is to monitor all logs and console logs for system health:</para>
<para> - Monitor logs on all servers and all clients.</para>
<para> - Invest in tools that allow you to condense logs from multiple systems.</para>
- <para> - Use the logging resources provided by Linux.</para>
+ <para> - Use the logging resources provided in the Linux distribution.</para>
</tip>
</partintro>
<?xml version="1.0" encoding="UTF-8"?>
<part version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='part.intro'>
- <title>Introducing the Lustre* File System</title>
+ <title>Introducing the Lustre<superscript>*</superscript> File System</title>
<partintro>
- <para>Part I provides background information to help you understand the Lustre* file system
+ <para>Part I provides background information to help you understand the Lustre file system
architecture and how the major components fit together. You will find information in
this section about: </para>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="installoverview">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="installoverview">
<title xml:id="installoverview.title">Installation Overview</title>
<para>This chapter provides on overview of the procedures required to set up, install and configure a Lustre file system.</para>
<note>
</listitem>
<listitem>
<para><emphasis>(Optional)</emphasis> <emphasis role="bold">Configure Lustre Networking (LNET).</emphasis></para>
- <para>See <xref linkend="configuringlnet"/> - Describes how to configure LNET if the default configuration is not sufficient. By default, LNET will use the first TCP/IP interface it discovers on a system. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
+ <para>See <xref linkend="configuringlnet"/> - Describes how to configure LNET if the default
+ configuration is not sufficient. By default, LNET will use the first TCP/IP interface it
+ discovers on a system. LNET configuration is required if you are using InfiniBand or
+ multiple Ethernet interfaces.</para>
</listitem>
<listitem>
<para><emphasis>(Required)</emphasis> <emphasis role="bold">Configure Lustre.</emphasis></para>
<?xml version='1.0' encoding='UTF-8'?>
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0"
xml:lang="en-US" xml:id="installinglustre">
- <title xml:id="installinglustre.title">Installing the Lustre* Software</title>
- <para>This chapter describes how to install the Lustre* software from RPM packages. It
+ <title xml:id="installinglustre.title">Installing the Lustre Software</title>
+ <para>This chapter describes how to install the Lustre software from RPM packages. It
includes:</para>
<itemizedlist>
<listitem>
</tgroup>
</table>
</para>
- <para><superscript>1</superscript>Red Hat* Enterprise Edition*, CentOS Enterprise Linux
- Distribution, SUSE Linux Enterprise Server, Fedora* F18 Linux kernel.</para>
+ <para><superscript>1</superscript>Red Hat Enterprise Edition (RHEL), CentOS Enterprise Linux
+ Distribution (CentOS), SUSE Linux Enterprise Server (SLES), Fedora<superscript>*</superscript>
+ F18 Linux kernel (FC18).</para>
<section xml:id="section_rqs_tjw_3k">
<title>Software Requirements</title>
<para>To install the Lustre software from RPMs, the following are required:<itemizedlist>
<listitem>
<para><emphasis role="bold"><emphasis role="italic">Lustre server
packages</emphasis></emphasis>. The required packages for Lustre servers are listed
- in the table below, where <replaceable>ver</replaceable> refers to the Linux* kernel
+ in the table below, where <replaceable>ver</replaceable> refers to the Linux kernel
distribution (e.g., 2.6.32-358.6.2.el6) and <replaceable>arch</replaceable> refers to
the processor architecture (e.g., x86_64). These packages are available in the <link
xl:href="https://wiki.hpdd.intel.com/display/PUB/Lustre+Releases">Lustre
<colspec colname="c2" colnum="2" colwidth="1*"/>
<thead>
<row>
- <entry>Package Name*</entry>
+ <entry>Package Name</entry>
<entry>Description</entry>
</row>
</thead>
IPoIB</entry>
</row>
<row>
- <entry>InfiniBand* network</entry>
+ <entry>InfiniBand network</entry>
<entry>OpenFabrics OFED (o2ib)</entry>
</row>
<row>
</row>
<row>
<entry>MX</entry>
- <entry>Myrinet* network</entry>
+ <entry>Myrinet network</entry>
</row>
<row>
<entry>ra</entry>
- <entry>RapidArray* interconnect</entry>
+ <entry>RapidArray<superscript>*</superscript> interconnect</entry>
</row>
<row>
<entry>Elan</entry>
</itemizedlist></para>
<para>
<note>
- <para>The InfiniBand* and TCP Lustre LNDs are routinely tested during release cycles. The
+ <para>The InfiniBand and TCP Lustre LNDs are routinely tested during release cycles. The
other LNDs are maintained by their respective owners</para>
</note>
<itemizedlist>
third party modules, such as network drivers.</para>
</listitem>
<listitem>
- <para><code>kernel-firmware</code> - Standard Red Hat Enterprise Linux package
- that has been recompiled to work with the Lustre kernel.</para>
+ <para><code>kernel-firmware</code> - Standard Red Hat Enterprise Linux
+ distribution that has been recompiled to work with the Lustre kernel.</para>
</listitem>
<listitem>
<para><code>kernel-headers</code> - Header files installed under /user/include and
requirements are met.<itemizedlist>
<listitem>
<para><emphasis role="italic">(Required)</emphasis>
- <emphasis role="italic"><emphasis role="bold">Disable Security-Enhanced Linux
- (SELinux) on all Lustre servers and clients.</emphasis></emphasis> The Lustre
- software does not support SELinux. Therefore, the SELinux system extension must be
- disabled on all Lustre nodes. Also, make sure other security extensions (such as the
- Novell AppArmor* security system) and network packet filtering tools (such as
- iptables) do not interfere with the Lustre software.</para>
+ <emphasis role="italic"><emphasis role="bold">Disable Security-Enhanced
+ Linux<superscript>*</superscript> (SELinux) on all Lustre servers and
+ clients.</emphasis></emphasis> The Lustre software does not support SELinux.
+ Therefore, the SELinux system extension must be disabled on all Lustre nodes. Also,
+ make sure other security extensions (such as the Novell
+ AppArmor<superscript>*</superscript> security system) and network packet filtering
+ tools (such as iptables) do not interfere with the Lustre software.</para>
</listitem>
<listitem>
<para><emphasis role="italic">(Required) </emphasis><emphasis role="italic"><emphasis
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="installinglustrefromsourcecode">
- <title xml:id="installinglustrefromsourcecode.title"><indexterm
- xmlns:xi="http://www.w3.org/2001/XInclude">
- <primary>installing</primary>
- <secondary>from source code</secondary>
- </indexterm> Installing a Lustre* File System from Source Code</title>
- <para>This chapter describes how to create a customized Lustre* server kernel from source code.
+ <title xml:id="installinglustrefromsourcecode.title">Installing a Lustre File System from Source
+ Code</title>
+ <para>This chapter describes how to create a customized Lustre server kernel from source code.
Sections included are:</para>
<itemizedlist>
<listitem>
<itemizedlist>
<listitem>
<para>A x86_64 machine with a fresh installation of a Linux operating system.</para>
- <note><para>This section of the Manual only describes the specific procedure against building on Red Hat* 6 Linux and derivatives. For a complete set of supported platforms please visit <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="LustreTestMatrixTable"/>.The server kernel should be one of the supported kernel versions but it is not necessary to run a vendor distribution to build the Lustre software.</para></note>
+ <note><para>This section of the Manual only describes the specific procedure against building on Red Hat
+ Enterprise Linux 6 and derivatives. For a complete set of supported platforms please
+ visit <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="LustreTestMatrixTable"
+ />.The server kernel should be one of the supported kernel versions but it is not
+ necessary to run a vendor distribution to build the Lustre software.</para></note>
</listitem>
<listitem>
<para>Access to the Lustre software git repository at <link xl:href="http://git.hpdd.intel.com/">http://git.hpdd.intel.com/</link>.</para>
<primary>quilt</primary>
</indexterm>Provisioning the Build Machine and Installing Dependencies</title>
<note>
- <para>This example is specific for building a modified Red Hat* Enteprise Linux* 6 kernel for the Lustre file system server. It is assumed that the operating system has been
- freshly installed on a machine with the hostname <literal>rhel6-master</literal>.</para>
+ <para>This example is specific for building a modified Red Hat Enteprise Linux 6 kernel for
+ the Lustre file system server. It is assumed that the operating system has been freshly
+ installed on a machine with the hostname <literal>rhel6-master</literal>.</para>
</note>
<para>To provision the build machine and install dependencies, complete these steps.</para>
<orderedlist>
</para>
<note>
<para>
- <package>newt-devel</package> may not be available for RHEL6. One option is to
- download the <link xmlns:xlink="http://www.w3.org/1999/xlink"
+ <package>newt-devel</package> may not be available for Red Hat Enterprise Linux 6. One
+ option is to download the <link xmlns:xlink="http://www.w3.org/1999/xlink"
xlink:href="http://mirror.centos.org/centos/6/os/x86_64/Packages/newt-devel-0.52.11-3.el6.x86_64.rpm"
>newt-devel</link>, <link xmlns:xlink="http://www.w3.org/1999/xlink"
xlink:href="http://mirror.centos.org/centos/6/os/x86_64/Packages/slang-devel-2.2.1-1.el6.x86_64.rpm"
>slang-devel</link>, and <link xmlns:xlink="http://www.w3.org/1999/xlink"
xlink:href="http://mirror.centos.org/centos/6/os/i386/Packages/asciidoc-8.4.5-4.1.el6.noarch.rpm"
- >asciidoc</link> RPMs from CentOS and install using:</para>
+ >asciidoc</link> RPMs from the CentOS Enterprise Linux Distribution and install
+ using:</para>
<para>
<screen>yum --nogpgcheck localinstall
./newt-devel-0.52.11-3.el6.x86_64.rpm
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lnetselftest">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lnetselftest">
<title xml:id="lnetselftest.title">Testing Lustre Network Performance (LNET Self-Test)</title>
<para><anchor xml:id="dbdoclet.50438223_36273" xreflabel=""/>This chapter describes the LNET self-test, which is used by site administrators to confirm that Lustre Networking (LNET) has been properly installed and configured, and that underlying network software and hardware are performing according to expectations. The chapter includes:</para>
<itemizedlist>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustredebugging">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustredebugging">
<title xml:id="lustredebugging.title">Lustre Debugging</title>
<para>This chapter describes tips and information to debug Lustre, and includes the following sections:</para>
<itemizedlist>
<para>The tools described in this section are provided in the Linux kernel or are available at an external website. For information about using some of these tools for Lustre debugging, see <xref linkend="dbdoclet.50438274_23607"/> and <xref linkend="dbdoclet.50438274_80443"/>.</para>
<section remap="h4">
<title><indexterm><primary>debugging</primary><secondary>admin tools</secondary></indexterm>Tools for Administrators and Developers</title>
- <para>Some general debugging tools provided as a part of the standard Linux distro are:</para>
+ <para>Some general debugging tools provided as a part of the standard Linux distribution
+ are:</para>
<itemizedlist>
<listitem>
<para><emphasis role="bold">
<listitem>
<para><emphasis role="bold">
<literal>netdump</literal>
- </emphasis>. A crash dump utility from Red Hat that allows memory images to be dumped over a network to a central server for analysis. The <literal>netdump</literal> utility was replaced by <literal>kdump</literal> in RHEL 5. For more information about <literal>netdump</literal>, see <link xl:href="http://www.redhat.com/support/wpapers/redhat/netdump/">Red Hat, Inc.'s Network Console and Crash Dump Facility</link>.</para>
+ </emphasis>. A crash dump utility from Red Hat that allows memory images to be dumped
+ over a network to a central server for analysis. The <literal>netdump</literal>
+ utility was replaced by <literal>kdump</literal> in Red Hat Enterprise Linux 5. For
+ more information about <literal>netdump</literal>, see <link
+ xl:href="http://www.redhat.com/support/wpapers/redhat/netdump/">Red Hat, Inc.'s
+ Network Console and Crash Dump Facility</link>.</para>
</listitem>
<listitem>
<para><emphasis><literal>wireshark</literal> </emphasis> . A network
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremaintenance">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremaintenance">
<title xml:id="lustremaintenance.title">Lustre Maintenance</title>
<para>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre maintenance tasks:</para>
<itemizedlist>
</section>
<section xml:id="dbdoclet.addingamdt" condition='l24'>
<title><indexterm><primary>maintenance</primary><secondary>adding an MDT</secondary></indexterm>Adding a new MDT to a Lustre file system</title>
- <para>Additional MDTs can be added to serve one or more remote sub-directories within the filesystem. It is possible to have multiple remote sub-directories reference the same MDT. However, the root directory will always be located on MDT0. To add a new MDT into the file system:</para>
+ <para>Additional MDTs can be added to serve one or more remote sub-directories within the
+ file system. It is possible to have multiple remote sub-directories reference the same MDT.
+ However, the root directory will always be located on MDT0. To add a new MDT into the file
+ system:</para>
<orderedlist>
<listitem>
<para>Discover the maximum MDT index. Each MDTs must have unique index.</para>
<para>Files existing prior to the expansion can optionally be rebalanced with an in-place copy, which can be done with a simple script. The basic method is to copy existing files to a temporary file, then move the temp file over the old one. This should not be attempted with files which are currently being written to by users or applications. This operation redistributes the stripes over the entire set of OSTs.</para>
<para>For example, to rebalance all files within <literal>/mnt/lustre/dir</literal>, enter:</para>
<screen>client# lfs_migrate /mnt/lustre/file</screen>
- <para>To migrate files within the <literal>/test</literal> filesystem on <literal>OST0004</literal> that are larger than 4GB in size, enter:</para>
+ <para>To migrate files within the <literal>/test</literal> file system on
+ <literal>OST0004</literal> that are larger than 4GB in size, enter:</para>
<screen>client# lfs find /test -obd test-OST0004 -size +4G | lfs_migrate -y</screen>
<para>See <xref linkend="dbdoclet.50438206_42260"/> for more details.</para>
</listitem>
corruption), an effort should be made to preserve these files. </para>
<orderedlist>
<listitem>
- <para>Mount the OST filesystem.
+ <para>Mount the OST file system.
<screen>oss# mkdir -p /mnt/ost
[oss]# mount -t ldiskfs <replaceable>/dev/ost_device</replaceable> /mnt/ost</screen>
</para>
</para>
</listitem>
<listitem>
- <para> Unmount the OST filesystem. <screen>oss# umount /mnt/ost</screen>
+ <para> Unmount the OST file system. <screen>oss# umount /mnt/ost</screen>
</para>
</listitem>
</orderedlist>
</para>
</listitem>
<listitem>
- <para> Mount the OST filesystem.
+ <para> Mount the OST file system.
<screen>oss# mkdir /mnt/ost
oss# mount -t ldiskfs <replaceable>/dev/new_ost_dev</replaceable> <replaceable>/mnt/ost</replaceable></screen>
</para>
oss0# dd if=/tmp/ldd of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 skip=5 conv=notrunc</screen></para>
</listitem>
<listitem>
- <para> Unmount the OST filesystem. <screen>oss# umount /mnt/ost</screen>
+ <para> Unmount the OST file system. <screen>oss# umount /mnt/ost</screen>
</para>
</listitem>
</orderedlist>
<listitem>
<para>Start the MGS.</para>
<screen>mgs# mount -t lustre <replaceable>/dev/mgs_device</replaceable> <replaceable>/mgs_mount_point</replaceable></screen>
- <para>Check to make sure it knows about all your filesystem</para>
+ <para>Check to make sure it knows about all your file system</para>
<screen>cat /proc/fs/lustre/mgs/MGS/filesystems</screen>
</listitem>
<listitem>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremonitoring">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremonitoring">
<title xml:id="lustremonitoring.title">Lustre Monitoring</title>
<para>This chapter provides information on monitoring Lustre and includes the following sections:</para>
<itemizedlist>
<indexterm><primary>monitoring</primary><secondary>jobstats</secondary></indexterm>
Lustre Jobstats</title>
- <para>The Lustre Jobstats feature is available starting in Lustre version 2.3. It collects filesystem operation statistics for the jobs running on Lustre clients, and exposes them via procfs on the server. Job schedulers known to be able to work with jobstats include: SLURM, SGE, LSF, Loadleveler, PBS and Maui/MOAB.</para>
+ <para>The Lustre Jobstats feature is available starting in Lustre version 2.3. It collects file
+ system operation statistics for the jobs running on Lustre clients, and exposes them via
+ procfs on the server. Job schedulers known to be able to work with jobstats include: SLURM,
+ SGE, LSF, Loadleveler, PBS and Maui/MOAB.</para>
<para>Since Jobstats is implemented in a scheduler-agnostic manner, it is likely that it will be able to work with other schedulers also.</para>
<section remap="h3">
<title><indexterm><primary>monitoring</primary><secondary>jobstats</secondary></indexterm>
$ lctl get_param jobid_var
jobid_var=disable
</screen>
- <para>The Lustre Jobstats code extracts the job identifier from an environment variable set by the scheduler when the job is started. To enable jobstats, specify the <literal>jobid_var</literal> to name the environment variable set by the scheduler. For example, SLURM sets the <literal>SLURM_JOB_ID</literal> environment variable with the unique job ID on each client. To permanently enable Jobstats on the <literal>testfs</literal> filesystem:</para>
+ <para>The Lustre Jobstats code extracts the job identifier from an environment variable set by
+ the scheduler when the job is started. To enable jobstats, specify the
+ <literal>jobid_var</literal> to name the environment variable set by the scheduler. For
+ example, SLURM sets the <literal>SLURM_JOB_ID</literal> environment variable with the unique
+ job ID on each client. To permanently enable Jobstats on the <literal>testfs</literal> file
+ system:</para>
<screen>$ lctl conf_param testfs.sys.jobid_var=SLURM_JOB_ID</screen>
<para>The value of <literal>jobid_var</literal> can be:</para>
<informaltable frame="all">
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustreoperations">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustreoperations">
<title xml:id="lustreoperations.title">Lustre Operations</title>
<para>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre administration tasks:</para>
<itemizedlist>
<para>In this example, the MDT, an OST (ost0) and file system (testfs) are mounted.</para>
<screen>LABEL=testfs-MDT0000 /mnt/test/mdt lustre defaults,_netdev,noauto 0 0
LABEL=testfs-OST0000 /mnt/test/ost0 lustre defaults,_netdev,noauto 0 0</screen>
- <para>In general, it is wise to specify noauto and let your high-availability (HA) package manage when to mount the device. If you are not using failover, make sure that networking has been started before mounting a Lustre server. RedHat, SuSE, Debian (and perhaps others) use the <literal>_netdev</literal> flag to ensure that these disks are mounted after the network is up.</para>
+ <para>In general, it is wise to specify noauto and let your high-availability (HA) package
+ manage when to mount the device. If you are not using failover, make sure that networking has
+ been started before mounting a Lustre server. If you are running Red Hat Enterprise Linux,
+ SUSE Linux Enterprise Server, Debian operating system (and perhaps others), use the
+ <literal>_netdev</literal> flag to ensure that these disks are mounted after the network is
+ up.</para>
<para>We are mounting by disk label here. The label of a device can be read with <literal>e2label</literal>. The label of a newly-formatted Lustre server may end in <literal>FFFF</literal> if the <literal>--index</literal> option is not specified to <literal>mkfs.lustre</literal>, meaning that it has yet to be assigned. The assignment takes place when the server is first started, and the disk label is updated. It is recommended that the <literal>--index</literal> option always be used, which will also ensure that the label is set at format time.</para>
<caution>
<para>Do not do this when the client and OSS are on the same node, as memory pressure between the client and OSS can lead to deadlocks.</para>
<para>By default, the <literal>mkfs.lustre</literal> command creates a file system named <literal>lustre</literal>. To specify a different file system name (limited to 8 characters) at format time, use the <literal>--fsname</literal> option:</para>
<para><screen>mkfs.lustre --fsname=<replaceable>file_system_name</replaceable></screen></para>
<note>
- <para>The MDT, OSTs and clients in the new file system must use the same filesystem name (prepended to the device name). For example, for a new file system named <literal>foo</literal>, the MDT and two OSTs would be named <literal>foo-MDT0000</literal>, <literal>foo-OST0000</literal>, and <literal>foo-OST0001</literal>.</para>
+ <para>The MDT, OSTs and clients in the new file system must use the same file system name
+ (prepended to the device name). For example, for a new file system named
+ <literal>foo</literal>, the MDT and two OSTs would be named
+ <literal>foo-MDT0000</literal>, <literal>foo-OST0000</literal>, and
+ <literal>foo-OST0001</literal>.</para>
</note>
<para>To mount a client on the file system, run:</para>
<screen>client# mount -t lustre <replaceable>mgsnode</replaceable>:<replaceable>/new_fsname</replaceable> <replaceable>/mount_point</replaceable></screen>
</section>
<section xml:id="dbdoclet.50438194_55253">
<title>Setting Parameters with <literal>tunefs.lustre</literal></title>
- <para>If a server (OSS or MDS) is stopped, parameters can be added to an existing filesystem using the <literal>--param</literal> option to the <literal>tunefs.lustre</literal> command. For example:</para>
+ <para>If a server (OSS or MDS) is stopped, parameters can be added to an existing file system
+ using the <literal>--param</literal> option to the <literal>tunefs.lustre</literal> command.
+ For example:</para>
<screen>oss# tunefs.lustre --param=failover.node=192.168.0.13@tcp0 /dev/sda</screen>
<para>With <literal>tunefs.lustre</literal>, parameters are <emphasis>additive</emphasis> -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old <literal>tunefs.lustre</literal> parameters and just use newly-specified parameters, run:</para>
<screen>mds# tunefs.lustre --erase-params --param=<replaceable>new_parameters</replaceable> </screen>
</section>
<section xml:id="dbdoclet.50438194_70905">
<title><indexterm><primary>operations</primary><secondary>erasing a file system</secondary></indexterm>Erasing a File System</title>
- <para>If you want to erase a file system and permanently delete all the
- data in the filesystem, run this command on your targets:</para>
+ <para>If you want to erase a file system and permanently delete all the data in the file system,
+ run this command on your targets:</para>
<screen>$ "mkfs.lustre --reformat"</screen>
<para>If you are using a separate MGS and want to keep other file systems defined on that MGS, then set the <literal>writeconf</literal> flag on the MDT for that file system. The <literal>writeconf</literal> flag causes the configuration logs to be erased; they are regenerated the next time the servers start.</para>
<para>To set the <literal>writeconf</literal> flag on the MDT:</para>
</section>
<section xml:id="dbdoclet.50438194_16954">
<title><indexterm><primary>operations</primary><secondary>reclaiming space</secondary></indexterm>Reclaiming Reserved Disk Space</title>
- <para>All current Lustre installations run the ldiskfs file system
- internally on service nodes. By default, ldiskfs reserves 5% of the disk
- space to avoid filesystem fragmentation. In order to reclaim this space,
- run the following command on your OSS for each OST in the filesystem:</para>
+ <para>All current Lustre installations run the ldiskfs file system internally on service nodes.
+ By default, ldiskfs reserves 5% of the disk space to avoid file system fragmentation. In order
+ to reclaim this space, run the following command on your OSS for each OST in the file
+ system:</para>
<screen>tune2fs [-m reserved_blocks_percent] /dev/<emphasis>{ostdev}</emphasis></screen>
<para>You do not need to shut down Lustre before running this command or restart it afterwards.</para>
<warning>
- <para>Reducing the space reservation can cause severe performance
- degradation as the OST filesystem becomes more than 95% full, due to
- difficulty in locating large areas of contiguous free space. This
- performance degradation may persist even if the space usage drops
- below 95% again. It is recommended NOT to reduce the reserved disk
- space below 5%.</para>
+ <para>Reducing the space reservation can cause severe performance degradation as the OST file
+ system becomes more than 95% full, due to difficulty in locating large areas of contiguous
+ free space. This performance degradation may persist even if the space usage drops below 95%
+ again. It is recommended NOT to reduce the reserved disk space below 5%.</para>
</warning>
</section>
<section xml:id="dbdoclet.50438194_69998">
<para>The header information includes:</para>
<itemizedlist>
<listitem>
- <para><literal>snapshot_time</literal> - UNIX* epoch instant the file was read.</para>
+ <para><literal>snapshot_time</literal> - UNIX epoch instant the file was read.</para>
</listitem>
<listitem>
<para><literal>read RPCs in flight</literal> - Number of read RPCs issued by the OSC, but
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustreprogramminginterfaces">
- <title xml:id="lustreprogramminginterfaces.title">Lustre* Programming Interfaces</title>
+ <title xml:id="lustreprogramminginterfaces.title">Lustre Programming Interfaces</title>
<para>This chapter describes public programming interfaces to that can be used to control various
- aspects of a Lustre* file system from userspace. This chapter includes the following
+ aspects of a Lustre file system from userspace. This chapter includes the following
sections:</para>
<itemizedlist>
<listitem>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustrerecovery">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustrerecovery">
<title xml:id="lustrerecovery.title">Lustre Recovery</title>
<para>This chapter describes how recovery is implemented in Lustre and includes the following sections:</para>
<itemizedlist>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretroubleshooting">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretroubleshooting">
<title xml:id="lustretroubleshooting.title">Lustre Troubleshooting</title>
<para>This chapter provides information to troubleshoot Lustre, submit a Lustre bug, and Lustre performance tips. It includes the following sections:</para>
<itemizedlist>
<para>Several resources are available to help troubleshoot Lustre. This section describes error numbers, error messages and logs.</para>
<section remap="h3">
<title><indexterm><primary>troubleshooting</primary><secondary>error numbers</secondary></indexterm>Error Numbers</title>
- <para>Error numbers for Lustre come from the Linux errno.h, and are located in <literal>/usr/include/asm-generic/errno.h</literal>. Lustre does not use all of the available Linux error numbers. The exact meaning of an error number depends on where it is used. Here is a summary of the basic errors that Lustre users may encounter.</para>
+ <para>Error numbers are generated by the Linux operating system and are located in
+ <literal>/usr/include/asm-generic/errno.h</literal>. The Lustre software does not use all
+ of the available Linux error numbers. The exact meaning of an error number depends on where
+ it is used. Here is a summary of the basic errors that Lustre users may encounter.</para>
<informaltable frame="all">
<tgroup cols="3">
<colspec colname="c1" colwidth="33*"/>
email list or search the archives for information about your issue.</para>
</listitem>
<listitem>
- <para>Submit a Jira ticket. The <link xmlns:xlink="http://www.w3.org/1999/xlink"
- xlink:href="https://jira.hpdd.intel.com/secure/Dashboard.jspa">Jira</link> project
- tracker is the bug tracking and project management tool used for the Lustre project. If
- you are a first-time user, you'll need to open an account by clicking on <emphasis
- role="bold">Sign up</emphasis> on the Welcome page.</para>
+ <para>Submit a ticket to the <link xmlns:xlink="http://www.w3.org/1999/xlink"
+ xlink:href="https://jira.hpdd.intel.com/secure/Dashboard.jspa"
+ >Jira</link><abbrev><superscript>*</superscript></abbrev> bug tracking and project
+ management tool used for the Lustre project. If you are a first-time user, you'll need
+ to open an account by clicking on <emphasis role="bold">Sign up</emphasis> on the
+ Welcome page.</para>
</listitem>
</itemizedlist> To submit a Jira ticket, follow these steps:<orderedlist>
<listitem>
</listitem>
</orderedlist></para>
<section xml:id="section_jj2_4b1_kk">
- <title>Searching Jira for Duplicate Tickets</title>
- <para>Before submitting a ticket, always search Jira for an existing ticket for your
- issue.This avoids duplicating effort and may immediately provide you with a solution to your
- problem. </para>
- <para>To do a search in Jira, select the <emphasis role="bold">Issues</emphasis> tab and click
- on <emphasis role="bold">New filter</emphasis>. Use the filters provided to select criteria
- for your search. To search for specific text, enter the text in the "Contains text" field
- and click the magnifying glass icon.</para>
+ <title>Searching the Jira<superscript>*</superscript> Bug Tracker for Duplicate
+ Tickets</title>
+ <para>Before submitting a ticket, always search the Jira bug tracker for an existing ticket
+ for your issue.This avoids duplicating effort and may immediately provide you with a
+ solution to your problem. </para>
+ <para>To do a search in the Jira bug tracker, select the <emphasis role="bold"
+ >Issues</emphasis> tab and click on <emphasis role="bold">New filter</emphasis>. Use the
+ filters provided to select criteria for your search. To search for specific text, enter the
+ text in the "Contains text" field and click the magnifying glass icon.</para>
<para>When searching for text such as an ASSERTION or LustreError message, you can remove NIDS
and other installation-specific text from your search string by following the example
below.</para>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretuning">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustretuning">
<title xml:id="lustretuning.title">Lustre Tuning</title>
<para>This chapter contains information about tuning Lustre for better performance and includes the following sections:</para>
<itemizedlist>
<para>By default, this parameter is off. As always, you should test the performance to compare the impact of changing this parameter.</para>
</section>
<section><title><indexterm><primary>tuning</primary><secondary>Network interface binding</secondary></indexterm>Binding Network Interface Against CPU Partitions</title>
- <para condition='l23'>Luster 2.3 and beyond provide enhanced network interface control. The enhancement means that an administrator can bind an interface to one or more CPU Partitions. Bindings are specified as options to the lnet modules. For more information on specifying module options, see <xref linkend="dbdoclet.50438293_15350"/></para>
+ <para condition="l23">Lustre 2.3 and beyond provide enhanced network interface control. The
+ enhancement means that an administrator can bind an interface to one or more CPU Partitions.
+ Bindings are specified as options to the lnet modules. For more information on specifying
+ module options, see <xref linkend="dbdoclet.50438293_15350"/></para>
<para>For example, <literal>o2ib0(ib0)[0,1]</literal> will ensure that all messages for <literal>o2ib0</literal> will be handled by LND threads executing on <literal>CPT0</literal> and <literal>CPT1</literal>. An additional example might be: <literal>tcp1(eth0)[0]</literal>. Messages for <literal>tcp1</literal> are handled by threads on <literal>CPT0</literal>.</para>
</section>
<section><title><indexterm><primary>tuning</primary><secondary>Network interface credits</secondary></indexterm>Network Interface Credits</title>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingfailover">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingfailover">
<title xml:id="managingfailover.title">Managing Failover</title>
<para>This chapter describes failover in a Lustre system and includes the following sections:</para>
<itemizedlist>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managinglnet">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managinglnet">
<title xml:id="managinglnet.title">Managing Lustre Networking (LNET)</title>
<para>This chapter describes some tools for managing Lustre Networking (LNET) and includes the following sections:</para>
<itemizedlist>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438203_78227">
- <title><indexterm><primary>LNET</primary><secondary>Infiniband load balancing</secondary></indexterm>Load Balancing with InfiniBand</title>
- <para>A Lustre file system contains OSSs with two InfiniBand HCAs. Lustre clients have only one InfiniBand HCA using OFED Infiniband ''o2ib'' drivers. Load balancing between the HCAs on the OSS is accomplished through LNET.</para>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>InfiniBand load balancing</secondary>
+ </indexterm>Load Balancing with an InfiniBand<superscript>*</superscript> Network</title>
+ <para>A Lustre file system contains OSSs with two InfiniBand HCAs. Lustre clients have only one
+ InfiniBand HCA using OFED-based Infiniband ''o2ib'' drivers. Load
+ balancing between the HCAs on the OSS is accomplished through LNET.</para>
<section remap="h3">
<title><indexterm><primary>LNET</primary><secondary>lustre.conf</secondary></indexterm>Setting Up <literal>lustre.conf</literal> for Load Balancing</title>
<para>To configure LNET for load balancing on clients and servers:</para>
192.168.10.101@o2ib0,192.168.10.102@o2ib1:/mds/client /mnt/lustre</screen>
</listitem>
</orderedlist>
- <para>As an example, consider a two-rail IB cluster running the OFA stack (OFED) with these IPoIB address assignments.</para>
+ <para>As an example, consider a two-rail IB cluster running the OFED stack with these IPoIB
+ address assignments.</para>
<screen> ib0 ib1
Servers 192.168.0.* 192.168.1.*
Clients 192.168.[2-127].* 192.168.[128-253].*</screen>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingsecurity">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managingsecurity">
<title xml:id="managingsecurity.title">Managing Lustre Security</title>
<para>This chapter describes Lustre security and includes the following sections:</para>
<itemizedlist>
<para>An access control list (ACL), is a set of data that informs an operating system about permissions or access rights that each user or group has to specific system objects, such as directories or files. Each object has a unique security attribute that identifies users who have access to it. The ACL lists each object and user access privileges such as read, write or execute.</para>
<section remap="h3">
<title><indexterm><primary>Access Control List (ACL)</primary><secondary>how they work</secondary></indexterm>How ACLs Work</title>
- <para>Implementing ACLs varies between operating systems. Systems that support the Portable Operating System Interface (POSIX) family of standards share a simple yet powerful file system permission model, which should be well-known to the Linux/Unix administrator. ACLs add finer-grained permissions to this model, allowing for more complicated permission schemes. For a detailed explanation of ACLs on Linux, refer to the SuSE Labs article, <emphasis>Posix Access Control Lists on Linux</emphasis>:</para>
+ <para>Implementing ACLs varies between operating systems. Systems that support the Portable
+ Operating System Interface (POSIX) family of standards share a simple yet powerful file
+ system permission model, which should be well-known to the Linux/UNIX administrator. ACLs
+ add finer-grained permissions to this model, allowing for more complicated permission
+ schemes. For a detailed explanation of ACLs on a Linux operating system, refer to the SUSE
+ Labs article, <emphasis>Posix Access Control Lists on Linux</emphasis>:</para>
<para><link xl:href="http://www.suse.de/~agruen/acl/linux-acls/online/">http://www.suse.de/~agruen/acl/linux-acls/online/</link></para>
<para>We have implemented ACLs according to this model. Lustre works with the standard Linux ACL tools, setfacl, getfacl, and the historical chacl, normally installed with the ACL package.</para>
<note>
<indexterm>
<primary>space</primary>
<secondary>striping</secondary>
- </indexterm>How Lustre* File System Striping Works</title>
- <para>In a Lustre* file system, the MDS allocates objects to OSTs using either a round-robin
+ </indexterm>How Lustre File System Striping Works</title>
+ <para>In a Lustre file system, the MDS allocates objects to OSTs using either a round-robin
algorithm or a weighted algorithm. When the amount of free space is well balanced (i.e., by
default, when the free space across OSTs differs by less than 17%), the round-robin algorithm
is used to select the next OST to which a stripe is to be written. Periodically, the MDS
<?xml version='1.0' encoding='UTF-8'?>
<preface xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US">
<title>Preface</title>
- <para>The <emphasis role="italic">Lustre* Operations Manual</emphasis> provides detailed
- information and procedures to install, configure and tune a Lustre file system. The manual
- covers topics such as failover, quotas, striping, and bonding. This manual also contains
- troubleshooting information and tips to improve the operation and performance of a Lustre file
- system.</para>
+ <para>The <emphasis role="italic">Lustre<superscript>*</superscript> Operations Manual</emphasis>
+ provides detailed information and procedures to install, configure and tune a Lustre file
+ system. The manual covers topics such as failover, quotas, striping, and bonding. This manual
+ also contains troubleshooting information and tips to improve the operation and performance of a
+ Lustre file system.</para>
<section remap="h2">
<title>About this Document</title>
<para>This document is maintained by Intel in Docbook format. The
<link xl:href="http://wiki.hpdd.intel.com/display/PUB/Documentation">http://wiki.hpdd.intel.com/display/PUB/Documentation</link>. </para>
<section remap="h2">
- <title>UNIX Commands</title>
- <para>This document may not contain information about basic UNIX* operating system commands
- and procedures such as shutting down the system, booting the system, and configuring
- devices. Refer to the following for this information:</para>
+ <title>UNIX<superscript>*</superscript> Commands</title>
+ <para>This document may not contain information about basic UNIX<superscript>*</superscript>
+ operating system commands and procedures such as shutting down the system, booting the
+ system, and configuring devices. Refer to the following for this information:</para>
<itemizedlist>
<listitem>
<para> Software documentation that you received with your system</para>
</listitem>
<listitem>
- <para> Red Hat* Enterprise Linux* documentation, which is at: <link
- xl:href="http://docs.redhat.com/docs/en-US/index.html"
+ <para> Red Hat<superscript>*</superscript> Enterprise Linux<superscript>*</superscript>
+ documentation, which is at: <link xl:href="http://docs.redhat.com/docs/en-US/index.html"
>http://docs.redhat.com/docs/en-US/index.html</link></para>
- <note><para> The Lustre client module is available for many different Linux* versions and distributions.
- Red Hat Enterprise is the best supported and tested platform for Lustre servers. </para></note>
+ <note><para> The Lustre client module is available for many different Linux<superscript>*</superscript>
+ versions and distributions. The Red Hat Enterprise Linux distribution is the best
+ supported and tested platform for Lustre servers. </para></note>
</listitem>
</itemizedlist>
</section>
</section>
<section remap="h2">
<title>Related Documentation</title>
- <para>The documents listed as online are available at:</para>
- <para><link xl:href="http://wiki.hpdd.intel.com/display/PUB/Documentation">http://wiki.hpdd.intel.com/display/PUB/Documentation</link></para>
<informaltable frame="all">
<tgroup cols="4">
<colspec colname="c1" colwidth="25*"/>
<colspec colname="c2" colwidth="25*"/>
- <colspec colname="c3" colwidth="25*"/>
- <colspec colname="c4" colwidth="25*"/>
+ <colspec colname="c3" colwidth="15*"/>
+ <colspec colname="c4" colwidth="35*"/>
<thead>
<row>
<entry>
</entry>
<entry>
<para>
- <emphasis>Lustre* 2.0 Release Notes</emphasis></para>
+ <emphasis>Lustre 2.0 Release Notes</emphasis></para>
</entry>
<entry>
<para> PDF</para>
</entry>
<entry>
- <para> Online</para>
+ <para> Online at <link
+ xl:href="http://wiki.hpdd.intel.com/display/PUB/Documentation"
+ >http://wiki.hpdd.intel.com/display/PUB/Documentation</link></para>
</entry>
</row>
<row>
</entry>
<entry>
<para>
- <emphasis>Lustre* 2.0 Operations Manual</emphasis></para>
+ <emphasis>Lustre 2.0 Operations Manual</emphasis></para>
</entry>
<entry>
<para> PDF</para>
<para>HTML</para>
</entry>
<entry>
- <para> Online</para>
+ <para> Online at <link
+ xl:href="http://wiki.hpdd.intel.com/display/PUB/Documentation"
+ >http://wiki.hpdd.intel.com/display/PUB/Documentation</link></para>
</entry>
</row>
</tbody>
<?xml version="1.0" encoding="UTF-8"?>
<section xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US">
<title>Revisions</title>
- <para>Note that the <emphasis role="italic">Lustre* Operations Manual 2.x</emphasis> is
- intended to be relevant for all 2.x releases of the Lustre software. Most of the manual
- content is relevant to all of the releases. However, any features and content that are added
- for new versions of the Lustre software are clearly marked with the version in which this
- functionality is added. Similarly, features that are no longer available are marked with
- the version in which they were deprecated or removed. <revhistory>
+ <para>Note that the <emphasis role="italic">Lustre Operations Manual 2.x</emphasis> is intended
+ to be relevant for all 2.x releases of the Lustre software. Most of the manual content is
+ relevant to all of the releases. However, any features and content that are added for new
+ versions of the Lustre software are clearly marked with the version in which this
+ functionality is added. Similarly, features that are no longer available are marked with the
+ version in which they were deprecated or removed. <revhistory>
<revision>
<revnumber>2.3</revnumber>
<date>November 2012</date>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settinglustreproperties">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settinglustreproperties">
<title xml:id="settinglustreproperties.title">Setting Lustre Properties in a C Program (<literal>llapi</literal>)</title>
<para>This chapter describes the <literal>llapi</literal> library of commands used for setting Lustre file properties within a C program running in a cluster environment, such as a data processing or MPI application. The commands described in this chapter are:</para>
<itemizedlist>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settingupbonding">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settingupbonding">
<title xml:id="settingupbonding.title">Setting Up Network Interface Bonding</title>
<para>This chapter describes how to use multiple network interfaces in parallel to increase bandwidth and/or redundancy. Topics include:</para>
<itemizedlist>
<section xml:id="dbdoclet.50438258_95824">
<title>Network Interface Bonding Overview</title>
<para>Bonding, also known as link aggregation, trunking and port trunking, is a method of aggregating multiple physical network links into a single logical link for increased bandwidth.</para>
- <para>Several different types of bonding are available in Linux. All these types are referred to as 'modes', and use the bonding kernel module.</para>
+ <para>Several different types of bonding are available in the Linux distribution. All these
+ types are referred to as 'modes', and use the bonding kernel module.</para>
<para>Modes 0 to 3 allow load balancing and fault tolerance by using multiple interfaces. Mode 4 aggregates a group of interfaces into a single virtual interface where all members of the group share the same speed and duplex settings. This mode is described under IEEE spec 802.3ad, and it is referred to as either 'mode 4' or '802.3ad.'</para>
</section>
<section xml:id="dbdoclet.50438258_29107">
<note>
<para>You must <literal>modprobe</literal> the bonding module for each bonded interface. If you wish to create bond0 and bond1, two entries in <literal>bond.conf</literal> file are required.</para>
</note>
- <para>The examples below are from RedHat systems. For setup use: <literal>/etc/sysconfig/networking-scripts/ifcfg-*</literal> The website referenced below includes detailed instructions for other configuration methods, instructions to use DHCP with bonding, and other setup details. We strongly recommend you use this website.</para>
+ <para>The examples below are from systems running Red Hat Enterprise Linux. For setup use:
+ <literal>/etc/sysconfig/networking-scripts/ifcfg-*</literal> The website referenced
+ below includes detailed instructions for other configuration methods, instructions to use
+ DHCP with bonding, and other setup details. We strongly recommend you use this
+ website.</para>
<para><link xl:href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding">http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding</link></para>
</listitem>
<listitem>
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settinguplustresystem">
<title xml:id="settinguplustresystem.title">Determining Hardware Configuration Requirements and
Formatting Options</title>
- <para>This chapter describes hardware configuration requirements for a Lustre* file system
+ <para>This chapter describes hardware configuration requirements for a Lustre file system
including:</para>
<itemizedlist>
<listitem>
nearly 1 PB. If each OST uses ten 1 TB SATA disks (8 data disks plus 2 parity disks in a
RAID 6 configuration), it may be possible to get 50 MB/sec from each drive, providing up to
400 MB/sec of disk bandwidth per OST. If this system is used as storage backend with a
- system network, such as the InfiniBand* network, that provides a similar bandwidth, then
- each OSS could provide 800 MB/sec of end-to-end I/O throughput. (Although the architectural
+ system network, such as the InfiniBand network, that provides a similar bandwidth, then each
+ OSS could provide 800 MB/sec of end-to-end I/O throughput. (Although the architectural
constraints described here are simple, in practice it takes careful hardware selection,
benchmarking and integration to obtain such results.)</para>
</section>
<para>To override the default formatting options, use arguments to
<literal>mkfs.lustre</literal> to pass formatting options to the backing file system:</para>
<screen>--mkfsoptions='backing fs options'</screen>
- <para>For other <literal>mkfs.lustre</literal> options, see the Linux* man page for
+ <para>For other <literal>mkfs.lustre</literal> options, see the Linux man page for
<literal>mke2fs(8)</literal>.</para>
<section xml:id="dbdoclet.50438256_pgfId-1293228">
<title><indexterm>
<para><emphasis role="bold">Service threads</emphasis> : The service threads on the OSS node pre-allocate a 4 MB I/O buffer for each ost_io service thread, so these buffers do not need to be allocated and freed for each I/O request.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">OSS read cache</emphasis> : OSS read cache provides read-only caching of data on an OSS, using the regular Linux page cache to store the data. Just like caching from a regular file system in Linux, OSS read cache uses as much physical memory as is available.</para>
+ <para><emphasis role="bold">OSS read cache</emphasis> : OSS read cache provides read-only
+ caching of data on an OSS, using the regular Linux page cache to store the data. Just
+ like caching from a regular file system in the Linux operating system, OSS read cache
+ uses as much physical memory as is available.</para>
</listitem>
</itemizedlist>
<para>The same calculation applies to files accessed from the OSS as for the MDS, but the load is distributed over many more OSSs nodes, so the amount of memory required for locks, inode cache, etc. listed under MDS is spread out over the OSS nodes.</para>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="systemconfigurationutilities">
<title xml:id="systemconfigurationutilities.title">System Configuration Utilities</title>
<para>This chapter includes system configuration utilities and includes the following sections:</para>
<para> </para>
</entry>
<entry>
- <para> Mounts the Lustre file system named <literal>fsname</literal> on the client by contacting the Management Service at <literal>mgsspec</literal> on the pathname given by <literal>directory</literal>. The format for <literal>mgsspec</literal> is defined below. A mounted client file system appears in fstab(5) and is usable, like any local file system, and provides a full POSIX-compliant interface.</para>
+ <para> Mounts the Lustre file system named <literal>fsname</literal> on the client
+ by contacting the Management Service at <literal>mgsspec</literal> on the pathname
+ given by <literal>directory</literal>. The format for <literal>mgsspec</literal>
+ is defined below. A mounted client file system appears in fstab(5) and is usable,
+ like any local file system, and provides a full POSIX standard-compliant
+ interface.</para>
</entry>
</row>
<row>
<para> <literal>--writeconf</literal></para>
</entry>
<entry>
- <para> Erases all configuration logs for the file system to which this MDT belongs, and regenerates them. This is dangerous operation. All clients must be unmounted and servers for this file system should be stopped. All targets (OSTs/MDTs) must then be restarted to regenerate the logs. No clients should be started until all targets have restarted.</para>
- <para> </para>
+ <para> Erases all configuration logs for the file system to which this MDT belongs,
+ and regenerates them. This is dangerous operation. All clients must be unmounted
+ and servers for this file system should be stopped. All targets (OSTs/MDTs) must
+ then be restarted to regenerate the logs. No clients should be started until all
+ targets have restarted.</para>
<para>The correct order of operations is:</para>
- <para>* Unmount all clients on the file system</para>
- <para>* Unmount the MDT and all OSTs on the file system</para>
- <para>* Run <literal>tunefs.lustre --writeconf <replaceable>device</replaceable></literal> on every server</para>
- <para>* Mount the MDT and OSTs</para>
- <para>* Mount the clients</para>
+ <orderedlist>
+ <listitem>
+ <para>Unmount all clients on the file system</para>
+ </listitem>
+ <listitem>
+ <para>Unmount the MDT and all OSTs on the file system</para>
+ </listitem>
+ <listitem>
+ <para>Run <literal>tunefs.lustre --writeconf
+ <replaceable>device</replaceable></literal> on every server</para>
+ </listitem>
+ <listitem>
+ <para>Mount the MDT and OSTs</para>
+ </listitem>
+ <listitem>
+ <para>Mount the clients</para>
+ </listitem>
+ </orderedlist>
</entry>
</row>
</tbody>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="troubleshootingrecovery">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="troubleshootingrecovery">
<title xml:id="troubleshootingrecovery.title">Troubleshooting Recovery</title>
<para>This chapter describes what to do if something goes wrong during recovery. It describes:</para>
<itemizedlist>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="understandingfailover">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="understandingfailover">
<title xml:id="understandingfailover.title">Understanding Failover in Lustre</title>
<para>This chapter describes failover in a Lustre system. It includes:</para>
<itemizedlist>
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0"
xml:lang="en-US" xml:id="understandinglustre">
<title xml:id="understandinglustre.title">Understanding Lustre Architecture</title>
- <info/>
<para>This chapter describes the Lustre architecture and features of Lustre. It includes the
following sections:</para>
<itemizedlist>
</indexterm>What a Lustre File System Is (and What It Isn't)</title>
<para>The Lustre architecture is a storage architecture for clusters. The central component of
the Lustre architecture is the Lustre file system, which is supported on the Linux operating
- system and provides a POSIX-compliant UNIX file system interface.</para>
+ system and provides a POSIX<superscript>*</superscript> standard-compliant UNIX file system
+ interface.</para>
<para>The Lustre storage architecture is used for many different kinds of clusters. It is best
known for powering many of the largest high-performance computing (HPC) clusters worldwide,
with tens of thousands of client systems, petabytes (PB) of storage and hundreds of gigabytes
<secondary>features</secondary>
</indexterm>Lustre Features</title>
<para>Lustre file systems run on a variety of vendor's kernels. For more details, see the
- <link xl:href="http://wiki.hpdd.intel.com/display/PUB/Lustre+Support+Matrix">Lustre Support
- Matrix</link> on the Intel Lustre community wiki.</para>
+ Lustre Test Matrix <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="dbdoclet.50438261_99193"/>.</para>
<para>A Lustre installation can be scaled up or down with respect to the number of client
nodes, disk storage and bandwidth. Scalability and performance are dependent on available
disk and network bandwidth and the processing power of the servers in the system. A Lustre
<para>
<emphasis>MDS count:</emphasis></para>
<para> 1 primary + 1 backup</para>
- <para condition="l24">Since Lustre* Release 2.4: up to 4096 MDSs and up to 4096
+ <para condition="l24">Since Lustre release 2.4: up to 4096 MDSs and up to 4096
MDTs.</para>
</entry>
<entry>
provide additional functionality needed by the Lustre file system.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">POSIX* compliance:</emphasis> The full POSIX test suite passes
- in an identical manner to a local ext4 filesystem, with limited exceptions on Lustre
- clients. In a cluster, most operations are atomic so that clients never see stale data
- or metadata. The Lustre software supports mmap() file I/O.</para>
+ <para><emphasis role="bold">POSIX standard compliance:</emphasis> The full POSIX test
+ suite passes in an identical manner to a local ext4 file system, with limited exceptions
+ on Lustre clients. In a cluster, most operations are atomic so that clients never see
+ stale data or metadata. The Lustre software supports mmap() file I/O.</para>
</listitem>
<listitem>
<para><emphasis role="bold">High-performance heterogeneous networking:</emphasis> The
Lustre software supports a variety of high performance, low latency networks and permits
- Remote Direct Memory Access (RDMA) for Infiniband* (OFED) and other advanced networks
- for fast and efficient network transport. Multiple RDMA networks can be bridged using
- Lustre routing for maximum performance. The Lustre software also includes integrated
- network diagnostics.</para>
+ Remote Direct Memory Access (RDMA) for InfiniBand<superscript>*</superscript> (utilizing
+ OpenFabrics Enterprise Distribution (OFED<superscript>*</superscript>) and other
+ advanced networks for fast and efficient network transport. Multiple RDMA networks can
+ be bridged using Lustre routing for maximum performance. The Lustre software also
+ includes integrated network diagnostics.</para>
</listitem>
<listitem>
<para><emphasis role="bold">High-availability:</emphasis> The Lustre file system supports
architecture.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">NFS and CIFS export:</emphasis> Lustre files can be re-exported using NFS (via Linux knfsd) or CIFS (via Samba) enabling them to be shared with non-Linux clients, such as Microsoft* Windows* and Apple* Mac OS X*.</para>
+ <para><emphasis role="bold">NFS and CIFS export:</emphasis> Lustre files can be
+ re-exported using NFS (via Linux knfsd) or CIFS (via Samba) enabling them to be shared
+ with non-Linux clients, such as Microsoft<superscript>*</superscript>
+ Windows<superscript>*</superscript> and Apple<superscript>*</superscript> Mac OS
+ X<superscript>*</superscript>.</para>
</listitem>
<listitem>
<para><emphasis role="bold">Disaster recovery tool:</emphasis> The Lustre file system
</listitem>
<listitem>
<para><emphasis role="bold">Open source:</emphasis> The Lustre software is licensed under
- the GPL 2.0 license for use with Linux.</para>
+ the GPL 2.0 license for use with the Linux operating system.</para>
</listitem>
</itemizedlist>
</section>
<para>A basic configuration of Lustre components is shown in <xref
linkend="understandinglustre.fig.cluster"/>.</para>
<figure>
- <title xml:id="understandinglustre.fig.cluster">Lustre* components in a basic cluster </title>
+ <title xml:id="understandinglustre.fig.cluster">Lustre components in a basic cluster </title>
<mediaobject>
<imageobject>
<imagedata scalefit="1" width="100%" fileref="./figures/Basic_Cluster.png"/>
</imageobject>
<textobject>
- <phrase> Lustre* components in a basic cluster </phrase>
+ <phrase> Lustre components in a basic cluster </phrase>
</textobject>
</mediaobject>
</figure>
<title xml:id="understandinglustre.tab.storagerequire"><indexterm>
<primary>Lustre</primary>
<secondary>requirements</secondary>
- </indexterm>Storage and hardware requirements for Lustre* components</title>
+ </indexterm>Storage and hardware requirements for Lustre components</title>
<tgroup cols="3">
<colspec colname="c1" colwidth="1*"/>
<colspec colname="c2" colwidth="3*"/>
<title xml:id="understandinglustre.fig.lustrescale"><indexterm>
<primary>Lustre</primary>
<secondary>at scale</secondary>
- </indexterm>Lustre* cluster at scale</title>
+ </indexterm>Lustre cluster at scale</title>
<mediaobject>
<imageobject>
<imagedata scalefit="1" width="100%" fileref="./figures/Scaled_Cluster.png"/>
</imageobject>
<textobject>
- <phrase> Lustre* clustre at scale </phrase>
+ <phrase> Lustre cluster at scale </phrase>
</textobject>
</mediaobject>
</figure>
in <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="Fig1.4_ClientReqstgData"
/><?oxy_custom_end?>.</para>
<figure xml:id="Fig1.4_ClientReqstgData">
- <title>Lustre* client requesting file data</title>
+ <title>Lustre client requesting file data</title>
<mediaobject>
<imageobject>
<imagedata scalefit="1" width="75%" fileref="./figures/File_Write.png"/>
</imageobject>
<textobject>
- <phrase> Lustre* client requesting file data </phrase>
+ <phrase> Lustre client requesting file data </phrase>
</textobject>
</mediaobject>
</figure>
<para>No space is reserved on the OST for unwritten data. File A in <xref
linkend="understandinglustre.fig.filestripe"/>.</para>
<figure>
- <title xml:id="understandinglustre.fig.filestripe">File striping on a Lustre* file
+ <title xml:id="understandinglustre.fig.filestripe">File striping on a Lustre file
system</title>
<mediaobject>
<imageobject>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="understandinglustrenetworking">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="understandinglustrenetworking">
<title xml:id="understandinglustrenetworking.title">Understanding Lustre Networking (LNET)</title>
<para>This chapter introduces Lustre Networking (LNET) and includes the following sections:</para>
<itemizedlist>
<para>Key features of LNET include:</para>
<itemizedlist>
<listitem>
- <para>RDMA, when supported by underlying networks such as InfiniBand or Myrinet MX</para>
+ <para>RDMA, when supported by underlying networks such as InfiniBand or
+ Myrinet<superscript>*</superscript> MX networks</para>
</listitem>
<listitem>
- <para>Support for many commonly-used network types such as InfiniBand and TCP/IP</para>
+ <para>Support for many commonly-used network types such as InfiniBand and TCP/IP
+ networks</para>
</listitem>
<listitem>
<para>High availability and recovery features enabling transparent recovery in conjunction with failover servers</para>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="upgradinglustre">
- <title xml:id="upgradinglustre.title">Upgrading a Lustre* File System</title>
- <para>This chapter describes interoperability between Lustre* software releases. It also provides
+ <title xml:id="upgradinglustre.title">Upgrading a Lustre File System</title>
+ <para>This chapter describes interoperability between Lustre software releases. It also provides
procedures for upgrading from a Lustre 1.8 release to a Lustre 2.x release, from a Lustre 2.x
release to a more recent Lustre 2.x release (major release upgrade), and from a Lustre 2.X.y
release to 2.X.y (minor release upgrade). It includes the following sections:</para>
xmlns:xlink="http://www.w3.org/1999/xlink" linkend="LustreTestMatrixTable"/>.</para>
</listitem>
<listitem>
- <para>Upgrade the Linux operating system on all clients to RHEL 6 or other compatible
- (tested) distribution and reboot. See the Linux Test Matrix at <xref
+ <para>Upgrade the Linux operating system on all clients to Red Hat Enterprise Linux 6 or
+ other compatible (tested) distribution and reboot. See the Linux Test Matrix at <xref
xmlns:xlink="http://www.w3.org/1999/xlink" linkend="LustreTestMatrixTable"/>.</para>
</listitem>
<listitem>
-<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="userutilities">
+<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="userutilities">
<title xml:id="userutilities.title">User Utilities</title>
<para>This chapter describes user utilities and includes the following sections:</para>
<itemizedlist>
<para> <literal>--type</literal></para>
</entry>
<entry>
- <para>File has the type (block, character, directory, pipe, file, symlink, socket or Door [Solaris]).</para>
+ <para>File has the type - block, character, directory, pipe, file, symlink, socket
+ or door (used in Solaris operating system).</para>
</entry>
</row>
<row>
<?xml version='1.0' encoding='UTF-8'?>
<book xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xmlns:xi="http://www.w3.org/2001/XInclude" version="5.0" xml:lang="en-US">
<info>
- <title>Lustre File System 2.x</title>
+ <title>Lustre<superscript>*</superscript> File System Release 2.x</title>
<subtitle>Operations Manual</subtitle>
<copyright>
<year>2010</year>
<year>2011</year>
- <holder>Oracle and/or its affiliates. (The original version of this Operations Manual without the Intel modifications.)</holder>
+ <holder>Oracle and/or its affiliates. (The original version of this Operations Manual without
+ the Intel modifications.)</holder>
</copyright>
<copyright>
<year>2011</year>
+ <year>2012</year>
<year>2013</year>
- <holder>Intel Corporation. (Intel modifications to the original version of this Operations Manual.)</holder>
+ <holder>Intel Corporation. (Intel modifications to the original version of this Operations
+ Manual.)</holder>
</copyright>
<legalnotice>
- <para>Notwithstanding Intel’s ownership of the copyright in the modifications to the original version of this Operations Manual, as between Intel and Oracle, Oracle and/or its affiliates retain sole ownership of the copyright in the unmodified portions of this Operations Manual.</para>
+ <para>Notwithstanding Intel’s ownership of the copyright in the modifications to the original
+ version of this Operations Manual, as between Intel and Oracle, Oracle and/or its affiliates
+ retain sole ownership of the copyright in the unmodified portions of this Operations
+ Manual.</para>
</legalnotice>
<xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="legalnoticeIntel.xml"/>
- <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="legalnoticeOracle.xml"/>
+ <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="legalnoticeOracle.xml"/>
</info>
<xi:include href="Preface.xml"/>
<xi:include href="I_LustreIntro.xml"/>
<para>Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. Lustre is a registered trademark of Oracle Corporation.</para>
- <para>* Other names and brands may be claimed as the property of others.</para>
+ <para><superscript>*</superscript>Other names and brands may be claimed as the property of
+ others.</para>
<para>THE ORIGINAL LUSTRE 2.x FILESYSTEM: OPERATIONS MANUAL HAS BEEN MODIFIED: THIS OPERATIONS MANUAL IS A MODIFIED VERSION OF, AND IS DERIVED FROM, THE LUSTRE 2.0 FILESYSTEM: OPERATIONS MANUAL PUBLISHED BY ORACLE AND AVAILABLE AT [http://www.lustre.org/]. MODIFICATIONS (collectively, the “Modifications”) HAVE BEEN MADE BY INTEL CORPORATION (“Intel”). ORACLE AND ITS AFFILIATES HAVE NOT REVIEWED, APPROVED, SPONSORED, OR ENDORSED THIS MODIFIED OPERATIONS MANUAL, OR ENDORSED INTEL, AND ORACLE AND ITS AFFILIATES ARE NOT RESPONSIBLE OR LIABLE FOR ANY MODIFICATIONS THAT INTEL HAS MADE TO THE ORIGINAL OPERATIONS MANUAL.</para>
git://git.whamcloud.com / doc / manual.git / commitdiff