-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='configuringfailover'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. -->
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringfailover">
<info>
- <title xml:id='configuringfailover.title'>Configuring Lustre Failover</title>
+ <title xml:id="configuringfailover.title">Configuring Lustre Failover</title>
</info>
<para>This chapter describes how to configure Lustre failover using the Heartbeat cluster infrastructure daemon. It includes:</para>
-
- <itemizedlist><listitem>
- <para><xref linkend='dbdoclet.50438188_82389'/></para>
- </listitem>
- <listitem>
- <para><xref linkend='dbdoclet.50438188_92688'/></para>
- </listitem>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438188_82389"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438188_92688"/></para>
+ </listitem>
</itemizedlist>
-
<note>
+ <para>
Using Lustre Failover is optional
</para>
</note>
-
- <section xml:id="dbdoclet.50438188_82389">
- <title>11.1 Creating a Failover Environment</title>
- <para>Lustre provides failover mechanisms only at the file system level. No failover functionality is provided for system-level components, such as node failure detection or power control, as would typically be provided in a complete failover solution. Additional tools are also needed to provide resource fencing, control and monitoring.</para>
- <section remap="h3">
- <title>11.1.1 Power Management Software</title>
- <para>Lustre failover requires power control and management capability to verify that a failed node is shut down before I/O is directed to the failover node. This avoids double-mounting the two nodes, and the risk of unrecoverable data corruption. A variety of power management tools will work, but two packages that are commonly used with Lustre are STONITH and PowerMan.</para>
- <para>Shoot The Other Node In The HEAD (STONITH), is a set of power management tools provided with the Linux-HA package. STONITH has native support for many power control devices and is extensible. It uses expect scripts to automate control.</para>
- <para>PowerMan, available from the Lawrence Livermore National Laboratory (LLNL), is used to control remote power control (RPC) devices from a central location. PowerMan provides native support for several RPC varieties and expect-like configuration simplifies the addition of new devices.</para>
- <para>The latest versions of PowerMan are available at:</para>
- <para><link xl:href="http://sourceforge.net/projects/powerman">http://sourceforge.net/projects/powerman</link></para>
- <para>For more information about PowerMan, go to:</para>
- <para><link xl:href="https://computing.llnl.gov/linux/powerman.html">https://computing.llnl.gov/linux/powerman.html</link></para>
- </section>
- <section remap="h3">
- <title>11.1.2 Power Equipment</title>
- <para>Lustre failover also requires the use of RPC devices, which come in different configurations. Lustre server nodes may be equipped with some kind of service processor that allows remote power control. If a Lustre server node is not equipped with a service processor, then a multi-port, Ethernet-addressable RPC may be used as an alternative. For recommended products, refer to the list of supported RPC devices on the PowerMan website.</para>
- <para><link xl:href="https://computing.llnl.gov/linux/powerman.html">https://computing.llnl.gov/linux/powerman.html</link></para>
- </section>
+ <section xml:id="dbdoclet.50438188_82389">
+ <title>11.1 Creating a Failover Environment</title>
+ <para>Lustre provides failover mechanisms only at the file system level. No failover functionality is provided for system-level components, such as node failure detection or power control, as would typically be provided in a complete failover solution. Additional tools are also needed to provide resource fencing, control and monitoring.</para>
+ <section remap="h3">
+ <title>11.1.1 Power Management Software</title>
+ <para>Lustre failover requires power control and management capability to verify that a failed node is shut down before I/O is directed to the failover node. This avoids double-mounting the two nodes, and the risk of unrecoverable data corruption. A variety of power management tools will work, but two packages that are commonly used with Lustre are STONITH and PowerMan.</para>
+ <para>Shoot The Other Node In The HEAD (STONITH), is a set of power management tools provided with the Linux-HA package. STONITH has native support for many power control devices and is extensible. It uses expect scripts to automate control.</para>
+ <para>PowerMan, available from the Lawrence Livermore National Laboratory (LLNL), is used to control remote power control (RPC) devices from a central location. PowerMan provides native support for several RPC varieties and expect-like configuration simplifies the addition of new devices.</para>
+ <para>The latest versions of PowerMan are available at:</para>
+ <para><ulink xl:href="http://sourceforge.net/projects/powerman">http://sourceforge.net/projects/powerman</ulink></para>
+ <para>For more information about PowerMan, go to:</para>
+ <para><ulink xl:href="https://computing.llnl.gov/linux/powerman.html">https://computing.llnl.gov/linux/powerman.html</ulink></para>
</section>
- <section xml:id="dbdoclet.50438188_92688">
- <title>11.2 Setting up High-Availability (HA) Software with Lustre</title>
- <para>Lustre must be combined with high-availability (HA) software to enable a complete Lustre failover solution. Lustre can be used with several HA packages including:</para>
- <itemizedlist><listitem>
- <para><emphasis>Red Hat Cluster Manager</emphasis> - For more information about setting up Lustre failover with Red Hat Cluster Manager, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Red_Hat_Cluster_Manager_with_Lustre">Using Red Hat Cluster Manager with Lustre</link>.</para>
- </listitem>
-<listitem>
- <para><emphasis>Pacemaker</emphasis> - For more information about setting up Lustre failover with Pacemaker, see the Lustre wiki topic <link xl:href="http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre">Using Pacemaker with Lustre</link>.<anchor xml:id="dbdoclet.50438188_61775" xreflabel=""/></para>
- </listitem>
-</itemizedlist>
+ <section remap="h3">
+ <title>11.1.2 Power Equipment</title>
+ <para>Lustre failover also requires the use of RPC devices, which come in different configurations. Lustre server nodes may be equipped with some kind of service processor that allows remote power control. If a Lustre server node is not equipped with a service processor, then a multi-port, Ethernet-addressable RPC may be used as an alternative. For recommended products, refer to the list of supported RPC devices on the PowerMan website.</para>
+ <para><ulink xl:href="https://computing.llnl.gov/linux/powerman.html">https://computing.llnl.gov/linux/powerman.html</ulink></para>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438188_92688">
+ <title>11.2 Setting up High-Availability (HA) Software with Lustre</title>
+ <para>Lustre must be combined with high-availability (HA) software to enable a complete Lustre failover solution. Lustre can be used with several HA packages including:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis>Red Hat Cluster Manager</emphasis> - For more information about setting up Lustre failover with Red Hat Cluster Manager, see the Lustre wiki topic <ulink xl:href="http://wiki.lustre.org/index.php/Using_Red_Hat_Cluster_Manager_with_Lustre">Using Red Hat Cluster Manager with Lustre</ulink>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis>Pacemaker</emphasis> - For more information about setting up Lustre failover with Pacemaker, see the Lustre wiki topic <ulink xl:href="http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre">Using Pacemaker with Lustre</ulink>.<anchor xml:id="dbdoclet.50438188_61775" xreflabel=""/></para>
+ </listitem>
+ </itemizedlist>
</section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='configuringlnet'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. -->
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlnet">
<info>
- <title xml:id='configuringlnet.title'>Configuring Lustre Networking (LNET)</title>
+ <title xml:id="configuringlnet.title">Configuring Lustre Networking (LNET)</title>
</info>
-
<para>This chapter describes how to configure Lustre Networking (LNET). It includes the following sections:</para>
- <toc/>
-
- <itemizedlist><listitem>
- <para><xref linkend='dbdoclet.50438216_33148'/>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438216_33148"/>
</para>
- </listitem>
- <listitem>
- <para><xref linkend='dbdoclet.50438216_46279'/>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438216_46279"/>
</para>
- </listitem>
- <listitem>
-
<para><xref linkend="dbdoclet.50438216_31414"/>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438216_31414"/>
</para>
- </listitem>
- <listitem>
-
<para><xref linkend="dbdoclet.50438216_71227"/>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438216_71227"/>
</para>
- </listitem>
- <listitem>
- <para><xref linkend="dbdoclet.50438216_10523"/>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438216_10523"/>
+ </para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438216_35668"/>
+ </para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438216_15200"/>
</para>
+ </listitem>
+ </itemizedlist>
+ <note>
+ <para>Configuring LNET is optional.</para>
+ <para> LNET will, by default, use the first TCP/IP interface it discovers on a system (<literal>eth0</literal>). If this network configuration is sufficient, you do not need to configure LNET. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
+ </note>
+ <section xml:id="dbdoclet.50438216_33148">
+ <title>9.1 Overview of LNET Module Parameters</title>
+ <para>LNET kernel module (lnet) parameters specify how LNET is to be configured to work with Lustre, including which NICs will be configured to work with Lustre and the routing to be used with Lustre.</para>
+ <para>Parameters for lnet are specified in the <literal>modprobe.conf</literal> or <literal>modules.conf</literal> file (depending on your Linux distribution) in one or more entries with the syntax:</para>
+ <screen>options lnet <<emphasis>parameter</emphasis>>=<<emphasis>parameter value</emphasis>>
+</screen>
+ <para>To specify the network interfaces that are to be used for Lustre, set either the networks parameter or the <literal>ip2nets</literal> parameter (only one of these parameters can be used at a time):</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>networks</literal> - Specifies the networks to be used.</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438216_35668"/>
- </para>
+ <para><literal>ip2nets</literal> - Lists globally-available networks, each with a range of IP addresses. LNET then identifies locally-available networks through address list-matching lookup.</para>
</listitem>
+ </itemizedlist>
+ <para>See <xref linkend="dbdoclet.50438216_46279"/> and <xref linkend="dbdoclet.50438216_31414"/>Setting the LNET Module ip2nets Parameter for more details.</para>
+ <para>To set up routing between networks, use:</para>
+ <itemizedlist>
<listitem>
- <para><xref linkend="dbdoclet.50438216_15200"/>
- </para>
+ <para>routes - Lists networks and the NIDs of routers that forward to them.</para>
</listitem>
- </itemizedlist>
-
+ </itemizedlist>
+ <para>See <xref linkend="dbdoclet.50438216_71227"/> for more details.</para>
+ <para>A <literal>router</literal> checker can be configured to enable Lustre nodes to detect router health status, avoid routers that appear dead, and reuse those that restore service after failures. See <xref linkend="dbdoclet.50438216_35668"/> for more details.</para>
+ <para>For a complete reference to the LNET module parameters, see <emphasis><xref linkend="configurationfilesmoduleparameters"/>LNET Options</emphasis>.</para>
<note>
-<para>Configuring LNET is optional.</para><para> LNET will, by default, use the first TCP/IP interface it discovers on a system (eth0). If this network configuration is sufficient, you do not need to configure LNET. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
-</note>
-
- <section xml:id="dbdoclet.50438216_33148" >
- <title>9.1 Overview of LNET Module Parameters</title>
- <para>LNET kernel module (lnet) parameters specify how LNET is to be configured to work with Lustre, including which NICs will be configured to work with Lustre and the routing to be used with Lustre.</para>
- <para>Parameters for lnet are specified in the modprobe.conf or modules.conf file (depending on your Linux distribution) in one or more entries with the syntax:</para>
- <screen>options lnet <<emphasis>parameter</emphasis>>=<<emphasis>parameter value</emphasis>>
+ <para>We recommend that you use 'dotted-quad' notation for IP addresses rather than host names to make it easier to read debug logs and debug configurations with multiple interfaces.</para>
+ </note>
+ <section remap="h3">
+ <title>9.1.1 Using a Lustre Network Identifier (NID) to Identify a Node</title>
+ <para>A Lustre network identifier (NID) is used to uniquely identify a Lustre network endpoint by node ID and network type. The format of the NID is:</para>
+ <screen><<emphasis>network</emphasis><emphasis>id</emphasis>>@<<emphasis>network</emphasis><emphasis>type</emphasis>></screen>
+ <para>Examples are:</para>
+ <screen>10.67.73.200@tcp0
+10.67.75.100@o2ib</screen>
+ <para>The first entry above identifes a TCP/IP node, while the second entry identifies an InfiniBand node.</para>
+ <para>When a mount command is run on a client, the client uses the NID of the MDS to retrieve configuration information. If an MDS has more than one NID, the client should use the appropriate NID for its local network.</para>
+ <para>To determine the appropriate NID to specify in the mount command, use the <literal>lctl</literal> command. To display MDS NIDs, run on the MDS :</para>
+ <screen>lctl list_nids
</screen>
- <para>To specify the network interfaces that are to be used for Lustre, set either the networks parameter or the ip2nets parameter (only one of these parameters can be used at a time):</para>
- <itemizedlist><listitem>
- <para>networks - Specifies the networks to be used.</para>
- </listitem>
-<listitem>
- <para>ip2nets - Lists globally-available networks, each with a range of IP addresses. LNET then identifies locally-available networks through address list-matching lookup.</para>
- </listitem>
-</itemizedlist>
-<para>See <xref linkend='dbdoclet.50438216_46279'/> and <xref linkend="dbdoclet.50438216_31414"/>Setting the LNET Module ip2nets Parameter for more details.</para>
- <para>To set up routing between networks, use:</para>
- <itemizedlist><listitem>
- <para>routes - Lists networks and the NIDs of routers that forward to them.</para>
- </listitem>
-</itemizedlist>
-<para>See <xref linkend="dbdoclet.50438216_71227"/> for more details.</para>
- <para>A router checker can be configured to enable Lustre nodes to detect router health status, avoid routers that appear dead, and reuse those that restore service after failures. See <xref linkend="dbdoclet.50438216_35668"/> for more details.</para>
- <para>For a complete reference to the LNET module parameters, see <emphasis><xref linkend="configurationfilesmoduleparameters"/>LNET Options</emphasis>.</para>
-
- <note>
-<para>We recommend that you use 'dotted-quad' notation for IP addresses rather than host names to make it easier to read debug logs and debug configurations with multiple interfaces.</para>
-</note>
-
- <section remap="h3">
- <title>9.1.1 Using a Lustre Network Identifier (NID) to Identify a Node</title>
- <para>A Lustre network identifier (NID) is used to uniquely identify a Lustre network endpoint by node ID and network type. The format of the NID is:</para>
- <screen><<emphasis>network</emphasis><emphasis>id</emphasis>>@<<emphasis>network</emphasis><emphasis>type</emphasis>></screen>
- <para>Examples are:</para>
- <screen>10.67.73.200@tcp010.67.75.100@o2ib</screen>
- <para>The first entry above identifes a TCP/IP node, while the second entry identifies an InfiniBand node.</para>
- <para>When a mount command is run on a client, the client uses the NID of the MDS to retrieve configuration information. If an MDS has more than one NID, the client should use the appropriate NID for its local network.</para>
- <para>To determine the appropriate NID to specify in the mount command, use the lctl command. To display MDS NIDs, run on the MDS :</para>
- <screen>lctl list_nids
-</screen>
- <para>To determine if a client can reach the MDS using a particular NID, run on the client:</para>
- <screen>lctl which_nid <<emphasis>MDS NID</emphasis>><anchor xml:id="dbdoclet.50438216_14112" xreflabel=""/></screen>
- </section>
+ <para>To determine if a client can reach the MDS using a particular NID, run on the client:</para>
+ <screen>lctl which_nid <<emphasis>MDS NID</emphasis>><anchor xml:id="dbdoclet.50438216_14112" xreflabel=""/></screen>
</section>
- <section xml:id="dbdoclet.50438216_46279">
- <title>9.2 Setting the LNET Module networks Parameter</title>
- <para>If a node has more than one network interface, you'll typically want to dedicate a specific interface to Lustre. You can do this by including an entry in the modprobe.conf file on the node that sets the LNET module networks parameter:</para>
- <screen>options lnet networks=<<emphasis>comma-separated list of networks</emphasis>>
-</screen>
- <para>This example specifies that a Lustre node will use a TCP/IP interface and an InfiniBand interface:</para>
- <screen>options lnet networks=tcp0(eth0),o2ib(ib0)
+ </section>
+ <section xml:id="dbdoclet.50438216_46279">
+ <title>9.2 Setting the LNET Module networks Parameter</title>
+ <para>If a node has more than one network interface, you'll typically want to dedicate a specific interface to Lustre. You can do this by including an entry in the <literal>modprobe.conf</literal> file on the node that sets the LNET module <literal>networks</literal> parameter:</para>
+ <screen>options lnet networks=<<emphasis>comma-separated list of networks</emphasis>>
</screen>
- <para>This example specifies that the Lustre node will use the TCP/IP interface eth1:</para>
- <screen>options lnet networks=tcp0(eth1)
+ <para>This example specifies that a Lustre node will use a TCP/IP interface and an InfiniBand interface:</para>
+ <screen>options lnet networks=tcp0(eth0),o2ib(ib0)
</screen>
- <para>Depending on the network design, it may be necessary to specify explicit interfaces. To explicitly specify that interface eth2 be used for network tcp0 and eth3 be used for tcp1 , use this entry:</para>
- <screen>options lnet networks=tcp0(eth2),tcp1(eth3)
+ <para>This example specifies that the Lustre node will use the TCP/IP interface <literal>eth1</literal>:</para>
+ <screen>options lnet networks=tcp0(eth1)
</screen>
- <para>When more than one interface is available during the network setup, Lustre chooses the best route based on the hop count. Once the network connection is established, Lustre expects the network to stay connected. In a Lustre network, connections do not fail over to another interface, even if multiple interfaces are available on the same node.</para>
-
- <note>
-<para>LNET lines in modprobe.conf are only used by the local node to determine what to call its interfaces. They are not used for routing decisions.</para>
-</note>
-
- <section remap="h3">
- <title>9.2.1 <anchor xml:id="dbdoclet.50438216_74334" xreflabel=""/>Multihome Server Example</title>
- <para>If a server with multiple IP addresses (multihome server) is connected to a Lustre network, certain configuration setting are required. An example illustrating these setting consists of a network with the following nodes:</para>
- <itemizedlist><listitem>
- <para> Server svr1 with three TCP NICs (eth0, eth1, and eth2) and an InfiniBand NIC.</para>
- </listitem>
-<listitem>
- <para> Server svr2 with three TCP NICs (eth0, eth1, and eth2) and an InfiniBand NIC. Interface eth2 will not be used for Lustre networking.</para>
- </listitem>
-<listitem>
- <para> TCP clients, each with a single TCP interface.</para>
- </listitem>
-<listitem>
- <para> InfiniBand clients, each with a single Infiniband interface and a TCP/IP interface for administration.</para>
- </listitem>
-</itemizedlist>
- <para>To set the networks option for this example:</para>
- <itemizedlist><listitem>
- <para> On each server, svr1 and svr2, include the following line in the modprobe.conf file:</para>
- </listitem>
-</itemizedlist>
- <screen>options lnet networks=tcp0(eth0),tcp1(eth1),o2ib
+ <para>Depending on the network design, it may be necessary to specify explicit interfaces. To explicitly specify that interface <literal>eth2</literal> be used for network <literal>tcp0</literal> and <literal>eth3</literal> be used for <literal>tcp1</literal> , use this entry:</para>
+ <screen>options lnet networks=tcp0(eth2),tcp1(eth3)
</screen>
- <itemizedlist><listitem>
- <para> For TCP-only clients, the first available non-loopback IP interface is used for tcp0. Thus, TCP clients with only one interface do not need to have options defined in the modprobe.conf file.</para>
- </listitem>
-<listitem>
- <para> On the InfiniBand clients, include the following line in the modprobe.conf file:</para>
- </listitem>
-</itemizedlist>
- <screen>options lnet networks=o2ib
-</screen>
-
-<note>
- <para>By default, Lustre ignores the loopback (lo0) interface. Lustre does not ignore IP addresses aliased to the loopback. If you alias IP addresses to the loopback interface, you must specify all Lustre networks using the LNET networks parameter.</para></note>
-
-<note>
- <para>If the server has multiple interfaces on the same subnet, the Linux kernel will send all traffic using the first configured interface. This is a limitation of Linux, not Lustre. In this case, network interface bonding should be used. For more information about network interface bonding, see <xref linkend='settingupbonding'/>.</para>
-</note>
-
- </section>
- </section>
- <section xml:id="dbdoclet.50438216_31414">
- <title>9.3 Setting the LNET Module ip2nets Parameter</title>
- <para>The ip2nets option is typically used when a single, universal modprobe.conf file is run on all servers and clients. Each node identifies the locally available networks based on the listed IP address patterns that match the node's local IP addresses.</para>
- <para>Note that the IP address patterns listed in the ip2nets option are <emphasis>only</emphasis> used to identify the networks that an individual node should instantiate. They are <emphasis>not</emphasis> used by LNET for any other communications purpose.</para>
- <para>For the example below, the nodes in the network have these IP addresses:</para>
- <itemizedlist><listitem>
- <para> Server svr1: eth0 IP address 192.168.0.2, IP over Infiniband (o2ib) address 132.6.1.2.</para>
+ <para>When more than one interface is available during the network setup, Lustre chooses the best route based on the hop count. Once the network connection is established, Lustre expects the network to stay connected. In a Lustre network, connections do not fail over to another interface, even if multiple interfaces are available on the same node.</para>
+ <note>
+ <para>LNET lines in <literal>modprobe.conf</literal> are only used by the local node to determine what to call its interfaces. They are not used for routing decisions.</para>
+ </note>
+ <section remap="h3">
+ <title>9.2.1 <anchor xml:id="dbdoclet.50438216_74334" xreflabel=""/>Multihome Server Example</title>
+ <para>If a server with multiple IP addresses (multihome server) is connected to a Lustre network, certain configuration setting are required. An example illustrating these setting consists of a network with the following nodes:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Server svr1 with three TCP NICs (<literal>eth0</literal>, <literal>eth1</literal>, and <literal>eth2</literal>) and an InfiniBand NIC.</para>
</listitem>
-<listitem>
- <para> Server svr2: eth0 IP address 192.168.0.4, IP over Infiniband (o2ib) address 132.6.1.4.</para>
+ <listitem>
+ <para> Server svr2 with three TCP NICs (<literal>eth0</literal>, <literal>eth1</literal>, and <literal>eth2</literal>) and an InfiniBand NIC. Interface eth2 will not be used for Lustre networking.</para>
</listitem>
-<listitem>
- <para> TCP clients have IP addresses 192.168.0.5-255.</para>
+ <listitem>
+ <para> TCP clients, each with a single TCP interface.</para>
</listitem>
-<listitem>
- <para> Infiniband clients have IP over Infiniband (o2ib) addresses 132.6.[2-3].2, .4, .6, .8.</para>
+ <listitem>
+ <para> InfiniBand clients, each with a single Infiniband interface and a TCP/IP interface for administration.</para>
</listitem>
-</itemizedlist>
- <para>The following entry is placed in the modprobe.conf file on each server and client:</para>
- <screen>options lnet 'ip2nets="tcp0(eth0) 192.168.0.[2,4]; \
-tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"'
+ </itemizedlist>
+ <para>To set the <literal>networks</literal> option for this example:</para>
+ <itemizedlist>
+ <listitem>
+ <para> On each server, <literal>svr1</literal> and <literal>svr2</literal>, include the following line in the modprobe.conf file:</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet networks=tcp0(eth0),tcp1(eth1),o2ib
</screen>
- <para>Each entry in ip2nets is referred to as a 'rule'.</para>
- <para>The order of LNET entries is important when configuring servers. If a server node can be reached using more than one network, the first network specified in modprobe.conf will be used.</para>
- <para>Because svr1 and svr2 match the first rule, LNET uses eth0 for tcp0 on those machines. (Although svr1 and svr2 also match the second rule, the first matching rule for a particular network is used).</para>
- <para>The [2-8/2] format indicates a range of 2-8 stepped by 2; that is 2,4,6,8. Thus, the clients at 132.6.3.5 will not find a matching o2ib network.</para>
+ <itemizedlist>
+ <listitem>
+ <para> For TCP-only clients, the first available non-loopback IP interface is used for <literal>tcp0</literal>. Thus, TCP clients with only one interface do not need to have options defined in the <literal>modprobe.conf</literal> file.</para>
+ </listitem>
+ <listitem>
+ <para> On the InfiniBand clients, include the following line in the <literal>modprobe.conf</literal> file:</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet networks=o2ib
+</screen>
+ <note>
+ <para>By default, Lustre ignores the loopback (<literal>lo0</literal>) interface. Lustre does not ignore IP addresses aliased to the loopback. If you alias IP addresses to the loopback interface, you must specify all Lustre networks using the LNET networks parameter.</para>
+ </note>
+ <note>
+ <para>If the server has multiple interfaces on the same subnet, the Linux kernel will send all traffic using the first configured interface. This is a limitation of Linux, not Lustre. In this case, network interface bonding should be used. For more information about network interface bonding, see <xref linkend="settingupbonding"/>.</para>
+ </note>
</section>
- <section xml:id="dbdoclet.50438216_71227">
- <title>9.4 Setting the LNET Module routes Parameter</title>
- <para>The LNET module routes parameter is used to identify routers in a Lustre configuration. These parameters are set in modprob.conf on each Lustre node.</para>
- <para>The LNET routes parameter specifies a colon-separated list of router definitions. Each route is defined as a network number, followed by a list of routers:</para>
- <screen>routes=<<emphasis>net type</emphasis>> <<emphasis>router NID(s)</emphasis>>
+ </section>
+ <section xml:id="dbdoclet.50438216_31414">
+ <title>9.3 Setting the LNET Module ip2nets Parameter</title>
+ <para>The <literal>ip2nets</literal> option is typically used when a single, universal <literal>modprobe.conf</literal> file is run on all servers and clients. Each node identifies the locally available networks based on the listed IP address patterns that match the node's local IP addresses.</para>
+ <para>Note that the IP address patterns listed in the <literal>ip2nets</literal> option are <emphasis>only</emphasis> used to identify the networks that an individual node should instantiate. They are <emphasis>not</emphasis> used by LNET for any other communications purpose.</para>
+ <para>For the example below, the nodes in the network have these IP addresses:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Server svr1: <literal>eth0</literal> IP address <literal>192.168.0.2</literal>, IP over Infiniband (<literal>o2ib</literal>) address <literal>132.6.1.2</literal>.</para>
+ </listitem>
+ <listitem>
+ <para> Server svr2: <literal>eth0</literal> IP address <literal>192.168.0.4</literal>, IP over Infiniband (<literal>o2ib</literal>) address <literal>132.6.1.4</literal>.</para>
+ </listitem>
+ <listitem>
+ <para> TCP clients have IP addresses <literal>192.168.0.5-255.</literal></para>
+ </listitem>
+ <listitem>
+ <para> Infiniband clients have IP over Infiniband (<literal>o2ib</literal>) addresses <literal>132.6.[2-3].2, .4, .6, .8</literal>.</para>
+ </listitem>
+ </itemizedlist>
+ <para>The following entry is placed in the <literal>modprobe.conf</literal> file on each server and client:</para>
+ <screen>options lnet 'ip2nets="tcp0(eth0) 192.168.0.[2,4]; \
+tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"'
</screen>
- <para>This example specifies bi-directional routing in which TCP clients can reach Lustre resources on the IB networks and IB servers can access the TCP networks:</para>
- <screen>options lnet 'ip2nets="tcp0 192.168.0.*; \
-o2ib0(ib0) 132.6.1.[1-128]"' 'routes="tcp0 132.6.1.[1-8]@o2ib0; \o2ib0 192.16\
-8.0.[1-8]@tcp0"'
+ <para>Each entry in <literal>ip2nets</literal> is referred to as a 'rule'.</para>
+ <para>The order of LNET entries is important when configuring servers. If a server node can be reached using more than one network, the first network specified in <literal>modprobe.conf</literal> will be used.</para>
+ <para>Because <literal>svr1</literal> and <literal>svr2</literal> match the first rule, LNET uses <literal>eth0</literal> for <literal>tcp0</literal> on those machines. (Although <literal>svr1</literal> and <literal>svr2</literal> also match the second rule, the first matching rule for a particular network is used).</para>
+ <para>The <literal>[2-8/2]</literal> format indicates a range of 2-8 stepped by 2; that is 2,4,6,8. Thus, the clients at <literal>132.6.3.5</literal> will not find a matching o2ib network.</para>
+ </section>
+ <section xml:id="dbdoclet.50438216_71227">
+ <title>9.4 Setting the LNET Module routes Parameter</title>
+ <para>The LNET module routes parameter is used to identify routers in a Lustre configuration. These parameters are set in <literal>modprob.conf</literal> on each Lustre node.</para>
+ <para>The LNET routes parameter specifies a colon-separated list of router definitions. Each route is defined as a network number, followed by a list of routers:</para>
+ <screen>routes=<<emphasis>net type</emphasis>> <<emphasis>router NID(s)</emphasis>>
</screen>
- <para>All LNET routers that bridge two networks are equivalent. They are not configured as primary or secondary, and the load is balanced across all available routers.</para>
- <para>The number of LNET routers is not limited. Enough routers should be used to handle the required file serving bandwidth plus a 25 percent margin for headroom.</para>
- <section remap="h3">
- <title>9.4.1 <anchor xml:id="dbdoclet.50438216_34481" xreflabel=""/><anchor xml:id="dbdoclet.50438216_56435" xreflabel=""/>Routing Example</title>
- <para>On the clients, place the following entry in the modprobe.conf file</para>
- <screen>lnet networks="tcp" routes="o2ib0 192.168.0.[1-8]@tcp0"
+ <para>This example specifies bi-directional routing in which TCP clients can reach Lustre resources on the IB networks and IB servers can access the TCP networks:</para>
+ <screen>options lnet 'ip2nets="tcp0 192.168.0.*; \
+ o2ib0(ib0) 132.6.1.[1-128]"' 'routes="tcp0 132.6.1.[1-8]@o2ib0; \
+ o2ib0 192.16.8.0.[1-8]@tcp0"'
</screen>
- <para>On the router nodes, use:</para>
- <screen>lnet networks="tcp o2ib" forwarding=enabled
+ <para>All LNET routers that bridge two networks are equivalent. They are not configured as primary or secondary, and the load is balanced across all available routers.</para>
+ <para>The number of LNET routers is not limited. Enough routers should be used to handle the required file serving bandwidth plus a 25 percent margin for headroom.</para>
+ <section remap="h3">
+ <title>9.4.1 <anchor xml:id="dbdoclet.50438216_34481" xreflabel=""/><anchor xml:id="dbdoclet.50438216_56435" xreflabel=""/>Routing Example</title>
+ <para>On the clients, place the following entry in the modprobe.conf file</para>
+ <screen>lnet networks="tcp" routes="o2ib0 192.168.0.[1-8]@tcp0"
</screen>
- <para>On the MDS, use the reverse as shown below:</para>
- <screen>lnet networks="o2ib0" rountes="tcp0 132.6.1.[1-8]@o2ib0"
+ <para>On the router nodes, use:</para>
+ <screen>lnet networks="tcp o2ib" forwarding=enabled
</screen>
- <para>To start the routers, run:</para>
- <screen>modprobe lnet
+ <para>On the MDS, use the reverse as shown below:</para>
+ <screen>lnet networks="o2ib0" rountes="tcp0 132.6.1.[1-8]@o2ib0"
+</screen>
+ <para>To start the routers, run:</para>
+ <screen>modprobe lnet
lctl network configure
</screen>
- </section>
- </section>
- <section xml:id="dbdoclet.50438216_10523">
- <title>9.5 Testing the LNET Configuration</title>
- <para>After configuring Lustre Networking, it is highly recommended that you test your LNET configuration using the LNET Self-Test provided with the Lustre software. For more information about using LNET Self-Test, see <xref linkend='lnetselftest'/>.</para>
</section>
- <section xml:id="dbdoclet.50438216_35668">
- <title>9.6 Configuring the Router Checker</title>
- <para>In a Lustre configuration in which different types of networks, such as a TCP/IP network and an Infiniband network, are connected by routers, a router checker can be run on the clients and servers in the routed configuration to monitor the status of the routers. In a multi-hop routing configuration, router checkers can be configured on routers to monitor the health of their next-hop routers.</para>
- <para>A router checker is configured by setting lnet parameters in modprobe.conf by including an entry in this form:</para>
- <screen>options lnet <<emphasis>router checker parameter</emphasis>>=<<emphasis>parameter value</emphasis>>
+ </section>
+ <section xml:id="dbdoclet.50438216_10523">
+ <title>9.5 Testing the LNET Configuration</title>
+ <para>After configuring Lustre Networking, it is highly recommended that you test your LNET configuration using the LNET Self-Test provided with the Lustre software. For more information about using LNET Self-Test, see <xref linkend="lnetselftest"/>.</para>
+ </section>
+ <section xml:id="dbdoclet.50438216_35668">
+ <title>9.6 Configuring the Router Checker</title>
+ <para>In a Lustre configuration in which different types of networks, such as a TCP/IP network and an Infiniband network, are connected by routers, a router checker can be run on the clients and servers in the routed configuration to monitor the status of the routers. In a multi-hop routing configuration, router checkers can be configured on routers to monitor the health of their next-hop routers.</para>
+ <para>A router checker is configured by setting lnet parameters in modprobe.conf by including an entry in this form:</para>
+ <screen>options lnet <<emphasis>router checker parameter</emphasis>>=<<emphasis>parameter value</emphasis>>
</screen>
- <para>The router checker parameters are:</para>
- <itemizedlist><listitem>
- <para>live_router_check_interval - Specifies a time interval in seconds after which the router checker will ping the live routers. The default value is 0, meaning no checking is done. To set the value to 60, enter:</para>
- </listitem>
-</itemizedlist>
- <screen>options lnet live_router_check_interval=60
+ <para>The router checker parameters are:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>live_router_check_interval</literal> - Specifies a time interval in seconds after which the router checker will ping the live routers. The default value is 0, meaning no checking is done. To set the value to 60, enter:</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet live_router_check_interval=60
</screen>
- <itemizedlist><listitem>
- <para>dead_router_check_interval - Specifies a time interval in seconds after which the router checker will check for dead routers. The default value is 0, meaning no checking is done. To set the value to 60, enter:</para>
- </listitem>
-</itemizedlist>
- <screen>options lnet dead_router_check_interval=60
+ <itemizedlist>
+ <listitem>
+ <para><literal>dead_router_check_interval</literal> - Specifies a time interval in seconds after which the router checker will check for dead routers. The default value is 0, meaning no checking is done. To set the value to 60, enter:</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet dead_router_check_interval=60
</screen>
- <itemizedlist><listitem>
- <para>auto_down - Enables/disables (1/0) the automatic marking of router state as up or down. The default value is 1. To disable router marking, enter:</para>
- </listitem>
-</itemizedlist>
- <screen>options lnet auto_down=0
+ <itemizedlist>
+ <listitem>
+ <para>auto_down - Enables/disables (1/0) the automatic marking of router state as up or down. The default value is 1. To disable router marking, enter:</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet auto_down=0
</screen>
- <itemizedlist><listitem>
- <para>router_ping_timeout - Specifies a timeout for the router checker when it checks live or dead routers. The router checker sends a ping message to each dead or live router once every dead_router_check_interval or live_router_check_interval respectively. The default value is 50. To set the value to 60, enter:</para>
- </listitem>
-</itemizedlist>
- <screen>options lnet router_ping_timeout=60
+ <itemizedlist>
+ <listitem>
+ <para><literal>router_ping_timeout</literal> - Specifies a timeout for the router checker when it checks live or dead routers. The router checker sends a ping message to each dead or live router once every dead_router_check_interval or live_router_check_interval respectively. The default value is 50. To set the value to 60, enter:</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet router_ping_timeout=60
</screen>
-
-<note>
-<para>The router_ping_timeout is consistent with the default LND timeouts. You may have to increase it on very large clusters if the LND timeout is also increased. For larger clusters, we suggest increasing the check interval.</para>
-</note>
-
-
- <itemizedlist><listitem>
- <para>check_routers_before_use - Specifies that routers are to be checked before use. Set to off by default. If this parameter is set to on, the dead_router_check_interval parameter must be given a positive integer value.</para>
- </listitem>
-</itemizedlist>
- <screen>options lnet check_routers_before_use=on
+ <note>
+ <para>The router_ping_timeout is consistent with the default LND timeouts. You may have to increase it on very large clusters if the LND timeout is also increased. For larger clusters, we suggest increasing the check interval.</para>
+ </note>
+ <itemizedlist>
+ <listitem>
+ <para>check_routers_before_use - Specifies that routers are to be checked before use. Set to off by default. If this parameter is set to on, the dead_router_check_interval parameter must be given a positive integer value.</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet check_routers_before_use=on
</screen>
- <para>The router checker obtains the following information from each router:</para>
- <itemizedlist><listitem>
- <para> Time the router was disabled</para>
- </listitem>
-<listitem>
- <para> Elapsed disable time</para>
- </listitem>
-</itemizedlist>
- <para>If the router checker does not get a reply message from the router within router_ping_timeout seconds, it considers the router to be down.</para>
- <para>If a router is marked 'up' and responds to a ping, the timeout is reset.</para>
- <para>If 100 packets have been sent successfully through a router, the sent-packets counter for that router will have a value of 100.</para>
+ <para>The router checker obtains the following information from each router:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Time the router was disabled</para>
+ </listitem>
+ <listitem>
+ <para> Elapsed disable time</para>
+ </listitem>
+ </itemizedlist>
+ <para>If the router checker does not get a reply message from the router within router_ping_timeout seconds, it considers the router to be down.</para>
+ <para>If a router is marked 'up' and responds to a ping, the timeout is reset.</para>
+ <para>If 100 packets have been sent successfully through a router, the sent-packets counter for that router will have a value of 100.</para>
+ </section>
+ <section xml:id="dbdoclet.50438216_15200">
+ <title>9.7 Best Practices for LNET Options</title>
+ <para>For the <literal>networks</literal>, <literal>ip2nets</literal>, and <literal>routes</literal> options, follow these best practices to avoid configuration errors.</para>
+ <section remap="h5">
+ <title>Escaping commas with quotes</title>
+ <para>Depending on the Linux distribution, commas may need to be escaped using single or double quotes. In the extreme case, the <literal>options</literal> entry would look like this:</para>
+ <para><screen>options lnet'networks="tcp0,elan0"' 'routes="tcp [2,10]@elan0"'</screen></para>
+ <para>Added quotes may confuse some distributions. Messages such as the following may indicate an issue related to added quotes:</para>
+ <para><screen>lnet: Unknown parameter 'networks'</screen></para>
+ <para>A <literal>'Refusing connection - no matching NID'</literal> message generally points to an error in the LNET module configuration.</para>
+ </section>
+ <section remap="h5">
+ <title>Including comments</title>
+ <para><emphasis>Place the semicolon terminating a comment immediately after the comment.</emphasis> LNET silently ignores everything between the <literal>#</literal> character at the beginning of the comment and the next semicolon.</para>
+ <para>In this <emphasis>incorrect</emphasis> example, LNET silently ignores <literal>pt11 192.168.0.[92,96]</literal>, resulting in these nodes not being properly initialized. No error message is generated.</para>
+ <screen>options lnet ip2nets=
+# comment with semicolon BEFORE comment \
+pt11 192.168.0.[92,96];
+</screen>
+ <para>This <emphasis role="italic">correct</emphasis> example shows the required syntax:
+</para>
+ <para><screen>options lnet ip2nets="pt10 192.168.0.[89,93] \
+# comment with semicolon AFTER comment; \
+pt11 192.168.0.[92,96] # comment</screen></para>
+ <para><emphasis role="italic">Do not add an excessive number of comments.</emphasis> The Linux kernel limits the length of
+character strings used in module options (usually to 1KB, but this may differ
+between vendor kernels). If you exceed this limit, errors result and the specified
+configuration may not be processed correctly.</para>
</section>
- <section xml:id="dbdoclet.50438216_15200">
- <title>9.7 Best Practices for LNET Options</title>
- <para>For the networks, ip2nets, and routes options, follow these best practices to avoid configuration errors.</para>
- <section remap="h5">
- <title>Escaping commas with quotes</title>
- <para>Depending on the Linux distribution, commas may need to be escaped using single or double quotes. In the extreme case, the options entry would look like this:</para>
- <para>options lnet'networks="tcp0,elan0"' 'routes="tcp [2,10]@elan0"'</para>
- <para>Added quotes may confuse some distributions. Messages such as the following may indicate an issue related to added quotes:</para>
- <para>lnet: Unknown parameter 'networks'</para>
- <para>A 'Refusing connection - no matching NID' message generally points to an error in the LNET module configuration.</para>
- </section>
- <section remap="h5">
- <title>Including comments</title>
- <para><emphasis>Place the semicolon terminating a comment immediately after the comment.</emphasis> LNET silently ignores everything between the # character at the beginning of the comment and the next semicolon.</para>
- <para>In this <emphasis>incorrect</emphasis> example, LNET silently ignores pt11 192.168.0.[92,96], resulting in these nodes not being properly initialized. No error message is generated.</para>
- <screen>options lnet ip2nets=</screen>
- </section>
</section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='configuringlustre'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlustre">
<info>
- <title xml:id='configuringlustre.title'>Configuring Lustre</title>
+ <title xml:id="configuringlustre.title">Configuring Lustre</title>
</info>
-
-
-
<para>This chapter shows how to configure a simple Lustre system comprised of a combined MGS/MDT, an OST and a client. It includes:</para>
- <itemizedlist><listitem>
- <para><xref linkend="dbdoclet.50438267_50692"/>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438267_50692"/>
</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438267_76752"/>
+ </para>
+ </listitem>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438267_50692">
+ <title>10.1 Configuring a Simple Lustre File System</title>
+ <para>A Lustre system can be set up in a variety of configurations by using the administrative utilities provided with Lustre. The procedure below shows how to to configure a simple Lustre file system consisting of a combined MGS/MDS, one OSS with two OSTs, and a client. For an overview of the entire Lustre installation procedure, see <xref linkend="installoverview"/>.</para>
+ <para>This configuration procedure assumes you have completed the following:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis>
+ <emphasis role="bold">Set up and configured your hardware</emphasis>
+ </emphasis>. For more information about hardware requirements, see <xref linkend="settinguplustresystem"/>.</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438267_76752"/>
- </para>
+ <para><emphasis><emphasis role="bold">Downloaded and installed the Lustre software</emphasis>.</emphasis> For more information about preparing for and installing the Lustre software, see <xref linkend="installinglustre"/>.</para>
</listitem>
- </itemizedlist>
-
- <section xml:id="dbdoclet.50438267_50692">
- <title>10.1 Configuring a Simple Lustre File System</title>
- <para>A Lustre system can be set up in a variety of configurations by using the administrative utilities provided with Lustre. The procedure below shows how to to configure a simple Lustre file system consisting of a combined MGS/MDS, one OSS with two OSTs, and a client. For an overview of the entire Lustre installation procedure, see <xref linkend='installoverview'/>.</para>
- <para>This configuration procedure assumes you have completed the following:</para>
- <itemizedlist><listitem>
- <para><emphasis>Set up and configured your hardware</emphasis> . For more information about hardware requirements, see <xref linkend='settinguplustresystem'/>.</para>
- </listitem>
-<listitem>
- <para><emphasis>Downloaded and installed the Lustre software.</emphasis> For more information about preparing for and installing the Lustre software, see <xref linkend='installinglustre'/>.</para>
- </listitem>
-</itemizedlist>
- <para>The following optional steps should also be completed, if needed, before the Lustre software is configured:</para>
- <itemizedlist><listitem>
- <para><emphasis>Set up a hardware or software RAID on block devices to be used as OSTs or MDTs.</emphasis> For information about setting up RAID, see the documentation for your RAID controller or <xref linkend='configuringstorage'/>.</para>
- </listitem>
-<listitem>
- <para><emphasis>Set up network interface bonding on Ethernet interfaces.</emphasis> For information about setting up network interface bonding, see <xref linkend='settingupbonding'/>.</para>
- </listitem>
-<listitem>
- <para><emphasis>Set</emphasis>lnet<emphasis>module parameters to specify how Lustre Networking (LNET) is to be configured to work with Lustre and test the LNET configuration.</emphasis> LNET will, by default, use the first TCP/IP interface it discovers on a system. If this network configuration is sufficient, you do not need to configure LNET. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
- </listitem>
-</itemizedlist>
-<para>For information about configuring LNET, see <xref linkend='configuringlnet'/>. For information about testing LNET, see <xref linkend='lnetselftest'/>.</para>
- <itemizedlist><listitem>
- <para><emphasis>Run the benchmark script sgpdd_survey to determine baseline performance of your hardware.</emphasis> Benchmarking your hardware will simplify debugging performance issues that are unrelated to Lustre and ensure you are getting the best possible performance with your installation. For information about running sgpdd_survey, see <xref linkend='benchmarkingtests'/>.</para>
- </listitem>
-</itemizedlist>
-
-<note>
- <para>
-The sgpdd_survey script overwrites the device being tested so it must be run before the OSTs are configured.</para>
-</note>
-
- <para>To configure a simple Lustre file system, complete these steps:</para>
- <orderedlist>
- <listitem>
- <para><emphasis role="bold">Create</emphasis> a combined MGS/MDT file system on a block device. On the MDS node, run:</para>
- <screen>mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgs --mdt <<emphasis>block device name</emphasis>>
+ </itemizedlist>
+ <para>The following optional steps should also be completed, if needed, before the Lustre software is configured:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis>Set up a hardware or software RAID on block devices to be used as OSTs or MDTs.</emphasis> For information about setting up RAID, see the documentation for your RAID controller or <xref linkend="configuringstorage"/>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis>Set up network interface bonding on Ethernet interfaces.</emphasis> For information about setting up network interface bonding, see <xref linkend="settingupbonding"/>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis>Set</emphasis>lnet<emphasis>module parameters to specify how Lustre Networking (LNET) is to be configured to work with Lustre and test the LNET configuration.</emphasis> LNET will, by default, use the first TCP/IP interface it discovers on a system. If this network configuration is sufficient, you do not need to configure LNET. LNET configuration is required if you are using Infiniband or multiple Ethernet interfaces.</para>
+ </listitem>
+ </itemizedlist>
+ <para>For information about configuring LNET, see <xref linkend="configuringlnet"/>. For information about testing LNET, see <xref linkend="lnetselftest"/>.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis>Run the benchmark script <literal>sgpdd_survey</literal> to determine baseline performance of your hardware.</emphasis> Benchmarking your hardware will simplify debugging performance issues that are unrelated to Lustre and ensure you are getting the best possible performance with your installation. For information about running sgpdd_survey, see <xref linkend="benchmarkingtests"/>.</para>
+ </listitem>
+ </itemizedlist>
+ <note>
+ <para>
+The <literal>sgpdd_survey</literal> script overwrites the device being tested so it must be run before the OSTs are configured.</para>
+ </note>
+ <para>To configure a simple Lustre file system, complete these steps:</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Create</emphasis> a combined MGS/MDT file system on a block device. On the MDS node, run:</para>
+ <screen>mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgs --mdt <<emphasis>block device name</emphasis>>
</screen>
-
-<note>
- <para>
-If you plan to generate multiple file systems, the MGS should be created separately on its own dedicated block device, by running:</para><para> mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgs <<emphasis>block device name</emphasis>></para>
-</note>
-
-
-
- </listitem>
- <listitem>
- <para>Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</para>
- <screen>mount -t lustre <<emphasis>block device name</emphasis>> <<emphasis>mount point</emphasis>>
+ <para>The default file system name (<literal>fsname</literal>) is <literal>lustre</literal>.</para>
+ <note>
+ <para>If you plan to generate multiple file systems, the MGS should be created separately on its own dedicated block device, by running:</para>
+ <para> mkfs.lustre --fsname=<<emphasis>fsname</emphasis>> --mgs <<emphasis>block device name</emphasis>></para>
+ </note>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</emphasis></para>
+ <screen>mount -t lustre <<emphasis>block device name</emphasis>> <<emphasis>mount point</emphasis>>
</screen>
-
-<note><para>
-If you have created and MGS and an MDT on separate block devices, mount them both.</para>
-</note>
-
-
- </listitem>
- <listitem xml:id="dbdoclet.50438267_pgfId-1290915">
- <para>Create the OST. On the OSS node, run:</para>
- <screen>mkfs.lustre --ost --fsname=<<emphasis>fsname</emphasis>> --mgsnode=<<emphasis>NID</emphasis>> <<emphasis>block device name</emphasis>>
+ <note>
+ <para>If you have created and MGS and an MDT on separate block devices, mount them both.</para>
+ </note>
+ </listitem>
+ <listitem xml:id="dbdoclet.50438267_pgfId-1290915">
+ <para><emphasis role="bold">Create the OST. On the OSS node, run:</emphasis></para>
+ <screen>mkfs.lustre --ost --fsname=<<emphasis>fsname</emphasis>> --mgsnode=<<emphasis>NID</emphasis>> <<emphasis>block device name</emphasis>>
</screen>
- <para>When you create an OST, you are formatting a ldiskfs file system on a block storage device like you would with any local file system.</para>
- <para>You can have as many OSTs per OSS as the hardware or drivers allow. For more information about storage and memory requirements for a Lustre file system, see <xref linkend='settinguplustresystem'/>.</para>
- <para>You can only configure one OST per block device. You should create an OST that uses the raw block device and does not use partitioning.</para>
- <para>If you are using block devices that are accessible from multiple OSS nodes, ensure that you mount the OSTs from only one OSS node at at time. It is strongly recommended that multiple-mount protection be enabled for such devices to prevent serious data corruption. For more information about multiple-mount protection, see <xref linkend='managingfailover'/>.</para>
-
+ <para>When you create an OST, you are formatting a <literal>ldiskfs</literal> file system on a block storage device like you would with any local file system.</para>
+ <para>You can have as many OSTs per OSS as the hardware or drivers allow. For more information about storage and memory requirements for a Lustre file system, see <xref linkend="settinguplustresystem"/>.</para>
+
<para>You can only configure one OST per block device. You should create an OST that uses the raw block device and does not use partitioning.</para>
+ <para>If you are using block devices that are accessible from multiple OSS nodes, ensure that you mount the OSTs from only one OSS node at at time. It is strongly recommended that multiple-mount protection be enabled for such devices to prevent serious data corruption. For more information about multiple-mount protection, see <xref linkend="managingfailover"/>.</para>
+ <note>
Lustre currently supports block devices up to 16 TB on OEL 5/RHEL 5 (up to 8 TB on other distributions). If the device size is only slightly larger that 16 TB, it is recommended that you limit the file system size to 16 TB at format time. If the size is significantly larger than 16 TB, you should reconfigure the storage into devices smaller than 16 TB. We recommend that you not place partitions on top of RAID 5/6 block devices due to negative impacts on performance.</para>
-</note>
-
-
- </listitem>
- <listitem xml:id="dbdoclet.50438267_pgfId-1293955">
- <para>Mount the OST. On the OSS node where the OST was created, run:</para>
- <screen>mount -t lustre <emphasis><block device name> <mount point></emphasis></screen>
-
-
- <note><para>
- To create additional OSTs, repeat <xref linkend='dbdoclet.50438267_pgfId-1290915'/>Step 3 and <xref linkend='dbdoclet.50438267_pgfId-1293955'/>Step 4.</para>
-</note>
-
- </listitem>
- <listitem xml:id="dbdoclet.50438267_pgfId-1290934">
- <para>Mount the Lustre file system on the client. On the client node, run:</para>
- <screen>mount -t lustre <<emphasis>MGS node</emphasis>>:/<<emphasis>fsname</emphasis>> <<emphasis>mount point</emphasis>>
+ </note>
+ </listitem>
+ <listitem xml:id="dbdoclet.50438267_pgfId-1293955">
+ <para><emphasis role="bold">Mount the OST. On the OSS node where the OST was created, run:</emphasis></para>
+ <screen>mount -t lustre <emphasis><block device name> <mount point></emphasis></screen>
+ <note>
+ <para>
+ To create additional OSTs, repeat <xref linkend="dbdoclet.50438267_pgfId-1290915"/>Step 3 and <xref linkend="dbdoclet.50438267_pgfId-1293955"/>Step 4.</para>
+ </note>
+ </listitem>
+ <listitem xml:id="dbdoclet.50438267_pgfId-1290934">
+ <para><emphasis role="bold">Mount the Lustre file system on the client. On the client node, run:</emphasis></para>
+ <screen>mount -t lustre <<emphasis>MGS node</emphasis>>:/<<emphasis>fsname</emphasis>> <<emphasis>mount point</emphasis>>
</screen>
-
-<note><para>
- To create additional clients, repeat <xref linkend='dbdoclet.50438267_pgfId-1290934'/>Step 5.</para>
-</note>
-
- </listitem>
- <listitem>
- <para>Verify that the file system started and is working correctly. Do this by running the lfs df, dd and ls commands on the client node.</para>
-
- <note><para>
-If you have a problem mounting the file system, check the syslogs on the client and all the servers for errors and also check the network settings. A common issue with newly-installed systems is that hosts.deny or firewall rules may prevent connections on port 988.</para>
-</note>
-
-
- </listitem>
- <listitem>
- <para><emphasis>(Optional) Run benchmarking to</emphasis>ols to validate the performance of hardware and software layers in the cluster. Available tools include:</para>
-
- <itemizedlist><listitem>
- <para>obdfilter_survey - Characterizes the storage performance of a Lustre file system. For details, see <xref linkend='benchmarkingtests'/><emphasis><link xl:href="BenchmarkingTests.html#50438212_40624">Testing OST Performance (obdfilter_survey)</link></emphasis>.</para>
- </listitem>
-<listitem>
- <para>ost_survey - Performs I/O against OSTs to detect anomalies between otherwise identical disk subsystems. For details, see <xref linkend='benchmarkingtests'/><emphasis><link xl:href="BenchmarkingTests.html#50438212_85136">Testing OST I/O Performance (ost_survey)</link></emphasis>.</para>
- </listitem>
-</itemizedlist>
- </listitem>
-</orderedlist>
-
- <section remap="h3">
- <title>10.1.1 Simple Lustre <anchor xml:id="dbdoclet.50438267_marker-1290955" xreflabel=""/>Configuration Example</title>
- <para>To see the steps in a simple Lustre configuration, follow this example in which a combined MGS/MDT and two OSTs are created. Three block devices are used, one for the combined MGS/MDS node and one for each OSS node. Common parameters used in the example are listed below, along with individual node parameters.</para>
- <informaltable frame="all">
- <tgroup cols="4">
- <colspec colname="c1" colwidth="2*"/>
- <colspec colname="c2" colwidth="25*"/>
- <colspec colname="c3" colwidth="25*"/>
- <colspec colname="c4" colwidth="25*"/>
- <thead>
- <row>
- <entry nameend="c2" namest="c1"><para><emphasis role="bold">Common Parameters</emphasis></para></entry>
- <entry><para><emphasis role="bold">Value</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">MGS node</emphasis></para></entry>
- <entry><para> 10.2.0.1@tcp0</para></entry>
- <entry><para> Node for the combined MGS/MDS</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">file system</emphasis></para></entry>
- <entry><para> temp</para></entry>
- <entry><para> Name of the Lustre file system</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">network type</emphasis></para></entry>
- <entry><para> TCP/IP</para></entry>
- <entry><para> Network type used for Lustre file system temp</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <informaltable frame="all">
- <tgroup cols="4">
- <colspec colname="c1" colwidth="25*"/>
- <colspec colname="c2" colwidth="25*"/>
- <colspec colname="c3" colwidth="25*"/>
- <colspec colname="c4" colwidth="25*"/>
- <thead>
- <row>
- <entry nameend="c2" namest="c1"><para><emphasis role="bold">Node Parameters</emphasis></para></entry>
- <entry><para><emphasis role="bold">Value</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry nameend="c4" namest="c1"><para> MGS/MDS node</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">MGS/MDS node</emphasis></para></entry>
- <entry><para> mdt1</para></entry>
- <entry><para> MDS in Lustre file system temp</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">block device</emphasis></para></entry>
- <entry><para> /dev/sdb</para></entry>
- <entry><para> Block device for the combined MGS/MDS node</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">mount point</emphasis></para></entry>
- <entry><para> /mnt/mdt</para></entry>
- <entry><para> Mount point for the mdt1 block device (/dev/sdb) on the MGS/MDS node</para></entry>
- </row>
- <row>
- <entry nameend="c4" namest="c1"><para> First OSS node</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">OSS node</emphasis></para></entry>
- <entry><para> oss1</para></entry>
- <entry><para> First OSS node in Lustre file system temp</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">OST</emphasis></para></entry>
- <entry><para> ost1</para></entry>
- <entry><para> First OST in Lustre file system temp</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">block device</emphasis></para></entry>
- <entry><para> /dev/sdc</para></entry>
- <entry><para> Block device for the first OSS node (oss1)</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">mount point</emphasis></para></entry>
- <entry><para> /mnt/ost1</para></entry>
- <entry><para> Mount point for the ost1 block device (/dev/sdc) on the oss1 node</para></entry>
- </row>
- <row>
- <entry nameend="c4" namest="c1"><para> Second OSS node</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">OSS node</emphasis></para></entry>
- <entry><para> oss2</para></entry>
- <entry><para> Second OSS node in Lustre file system temp</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">OST</emphasis></para></entry>
- <entry><para> ost2</para></entry>
- <entry><para> Second OST in Lustre file system temp</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">block device</emphasis></para></entry>
- <entry><para> /dev/sdd</para></entry>
- <entry><para> Block device for the second OSS node (oss2)</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">mount point</emphasis></para></entry>
- <entry><para> /mnt/ost2</para></entry>
- <entry><para> Mount point for the ost2 block device (/dev/sdd) on the oss2 node</para></entry>
- </row>
- <row>
- <entry nameend="c4" namest="c1"><para> Client node</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">client node</emphasis></para></entry>
- <entry><para> client1</para></entry>
- <entry><para> Client in Lustre file system temp</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> <emphasis role="bold">mount point</emphasis></para></entry>
- <entry><para> /lustre</para></entry>
- <entry><para> Mount point for Lustre file system temp on the client1 node</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
-
-
-
<note>
+ <para>
+ To create additional clients, repeat <xref linkend="dbdoclet.50438267_pgfId-1290934"/>Step 5.</para>
+ </note>
+ </listitem>
+ <listitem>
+ <para><note>
<para>
-We recommend that you use 'dotted-quad' notation for IP addresses rather than host names to make it easier to read debug logs and debug configurations with multiple interfaces.</para>
- </note>
-
- <para>For this example, complete the steps below:</para>
-
- <orderedlist>
- <listitem>
-
- <para>Create a combined MGS/MDT file system on the block device. On the MDS node, run:</para>
- <screen>[root@mds /]# mkfs.lustre --fsname=temp --mgs --mdt /dev/sdb
+If you have a problem mounting the file system, check the syslogs on the client and all the servers for errors and also check the network settings. A common issue with newly-installed systems is that hosts.deny or firewall rules may prevent connections on port 988.</para>
+ </note><emphasis role="bold">Verify that the file system started and is working correctly. Do this by running the <literal>lfs</literal><literal> df</literal>, <literal>dd</literal> and <literal>ls</literal> commands on the client node.</emphasis></para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold"><emphasis><emphasis role="italic">(Optional)</emphasis> Run benchmarking to</emphasis>ols to validate the performance of hardware and software layers in the cluster. Available tools include:</emphasis></para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>obdfilter_survey</literal> - Characterizes the storage performance of a Lustre file system. For details, see <xref linkend="dbdoclet.50438212_40624"/>.</para>
+ </listitem>
+ <listitem>
+ <para><literal>ost_survey</literal> - Performs I/O against OSTs to detect anomalies between otherwise identical disk subsystems. For details, see <xref linkend="dbdoclet.50438212_85136"/>.</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ </orderedlist>
+ <section remap="h3">
+ <title>10.1.1 Simple Lustre <anchor xml:id="dbdoclet.50438267_marker-1290955" xreflabel=""/>Configuration Example</title>
+ <para>To see the steps in a simple Lustre configuration, follow this example in which a combined MGS/MDT and two OSTs are created. Three block devices are used, one for the combined MGS/MDS node and one for each OSS node. Common parameters used in the example are listed below, along with individual node parameters.</para>
+ <informaltable frame="all">
+ <tgroup cols="4">
+ <colspec colname="c1" colwidth="2*"/>
+ <colspec colname="c2" colwidth="25*"/>
+ <colspec colname="c3" colwidth="25*"/>
+ <colspec colname="c4" colwidth="25*"/>
+ <thead>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para><emphasis role="bold">Common Parameters</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Value</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">MGS node</emphasis></para>
+ </entry>
+ <entry>
+ <para> <literal>10.2.0.1@tcp0</literal></para>
+ </entry>
+ <entry>
+ <para> Node for the combined MGS/MDS</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">file system</emphasis></para>
+ </entry>
+ <entry>
+ <para><literal> temp</literal></para>
+ </entry>
+ <entry>
+ <para> Name of the Lustre file system</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">network type</emphasis></para>
+ </entry>
+ <entry>
+ <para> <literal>TCP/IP</literal></para>
+ </entry>
+ <entry>
+ <para> Network type used for Lustre file system temp</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <informaltable frame="all">
+ <tgroup cols="4">
+ <colspec colname="c1" colwidth="25*"/>
+ <colspec colname="c2" colwidth="25*"/>
+ <colspec colname="c3" colwidth="25*"/>
+ <colspec colname="c4" colwidth="25*"/>
+ <thead>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para><emphasis role="bold">Node Parameters</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Value</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry nameend="c4" namest="c1">
+ <para> MGS/MDS node</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">MGS/MDS node</emphasis></para>
+ </entry>
+ <entry>
+ <para> <literal>mdt1</literal></para>
+ </entry>
+ <entry>
+ <para> MDS in Lustre file system temp</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">block device</emphasis></para>
+ </entry>
+ <entry>
+ <para> <literal>/dev/sdb</literal></para>
+ </entry>
+ <entry>
+ <para> Block device for the combined MGS/MDS node</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">mount point</emphasis></para>
+ </entry>
+ <entry>
+ <para> <literal>/mnt/mdt</literal></para>
+ </entry>
+ <entry>
+ <para> Mount point for the <literal>mdt1</literal> block device (<literal>/dev/sdb</literal>) on the MGS/MDS node</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c4" namest="c1">
+ <para> First OSS node</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">OSS node</emphasis></para>
+ </entry>
+ <entry>
+ <para><literal> oss1</literal></para>
+ </entry>
+ <entry>
+ <para>First OSS node in Lustre file system temp</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">OST</emphasis></para>
+ </entry>
+ <entry>
+ <para><literal> ost1</literal></para>
+ </entry>
+ <entry>
+ <para>First OST in Lustre file system temp</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">block device</emphasis></para>
+ </entry>
+ <entry>
+ <para> <literal>/dev/sdc</literal></para>
+ </entry>
+ <entry>
+ <para>Block device for the first OSS node (<literal>oss1</literal>)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">mount point</emphasis></para>
+ </entry>
+ <entry>
+ <para> <literal>/mnt/ost1</literal></para>
+ </entry>
+ <entry>
+ <para> Mount point for the <literal>ost1</literal> block device (<literal>/dev/sdc</literal>) on the <literal>oss1</literal> node</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c4" namest="c1">
+ <para> Second OSS node</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">OSS node</emphasis></para>
+ </entry>
+ <entry>
+ <para><literal>oss2</literal></para>
+ </entry>
+ <entry>
+ <para>Second OSS node in Lustre file system temp</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">OST</emphasis></para>
+ </entry>
+ <entry>
+ <para> <literal>ost2</literal></para>
+ </entry>
+ <entry>
+ <para>Second OST in Lustre file system temp</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">block device</emphasis></para>
+ </entry>
+ <entry>
+ <para><literal>/dev/sdd</literal></para>
+ </entry>
+ <entry>
+ <para>Block device for the second OSS node (oss2)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">mount point</emphasis></para>
+ </entry>
+ <entry>
+ <para><literal>/mnt/ost2</literal></para>
+ </entry>
+ <entry>
+ <para> Mount point for the <literal>ost2</literal> block device (<literal>/dev/sdd</literal>) on the <literal>oss2</literal> node</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c4" namest="c1">
+ <para> Client node</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">client node</emphasis></para>
+ </entry>
+ <entry>
+ <para> <literal>client1</literal></para>
+ </entry>
+ <entry>
+ <para>Client in Lustre file system temp</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">mount point</emphasis></para>
+ </entry>
+ <entry>
+ <para> <literal>/lustre</literal></para>
+ </entry>
+ <entry>
+ <para>Mount point for Lustre file system temp on the <literal>client1</literal> node</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <note>
+ <para>We recommend that you use 'dotted-quad' notation for IP addresses rather than host names to make it easier to read debug logs and debug configurations with multiple interfaces.</para>
+ </note>
+ <para>For this example, complete the steps below:</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Create a combined MGS/MDT file system on the block device. On the MDS node, run:</emphasis></para>
+ <screen>[root@mds /]# mkfs.lustre --fsname=temp --mgs --mdt /dev/sdb
</screen>
- <para>This command generates this output:</para>
- <screen> Permanent disk data:
+
<para>This command generates this output:</para>
+ <screen> Permanent disk data:
Target: temp-MDTffff
Index: unassigned
Lustre FS: temp
dir_index,uninit_groups -F /dev/sdb
Writing CONFIGS/mountdata
</screen>
- </listitem>
- <listitem>
-
- <para>Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</para>
- <screen>[root@mds /]# mount -t lustre /dev/sdb /mnt/mdt
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Mount the combined MGS/MDT file system on the block device. On the MDS node, run:</emphasis></para>
+ <screen>[root@mds /]# mount -t lustre /dev/sdb /mnt/mdt
</screen>
- <para>This command generates this output:</para>
- <screen>Lustre: temp-MDT0000: new disk, initializing
+
<para>This command generates this output:</para>
+ <screen>Lustre: temp-MDT0000: new disk, initializing
Lustre: 3009:0:(lproc_mds.c:262:lprocfs_wr_group_upcall()) temp-MDT0000: gr\
oup upcall set to /usr/sbin/l_getgroups
Lustre: temp-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups
Lustre: Server temp-MDT0000 on device /dev/sdb has started
</screen>
- </listitem>
- <listitem xml:id="dbdoclet.50438267_pgfId-1291170">
- <para>Create and mount ost1.</para>
- <para>In this example, the OSTs (ost1 and ost2) are being created on different OSSs (oss1 and oss2 respectively).</para>
- <orderedlist>
+ </listitem>
+ <listitem xml:id="dbdoclet.50438267_pgfId-1291170">
+ <para><emphasis role="bold">Create and mount <literal>ost1</literal>.</emphasis></para>
+ <para>In this example, the OSTs (<literal>ost1</literal> and <literal>ost2</literal>) are being created on different OSSs (<literal>oss1</literal> and <literal>oss2</literal> respectively).</para>
+ <orderedlist>
<listitem>
- <para>a. Create ost1. On oss1 node, run:</para>
- <screen>[root@oss1 /]# mkfs.lustre --ost --fsname=temp --mgsnode=10.2.0.1@tcp0 /dev\
+ <para><emphasis role="bold">Create <literal>ost1</literal>. On <literal>oss1</literal> node, run:</emphasis></para>
+ <screen>[root@oss1 /]# mkfs.lustre --ost --fsname=temp --mgsnode=10.2.0.1@tcp0 /dev\
/sdc
</screen>
- <para>The command generates this output:</para>
- <screen> Permanent disk data:
+
<para>The command generates this output:</para>
+ <screen> Permanent disk data:
Target: temp-OSTffff
Index: unassigned
Lustre FS: temp
</screen>
</listitem>
<listitem>
- <para>b. Mount ost1 on the OSS on which it was created. On oss1 node, run:</para>
- <screen>root@oss1 /] mount -t lustre /dev/sdc /mnt/ost1
+ <para><emphasis role="bold">Mount ost1 on the OSS on which it was created. On <literal>oss1</literal> node, run:</emphasis></para>
+ <screen>root@oss1 /] mount -t lustre /dev/sdc /mnt/ost1
</screen>
- <para>The command generates this output:</para>
- <screen>LDISKFS-fs: file extents enabled
+
<para>The command generates this output:</para>
+ <screen>LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: temp-OST0000: new disk, initializing
Lustre: Server temp-OST0000 on device /dev/sdb has started
</screen>
- <para>Shortly afterwards, this output appears:</para>
- <screen>Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0
+
<para>Shortly afterwards, this output appears:</para>
+ <screen>Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0
Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphans
</screen>
</listitem>
- </orderedlist>
- </listitem>
- <listitem>
- <para>Create and mount ost2.</para>
- <orderedlist>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Create and mount <literal>ost2</literal>.</emphasis></para>
+ <orderedlist>
<listitem>
- <para>Create ost2. On oss2 node, run:</para>
- <screen>[root@oss2 /]# mkfs.lustre --ost --fsname=temp --mgsnode=10.2.0.1@tcp0 /dev\
+ <para><emphasis role="bold">Create ost2. On <literal>oss2</literal> node, run:</emphasis></para>
+ <screen>[root@oss2 /]# mkfs.lustre --ost --fsname=temp --mgsnode=10.2.0.1@tcp0 /dev\
/sdd
</screen>
- <para>The command generates this output:</para>
- <screen> Permanent disk data:
+
<para>The command generates this output:</para>
+ <screen> Permanent disk data:
Target: temp-OSTffff
Index: unassigned
Lustre FS: temp
</screen>
</listitem>
<listitem>
- <para>Mount ost2 on the OSS on which it was created. On oss2 node, run:</para>
- <screen>root@oss2 /] mount -t lustre /dev/sdd /mnt/ost2
+ <para><emphasis role="bold">Mount ost2 on the OSS on which it was created. On <literal>oss2</literal> node, run:</emphasis></para>
+ <screen>root@oss2 /] mount -t lustre /dev/sdd /mnt/ost2
</screen>
- <para>The command generates this output:</para>
- <screen>LDISKFS-fs: file extents enabled
+
<para>The command generates this output:</para>
+ <screen>LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: temp-OST0000: new disk, initializing
Lustre: Server temp-OST0000 on device /dev/sdb has started
</screen>
- <para>Shortly afterwards, this output appears:</para>
- <screen>Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0
+
<para>Shortly afterwards, this output appears:</para>
+ <screen>Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0
Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphans
</screen>
</listitem>
- </orderedlist>
- </listitem>
- <listitem>
- <para>Mount the Lustre file system on the client. On the client node, run:</para>
- <screen>root@client1 /] mount -t lustre 10.2.0.1@tcp0:/temp /lustre
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Mount the Lustre file system on the client. On the client node, run:</emphasis></para>
+ <screen>root@client1 /] mount -t lustre 10.2.0.1@tcp0:/temp /lustre
</screen>
- <para>This command generates this output:</para>
- <screen>Lustre: Client temp-client has started
+
<para>This command generates this output:</para>
+ <screen>Lustre: Client temp-client has started
</screen>
- </listitem>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Verify that the file system started and is working by running the <literal>df</literal>, <literal>dd</literal> and <literal>ls</literal> commands on the client node.</emphasis></para>
+ <orderedlist>
<listitem>
- <para>Verify that the file system started and is working by running the df, dd and ls commands on the client node.</para>
-<orderedlist>
-<listitem>
- <para>Run the lfsdf -h command:</para>
- <screen>[root@client1 /] lfs df -h
+ <para><emphasis role="bold">Run the <literal>lfs df -h</literal> command:</emphasis></para>
+ <screen>[root@client1 /] lfs df -h
</screen>
- <para>The lfsdf-h command lists space usage per OST and the MDT in human-readable format. This command generates output similar to this:</para>
- <screen>UUID bytes Used Available \
+ <para>The <literal>lfs df -h</literal> command lists space usage per OST and the MDT in human-readable format. This command generates output similar to this:</para>
+ <screen>UUID bytes Used Available \
Use% Mounted on
temp-MDT0000_UUID 8.0G 400.0M 7.6G 0% \
/lustre[MDT:0]
0% /lustre
</screen>
-</listitem>
-<listitem>
- <para>Run the lfsdf-ih command.</para>
- <screen>[root@client1 /] lfs df -ih
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Run the <literal>lfs df -ih</literal> command.</emphasis></para>
+ <screen>[root@client1 /] lfs df -ih
</screen>
- <para>The lfsdf-ih command lists inode usage per OST and the MDT. This command generates output similar to this:</para>
- <screen>UUID Inodes IUsed IFree IUse% Mounted\
+ <para>The <literal>lfs df -ih</literal> command lists inode usage per OST and the MDT. This command generates output similar to this:</para>
+ <screen>UUID Inodes IUsed IFree IUse% Mounted\
on
temp-MDT0000_UUID 2.5M 32 2.5M 0% /lustre[MDT:0]
temp-OST0000_UUID 5.5M 54 5.5M 0% /lustre[OST:0]
filesystem summary: 2.5M 32 2.5M 0% /lustre
</screen>
-</listitem>
-<listitem>
- <para>c. Run the dd command:</para>
- <screen>[root@client1 /] cd /lustre
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Run the <literal>dd</literal> command:</emphasis></para>
+ <screen>[root@client1 /] cd /lustre
[root@client1 /lustre] dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2
</screen>
- <para>The dd command verifies write functionality by creating a file containing all zeros (0s). In this command, an 8 MB file is created. This command generates output similar to this:</para>
- <screen>2+0 records in
+ <para>The <literal>dd</literal> command verifies write functionality by creating a file containing all zeros (<literal>0</literal>s). In this command, an 8 MB file is created. This command generates output similar to this:</para>
+ <screen>2+0 records in
2+0 records out
8388608 bytes (8.4 MB) copied, 0.159628 seconds, 52.6 MB/s
</screen>
-
-</listitem>
-<listitem>
-
- <para>d. Run the ls command:</para>
- <screen>[root@client1 /lustre] ls -lsah
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Run the <literal>ls</literal> command:</emphasis></para>
+ <screen>[root@client1 /lustre] ls -lsah
</screen>
- <para>The ls-lsah command lists files and directories in the current working directory. This command generates output similar to this:</para>
- <screen>total 8.0M
+ <para>The <literal>ls -lsah</literal> command lists files and directories in the current working directory. This command generates output similar to this:</para>
+ <screen>total 8.0M
4.0K drwxr-xr-x 2 root root 4.0K Oct 16 15:27 .
8.0K drwxr-xr-x 25 root root 4.0K Oct 16 15:27 ..
8.0M -rw-r--r-- 1 root root 8.0M Oct 16 15:27 zero.dat
</screen>
</listitem>
- </orderedlist>
- </listitem>
- </orderedlist>
- <para>Once the Lustre file system is configured, it is ready for use.</para>
- </section>
+ </orderedlist>
+ </listitem>
+ </orderedlist>
+ <para>Once the Lustre file system is configured, it is ready for use.</para>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438267_76752">
+ <title>10.2 Additional Configuration Options</title>
+ <para>This section describes how to scale the Lustre file system or make configuration changes using the Lustre configuration utilities.</para>
+ <section remap="h3">
+ <title>10.2.1 Scaling the <anchor xml:id="dbdoclet.50438267_marker-1292440" xreflabel=""/>Lustre File System</title>
+ <para>A Lustre file system can be scaled by adding OSTs or clients. For instructions on creating additional OSTs repeat Step <xref linkend="dbdoclet.50438267_pgfId-1291170"/> and Step <xref linkend="dbdoclet.50438267_pgfId-1293955"/> above. For mounting additional clients, repeat Step <xref linkend="dbdoclet.50438267_pgfId-1290934"/> for each client.</para>
+ </section>
+ <section remap="h3">
+ <title>10.2.2 <anchor xml:id="dbdoclet.50438267_50212" xreflabel=""/>Changing Striping Defaults</title>
+ <para>The default settings for the file layout stripe pattern are shown in <xref linkend="configuringlustre.tab.stripe"/>.</para>
+ <table frame="none" xml:id="configuringlustre.tab.stripe">
+ <title>Default stripe pattern</title>
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="3*"/>
+ <colspec colname="c2" colwidth="13*"/>
+ <colspec colname="c3" colwidth="13*"/>
+ <tbody>
+ <row>
+ <entry>
+ <para><emphasis role="bold">File Layout Parameter</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Default</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>stripe_size</literal></para>
+ </entry>
+ <entry>
+ <para> 1 MB</para>
+ </entry>
+ <entry>
+ <para> Amount of data to write to one OST before moving to the next OST.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>stripe_count</literal></para>
+ </entry>
+ <entry>
+ <para> 1</para>
+ </entry>
+ <entry>
+ <para> The number of OSTs to use for a single file.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>start_ost</literal></para>
+ </entry>
+ <entry>
+ <para> -1</para>
+ </entry>
+ <entry>
+ <para> The first OST where objects are created for each file. The default -1 allows the MDS to choose the starting index based on available space and load balancing. <emphasis>It's strongly recommended not to change the default for this parameter to a value other than -1.</emphasis></para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ <para>Use the <literal>lfs setstripe</literal> command described in <xref linkend="managingstripingfreespace"/> to change the file layout configuration.</para>
+ </section>
+ <section remap="h3">
+ <title>10.2.3 Using the Lustre Configuration Utilities</title>
+ <para>If additional configuration is necessary, several configuration utilities are available:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>mkfs.lustre</literal> - Use to format a disk for a Lustre service.</para>
+ </listitem>
+ <listitem>
+ <para><literal>tunefs.lustre</literal> - Use to modify configuration information on a Lustre target disk.</para>
+ </listitem>
+ <listitem>
+ <para><literal>lctl</literal> - Use to directly control Lustre via an ioctl interface, allowing various configuration, maintenance and debugging features to be accessed.</para>
+ </listitem>
+ <listitem>
+ <para><literal>mount.lustre</literal> - Use to start a Lustre client or target service.</para>
+ </listitem>
+ </itemizedlist>
+ <para>For examples using these utilities, see the topic <xref linkend="systemconfigurationutilities"/></para>
+ <para>The lfs utility is usful for configuring and querying a variety of options related to files. For more information, see <xref linkend="userutilities"/>.</para>
+ <note>
+ <para>
+Some sample scripts are included in the directory where Lustre is installed. If you have installed the Lustre source code, the scripts are located in the <literal>lustre/tests</literal> sub-directory. These scripts enable quick setup of some simple standard Lustre configurations.</para>
+ </note>
</section>
- <section xml:id="dbdoclet.50438267_76752">
- <title>10.2 Additional Configuration Options</title>
- <para>This section describes how to scale the Lustre file system or make configuration changes using the Lustre configuration utilities.</para>
- <section remap="h3">
- <title>10.2.1 Scaling the <anchor xml:id="dbdoclet.50438267_marker-1292440" xreflabel=""/>Lustre File System</title>
- <para>A Lustre file system can be scaled by adding OSTs or clients. For instructions on creating additional OSTs repeat <xref linkend="dbdoclet.50438267_pgfId-1291170"/>Step 3 and <xref linkend="dbdoclet.50438267_pgfId-1293955"/>Step 4 above. For mounting additional clients, repeat <xref linkend="dbdoclet.50438267_pgfId-1290934"/>Step 5 for each client.</para>
- </section>
- <section remap="h3">
- <title>10.2.2 <anchor xml:id="dbdoclet.50438267_50212" xreflabel=""/>Changing Striping Defaults</title>
- <para>The default settings for the file layout stripe pattern are shown in <xref linkend='configuringlustre.tab.stripe'/>.</para>
- <table frame="none" xml:id='configuringlustre.tab.stripe'>
- <title>Default stripe pattern</title>
- <tgroup cols="3">
- <colspec colname="c1" colwidth="3*"/>
- <colspec colname="c2" colwidth="13*"/>
- <colspec colname="c3" colwidth="13*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">File Layout Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold">Default</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- <row>
- <entry><para> stripe_size</para></entry>
- <entry><para> 1 MB</para></entry>
- <entry><para> Amount of data to write to one OST before moving to the next OST.</para></entry>
- </row>
- <row>
- <entry><para> stripe_count</para></entry>
- <entry><para> 1</para></entry>
- <entry><para> The number of OSTs to use for a single file.</para></entry>
- </row>
- <row>
- <entry><para> start_ost</para></entry>
- <entry><para> -1</para></entry>
- <entry><para> The first OST where objects are created for each file. The default -1 allows the MDS to choose the starting index based on available space and load balancing. <emphasis>It's strongly recommended not to change the default for this parameter to a value other than -1.</emphasis></para></entry>
- </row>
- </tbody>
- </tgroup>
- </table>
- <para>Use the lfs setstripe command described in <xref linkend='managingstripingfreespace'/> to change the file layout configuration.</para>
- </section>
- <section remap="h3">
- <title>10.2.3 Using the Lustre Configuration Utilities</title>
- <para>If additional configuration is necessary, several configuration utilities are available:</para>
- <itemizedlist><listitem>
- <para>mkfs.lustre - Use to format a disk for a Lustre service.</para>
- </listitem>
-<listitem>
- <para>tunefs.lustre - Use to modify configuration information on a Lustre target disk.</para>
- </listitem>
-<listitem>
- <para>lctl - Use to directly control Lustre via an ioctl interface, allowing various configuration, maintenance and debugging features to be accessed.</para>
- </listitem>
-<listitem>
- <para>mount.lustre - Use to start a Lustre client or target service.</para>
- </listitem>
-</itemizedlist>
-<para>For examples using these utilities, see the topic <xref linkend='systemconfigurationutilities'/></para>
- <para>The lfs utility is usful for configuring and querying a variety of options related to files. For more information, see <xref linkend='userutilities'/>.</para>
-
- <note><para>
-Some sample scripts are included in the directory where Lustre is installed. If you have installed the Lustre source code, the scripts are located in the lustre/tests sub-directory. These scripts enable quick setup of some simple standard Lustre configurations.</para>
-</note>
-
- </section>
</section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='configuringstorage'>
-<info>
-<title xml:id='configuringstorage.title'>Configuring Storage on a Lustre File System</title>
-</info>
-<para>This chapter describes best practices for storage selection and file system options to optimize perforance on RAID, and includes the following sections:</para>
-
-<itemizedlist>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringstorage">
+ <info>
+ <title xml:id="configuringstorage.title">Configuring Storage on a Lustre File System</title>
+ </info>
+ <para>This chapter describes best practices for storage selection and file system options to optimize perforance on RAID, and includes the following sections:</para>
+ <itemizedlist>
<listitem>
- <xref linkend='dbdoclet.50438208_60972'/>
+ <para>
+ <xref linkend="dbdoclet.50438208_60972"/>
</para>
</listitem>
<listitem>
- <xref linkend='dbdoclet.50438208_23285'/>
+ <para>
+ <xref linkend="dbdoclet.50438208_23285"/>
</para>
</listitem>
<listitem>
- <xref linkend='dbdoclet.50438208_40705'/>
+ <para>
+ <xref linkend="dbdoclet.50438208_40705"/>
</para>
</listitem>
<listitem>
- <xref linkend='dbdoclet.50438208_51921'/>
+ <para>
+ <xref linkend="dbdoclet.50438208_51921"/>
</para>
</listitem>
<listitem>
- <xref linkend='dbdoclet.50438208_88516'/>
+ <para>
+ <xref linkend="dbdoclet.50438208_88516"/>
</para>
</listitem>
-</itemizedlist>
-
-<note><para><emphasis role="bold">It is strongly recommended that hardware RAID be used with Lustre.</emphasis> Lustre currently does not support any redundancy at the file system level and RAID is required to protect agains disk failure.</para></note>
-
-
-<section xml:id='dbdoclet.50438208_60972'>
- <title>6.1 Selecting Storage for the MDT and OSTs</title>
- <para>The Lustre architecture allows the use of any kind of block device as backend storage. The characteristics of such devices, particularly in the case of failures, vary significantly and have an impact on configuration choices.</para>
- <para>This section describes issues and recommendations regarding backend storage.</para>
- <section remap="h3">
- <title>6.1.1 Metadata Target (MDT)</title>
- <para>I/O on the MDT is typically mostly reads and writes of small amounts of data. For this reason, we recommend that you use RAID 1 for MDT storage. If you require more capacity for an MDT than one disk provides, we recommend RAID 1 + 0 or RAID 10.</para>
+ </itemizedlist>
+ <note>
+ <para><emphasis role="bold">It is strongly recommended that hardware RAID be used with Lustre.</emphasis> Lustre currently does not support any redundancy at the file system level and RAID is required to protect agains disk failure.</para>
+ </note>
+ <section xml:id="dbdoclet.50438208_60972">
+ <title>6.1 Selecting Storage for the MDT and OSTs</title>
+ <para>The Lustre architecture allows the use of any kind of block device as backend storage. The characteristics of such devices, particularly in the case of failures, vary significantly and have an impact on configuration choices.</para>
+ <para>This section describes issues and recommendations regarding backend storage.</para>
+ <section remap="h3">
+ <title>6.1.1 Metadata Target (MDT)</title>
+ <para>I/O on the MDT is typically mostly reads and writes of small amounts of data. For this reason, we recommend that you use RAID 1 for MDT storage. If you require more capacity for an MDT than one disk provides, we recommend RAID 1 + 0 or RAID 10.</para>
+ </section>
+ <section remap="h3">
+ <title>6.1.2 Object Storage Server (OST)</title>
+ <para>A quick calculation makes it clear that without further redundancy, RAID 6 is required for large clusters and RAID 5 is not acceptable:</para>
+ <blockquote>
+ <para>For a 2 PB file system (2,000 disks of 1 TB capacity) assume the mean time to failure (MTTF) of a disk is about 1,000 days. This means that the expected failure rate is 2000/1000 = 2 disks per day. Repair time at 10% of disk bandwidth is 1000 GB at 10MB/sec = 100,000 sec, or about 1 day.</para>
+ <para>For a RAID 5 stripe that is 10 disks wide, during 1 day of rebuilding, the chance that a second disk in the same array will fail is about 9/1000 or about 1% per day. After 50 days, you have a 50% chance of a double failure in a RAID 5 array leading to data loss.</para>
+ <para>Therefore, RAID 6 or another double parity algorithm is needed to provide sufficient redundancy for OST storage.</para>
+ </blockquote>
+ <para>For better performance, we recommend that you create RAID sets with 4 or 8 data disks plus one or two parity disks. Using larger RAID sets will negatively impact performance compared to having multiple independent RAID sets.</para>
+ <para>To maximize performance for small I/O request sizes, storage configured as RAID 1+0 can yield much better results but will increase cost or reduce capacity.</para>
+ </section>
</section>
- <section remap="h3">
- <title>6.1.2 Object Storage Server (OST)</title>
- <para>A quick calculation makes it clear that without further redundancy, RAID 6 is required for large clusters and RAID 5 is not acceptable:</para>
- <para>For a 2 PB file system (2,000 disks of 1 TB capacity) assume the mean time to failure (MTTF) of a disk is about 1,000 days. This means that the expected failure rate is 2000/1000 = 2 disks per day. Repair time at 10% of disk bandwidth is 1000 GB at 10MB/sec = 100,000 sec, or about 1 day.</para>
- <para>For a RAID 5 stripe that is 10 disks wide, during 1 day of rebuilding, the chance that a second disk in the same array will fail is about 9/1000 or about 1% per day. After 50 days, you have a 50% chance of a double failure in a RAID 5 array leading to data loss.</para>
- <para>Therefore, RAID 6 or another double parity algorithm is needed to provide sufficient redundancy for OST storage.</para>
- <para>For better performance, we recommend that you create RAID sets with 4 or 8 data disks plus one or two parity disks. Using larger RAID sets will negatively impact performance compared to having multiple independent RAID sets.</para>
- <para>To maximize performance for small I/O request sizes, storage configured as RAID 1+0 can yield much better results but will increase cost or reduce capacity.</para>
+ <section xml:id="dbdoclet.50438208_23285">
+ <title>6.2 Reliability Best Practices</title>
+ <para>RAID monitoring software is recommended to quickly detect faulty disks and allow them to be replaced to avoid double failures and data loss. Hot spare disks are recommended so that rebuilds happen without delays.</para>
+ <para>Backups of the metadata file systems are recommended. For details, see <xref linkend="backupandrestore"/>.</para>
</section>
-</section>
-<section xml:id="dbdoclet.50438208_23285">
- <title>6.2 Reliability Best Practices</title>
- <para>RAID monitoring software is recommended to quickly detect faulty disks and allow them to be replaced to avoid double failures and data loss. Hot spare disks are recommended so that rebuilds happen without delays.</para>
- <para>Backups of the metadata file systems are recommended. For details, see <xref linkend='backupandrestore'/>.</para>
-</section>
-<section xml:id="dbdoclet.50438208_40705">
- <title>6.3 Performance Tradeoffs</title>
- <para>A writeback cache can dramatically increase write performance on many types of RAID arrays if the writes are not done at full stripe width. Unfortunately, unless the RAID array has battery-backed cache (a feature only found in some higher-priced hardware RAID arrays), interrupting the power to the array may result in out-of-sequence writes or corruption of RAID parity and future data loss.</para>
- <para>If writeback cache is enabled, a file system check is required after the array loses power. Data may also be lost because of this.</para>
- <para>Therefore, we recommend against the use of writeback cache when data integrity is critical. You should carefully consider whether the benefits of using writeback cache outweigh the risks.</para>
-</section>
-<section xml:id="dbdoclet.50438208_51921">
- <title>6.4 Formatting Options for RAID Devices</title>
- <para>When formatting a file system on a RAID device, it is beneficial to ensure that I/O requests are aligned with the underlying RAID geometry. This ensures that the Lustre RPCs do not generate unnecessary disk operations which may reduce performance dramatically. Use the --mkfsoptions parameter to specify additional parameters when formatting the OST or MDT.</para>
- <para>For RAID 5, RAID 6, or RAID 1+0 storage, specifying the following option to the --mkfsoptions parameter option improves the layout of the file system metadata, ensuring that no single disk contains all of the allocation bitmaps:</para>
- <screen>-Estride=<chunk_blocks>
+ <section xml:id="dbdoclet.50438208_40705">
+ <title>6.3 Performance Tradeoffs</title>
+ <para>A writeback cache can dramatically increase write performance on many types of RAID arrays if the writes are not done at full stripe width. Unfortunately, unless the RAID array has battery-backed cache (a feature only found in some higher-priced hardware RAID arrays), interrupting the power to the array may result in out-of-sequence writes or corruption of RAID parity and future data loss.</para>
+ <para>If writeback cache is enabled, a file system check is required after the array loses power. Data may also be lost because of this.</para>
+ <para>Therefore, we recommend against the use of writeback cache when data integrity is critical. You should carefully consider whether the benefits of using writeback cache outweigh the risks.</para>
+ </section>
+ <section xml:id="dbdoclet.50438208_51921">
+ <title>6.4 Formatting Options for RAID Devices</title>
+ <para>When formatting a file system on a RAID device, it is beneficial to ensure that I/O requests are aligned with the underlying RAID geometry. This ensures that the Lustre RPCs do not generate unnecessary disk operations which may reduce performance dramatically. Use the <literal>--mkfsoptions</literal> parameter to specify additional parameters when formatting the OST or MDT.</para>
+ <para>For RAID 5, RAID 6, or RAID 1+0 storage, specifying the following option to the <literal>--mkfsoptions</literal> parameter option improves the layout of the file system metadata, ensuring that no single disk contains all of the allocation bitmaps:</para>
+ <screen>-E stride = <chunk_blocks>
</screen>
- <para>The <chunk_blocks> variable is in units of 4096-byte blocks and represents the amount of contiguous data written to a single disk before moving to the next disk. This is alternately referred to as the RAID stripe size. This is applicable to both MDT and OST file systems.</para>
- <para>For more information on how to override the defaults while formatting MDT or OST file systems, see <link xl:href="SettingUpLustreSystem.html#50438256_84701">Setting File System Formatting Options</link>.</para>
- <section remap="h3">
- <title>6.4.1 Computing file system parameters for mkfs</title>
- <para>For best results, use RAID 5 with 5 or 9 disks or RAID 6 with 6 or 10 disks, each on a different controller. The stripe width is the optimal minimum I/O size. Ideally, the RAID configuration should allow 1 MB Lustre RPCs to fit evenly on a single RAID stripe without an expensive read-modify-write cycle. Use this formula to determine the <emphasis><stripe_width></emphasis>, where <emphasis><number_of_data_disks></emphasis> does <emphasis>not</emphasis> include the RAID parity disks (1 for RAID 5 and 2 for RAID 6):</para>
- <screen><emphasis><stripe_width_blocks></emphasis> = <emphasis><chunk_blocks></emphasis> * <emphasis><number_of_data_disks></emphasis> = 1 MB
+ <para>The <literal><chunk_blocks></literal> variable is in units of 4096-byte blocks and represents the amount of contiguous data written to a single disk before moving to the next disk. This is alternately referred to as the RAID stripe size. This is applicable to both MDT and OST file systems.</para>
+ <para>For more information on how to override the defaults while formatting MDT or OST file systems, see <xref linkend="dbdoclet.50438256_84701"/>.</para>
+ <section remap="h3">
+ <title>6.4.1 Computing file system parameters for mkfs</title>
+ <para>For best results, use RAID 5 with 5 or 9 disks or RAID 6 with 6 or 10 disks, each on a different controller. The stripe width is the optimal minimum I/O size. Ideally, the RAID configuration should allow 1 MB Lustre RPCs to fit evenly on a single RAID stripe without an expensive read-modify-write cycle. Use this formula to determine the <emphasis>
+ <literal><stripe_width></literal>
+ </emphasis>, where <emphasis>
+ <literal><number_of_data_disks></literal>
+ </emphasis> does <emphasis>not</emphasis> include the RAID parity disks (1 for RAID 5 and 2 for RAID 6):</para>
+ <screen><emphasis><stripe_width_blocks></emphasis> = <emphasis><chunk_blocks></emphasis> * <emphasis><number_of_data_disks></emphasis> = 1 MB
</screen>
- <para>If the RAID configuration does not allow <emphasis><chunk_blocks></emphasis> to fit evenly into 1 MB, select <emphasis><chunkblocks></emphasis>, such that <emphasis><stripe_width_blocks></emphasis> is close to 1 MB, but not larger.</para>
- <para>The <emphasis><stripe_width_blocks></emphasis>value must equal <emphasis><chunk_blocks></emphasis>*<emphasis><number_of_data_disks></emphasis>. Specifying the <emphasis><stripe_width_blocks></emphasis> parameter is only relevant for RAID 5 or RAID 6, and is not needed for RAID 1 plus 0.</para>
- <para>Run --reformat on the file system device (/dev/sdc), specifying the RAID geometry to the underlying ldiskfs file system, where:</para>
- <screen>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, stripe_width=<emphasis><stripe_width_blocks></emphasis>"
+ <para>If the RAID configuration does not allow <emphasis>
+ <literal><chunk_blocks></literal>
+ </emphasis> to fit evenly into 1 MB, select <literal>
+ <emphasis><chunkblocks></emphasis>
+ </literal>, such that <literal>
+ <emphasis><stripe_width_blocks></emphasis>
+ </literal> is close to 1 MB, but not larger.</para>
+ <para>The <emphasis>
+ <literal><stripe_width_blocks></literal>
+ </emphasis> value must equal <literal><emphasis><chunk_blocks></emphasis>*<emphasis><number_of_data_disks></emphasis></literal>. Specifying the <emphasis>
+ <literal><stripe_width_blocks></literal>
+ </emphasis> parameter is only relevant for RAID 5 or RAID 6, and is not needed for RAID 1 plus 0.</para>
+ <para>Run <literal>--reformat</literal> on the file system device (<literal>/dev/sdc</literal>), specifying the RAID geometry to the underlying ldiskfs file system, where:</para>
+ <screen>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, stripe_width=<emphasis><stripe_width_blocks></emphasis>"
</screen>
- <para>Example:</para>
- <para>A RAID 6 configuration with 6 disks has 4 data and 2 parity disks. The <emphasis><chunk_blocks></emphasis> <= 1024KB/4 = 256KB.</para>
- <para>Because the number of data disks is equal to the power of 2, the stripe width is equal to 1 MB.</para>
- <screen>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, stripe_width=<emphasis><stripe_width_blocks></emphasis>"...
+ <para><example>A RAID 6 configuration with 6 disks has 4 data and 2 parity disks. The <literal>
+ <emphasis><chunk_blocks></emphasis>
+ </literal> <= 1024KB/4 = 256KB.</example></para>
+ <para>Because the number of data disks is equal to the power of 2, the stripe width is equal to 1 MB.</para>
+ <screen>--mkfsoptions "<emphasis><other options></emphasis> -E stride=<emphasis><chunk_blocks></emphasis>, stripe_width=<emphasis><stripe_width_blocks></emphasis>"...
</screen>
- </section>
- <section remap="h3">
- <title>6.4.2 Choosing Parameters for an External <anchor xml:id="dbdoclet.50438208_marker-1289927" xreflabel=""/>Journal</title>
- <para>If you have configured a RAID array and use it directly as an OST, it contains both data and metadata. For better performance, we recommend putting the OST journal on a separate device, by creating a small RAID 1 array and using it as an external journal for the OST.</para>
- <para>Lustre's default journal size is 400 MB. A journal size of up to 1 GB has shown increased performance but diminishing returns are seen for larger journals. Additionally, a copy of the journal is kept in RAM. Therefore, make sure you have enough memory available to hold copies of all the journals.</para>
- <para>The file system journal options are specified to mkfs.luster using the --mkfsoptions parameter. For example:</para>
- <screen>--mkfsoptions "<other options> -j -J device=/dev/mdJ"
+ </section>
+ <section remap="h3">
+
<title>6.4.2 Choosing Parameters for an External <anchor xml:id="dbdoclet.50438208_marker-1289927" xreflabel=""/>Journal</title>
+
<para>If you have configured a RAID array and use it directly as an OST, it contains both data and metadata. For better performance, we recommend putting the OST journal on a separate device, by creating a small RAID 1 array and using it as an external journal for the OST.</para>
+
<para>Lustre's default journal size is 400 MB. A journal size of up to 1 GB has shown increased performance but diminishing returns are seen for larger journals. Additionally, a copy of the journal is kept in RAM. Therefore, make sure you have enough memory available to hold copies of all the journals.</para>
+ <para>The file system journal options are specified to mkfs.luster using the <literal>--mkfsoptions</literal> parameter. For example:</para>
+ <screen>--mkfsoptions "<other options> -j -J device=/dev/mdJ"
</screen>
- <para>To create an external journal, perform these steps for each OST on the OSS:</para>
- <para> 1. Create a 400 MB (or larger) journal partition (RAID 1 is recommended).</para>
- <para>In this example, /dev/sdb is a RAID 1 device.</para>
- <para> 2. Create a journal device on the partition. Run:</para>
- <screen>[oss#] mke2fs -b 4096 -O journal_dev /dev/sdb <emphasis><journal_size></emphasis></screen>
- <para>The value of <emphasis><journal_size></emphasis> is specified in units of 4096-byte blocks. For example, 262144 for a 1 GB journal size.</para>
- <para> 3. Create the OST.</para>
- <para>In this example, /dev/sdc is the RAID 6 device to be used as the OST, run:</para>
- <screen>[oss#] mkfs.lustre --ost --mgsnode=mds@osib --mkfsoptions="-J device=/dev/sd\
+ <para>To create an external journal, perform these steps for each OST on the OSS:</para>
+ <orderedlist>
+ <listitem>
+ <para>Create a 400 MB (or larger) journal partition (RAID 1 is recommended).</para>
+ <para>In this example, <literal>/dev/sdb</literal> is a RAID 1 device.</para>
+ </listitem>
+ <listitem>
+ <para>Create a journal device on the partition. Run:</para>
+ <screen>[oss#] mke2fs -b 4096 -O journal_dev /dev/sdb <emphasis><journal_size></emphasis></screen>
+ <para>The value of <literal>
+ <emphasis><journal_size></emphasis>
+ </literal> is specified in units of 4096-byte blocks. For example, 262144 for a 1 GB journal size.</para>
+ </listitem>
+ <listitem>
+ <para>Create the OST.</para>
+ <para>In this example, <literal>/dev/sdc</literal> is the RAID 6 device to be used as the OST, run:</para>
+ <screen>[oss#] mkfs.lustre --ost --mgsnode=mds@osib --mkfsoptions="-J device=/dev/sd\
b1" /dev/sdc
</screen>
- <para> 4. Mount the OST as usual.</para>
+ </listitem>
+ <listitem>
+ <para>Mount the OST as usual.</para>
+ </listitem>
+ </orderedlist>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438208_88516">
+ <title>6.5 Connecting a SAN to a Lustre File System</title>
+ <para>Depending on your cluster size and workload, you may want to connect a SAN to a Lustre file system. Before making this connection, consider the following:</para>
+ <itemizedlist>
+ <listitem>
+ <para>In many SAN file systems without Lustre, clients allocate and lock blocks or inodes individually as they are updated. The Lustre design avoids the high contention that some of these blocks and inodes may have.</para>
+ </listitem>
+ <listitem>
+ <para>Lustre is highly scalable and can have a very large number of clients. SAN switches do not scale to a large number of nodes, and the cost per port of a SAN is generally higher than other networking.</para>
+ </listitem>
+ <listitem>
+ <para>File systems that allow direct-to-SAN access from the clients have a security risk because clients can potentially read any data on the SAN disks, and misbehaving clients can corrupt the file system for many reasons like improper file system, network, or other kernel software, bad cabling, bad memory, and so on. The risk increases with increase in the number of clients directly accessing the storage.</para>
+ </listitem>
+ </itemizedlist>
</section>
-</section>
-<section xml:id="dbdoclet.50438208_88516">
- <title>6.5 Connecting a SAN to a Lustre File System</title>
- <para>Depending on your cluster size and workload, you may want to connect a SAN to a Lustre file system. Before making this connection, consider the following:</para>
- <itemizedlist><listitem>
- <para> In many SAN file systems without Lustre, clients allocate and lock blocks or inodes individually as they are updated. The Lustre design avoids the high contention that some of these blocks and inodes may have.</para>
- </listitem>
-<listitem>
- <para> Lustre is highly scalable and can have a very large number of clients. SAN switches do not scale to a large number of nodes, and the cost per port of a SAN is generally higher than other networking.</para>
- </listitem>
-<listitem>
- <para> File systems that allow direct-to-SAN access from the clients have a security risk because clients can potentially read any data on the SAN disks, and misbehaving clients can corrupt the file system for many reasons like improper file system, network, or other kernel software, bad cabling, bad memory, and so on. The risk increases with increase in the number of clients directly accessing the storage.</para>
- </listitem>
-</itemizedlist>
-</section>
</chapter>
<partintro>
<para>Part III provides information about tools and procedures to use to administer a Lustre file system. You will find information in this section about:</para>
- <tip>
- <para>The starting point for administering Lustre is to monitor all logs and console logs for system health:</para>
- <para> - Monitor logs on all servers and all clients.</para>
- <para> - Invest in tools that allow you to condense logs from multiple systems.</para>
- <para> - Use the logging resources provided by Linux.</para>
- </tip>
-
-
<itemizedlist>
<listitem>
<para>
</listitem>
</itemizedlist>
+ <tip>
+ <para>The starting point for administering Lustre is to monitor all logs and console logs for system health:</para>
+ <para> - Monitor logs on all servers and all clients.</para>
+ <para> - Invest in tools that allow you to condense logs from multiple systems.</para>
+ <para> - Use the logging resources provided by Linux.</para>
+ </tip>
+
+
</partintro>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='installinglustre'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="installinglustre">
<info>
- <title xml:id='installinglustre.title'>Installing the Lustre Software</title>
+ <title xml:id="installinglustre.title">Installing the Lustre Software</title>
</info>
-
<para>This chapter describes how to install the Lustre software. It includes:</para>
-
<itemizedlist>
- <listitem>
- <xref linkend='dbdoclet.50438261_99193'/>
+ <listitem>
+ <para>
+ <xref linkend="dbdoclet.50438261_99193"/>
</para>
- </listitem>
- <listitem>
- <xref linkend='dbdoclet.50438261_41900'/>
+ </listitem>
+ <listitem>
+ <para>
+ <xref linkend="dbdoclet.50438261_41900"/>
</para>
- </listitem>
+ </listitem>
</itemizedlist>
-
-
-
-
- <para>For hardware/system requirements, see <xref linkend='settinguplustresystem'/>.</para>
- <section xml:id="dbdoclet.50438261_99193">
- <title>8.1 Preparing to Install the Lustre Software</title>
- <para>If you are using a supported Linux distribution and architecture, you can install Lustre from downloaded packages (RPMs). For a list of supported configurations, see the <link xl:href="http://wiki.whamcloud.com/">Lustre 2.0</link> topic on the Lustre wiki.</para>
- <para>A list of supported Lustre LNET drivers can be found in <xref linkend='understandinglustrenetworking'/>.</para>
- <para>If you are not using a supported configuration, you can install Lustre directly from the source code. For more information on this installation method, see <xref linkend='installinglustrefromsourcecode'/>.</para>
- <section remap="h3">
- <title>8.1.1 <anchor xml:id="dbdoclet.50438261_37079" xreflabel=""/>Required Software</title>
- <para>To install Lustre, the following are required:</para>
- <itemizedlist><listitem>
- <para><emphasis>(On Linux servers only)</emphasis> Linux kernel patched with Lustre-specific patches for your platform and architecture. A Linux patched kernel can be installed on a client if it is desireable to use the same kernel on all nodes, but this is not required.</para>
- </listitem>
-<listitem>
- <para> Lustre modules compiled for the Linux kernel (see <xref linkend='installinglustre.tab.req'/> for which packages are required for servers and clients in your configuration).</para>
- </listitem>
-<listitem>
- <para> Lustre utilities required for configuring Lustre (see <xref linkend='installinglustre.tab.req'/> for which packages are required for servers and clients in your configuration).</para>
- </listitem>
-<listitem>
- <para><emphasis>(On Linux servers only)</emphasis>e2fsprogs package containing Lustre-specific tools (e2fsck and lfsck) used to repair a backing file system. This replaces the existing e2fsprogs package but provides complete e2fsprogs functionality</para>
- </listitem>
-<listitem>
- <para><emphasis>(Optional)</emphasis> Network-specific kernel modules and libraries such as the Lustre-specific OFED package required for an InfiniBand network interconnect.</para>
+ <para>For hardware/system requirements, see <xref linkend="settinguplustresystem"/>.</para>
+ <section xml:id="dbdoclet.50438261_99193">
+ <title>8.1 Preparing to Install the Lustre Software</title>
+ <para>If you are using a supported Linux distribution and architecture, you can install Lustre from downloaded packages (RPMs). For a list of supported configurations, see the <link xl:href="http://wiki.whamcloud.com/">Lustre 2.0</link> topic on the Lustre wiki.</para>
+ <para>A list of supported Lustre LNET drivers can be found in <xref linkend="understandinglustrenetworking"/>.</para>
+ <para>If you are not using a supported configuration, you can install Lustre directly from the source code. For more information on this installation method, see <xref linkend="installinglustrefromsourcecode"/>.</para>
+ <section remap="h3">
+ <title>8.1.1 <anchor xml:id="dbdoclet.50438261_37079" xreflabel=""/>Required Software</title>
+ <para>To install Lustre, the following are required:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis>(On Linux servers only)</emphasis> Linux kernel patched with Lustre-specific patches for your platform and architecture. A Linux patched kernel can be installed on a client if it is desireable to use the same kernel on all nodes, but this is not required.</para>
+ </listitem>
+ <listitem>
+ <para> Lustre modules compiled for the Linux kernel (see <xref linkend="installinglustre.tab.req"/> for which packages are required for servers and clients in your configuration).</para>
+ </listitem>
+ <listitem>
+ <para> Lustre utilities required for configuring Lustre (see <xref linkend="installinglustre.tab.req"/> for which packages are required for servers and clients in your configuration).</para>
+ </listitem>
+ <listitem>
+ <para><emphasis>(On Linux servers only) </emphasis><literal>e2fsprogs</literal> package containing Lustre-specific tools (<literal>e2fsck</literal> and <literal>lfsck</literal>) used to repair a backing file system. This replaces the existing <literal>e2fsprogs</literal> package but provides complete <literal>e2fsprogs</literal> functionality</para>
+ </listitem>
+ <listitem>
+ <para><emphasis>(Optional)</emphasis> Network-specific kernel modules and libraries such as the Lustre-specific OFED package required for an InfiniBand network interconnect.</para>
+ </listitem>
+ </itemizedlist>
+ <para>At least one Lustre RPM must be installed on each server and on each client in a Lustre file system. <xref linkend="installinglustre.tab.req"/> lists required Lustre packages and indicates where they are to be installed. Some Lustre packages are installed on Lustre servers (MGS, MDS, and OSSs), some are installed on Lustre clients, and some are installed on all Lustre nodes.</para>
+ <para>.</para>
+ <table xml:id="installinglustre.tab.req" frame="all">
+ <title>Lustre required packages, descriptions and installation guidance</title>
+ <tgroup cols="5">
+ <colspec colname="c1" colwidth="2*"/>
+ <colspec colname="c2" colwidth="12*"/>
+ <colspec colname="c3" colwidth="20*"/>
+ <colspec colname="c4" colwidth="6*"/>
+ <colspec colname="c5" colwidth="6*"/>
+ <thead>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para><emphasis role="bold"><anchor xml:id="dbdoclet.50438261_97803" xreflabel=""/>Lustre Package</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Install on servers</emphasis>
+ <footnote>
+ <para>Installing a patched kernel on a client node is not required. However, if a client node will be used as both a client and a server, or if you want to install the same kernel on all nodes for any reason, install the server packages designated with an asterisk (*) on the client node.</para>
+ </footnote></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Install on clients</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry nameend="c3" namest="c1">
+ <para> <emphasis role="bold"> Lustre patched kernel RPMs:</emphasis></para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> kernel-<ver>_lustre.<ver></para>
+ </entry>
+ <entry>
+ <para> For OEL 5 or RHEL 5 server platform</para>
+ </entry>
+ <entry>
+ <para> X*</para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> kernel-ib-<ver></para>
+ </entry>
+ <entry>
+ <para> Lustre OFED package. Required only if the network interconnect is InfiniBand.</para>
+ </entry>
+ <entry>
+ <para> X*</para>
+ </entry>
+ <entry>
+ <para> X</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis> Lustre module RPMs:</emphasis></para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> lustre-modules-<ver></para>
+ </entry>
+ <entry>
+ <para> For Lustre-patched kernel.</para>
+ </entry>
+ <entry>
+ <para> X*</para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> lustre-client-modules-<ver></para>
+ </entry>
+ <entry>
+ <para> For clients.</para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> X</para>
+ </entry>
+ </row>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para> <emphasis> Lustre utilities:</emphasis></para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> lustre-<ver></para>
+ </entry>
+ <entry>
+ <para> Lustre utilities package. This includes userspace utilities to configure and run Lustre.</para>
+ </entry>
+ <entry>
+ <para>  </para>
+ <para> </para>
+ <para>X*</para>
+ </entry>
+ <entry>
+ <para>  </para>
+ <para> </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> lustre-client-<ver></para>
+ </entry>
+ <entry>
+ <para> Lustre utilities for clients.</para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> X</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> lustre-ldiskfs-<ver></para>
+ </entry>
+ <entry>
+ <para> Lustre-patched backing file system kernel module package for the ldiskfs file system.</para>
+ </entry>
+ <entry>
+ <para>  </para>
+ <para>X</para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>  </para>
+ </entry>
+ <entry>
+ <para> e2fsprogs-<ver></para>
+ </entry>
+ <entry>
+ <para> Utilities package used to maintain the ldiskfs backing file system.</para>
+ </entry>
+ <entry>
+ <para>  </para>
+ <para>X</para>
+ </entry>
+ <entry>
+ <para>  </para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ <para>In all supported Lustre installations, a patched kernel must be run on each server, including the the MGS, the MDS, and all OSSs. Running a patched kernel on a Lustre client is only required if the client will be used for multiple purposes, such as running as both a client and an OST or if you want to use the same kernel on all nodes.</para>
+ <para>Lustre RPM packages are available on the <link xl:href="http://build.whamcloud.com">Whamcloud download site</link>. They must be installed in the order described in the <xref linkend="dbdoclet.50438261_41900"/>.</para>
+ <section remap="h4">
+ <title>8.1.1.1 Network-specific Kernel Modules and Libraries</title>
+ <para>Network-specific kernel modules and libraries may be needed such as the Lustre-specific OFED package required for an InfiniBand network interconnect.</para>
+ </section>
+ <section remap="h4">
+ <title>8.1.1.2 Lustre-Specific Tools <anchor xml:id="dbdoclet.50438261_marker-1292890" xreflabel=""/>and Utilities</title>
+ <para>Several third-party utilities are must be installed on servers:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">e2fsprogs</emphasis><anchor xml:id="dbdoclet.50438261_marker-1292893" xreflabel=""/> : Lustre requires a recent version of e2fsprogs that understands extents. Use <literal>e2fsprogs-1.41-90-wc1</literal> or later, available at:</para>
+ <para><ulink url="http://build.whamcloud.com/">http://build.whamcloud.com/</ulink></para>
+ <para>A quilt patchset of all changes to the vanilla e2fsprogs is available in <literal>e2fsprogs-{version}-patches.tgz</literal>.</para>
+ <note>
+ <para>The Lustre-patched <literal>e2fsprogs</literal> utility is only required on machines that mount backend (<literal>ldiskfs</literal>) file systems, such as the OSS, MDS and MGS nodes. It does not need to be loaded on clients.</para>
+ </note>
</listitem>
-</itemizedlist>
-<para>At least one Lustre RPM must be installed on each server and on each client in a Lustre file system. <xref linkend='installinglustre.tab.req'/> lists required Lustre packages and indicates where they are to be installed. Some Lustre packages are installed on Lustre servers (MGS, MDS, and OSSs), some are installed on Lustre clients, and some are installed on all Lustre nodes.</para>
- <para>.</para>
-
- <table xml:id='installinglustre.tab.req' frame="all">
- <title>Lustre required packages, descriptions and installation guidance</title>
- <tgroup cols="5">
- <colspec colname="c1" colwidth="2*"/>
- <colspec colname="c2" colwidth="12*"/>
- <colspec colname="c3" colwidth="20*"/>
- <colspec colname="c4" colwidth="6*"/>
- <colspec colname="c5" colwidth="6*"/>
- <thead>
- <row>
- <entry nameend="c2" namest="c1"><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438261_97803" xreflabel=""/>Lustre Package</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- <entry><para><emphasis role="bold">Install on servers</emphasis>
- <footnote><para>Installing a patched kernel on a client node is not required. However, if a client node will be used as both a client and a server, or if you want to install the same kernel on all nodes for any reason, install the server packages designated with an asterisk (*) on the client node.</para></footnote></para></entry>
- <entry><para><emphasis role="bold">Install on clients</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry nameend="c3" namest="c1"><para> <emphasis role="bold"> Lustre patched kernel RPMs:</emphasis></para></entry>
- <entry><para> </para></entry>
- <entry><para> </para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> kernel-<ver>_lustre.<ver></para></entry>
- <entry><para> For OEL 5 or RHEL 5 server platform</para></entry>
- <entry><para> X*</para></entry>
- <entry><para> </para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> kernel-ib-<ver></para></entry>
- <entry><para> Lustre OFED package. Required only if the network interconnect is InfiniBand.</para></entry>
- <entry><para> X*</para></entry>
- <entry><para> X</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis> Lustre module RPMs:</emphasis></para></entry>
- <entry><para> </para></entry>
- <entry><para> </para></entry>
- <entry><para> </para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> lustre-modules-<ver></para></entry>
- <entry><para> For Lustre-patched kernel.</para></entry>
- <entry><para> X*</para></entry>
- <entry><para> </para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> lustre-client-modules-<ver></para></entry>
- <entry><para> For clients.</para></entry>
- <entry><para> </para></entry>
- <entry><para> X</para></entry>
- </row>
- <row>
- <entry nameend="c2" namest="c1"><para> <emphasis> Lustre utilities:</emphasis></para></entry>
- <entry><para> </para></entry>
- <entry><para> </para></entry>
- <entry><para> </para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> lustre-<ver></para></entry>
- <entry><para> Lustre utilities package. This includes userspace utilities to configure and run Lustre.</para></entry>
- <entry><para> </para><para> </para><para>X*</para></entry>
- <entry><para> </para><para> </para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> lustre-client-<ver></para></entry>
- <entry><para> Lustre utilities for clients.</para></entry>
- <entry><para> </para></entry>
- <entry><para> X</para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> lustre-ldiskfs-<ver></para></entry>
- <entry><para> Lustre-patched backing file system kernel module package for the ldiskfs file system.</para></entry>
- <entry><para> </para><para>X</para></entry>
- <entry><para> </para></entry>
- </row>
- <row>
- <entry><para> </para></entry>
- <entry><para> e2fsprogs-<ver></para></entry>
- <entry><para> Utilities package used to maintain the ldiskfs backing file system.</para></entry>
- <entry><para> </para><para>X</para></entry>
- <entry><para> </para></entry>
- </row>
- </tbody>
- </tgroup>
- </table>
- <para>In all supported Lustre installations, a patched kernel must be run on each server, including the the MGS, the MDS, and all OSSs. Running a patched kernel on a Lustre client is only required if the client will be used for multiple purposes, such as running as both a client and an OST or if you want to use the same kernel on all nodes.</para>
- <para>Lustre RPM packages are available on the <link xl:href="http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L">Lustre download site</link>. They must be installed in the order described in the <link xl:href="InstallingLustre.html#50438261_41900">Lustre Installation Procedure</link>.</para>
- <section remap="h4">
- <title>8.1.1.1 Network-specific Kernel Modules and Libraries</title>
- <para>Network-specific kernel modules and libraries may be needed such as the Lustre-specific OFED package required for an InfiniBand network interconnect.</para>
- </section>
- <section remap="h4">
- <title>8.1.1.2 Lustre-Specific Tools <anchor xml:id="dbdoclet.50438261_marker-1292890" xreflabel=""/>and Utilities</title>
- <para>Several third-party utilities are must be installed on servers:</para>
- <itemizedlist><listitem>
- <para><emphasis role="bold">e2fsprogs</emphasis><anchor xml:id="dbdoclet.50438261_marker-1292893" xreflabel=""/> : Lustre requires a recent version of e2fsprogs that understands extents. Use e2fsprogs-1.41-10-sun2 or later, available at:</para>
- <para><link xl:href="http://downloads.lustre.org/public/tools/e2fsprogs/">http://downloads.lustre.org/public/tools/e2fsprogs/</link></para>
- <para>A quilt patchset of all changes to the vanilla e2fsprogs is available in e2fsprogs-{version}-patches.tgz.</para>
- <note><para>The Lustre-patched e2fsprogs utility is only required on machines that mount backend (ldiskfs) file systems, such as the OSS, MDS and MGS nodes. It does not need to be loaded on clients.</para>
- </note>
- </listitem>
-
<listitem>
-
<para><emphasis role="bold">Perl</emphasis><anchor xml:id="dbdoclet.50438261_marker-1292899" xreflabel=""/> - Various userspace utilities are written in Perl. Any recent version of Perl will work with Lustre.</para>
- </listitem>
+ <para><emphasis role="bold">Perl</emphasis><anchor xml:id="dbdoclet.50438261_marker-1292899" xreflabel=""/> - Various userspace utilities are written in Perl. Any recent version of Perl will work with Lustre.</para>
+ </listitem>
</itemizedlist>
-
- </section>
- <section remap="h4">
- <title>8.1.1.3 <anchor xml:id="dbdoclet.50438261_29838" xreflabel=""/>(Optional) <anchor xml:id="dbdoclet.50438261_marker-1292902" xreflabel=""/>High-Availability Software</title>
- <para>If you plan to enable failover server functionality with Lustre (either on an OSS or the MDS), you must add high-availability (HA) software to your cluster software. For more information, see <xref linkend='configuringfailover'/>Setting up High-Availability (HA) Software with Lustre.</para>
- </section>
- <section remap="h4">
- <title>8.1.1.4 (Optional) Debugging Tools and Other Optional Packages</title>
- <para>A variety of optional packages are provided on the <emphasis><link xl:href="http://downloads.whamcloud.com">Whamcloud download site</link></emphasis>. These include debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages.</para>
- <para>For more information about debugging tools, see the topic <emphasis><link xl:href="http://wiki.lustre.org/index.php/Debugging_Lustre">Debugging Lustre</link></emphasis> on the Lustre wiki.</para>
- </section>
</section>
- <section remap="h3">
- <title>8.1.2 <anchor xml:id="dbdoclet.50438261_19503" xreflabel=""/>Environmental <anchor xml:id="dbdoclet.50438261_marker-1292914" xreflabel=""/>Requirements</title>
- <para>Make sure the following environmental requirements are met before installing Lustre:</para>
- <itemizedlist><listitem>
- <para><emphasis>(Required)</emphasis><emphasis role="bold">Disable Security-Enhanced Linux (SELinux) on servers and clients</emphasis> . Lustre does not support SELinux. Therefore, disable the SELinux system extension on all Lustre nodes and make sure other security extensions, like Novell AppArmorand network packet filtering tools (such as iptables) do not interfere with Lustre. See <link xl:href="InstallingLustre.html#50438261_88296">Step 3</link> in the <link xl:href="InstallingLustre.html#50438261_41900">Lustre Installation Procedure</link> below.</para>
- </listitem>
-<listitem>
- <para><emphasis>(Required)</emphasis><emphasis role="bold">Maintain uniform user and group databases on all cluster nodes</emphasis> . Use the same user IDs (UID) and group IDs (GID) on all clients. If use of supplemental groups is required, verify that the group_upcall requirements have been met. See <link xl:href="LustreProgrammingInterfaces.html#50438291_32926">User/Group Cache Upcall</link>.</para>
- </listitem>
-<listitem>
- <para><emphasis>(Recommended)</emphasis><emphasis role="bold">Provide remote shell access to clients</emphasis> . Although not strictly required to run Lustre, we recommend that all cluster nodes have remote shell client access to facilitate the use of Lustre configuration and monitoring scripts. Parallel Distributed SHell (pdsh) is preferable, although Secure SHell (SSH) is acceptable.</para>
- </listitem>
-<listitem>
- <para><emphasis>(Recommended)</emphasis><emphasis role="bold">Ensure client clocks are synchronized</emphasis> . Lustre uses client clocks for timestamps. If clocks are out of sync between clients, files will appear with different time stamps when accessed by different clients. Drifting clocks can also cause problems, for example, by making it difficult to debug multi-node issues or correlate logs, which depend on timestamps. We recommend that you use Network Time Protocol (NTP) to keep client and server clocks in sync with each other. For more information about NTP, see: <link xl:href="http://www.ntp.org/">http://www.ntp.org</link>.</para>
- </listitem>
-</itemizedlist>
+ <section remap="h4">
+ <title>8.1.1.3 <anchor xml:id="dbdoclet.50438261_29838" xreflabel=""/>(Optional) <anchor xml:id="dbdoclet.50438261_marker-1292902" xreflabel=""/>High-Availability Software</title>
+ <para>If you plan to enable failover server functionality with Lustre (either on an OSS or the MDS), you must add high-availability (HA) software to your cluster software. For more information, see <xref linkend="dbdoclet.50438188_92688"/>.</para>
+ </section>
+ <section remap="h4">
+ <title>8.1.1.4 (Optional) Debugging Tools and Other Optional Packages</title>
+ <para>A variety of optional packages are provided on the <emphasis>
+ <ulink xl:href="http://downloads.whamcloud.com">Whamcloud download site</ulink>
+ </emphasis>. These include debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages.</para>
+ <para>For more information about debugging tools, see the topic <emphasis>
+ <ulink xl:href="http://wiki.lustre.org/index.php/Debugging_Lustre">Debugging Lustre</ulink>
+ </emphasis> on the Lustre wiki.</para>
</section>
</section>
- <section xml:id="dbdoclet.50438261_41900">
- <title>8.2 Lustre Installation Procedure</title>
-
- <caution>
-<para>Before installing Lustre, back up ALL data. Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured or administered properly.</para>
-</caution>
-
- <para>Use this procedure to install Lustre from RPMs.</para>
- <orderedlist>
- <listitem>
- <para>Verify that all Lustre installation requirements have been met.</para>
- <para>For more information on these prerequisites, see:</para>
- <itemizedlist><listitem>
- <para> Hardware requirements in <xref linkend='settinguplustresystem'/></para>
+ <section remap="h3">
+ <title>8.1.2 <anchor xml:id="dbdoclet.50438261_19503" xreflabel=""/>Environmental <anchor xml:id="dbdoclet.50438261_marker-1292914" xreflabel=""/>Requirements</title>
+ <para>Make sure the following environmental requirements are met before installing Lustre:</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">
+ <emphasis role="italic">
+ <emphasis>(Required)</emphasis>
+ </emphasis>
+ </emphasis><emphasis role="bold"> Disable Security-Enhanced Linux (SELinux) on servers and clients</emphasis> . Lustre does not support SELinux. Therefore, disable the SELinux system extension on all Lustre nodes and make sure other security extensions, like Novell AppArmorand network packet filtering tools (such as iptables) do not interfere with Lustre. See <xref linkend="dbdoclet.50438261_88296"/> in <xref linkend="dbdoclet.50438261_41900"/> below.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">
+ <emphasis role="italic">
+ <emphasis>(Required)</emphasis>
+ </emphasis>
+ </emphasis><emphasis role="bold"> Maintain uniform user and group databases on all cluster nodes</emphasis> . Use the same user IDs (UID) and group IDs (GID) on all clients. If use of supplemental groups is required, verify that the group_upcall requirements have been met. See <xref linkend="dbdoclet.50438291_32926"/>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">
+ <emphasis role="italic">
+ <emphasis>(Recommended)</emphasis>
+ </emphasis>
+ </emphasis><emphasis role="bold"> Provide remote shell access to clients</emphasis> . Although not strictly required to run Lustre, we recommend that all cluster nodes have remote shell client access to facilitate the use of Lustre configuration and monitoring scripts. Parallel Distributed SHell (pdsh) is preferable, although Secure SHell (SSH) is acceptable.</para>
</listitem>
-<listitem>
- <para> Software and environmental requirements in <xref linkend='installinglustre'/></para>
+ <listitem>
+ <para><emphasis role="bold">
+ <emphasis role="italic">
+ <emphasis>(Recommended)</emphasis>
+ </emphasis>
+ </emphasis><emphasis role="bold"> Ensure client clocks are synchronized</emphasis> . Lustre uses client clocks for timestamps. If clocks are out of sync between clients, files will appear with different time stamps when accessed by different clients. Drifting clocks can also cause problems, for example, by making it difficult to debug multi-node issues or correlate logs, which depend on timestamps. We recommend that you use Network Time Protocol (NTP) to keep client and server clocks in sync with each other. For more information about NTP, see: <ulink xl:href="http://www.ntp.org/">http://www.ntp.org</ulink>.</para>
</listitem>
-</itemizedlist>
- </listitem>
- <listitem>
- <para>Download the Lustre RPMs.</para>
- <orderedlist>
- <listitem>
- <para>On the <link xl:href='http://build.whamcloud.com/'>Lustre download site</link>, select your platform.</para>
- <para>The files required to install Lustre (kernels, modules and utilities RPMs) are listed for the selected platform.</para>
- </listitem>
- <listitem>
- <para>Download the required files.</para>
- </listitem>
- </orderedlist>
- </listitem>
- <listitem>
- <para>Install the Lustre packages on the servers.</para>
- <orderedlist>
- <listitem>
- <para>Refer to <xref linkend='installinglustre.tab.req'/> to determine which Lustre packages are to be installed on servers for your platform and architecture.</para>
- <para>Some Lustre packages are installed on the Lustre servers (MGS, MDS, and OSSs). Others are installed on Lustre clients.</para>
- <para>Lustre packages must be installed in the order specified in the following steps.</para>
- </listitem>
- <listitem>
- <para>Install the kernel, modules and ldiskfs packages.</para>
- <para>Use the rpm -ivh command to install the kernel, module and ldiskfs packages.</para>
-
- <note>
- <para>It is not recommended that you use the rpm -Uvh command to install a kernel, because this may leave you with an unbootable system if the new kernel doesn't work for some reason.</para>
- </note>
-
- <para>For example, the command in the following example would install required packages on a server with Infiniband networking</para>
- <screen>$ rpm -ivh kernel-<ver>_lustre-<ver> kernel-ib-<ver> \
+ </itemizedlist>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438261_41900">
+ <title>8.2 Lustre Installation Procedure</title>
+ <caution>
+ <para>Before installing Lustre, back up ALL data. Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured or administered properly.</para>
+ </caution>
+ <para>Use this procedure to install Lustre from RPMs.</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Verify that all Lustre installation requirements have been met.</emphasis></para>
+ <para>For more information on these prerequisites, see:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Hardware requirements in <xref linkend="settinguplustresystem"/></para>
+ </listitem>
+ <listitem>
+ <para> Software and environmental requirements in <xref linkend="installinglustre"/></para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Download the Lustre RPMs.</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">On the <ulink xl:href="http://build.whamcloud.com/">Lustre download site</ulink>, select your platform.</emphasis></para>
+ <para>The files required to install Lustre (kernels, modules and utilities RPMs) are listed for the selected platform.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Download the required files.</emphasis></para>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem xml:id="dbdoclet.50438261_88296">
+ <para><emphasis role="bold">Install the Lustre packages on the servers.</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Refer to <xref linkend="installinglustre.tab.req"/> to determine which Lustre packages are to be installed on servers for your platform and architecture.</emphasis></para>
+ <para>Some Lustre packages are installed on the Lustre servers (MGS, MDS, and OSSs). Others are installed on Lustre clients.</para>
+ <para>Lustre packages must be installed in the order specified in the following steps.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Install the kernel, modules and ldiskfs packages.</emphasis></para>
+ <para>Use the <literal>rpm -ivh</literal> command to install the kernel, module and ldiskfs packages.</para>
+ <note>
+ <para>It is not recommended that you use the rpm -Uvh command to install a kernel, because this may leave you with an unbootable system if the new kernel doesn't work for some reason.</para>
+ </note>
+ <para>For example, the command in the following example would install required packages on a server with Infiniband networking</para>
+ <screen>$ rpm -ivh kernel-<ver>_lustre-<ver> kernel-ib-<ver> \
lustre-modules-<ver> lustre-ldiskfs-<ver>
</screen>
- </listitem>
- <listitem>
- <para>Update the bootloader (grub.conf or lilo.conf) configuration file as needed.</para>
- <orderedlist>
- <listitem>
- <para><emphasis role="bold">Verify that the bootloader configuration file has been updated with an entry for the patched kernel.</emphasis></para>
- <para>Before you can boot to a new distribution or kernel, there must be an entry for it in the bootloader configuration file. Often this is added automatically when the kernel RPM is installed.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below:</emphasis></para>
- <screen>selinux=0
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Update the bootloader (<literal>grub.conf</literal> or <literal>lilo.conf</literal>) configuration file as needed.</emphasis></para>
+ <orderedlist>
+ <listitem>
+
<para><emphasis role="bold">Verify that the bootloader configuration file has been updated with an entry for the patched kernel.</emphasis></para>
+
<para>Before you can boot to a new distribution or kernel, there must be an entry for it in the bootloader configuration file. Often this is added automatically when the kernel RPM is installed.</para>
+ </listitem>
+ <listitem>
+
<para><emphasis role="bold">Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below:</emphasis></para>
+ <screen>selinux=0
</screen>
- </listitem>
- </orderedlist>
- </listitem>
- <listitem>
- <para>Install the utilities/userspace packages.</para>
- <para>Use the rpm -ivh command to install the utilities packages. For example:</para>
- <screen>$ rpm -ivh lustre-<ver>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Install the utilities/userspace packages.</emphasis></para>
+ <para>Use the <literal>rpm -ivh</literal> command to install the utilities packages. For example:</para>
+ <screen>$ rpm -ivh lustre-<ver>
</screen>
- </listitem>
- <listitem>
- <para>Install the e2fsprogs package.</para>
- <para>Use the rpm -ivh command to install the e2fsprogs package. For example:</para>
- <screen>$ rpm -ivh e2fsprogs-<ver>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Install the e2fsprogs package.</emphasis></para>
+ <para>Use the <literal>rpm -ivh</literal> command to install the <literal>e2fsprogs</literal> package. For example:</para>
+ <screen>$ rpm -ivh e2fsprogs-<ver>
</screen>
- <para>If e2fsprogs is already installed on your Linux system, install the Lustre-specific e2fsprogs version by using rpm -Uvh to upgrade the existing e2fsprogs package. For example:</para>
- <screen>$ rpm -Uvh e2fsprogs-<ver> </screen>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis>The rpm command options --force or --nodeps should not be used to install or update the Lustre-specific e2fsprogs package. If errors are reported, file a bug (for instructions see the topic <emphasis><link xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</link></emphasis> on the Lustre wiki).</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </listitem>
- <listitem>
- <para>(Optional) To add optional packages to your Lustre file system, install them now.</para>
- <para>Optional packages include file system creation and repair tools, debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages. A complete list of optional packages for your platform is provided on the <link xl:href="http://wiki.whamcloud.com">Whamcloud wiki</link>.</para>
- </listitem>
- </orderedlist>
- </listitem>
- <listitem>
- <para>Install the Lustre packages on the clients.</para>
- <para>Refer to <xref linkend='installinglustre.tab.req'/> to determine which Lustre packages are to be installed on clients for your platform and architecture.</para>
- <para>Install the module packages for clients.</para>
- <para>Use the rpm -ivh command to install the lustre-client and lustre-client-modules-<ver> packages. For example:</para>
- <screen>$ rpm -ivh lustre-client-modules-<ver> kernel-ib-<ver>
+ <para>If <literal>e2fsprogs</literal> is already installed on your Linux system, install the Lustre-specific <literal>e2fsprogs</literal> version by using <literal>rpm -Uvh</literal> to upgrade the existing <literal>e2fsprogs</literal> package. For example:</para>
+ <screen>$ rpm -Uvh e2fsprogs-<ver> </screen>
+ <informaltable frame="none">
+ <tgroup cols="1">
+ <colspec colname="c1" colwidth="100*"/>
+ <tbody>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Note -</emphasis>The <literal>rpm</literal> command options <literal>--force</literal> or <literal>--nodeps</literal> should not be used to install or update the Lustre-specific <literal>e2fsprogs</literal> package. If errors are reported, file a bug (for instructions see the topic <emphasis>
+ <ulink xl:href="http://wiki.lustre.org/index.php/Reporting_Bugs">Reporting Bugs</ulink>
+ </emphasis> on the Lustre wiki).</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold"><emphasis role="italic">
+ <emphasis role="italic">(Optional)</emphasis>
+ </emphasis> To add optional packages to your Lustre file system, install them now.</emphasis></para>
+ <para>Optional packages include file system creation and repair tools, debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages. A complete list of optional packages for your platform is provided on the <link xl:href="http://wiki.whamcloud.com">Whamcloud wiki</link>.</para>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Install the Lustre packages on the clients.</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Refer to <xref linkend="installinglustre.tab.req"/> to determine which Lustre packages are to be installed on clients for your platform and architecture.</emphasis></para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Install the module packages for clients.</emphasis></para>
+ <para>Use the <literal>rpm -ivh</literal> command to install the lustre-client and <literal>lustre-client-modules-<ver></literal> packages. For example:</para>
+ <screen>$ rpm -ivh lustre-client-modules-<ver> kernel-ib-<ver>
</screen>
- <para>Install the utilities/userspace packages for clients.</para>
- <para>Use the rpm -ivh command to install the utilities packages. For example:</para>
- <screen>$ rpm -ivh lustre-client
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Install the utilities/userspace packages for clients.</emphasis></para>
+ <para>Use the <literal>rpm -ivh</literal> command to install the utilities packages. For example:</para>
+ <screen>$ rpm -ivh lustre-client
</screen>
- <para>Update the bootloader (grub.conf or lilo.conf) configuration file as needed.</para>
- <orderedlist>
- <listitem>
- <para><emphasis role="bold">Verify that the bootloader configuration file has been updated with an entry for the patched kernel.</emphasis></para>
- <para>Before you can boot to a new distribution or kernel, there must be an entry for it in the bootloader configuration file. Often this is added automatically when the kernel RPM is installed.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below:</emphasis></para>
- <screen>selinux=0
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Update the bootloader (grub.conf or lilo.conf) configuration file as needed.</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Verify that the bootloader configuration file has been updated with an entry for the patched kernel.</emphasis></para>
+ <para>Before you can boot to a new distribution or kernel, there must be an entry for it in the bootloader configuration file. Often this is added automatically when the kernel RPM is installed.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below:</emphasis></para>
+ <screen>selinux=0
</screen>
- </listitem>
- </orderedlist>
- </listitem>
- <listitem>
- <para>Reboot the patched clients and the servers.</para>
- <orderedlist>
- <listitem>
- <para>If you applied the patched kernel to any clients, reboot them.</para>
- <para>Unpatched clients do not need to be rebooted.</para>
- </listitem>
- <listitem>
- <para>Reboot the servers.</para>
- </listitem>
- </orderedlist>
- <para>To configure LNET, go next to <xref linkend='configuringlnet'/>. If default settings will be used for LNET, go to <xref linkend='configuringlustre'/>.</para>
- </listitem>
- </orderedlist>
- </section>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Reboot the patched clients and the servers.</emphasis></para>
+ <orderedlist>
+ <listitem>
+ <para>If you applied the patched kernel to any clients, reboot them.</para>
+ <para>Unpatched clients do not need to be rebooted.</para>
+ </listitem>
+ <listitem>
+ <para>Reboot the servers.</para>
+ </listitem>
+ </orderedlist>
+ <para>To configure LNET, go next to <xref linkend="configuringlnet"/>. If default settings will be used for LNET, go to <xref linkend="configuringlustre"/>.</para>
+ </listitem>
+ </orderedlist>
+ </section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='lustremonitoring'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. -->
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremonitoring">
<info>
- <title xml:id='lustremonitoring.title'>Lustre Monitoring</title>
+ <title xml:id="lustremonitoring.title">Lustre Monitoring</title>
</info>
<para>This chapter provides information on monitoring Lustre and includes the following sections:</para>
-
- <itemizedlist><listitem>
- <para><xref linkend='dbdoclet.50438273_18711'/>Lustre Changelogs</para>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438273_18711"/>Lustre Changelogs</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438273_81684"/>Lustre Monitoring Tool</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438273_80593"/>CollectL</para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438273_44185"/>Other Monitoring Options</para>
+ </listitem>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438273_18711">
+ <title>12.1 Lustre <anchor xml:id="dbdoclet.50438273_marker-1297383" xreflabel=""/>Changelogs</title>
+ <para>The changelogs feature records events that change the file system namespace or file metadata. Changes such as file creation, deletion, renaming, attribute changes, etc. are recorded with the target and parent file identifiers (FIDs), the name of the target, and a timestamp. These records can be used for a variety of purposes:</para>
+ <itemizedlist>
+ <listitem>
+ <para> Capture recent changes to feed into an archiving system.</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438273_81684"/>Lustre Monitoring Tool</para>
+ <para> Use changelog entries to exactly replicate changes in a file system mirror.</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438273_80593"/>CollectL</para>
+ <para> Set up "watch scripts" that take action on certain events or directories.</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438273_44185"/>Other Monitoring Options</para>
- </listitem>
- </itemizedlist>
-
- <section xml:id="dbdoclet.50438273_18711">
- <title>12.1 Lustre <anchor xml:id="dbdoclet.50438273_marker-1297383" xreflabel=""/>Changelogs</title>
- <para>The changelogs feature records events that change the file system namespace or file metadata. Changes such as file creation, deletion, renaming, attribute changes, etc. are recorded with the target and parent file identifiers (FIDs), the name of the target, and a timestamp. These records can be used for a variety of purposes:</para>
- <itemizedlist><listitem>
- <para> Capture recent changes to feed into an archiving system.</para>
- </listitem>
-<listitem>
- <para> Use changelog entries to exactly replicate changes in a file system mirror.</para>
- </listitem>
-<listitem>
- <para> Set up "watch scripts" that take action on certain events or directories.</para>
- </listitem>
-<listitem>
- <para> Maintain a rough audit trail (file/directory changes with timestamps, but no user information).</para>
- </listitem>
-</itemizedlist>
- <para>Changelogs record types are:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Value</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> MARK</para></entry>
- <entry><para> Internal recordkeeping</para></entry>
- </row>
- <row>
- <entry><para> CREAT</para></entry>
- <entry><para> Regular file creation</para></entry>
- </row>
- <row>
- <entry><para> MKDIR</para></entry>
- <entry><para> Directory creation</para></entry>
- </row>
- <row>
- <entry><para> HLINK</para></entry>
- <entry><para> Hard link</para></entry>
- </row>
- <row>
- <entry><para> SLINK</para></entry>
- <entry><para> Soft link</para></entry>
- </row>
- <row>
- <entry><para> MKNOD</para></entry>
- <entry><para> Other file creation</para></entry>
- </row>
- <row>
- <entry><para> UNLNK</para></entry>
- <entry><para> Regular file removal</para></entry>
- </row>
- <row>
- <entry><para> RMDIR</para></entry>
- <entry><para> Directory removal</para></entry>
- </row>
- <row>
- <entry><para> RNMFM</para></entry>
- <entry><para> Rename, original</para></entry>
- </row>
- <row>
- <entry><para> RNMTO</para></entry>
- <entry><para> Rename, final</para></entry>
- </row>
- <row>
- <entry><para> IOCTL</para></entry>
- <entry><para> ioctl on file or directory</para></entry>
- </row>
- <row>
- <entry><para> TRUNC</para></entry>
- <entry><para> Regular file truncated</para></entry>
- </row>
- <row>
- <entry><para> SATTR</para></entry>
- <entry><para> Attribute change</para></entry>
- </row>
- <row>
- <entry><para> XATTR</para></entry>
- <entry><para> Extended attribute change</para></entry>
- </row>
- <row>
- <entry><para> UNKNW</para></entry>
- <entry><para> Unknown operation</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para>FID-to-full-pathname and pathname-to-FID functions are also included to map target and parent FIDs into the file system namespace.</para>
- <section remap="h3">
- <title>12.1.1 Working with Changelogs</title>
- <para>Several commands are available to work with changelogs.</para>
- <section remap="h5">
- <title>lctl changelog_register</title>
- <para>Because changelog records take up space on the MDT, the system administration must register changelog users. The registrants specify which records they are "done with", and the system purges up to the greatest common record.</para>
- <para>To register a new changelog user, run:</para>
- <screen>lctl --device <mdt_device> changelog_register
+ <para> Maintain a rough audit trail (file/directory changes with timestamps, but no user information).</para>
+ </listitem>
+ </itemizedlist>
+ <para>Changelogs record types are:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Value</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> MARK</para>
+ </entry>
+ <entry>
+ <para> Internal recordkeeping</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> CREAT</para>
+ </entry>
+ <entry>
+ <para> Regular file creation</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> MKDIR</para>
+ </entry>
+ <entry>
+ <para> Directory creation</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> HLINK</para>
+ </entry>
+ <entry>
+ <para> Hard link</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> SLINK</para>
+ </entry>
+ <entry>
+ <para> Soft link</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> MKNOD</para>
+ </entry>
+ <entry>
+ <para> Other file creation</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> UNLNK</para>
+ </entry>
+ <entry>
+ <para> Regular file removal</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> RMDIR</para>
+ </entry>
+ <entry>
+ <para> Directory removal</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> RNMFM</para>
+ </entry>
+ <entry>
+ <para> Rename, original</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> RNMTO</para>
+ </entry>
+ <entry>
+ <para> Rename, final</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> IOCTL</para>
+ </entry>
+ <entry>
+ <para> ioctl on file or directory</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> TRUNC</para>
+ </entry>
+ <entry>
+ <para> Regular file truncated</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> SATTR</para>
+ </entry>
+ <entry>
+ <para> Attribute change</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> XATTR</para>
+ </entry>
+ <entry>
+ <para> Extended attribute change</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> UNKNW</para>
+ </entry>
+ <entry>
+ <para> Unknown operation</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para>FID-to-full-pathname and pathname-to-FID functions are also included to map target and parent FIDs into the file system namespace.</para>
+ <section remap="h3">
+ <title>12.1.1 Working with Changelogs</title>
+ <para>Several commands are available to work with changelogs.</para>
+ <section remap="h5">
+ <title>lctl changelog_register</title>
+ <para>Because changelog records take up space on the MDT, the system administration must register changelog users. The registrants specify which records they are "done with", and the system purges up to the greatest common record.</para>
+ <para>To register a new changelog user, run:</para>
+ <screen>lctl --device <mdt_device> changelog_register
</screen>
- <para>Changelog entries are not purged beyond a registered user's set point (see lfs changelog_clear).</para>
- </section>
- <section remap="h5">
- <title>lfs changelog</title>
-
<para>To display the metadata changes on an MDT (the changelog records), run:</para>
- <screen>lfs changelog <MDT name> [startrec [endrec]]
+ <para>Changelog entries are not purged beyond a registered user's set point (see <literal>lfs changelog_clear</literal>).</para>
+ </section>
+ <section remap="h5">
+ <title>lfs changelog</title>
+ <para>To display the metadata changes on an MDT (the changelog records), run:</para>
+ <screen>lfs changelog <MDT name> [startrec [endrec]]
</screen>
-
<para>It is optional whether to specify the start and end records.</para>
-
<para>These are sample changelog records:</para>
- <screen>2 02MKDIR 4298396676 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0]\
+ <para>It is optional whether to specify the start and end records.</para>
+ <para>These are sample changelog records:</para>
+ <screen>2 02MKDIR 4298396676 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0]\
pics
3 01CREAT 4298402264 0x0 t=[0x200000405:0x15fa:0x0] p=[0x200000405:0x15f9:0\
x0] chloe.jpg
5 07RMDIR 4298405394 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0]\
pics
</screen>
- </section>
- <section remap="h5">
- <title>lfs changelog_clear</title>
- <para>To clear old changelog records for a specific user (records that the user no longer needs), run:</para>
- <screen>lfs changelog_clear <MDT name> <user ID> <endrec>
-</screen>
- <para>The changelog_clear command indicates that changelog records previous to <endrec> are no longer of interest to a particular user <user ID>, potentially allowing the MDT to free up disk space. An <endrec> value of 0 indicates the current last record. To run changelog_clear, the changelog user must be registered on the MDT node using lctl.</para>
- <para>When all changelog users are done with records < X, the records are deleted.</para>
- </section>
- <section remap="h5">
- <title>lctl changelog_deregister</title>
- <para>To deregister (unregister) a changelog user, run:</para>
- <screen>lctl --device <mdt_device> changelog_deregister <user ID>
+ </section>
+ <section remap="h5">
+ <title>lfs changelog_clear</title>
+ <para>To clear old changelog records for a specific user (records that the user no longer needs), run:</para>
+ <screen>lfs changelog_clear <MDT name> <user ID> <endrec>
</screen>
- <para> Changelog_deregister cl1 effectively does a changelog_clear cl10 as it deregisters.</para>
- </section>
+ <para>The <literal>changelog_clear</literal> command indicates that changelog records previous to <endrec> are no longer of interest to a particular user <user ID>, potentially allowing the MDT to free up disk space. An <literal><endrec></literal> value of 0 indicates the current last record. To run <literal>changelog_clear</literal>, the changelog user must be registered on the MDT node using <literal>lctl</literal>.</para>
+ <para>When all changelog users are done with records < X, the records are deleted.</para>
+ </section>
+ <section remap="h5">
+ <title>lctl changelog_deregister</title>
+ <para>To deregister (unregister) a changelog user, run:</para>
+ <screen>lctl --device <mdt_device> changelog_deregister <user ID> </screen>
+ <para> <literal>changelog_deregister cl1</literal> effectively does a <literal>changelog_clear cl1 0</literal> as it deregisters.</para>
</section>
- <section remap="h3">
- <title>12.1.2 Changelog Examples</title>
- <para>This section provides examples of different changelog commands.</para>
- <section remap="h5">
- <title>Registering a Changelog User</title>
- <para>To register a new changelog user for a device (lustre-MDT0000):</para>
- <screen># lctl --device lustre-MDT0000 changelog_register
+ </section>
+ <section remap="h3">
+ <title>12.1.2 Changelog Examples</title>
+ <para>This section provides examples of different changelog commands.</para>
+ <section remap="h5">
+ <title>Registering a Changelog User</title>
+ <para>To register a new changelog user for a device (<literal>lustre-MDT0000</literal>):</para>
+ <screen># lctl --device lustre-MDT0000 changelog_register
lustre-MDT0000: Registered changelog userid 'cl1'
</screen>
- </section>
- <section remap="h5">
- <title>Displaying Changelog Records</title>
- <para>To display changelog records on an MDT (lustre-MDT0000):</para>
- <screen>$ lfs changelog lustre-MDT0000
+ </section>
+ <section remap="h5">
+ <title>Displaying Changelog Records</title>
+ <para>To display changelog records on an MDT (<literal>lustre-MDT0000</literal>):</para>
+ <screen>$ lfs changelog lustre-MDT0000
1 00MARK 19:08:20.890432813 2010.03.24 0x0 t=[0x10001:0x0:0x0] p=[0:0x0:0x\
0] mdd_obd-lustre-MDT0000-0
2 02MKDIR 19:10:21.509659173 2010.03.24 0x0 t=[0x200000420:0x3:0x0] p=[0x61\
4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
0000420:0x3:0x0] hosts
</screen>
-
<para>Changelog records include this information:</para>
- <screen>rec#
+ <para>Changelog records include this information:</para>
+ <screen>rec#
operation_type(numerical/text)
timestamp
datestamp
p=parent_FID
target_name
</screen>
-
<para>Displayed in this format:</para>
- <screen>rec# operation_type(numerical/text) timestamp datestamp flags t=target_FID \
+ <para>Displayed in this format:</para>
+ <screen>rec# operation_type(numerical/text) timestamp datestamp flags t=target_FID \
p=parent_FID target_name
</screen>
-
<para>For example:</para>
- <screen>4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
+ <para>For example:</para>
+ <screen>4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
0000420:0x3:0x0] hosts
</screen>
- </section>
- <section remap="h5">
- <title>Clearing Changelog Records</title>
- <para>To notify a device that a specific user (cl1) no longer needs records (up to and including 3):</para>
- <screen>$ lfs changelog_clear lustre-MDT0000 cl1 3
+ </section>
+ <section remap="h5">
+ <title>Clearing Changelog Records</title>
+ <para>To notify a device that a specific user (<literal>cl1</literal>) no longer needs records (up to and including 3):</para>
+ <screen>$ lfs changelog_clear lustre-MDT0000 cl1 3
</screen>
- <para>To confirm that the changelog_clear operation was successful, run lfs changelog; only records after id-3 are listed:</para>
- <screen>$ lfs changelog lustre-MDT0000
+ <para>To confirm that the <literal>changelog_clear</literal> operation was successful, run <literal>lfs changelog</literal>; only records after id-3 are listed:</para>
+ <screen>$ lfs changelog lustre-MDT0000
4 01CREAT 19:10:37.113847713 2010.03.24 0x0 t=[0x200000420:0x4:0x0] p=[0x20\
0000420:0x3:0x0] hosts
</screen>
- </section>
- <section remap="h5">
- <title>Deregistering a Changelog User</title>
- <para>To deregister a changelog user (cl1) for a specific device (lustre-MDT0000):</para>
- <screen># lctl --device lustre-MDT0000 changelog_deregister cl1
+ </section>
+ <section remap="h5">
+ <title>Deregistering a Changelog User</title>
+ <para>To deregister a changelog user (<literal>cl1</literal>) for a specific device (<literal>lustre-MDT0000</literal>):</para>
+ <screen># lctl --device lustre-MDT0000 changelog_deregister cl1
lustre-MDT0000: Deregistered changelog user 'cl1'
</screen>
- <para>The deregistration operation clears all changelog records for the specified user (cli).</para>
- <screen>$ lfs changelog lustre-MDT0000
+ <para>The deregistration operation clears all changelog records for the specified user (<literal>cl1</literal>).</para>
+ <screen>$ lfs changelog lustre-MDT0000
5 00MARK 19:13:40.858292517 2010.03.24 0x0 t=[0x40001:0x0:0x0] p=[0:0x0:0x\
0] mdd_obd-lustre-MDT0000-0
</screen>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis>MARK records typically indicate changelog recording status changes.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Displaying the Changelog Index and Registered Users</title>
- <para>To display the current, maximum changelog index and registered changelog users for a specific device (lustre-MDT0000):</para>
- <screen># lctl get_param mdd.lustre-MDT0000.changelog_users
+ <informaltable frame="none">
+ <tgroup cols="1">
+ <colspec colname="c1" colwidth="100*"/>
+ <tbody>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Note -</emphasis>MARK records typically indicate changelog recording status changes.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Displaying the Changelog Index and Registered Users</title>
+ <para>To display the current, maximum changelog index and registered changelog users for a specific device (<literal>lustre-MDT0000</literal>):</para>
+ <screen># lctl get_param mdd.lustre-MDT0000.changelog_users
mdd.lustre-MDT0000.changelog_users=current index: 8
ID index
cl2 8
</screen>
- </section>
- <section remap="h5">
- <title>Displaying the Changelog Mask</title>
- <para>To show the current changelog mask on a specific device (lustre-MDT0000):</para>
- <screen># lctl get_param mdd.lustre-MDT0000.changelog_mask
+ </section>
+ <section remap="h5">
+ <title>Displaying the Changelog Mask</title>
+ <para>To show the current changelog mask on a specific device (<literal>lustre-MDT0000</literal>):</para>
+ <screen># lctl get_param mdd.lustre-MDT0000.changelog_mask
+
mdd.lustre-MDT0000.changelog_mask=
MARK CREAT MKDIR HLINK SLINK MKNOD UNLNK RMDIR RNMFM RNMTO OPEN CLOSE IOCTL\
TRUNC SATTR XATTR HSM
</screen>
- </section>
- <section remap="h5">
- <title>Setting the Changelog Mask</title>
- <para>To set the current changelog mask on a specific device (lustre-MDT0000):</para>
- <screen># lctl set_param mdd.lustre-MDT0000.changelog_mask=HLINK
+ </section>
+ <section remap="h5">
+ <title>Setting the Changelog Mask</title>
+ <para>To set the current changelog mask on a specific device (<literal>lustre-MDT0000</literal>):</para>
+ <screen># lctl set_param mdd.lustre-MDT0000.changelog_mask=HLINK
mdd.lustre-MDT0000.changelog_mask=HLINK
$ lfs changelog_clear lustre-MDT0000 cl1 0
$ mkdir /mnt/lustre/mydir/foo
$ cp /etc/hosts /mnt/lustre/mydir/foo/file
$ ln /mnt/lustre/mydir/foo/file /mnt/lustre/mydir/myhardlink
</screen>
-
<para> Only item types that are in the mask show up in the changelog.</para>
- <screen>$ lfs changelog lustre-MDT0000
+ <para> Only item types that are in the mask show up in the changelog.</para>
+ <screen>$ lfs changelog lustre-MDT0000
9 03HLINK 19:19:35.171867477 2010.03.24 0x0 t=[0x200000420:0x6:0x0] p=[0x20\
0000420:0x3:0x0] myhardlink
</screen>
- </section>
</section>
</section>
- <section xml:id="dbdoclet.50438273_81684">
- <title>12.2 Lustre <anchor xml:id="dbdoclet.50438273_marker-1297386" xreflabel=""/>Monitoring Tool</title>
- <para>The Lustre Monitoring Tool (LMT) is a Python-based, distributed system developed and maintained by Lawrence Livermore National Lab (LLNL)). It provides a ''top'' like display of activity on server-side nodes (MDS, OSS and portals routers) on one or more Lustre file systems. It does not provide support for monitoring clients. For more information on LMT, including the setup procedure, see:</para>
- <para><link xl:href="http://code.google.com/p/lmt/">http://code.google.com/p/lmt/</link></para>
- <para>LMT questions can be directed to:</para>
- <para><link xl:href="mailto:lmt-discuss@googlegroups.com">lmt-discuss@googlegroups.com</link></para>
- </section>
- <section xml:id="dbdoclet.50438273_80593">
- <title>12.3 Collect<anchor xml:id="dbdoclet.50438273_marker-1297391" xreflabel=""/>L</title>
- <para>CollectL is another tool that can be used to monitor Lustre. You can run CollectL on a Lustre system that has any combination of MDSs, OSTs and clients. The collected data can be written to a file for continuous logging and played back at a later time. It can also be converted to a format suitable for plotting.</para>
- <para>For more information about CollectL, see:</para>
- <para><link xl:href="http://collectl.sourceforge.net">http://collectl.sourceforge.net</link></para>
- <para>Lustre-specific documentation is also available. See:</para>
- <para><link xl:href="http://collectl.sourceforge.net/Tutorial-Lustre.html">http://collectl.sourceforge.net/Tutorial-Lustre.html</link></para>
- </section>
- <section xml:id="dbdoclet.50438273_44185">
- <title>12.4 Other Monitoring Options</title>
- <para>A variety of standard tools are available publically.</para>
- <para>Another option is to script a simple monitoring solution that looks at various reports from ipconfig, as well as the procfs files generated by Lustre.</para>
- <para><anchor xml:id="dbdoclet.50438273_67514" xreflabel=""/> </para>
- </section>
+ </section>
+ <section xml:id="dbdoclet.50438273_81684">
+ <title>12.2 Lustre <anchor xml:id="dbdoclet.50438273_marker-1297386" xreflabel=""/>Monitoring Tool</title>
+ <para>The Lustre Monitoring Tool (LMT) is a Python-based, distributed system developed and maintained by Lawrence Livermore National Lab (LLNL)). It provides a ''top'' like display of activity on server-side nodes (MDS, OSS and portals routers) on one or more Lustre file systems. It does not provide support for monitoring clients. For more information on LMT, including the setup procedure, see:</para>
+ <para><ulink xl:href="http://code.google.com/p/lmt/">http://code.google.com/p/lmt/</ulink></para>
+ <para>LMT questions can be directed to:</para>
+ <para><ulink xl:href="mailto:lmt-discuss@googlegroups.com">lmt-discuss@googlegroups.com</ulink></para>
+ </section>
+ <section xml:id="dbdoclet.50438273_80593">
+ <title>12.3 Collect<anchor xml:id="dbdoclet.50438273_marker-1297391" xreflabel=""/>L</title>
+ <para>CollectL is another tool that can be used to monitor Lustre. You can run CollectL on a Lustre system that has any combination of MDSs, OSTs and clients. The collected data can be written to a file for continuous logging and played back at a later time. It can also be converted to a format suitable for plotting.</para>
+ <para>For more information about CollectL, see:</para>
+ <para><ulink xl:href="http://collectl.sourceforge.net">http://collectl.sourceforge.net</ulink></para>
+ <para>Lustre-specific documentation is also available. See:</para>
+ <para><ulink xl:href="http://collectl.sourceforge.net/Tutorial-Lustre.html">http://collectl.sourceforge.net/Tutorial-Lustre.html</ulink></para>
+ </section>
+ <section xml:id="dbdoclet.50438273_44185">
+ <title>12.4 Other Monitoring Options</title>
+ <para>A variety of standard tools are available publically.</para>
+ <para>Another option is to script a simple monitoring solution that looks at various reports from ipconfig, as well as the procfs files generated by Lustre.</para>
+ <para><anchor xml:id="dbdoclet.50438273_67514" xreflabel=""/> </para>
+ </section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='settingupbonding'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. -->
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settingupbonding">
<info>
- <title xml:id='settingupbonding.title'>Setting Up Network Interface Bonding</title>
+ <title xml:id="settingupbonding.title">Setting Up Network Interface Bonding</title>
</info>
-
<para>This chapter describes how to use multiple network interfaces in parallel to increase bandwidth and/or redundancy. Topics include:</para>
-
<itemizedlist>
- <listitem>
- <xref linkend='dbdoclet.50438258_95824'/>
+ <listitem>
+ <para>
+ <xref linkend="dbdoclet.50438258_95824"/>
</para>
- </listitem>
- <listitem>
- <xref linkend='dbdoclet.50438258_29107'/>
+ </listitem>
+ <listitem>
+ <para>
+ <xref linkend="dbdoclet.50438258_29107"/>
</para>
- </listitem>
- <listitem>
- <xref linkend='dbdoclet.50438258_25353'/>
+ </listitem>
+ <listitem>
+ <para>
+ <xref linkend="dbdoclet.50438258_25353"/>
</para>
- </listitem>
- <listitem>
- <xref linkend='dbdoclet.50438258_26899'/>
+ </listitem>
+ <listitem>
+ <para>
+ <xref linkend="dbdoclet.50438258_26899"/>
</para>
- </listitem>
- <listitem>
- <xref linkend='dbdoclet.50438258_54769'/>
+ </listitem>
+ <listitem>
+ <para>
+ <xref linkend="dbdoclet.50438258_54769"/>
</para>
- </listitem>
- <listitem>
- <xref linkend='dbdoclet.50438258_92244'/>
+ </listitem>
+ <listitem>
+ <para>
+ <xref linkend="dbdoclet.50438258_92244"/>
</para>
- </listitem>
+ </listitem>
</itemizedlist>
-
- <note><para>Using network interface bonding is optional.</para></note>
-
-
- <section xml:id="dbdoclet.50438258_95824">
- <title>7.1 Network Interface Bonding Overview</title>
- <para>Bonding, also known as link aggregation, trunking and port trunking, is a method of aggregating multiple physical network links into a single logical link for increased bandwidth.</para>
- <para>Several different types of bonding are available in Linux. All these types are referred to as 'modes', and use the bonding kernel module.</para>
- <para>Modes 0 to 3 allow load balancing and fault tolerance by using multiple interfaces. Mode 4 aggregates a group of interfaces into a single virtual interface where all members of the group share the same speed and duplex settings. This mode is described under IEEE spec 802.3ad, and it is referred to as either 'mode 4' or '802.3ad.'</para>
- </section>
- <section xml:id="dbdoclet.50438258_29107">
- <title>7.2 Requirements</title>
- <para>The most basic requirement for successful bonding is that both endpoints of the connection must be capable of bonding. In a normal case, the non-server endpoint is a switch. (Two systems connected via crossover cables can also use bonding.) Any switch used must explicitly handle 802.3ad Dynamic Link Aggregation.</para>
- <para>The kernel must also be configured with bonding. All supported Lustre kernels have bonding functionality. The network driver for the interfaces to be bonded must have the ethtool functionality to determine slave speed and duplex settings. All recent network drivers implement it.</para>
- <para>To verify that your interface works with ethtool, run:</para>
- <screen># which ethtool
+ <note>
+ <para>Using network interface bonding is optional.</para>
+ </note>
+ <section xml:id="dbdoclet.50438258_95824">
+ <title>7.1 Network Interface Bonding Overview</title>
+ <para>Bonding, also known as link aggregation, trunking and port trunking, is a method of aggregating multiple physical network links into a single logical link for increased bandwidth.</para>
+ <para>Several different types of bonding are available in Linux. All these types are referred to as 'modes', and use the bonding kernel module.</para>
+ <para>Modes 0 to 3 allow load balancing and fault tolerance by using multiple interfaces. Mode 4 aggregates a group of interfaces into a single virtual interface where all members of the group share the same speed and duplex settings. This mode is described under IEEE spec 802.3ad, and it is referred to as either 'mode 4' or '802.3ad.'</para>
+ </section>
+ <section xml:id="dbdoclet.50438258_29107">
+ <title>7.2 Requirements</title>
+ <para>The most basic requirement for successful bonding is that both endpoints of the connection must be capable of bonding. In a normal case, the non-server endpoint is a switch. (Two systems connected via crossover cables can also use bonding.) Any switch used must explicitly handle 802.3ad Dynamic Link Aggregation.</para>
+ <para>The kernel must also be configured with bonding. All supported Lustre kernels have bonding functionality. The network driver for the interfaces to be bonded must have the ethtool functionality to determine slave speed and duplex settings. All recent network drivers implement it.</para>
+ <para>To verify that your interface works with ethtool, run:</para>
+ <screen># which ethtool
/sbin/ethtool
# ethtool eth0
# which ifenslave
/sbin/ifenslave
</screen>
- </section>
- <section xml:id="dbdoclet.50438258_25353">
- <title>7.3 Bonding Module Parameters</title>
-
<para>Bonding module parameters control various aspects of bonding.</para>
-
<para>Outgoing traffic is mapped across the slave interfaces according to the transmit hash policy. For Lustre, we recommend that you set the xmit_hash_policy option to the layer3+4 option for bonding. This policy uses upper layer protocol information if available to generate the hash. This allows traffic to a particular network peer to span multiple slaves, although a single connection does not span multiple slaves.</para>
- <screen>$ xmit_hash_policy=layer3+4
+ </section>
+ <section xml:id="dbdoclet.50438258_25353">
+ <title>7.3 Bonding Module Parameters</title>
+ <para>Bonding module parameters control various aspects of bonding.</para>
+ <para>Outgoing traffic is mapped across the slave interfaces according to the transmit hash policy. For Lustre, we recommend that you set the xmit_hash_policy option to the layer3+4 option for bonding. This policy uses upper layer protocol information if available to generate the hash. This allows traffic to a particular network peer to span multiple slaves, although a single connection does not span multiple slaves.</para>
+ <screen>$ xmit_hash_policy=layer3+4
</screen>
-
<para>The miimon option enables users to monitor the link status. (The parameter is a time interval in milliseconds.) It makes an interface failure transparent to avoid serious network degradation during link failures. A reasonable default setting is 100 milliseconds; run:</para>
- <screen>$ miimon=100
+ <para>The miimon option enables users to monitor the link status. (The parameter is a time interval in milliseconds.) It makes an interface failure transparent to avoid serious network degradation during link failures. A reasonable default setting is 100 milliseconds; run:</para>
+ <screen>$ miimon=100
</screen>
- <para>For a busy network, increase the timeout.</para>
- </section>
- <section xml:id="dbdoclet.50438258_26899">
- <title>7.4 Setting Up Bonding</title>
- <para>To set up bonding:</para>
-
- <orderedlist>
- <listitem>
- <para>Create a virtual 'bond' interface by creating a configuration file in:</para>
- <screen>/etc/sysconfig/network-scripts/ # vi /etc/sysconfig/ network-scripts/ifcfg-\
+ <para>For a busy network, increase the timeout.</para>
+ </section>
+ <section xml:id="dbdoclet.50438258_26899">
+ <title>7.4 Setting Up Bonding</title>
+ <para>To set up bonding:</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">Create a virtual 'bond' interface by creating a configuration file in:</emphasis></para>
+ <screen>/etc/sysconfig/network-scripts/ # vi /etc/sysconfig/ network-scripts/ifcfg-\
bond0
</screen>
- </listitem>
- <listitem>
- <para>Append the following lines to the file.</para>
- <screen>DEVICE=bond0
+ </listitem>
+ <listitem>
+ <emphasis role="bold">
+ <para>Append the following lines to the file.</para>
+ </emphasis>
+ <screen>DEVICE=bond0
IPADDR=192.168.10.79 # Use the free IP Address of your network
NETWORK=192.168.10.0
NETMASK=255.255.255.0
BOOTPROTO=none
ONBOOT=yes
</screen>
- </listitem>
- <listitem>
- <para> Attach one or more slave interfaces to the bond interface. Modify the eth0 and eth1 configuration files (using a VI text editor).</para>
- <orderedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Attach one or more slave interfaces to the bond interface. Modify the eth0 and eth1 configuration files (using a VI text editor).</emphasis></para>
+ <orderedlist>
<listitem>
- <para>Use the VI text editor to open the eth0 configuration file.</para>
- <screen># vi /etc/sysconfig/network-scripts/ifcfg-eth0
+ <para><emphasis role="bold">Use the VI text editor to open the eth0 configuration file.</emphasis></para>
+ <screen># vi /etc/sysconfig/network-scripts/ifcfg-eth0
</screen>
</listitem>
<listitem>
- <para>Modify/append the eth0 file as follows:</para>
- <screen>DEVICE=eth0
+ <para><emphasis role="bold">Modify/append the eth0 file as follows:</emphasis></para>
+ <screen>DEVICE=eth0
USERCTL=no
ONBOOT=yes
MASTER=bond0
</screen>
</listitem>
<listitem>
- <para>Use the VI text editor to open the eth1 configuration file.</para>
- <screen># vi /etc/sysconfig/network-scripts/ifcfg-eth1
+ <para><emphasis role="bold">Use the VI text editor to open the eth1 configuration file.</emphasis></para>
+ <screen># vi /etc/sysconfig/network-scripts/ifcfg-eth1
</screen>
</listitem>
<listitem>
- <para>Modify/append the eth1 file as follows:</para>
- <screen>DEVICE=eth1
+ <para><emphasis role="bold">Modify/append the eth1 file as follows:</emphasis></para>
+ <screen>DEVICE=eth1
USERCTL=no
ONBOOT=yes
MASTER=bond0
BOOTPROTO=none
</screen>
</listitem>
- </orderedlist>
- </listitem>
- <listitem>
- <para>Set up the bond interface and its options in /etc/modprobe.conf. Start the slave interfaces by your normal network method.</para>
- <screen># vi /etc/modprobe.conf
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Set up the bond interface and its options in /etc/modprobe.conf. Start the slave interfaces by your normal network method.</emphasis></para>
+ <screen># vi /etc/modprobe.conf
</screen>
- <orderedlist>
+ <orderedlist>
<listitem>
- <para>Append the following lines to the file.</para>
- <screen>alias bond0 bonding
+ <para><emphasis role="bold">Append the following lines to the file.</emphasis></para>
+ <screen>alias bond0 bonding
options bond0 mode=balance-alb miimon=100
</screen>
</listitem>
<listitem>
- <para>Load the bonding module.</para>
- <screen># modprobe bonding
+ <para><emphasis role="bold">Load the bonding module.</emphasis></para>
+ <screen># modprobe bonding
# ifconfig bond0 up
# ifenslave bond0 eth0 eth1
</screen>
</listitem>
- </orderedlist>
- </listitem>
- <listitem>
- <para>Start/restart the slave interfaces (using your normal network method).</para>
-
- <note>
- <para>You must modprobe the bonding module for each bonded interface. If you wish to create bond0 and bond1, two entries in modprobe.conf are required.</para></note>
-
- <para>The examples below are from RedHat systems. For setup use: /etc/sysconfig/networking-scripts/ifcfg-* The website referenced below includes detailed instructions for other configuration methods, instructions to use DHCP with bonding, and other setup details. We strongly recommend you use this website.</para>
- <para><link xl:href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding">http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding</link></para>
- </listitem>
- <listitem>
- <para>Check /proc/net/bonding to determine status on bonding. There should be a file there for each bond interface.</para>
- <screen># cat /proc/net/bonding/bond0
+ </orderedlist>
+ </listitem>
+ <listitem>
+ <emphasis role="bold">
+ <para>Start/restart the slave interfaces (using your normal network method).</para>
+ </emphasis>
+ <note>
+ <para>You must modprobe the bonding module for each bonded interface. If you wish to create bond0 and bond1, two entries in modprobe.conf are required.</para>
+ </note>
+ <para>The examples below are from RedHat systems. For setup use: <literal>/etc/sysconfig/networking-scripts/ifcfg-*</literal> The website referenced below includes detailed instructions for other configuration methods, instructions to use DHCP with bonding, and other setup details. We strongly recommend you use this website.</para>
+ <para><link xl:href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding">http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding</link></para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Check /proc/net/bonding to determine status on bonding. There should be a file there for each bond interface.</emphasis></para>
+ <screen># cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.0.3 (March 23, 2006)
Bonding Mode: load balancing (round-robin)
Link Failure Count: 0
Permanent HW addr: 00:14:2a:7c:40:1d
</screen>
- </listitem>
- <listitem>
- <para>Use ethtool or ifconfig to check the interface state. ifconfig lists the first bonded interface as 'bond0.'</para>
- <screen>ifconfig
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Use ethtool or ifconfig to check the interface state. ifconfig lists the first bonded interface as 'bond0.'</emphasis></para>
+ <screen>ifconfig
bond0 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0
inet addr:192.168.10.79 Bcast:192.168.10.255 \ Mask:255.255.255.0
inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link
RX bytes:152299 (148.7 KiB) TX bytes:64517 (63.0 KiB)
Interrupt:185 Base address:0x6000
</screen>
- </listitem>
- </orderedlist>
-
-
- <section remap="h3">
- <title>7.4.1 Examples</title>
- <para>This is an example showing modprobe.conf entries for bonding Ethernet interfaces eth1 and eth2 to bond0:</para>
- <screen># cat /etc/modprobe.conf
+ </listitem>
+ </orderedlist>
+ <section remap="h3">
+ <title>7.4.1 Examples</title>
+ <para>This is an example showing <literal>modprobe.conf</literal> entries for bonding Ethernet interfaces <literal>eth1</literal> and <literal>eth2</literal> to <literal>bond0</literal>:</para>
+ <screen># cat /etc/modprobe.conf
alias eth0 8139too
alias scsi_hostadapter sata_via
alias scsi_hostadapter1 usb-storage
MASTER=bond0
SLAVE=yes
</screen>
- <para>In the following example, the bond0 interface is the master (MASTER) while eth0 and eth1 are slaves (SLAVE).</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis>All slaves of bond0 have the same MAC address (Hwaddr) - bond0. All modes, except TLB and ALB, have this MAC address. TLB and ALB require a unique MAC address for each slave.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <screen>$ /sbin/ifconfig
+ <para>In the following example, the <literal>bond0</literal> interface is the master (MASTER) while <literal>eth0</literal> and <literal>eth1</literal> are slaves (SLAVE).</para>
+ <informaltable frame="none">
+ <tgroup cols="1">
+ <colspec colname="c1" colwidth="100*"/>
+ <tbody>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Note -</emphasis>All slaves of <literal>bond0</literal> have the same MAC address (Hwaddr) - <literal>bond0</literal>. All modes, except TLB and ALB, have this MAC address. TLB and ALB require a unique MAC address for each slave.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <screen>$ /sbin/ifconfig
bond0Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4
inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
collisions:0 txqueuelen:100
Interrupt:9 Base address:0x1400
</screen>
- </section>
</section>
- <section xml:id="dbdoclet.50438258_54769">
- <title>7.5 Configuring Lustre with Bonding</title>
- <para>Lustre uses the IP address of the bonded interfaces and requires no special configuration. It treats the bonded interface as a regular TCP/IP interface. If needed, specify bond0 using the Lustre networks parameter in /etc/modprobe.</para>
- <screen>options lnet networks=tcp(bond0)
+ </section>
+ <section xml:id="dbdoclet.50438258_54769">
+ <title>7.5 Configuring Lustre with Bonding</title>
+ <para>Lustre uses the IP address of the bonded interfaces and requires no special configuration. It treats the bonded interface as a regular TCP/IP interface. If needed, specify bond0 using the Lustre networks parameter in /etc/modprobe.</para>
+ <screen>options lnet networks=tcp(bond0)
</screen>
- </section>
- <section xml:id="dbdoclet.50438258_92244">
- <title>7.6 Bonding References</title>
- <para>We recommend the following bonding references:</para>
- <itemizedlist><listitem>
- <para> In the Linux kernel source tree, see documentation/networking/bonding.txt</para>
- </listitem>
-<listitem>
- <para><link xl:href="http://linux-ip.net/html/ether-bonding.html">http://linux-ip.net/html/ether-bonding.html</link></para>
- </listitem>
-<listitem>
- <para><link xl:href="http://www.sourceforge.net/projects/bonding">http://www.sourceforge.net/projects/bonding</link></para>
- </listitem>
-<listitem>
- <para> Linux Foundation bonding website: <link xl:href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding">http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding</link></para>
- </listitem>
-</itemizedlist>
- <para>This is the most extensive reference and we highly recommend it. This website includes explanations of more complicated setups, including the use of DHCP with bonding.</para>
- <screen>
+ </section>
+ <section xml:id="dbdoclet.50438258_92244">
+ <title>7.6 Bonding References</title>
+ <para>We recommend the following bonding references:</para>
+ <itemizedlist>
+ <listitem>
+ <para>In the Linux kernel source tree, see documentation/networking/bonding.txt</para>
+ </listitem>
+ <listitem>
+ <para><link xl:href="http://linux-ip.net/html/ether-bonding.html">http://linux-ip.net/html/ether-bonding.html</link></para>
+ </listitem>
+ <listitem>
+ <para><link xl:href="http://www.sourceforge.net/projects/bonding">http://www.sourceforge.net/projects/bonding</link></para>
+ </listitem>
+ <listitem>
+ <para>Linux Foundation bonding website: <link xl:href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding">http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding</link></para>
+ </listitem>
+ </itemizedlist>
+ <para>This is the most extensive reference and we highly recommend it. This website includes explanations of more complicated setups, including the use of DHCP with bonding.</para>
+ <screen>
</screen>
- </section>
+ </section>
</chapter>
git://git.whamcloud.com / doc / manual.git / commitdiff